Kevin's Worklog

Documenting the Daily Stream

Configuring WebDriver PhantomJS Testing in Maven

I’ve used JUnit for testing for awhile now, but recently I’ve been wanting to learn a little more about other testing tools and, in particular, tools that support the testing of JavaScript applications. I’ve dabbled with Selenium in the past, but the learning curve (including setup time) has been daunting.

I’m a big fan of things that run as a part of a Maven build with very little effort on my part. To this end, I started experimenting with Selenium’s WebDriver implementation. After trying out the HTMLUnitDriver, I decided that I wanted to use PhantomJS instead (since its code is based on an actual browser’s JavaScript implementation).

What follows is what I’ve come up with as a workable Maven configuration.

For the purposes of this post, I’m going to ignore the unit testing configuration and just focus on the integration testing. As a part of the integration testing, I’ll run code that tests content that’s being served from a Jetty server within the processes of the Maven build.

Usually PhantomJS requires that you download its binary and point your integration test runner to it, but luckily I found a Maven plugin that will do that work for me. The first thing I want to do, though, is to configure the WebDriver that I plan to use, GhostDriver.

I’ll also configure JUnit because I’m going to use that to write my integration tests. Here are the relevant parts of my pom.xml file:

<properties>
    <junit.version>4.12-beta-1</junit.version>
    <ghostdriver.version>1.1.0</ghostdriver.version>
</properties>

<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>${junit.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.github.detro.ghostdriver</groupId>
        <artifactId>phantomjsdriver</artifactId>
        <version>${ghostdriver.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Next, I’ll want to configure the phantomjs-maven-plugin (that will download PhantomJS for me). What the below does, in addition to downloading PhantomJS, is put the path to its executable in a phantomjs.binary property. This can then be passed via configuration as a system property for the integration test framework (an example of this can be seen in the code block after the one below).

<properties>
    <junit.version>4.12-beta-1</junit.version>
    <ghostdriver.version>1.1.0</ghostdriver.version>
    <phantomjs.plugin.version>0.4</phantomjs.plugin.version>
</properties>

<build>
    <plugins>
        <!-- Installs PhantomJS so it doesn't have to be pre-installed -->
        <plugin>
            <groupId>com.github.klieber</groupId>
            <artifactId>phantomjs-maven-plugin</artifactId>
            <version>${phantomjs.plugin.version}</version>
            <executions>
                <execution>
                    <goals>
                        <goal>install</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <version>${phantomjs.version}</version>
            </configuration>
        </plugin>
    </plugins>
</build>

In order to run the tests, we’ll need an integration test framework (in our case, Failsafe) and Jetty to serve our test files. We can also use the build-helper-maven-plugin to find ports on the local system that are available for Jetty to use.

The following plugins should be added to the pom.xml file’s build/plugins element. The configuration of the phantomjs.binary property can also be seen below.

<!-- Get two free ports for our test server to use -->
<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <version>1.6</version>
    <configuration>
        <portNames>
            <portName>jetty.port</portName>
            <portName>jetty.port.stop</portName>
        </portNames>
    </configuration>
    <executions>
        <execution>
            <id>reserve-port</id>
            <phase>initialize</phase>
            <goals>
                <goal>reserve-network-port</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<!-- Use failsafe to run our integration tests -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>${failsafe.version}</version>
    <configuration>
        <systemPropertyVariables>
            <phantomjs.binary>${phantomjs.binary}</phantomjs.binary>
        </systemPropertyVariables>
    </configuration>
    <executions>
        <execution>
            <id>integration-test</id>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
        <execution>
            <id>verify</id>
            <goals>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<!-- Use a Jetty test server to serve our test content -->
<plugin>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-maven-plugin</artifactId>
    <version>${jetty.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-servlet</artifactId>
            <version>${jetty.version}</version>
        </dependency>
    </dependencies>
    <configuration>
        <jvmArgs>-Djetty.port=${jetty.port}</jvmArgs>
        <stopKey>ANY_KEY_HERE_IS_FINE</stopKey>
        <stopPort>${jetty.port.stop}</stopPort>
    </configuration>
    <executions>
        <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>run</goal>
            </goals>
            <configuration>
                <daemon>true</daemon>
            </configuration>
        </execution>
        <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>stop</goal>
            </goals>
        </execution>
    </executions>
</plugin>

We’ll also need to add some more properties to the pom.xml file to handle the variables in the above plugin configurations.

<properties>
    <jetty.version>9.2.2.v20140723</jetty.version>
    <failsafe.version>2.17</failsafe.version>
    <junit.version>4.12-beta-1</junit.version>
    <ghostdriver.version>1.1.0</ghostdriver.version>
    <phantomjs.version>1.9.7</phantomjs.version>
    <phantomjs.plugin.version>0.4</phantomjs.plugin.version>
</properties>

Once this is all configured, the PhantomJS driver can be accessed via your JUnit-based Java code.

private static String PHANTOMJS_BINARY;

/**
 * Check that the PhantomJS binary was installed successfully.
 */
@BeforeClass
public static void beforeTest() {
    PHANTOMJS_BINARY = System.getProperty("phantomjs.binary");

    assertNotNull(PHANTOMJS_BINARY);
    assertTrue(new File(PHANTOMJS_BINARY).exists());
}

/**
 * Test something.
 */
@Test
public void testSomething() {
    final DesiredCapabilities capabilities = new DesiredCapabilities();
    final String port = System.getProperty("jetty.port");

    // Configure our WebDriver to support JavaScript and be able to find the PhantomJS binary
    capabilities.setJavascriptEnabled(true);
    capabilities.setCapability("takesScreenshot", false);
    capabilities.setCapability(
        PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,
        PHANTOMJS_BINARY
    );

    final WebDriver driver = new PhantomJSDriver(capabilities);
    final String baseURL = "http://localhost:" + port;

    if (port == null) {
        fail("System property 'jetty.port' is not set");
    }

    // If the referenced JavaScript files fail to load, the test fails at this point
    driver.navigate().to(baseURL + "/index.html");

    // Then do some more tests using WebDriver methods...
}

That’s about it… a relatively simple way to run integration tests against JavaScript sources using Maven and JUnit.

Access Forbidden on Omeka Themes Directory

This has bitten me twice now so I’m (finally) documenting it.

The situation arises when an Omeka site that’s working fine on a CentOS box is moved to an Ubuntu box. Everything will work fine except for viewing the site’s themes from within the administrative interface. When /admin/themes is accessed, the user is greeted with a “Directory access is forbidden” message.

It’s not a mod_rewrite issue (as one might first expect) because every other aspect of the site works fine.

The problem is that Ubuntu’s Apache configuration default has index.html as one of the DirectoryIndex values. This seems like a reasonable default, but it causes problems with Omeka’s rewriting rules in the site’s .htaccess file:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^admin/ - [C]
RewriteRule .* admin/index.php [L]

When index.html is a DirectoryIndex value in the Apache configuration, Apache will check /admin/themes/index.html and find a file there. That file is intended to be the fallback for the PHP script, but Apache bypasses the PHP because the RewriteCond tells it to skip the rewrite when a file is found. So, what can be done? Tweaking the .htaccess file won’t do you any good.

What needs to be done is to reconfigure the DirectoryIndex value for Apache in the virtual host configuration for your Omeka site. So, the new virtual host configuration might look like:

<VirtualHost *:80>
    ServerName omeka
    ServerAdmin ksclarke@gmail.com
    DocumentRoot /var/www/omeka

    LogLevel info

    ErrorLog ${APACHE_LOG_DIR}/omeka-error.log
    CustomLog ${APACHE_LOG_DIR}/omeka-access.log combined

    <Location />
        DirectoryIndex index.php
        Order deny,allow
        Allow from all
    </Location>
</VirtualHost>

By telling Apache to only look for an index.php file and not for an index.html file, the rewriting in the .htaccess file will work and Omeka will redirect the user to the index.html file only when it’s absolutely necessary. Once this is changed, the /admin/themes URL will work again. Hooray!

Creating New Linux Users

This is something very basic but I, not being a sysadmin, so rarely do it that I thought I’d document it.

To add a new user:

sudo adduser --home /home/ksclarke --shell /bin/bash --groups wheel,apache,users ksclarke

To remove that user (including their home directory):

sudo userdel -r ksclarke

Lastly, you might want to add that user’s public key to their authorized keys list (using nano to copy and paste it in):

sudo mkdir /home/ksclarke/.ssh
sudo nano /home/ksclarke/.ssh/authorized_keys
sudo chown -R ksclarke:ksclarke /home/ksclarke/.ssh
sudo chmod 700 /home/ksclarke/.ssh
sudo chmod 600 /home/ksclarke/.ssh/authorized_keys

Adding Pidgin to Gnome-Shell’s Status Tray

This worked previously, but it seems a recent update once again broke Pidgin’s ability to display in Gnome-Shell’s status bar. I need to add this back because without it I have to keep Pidgin open in the background for it to be accessible. Documenting for future use…

First, create a new directory:

mkdir ~/.local/share/gnome-shell/extensions/pidgin.status@gnome-shell.lisforge.net

Next, create a new file in that directory named metadata.json. Its contents should be:

{
  "shell-version": ["3.10.4"], 
  "uuid": "pidgin.status@gnome-shell.lisforge.net", 
  "name": "Pidgin Status Icon", 
  "description": "Integrates Pidgin Messaging Client into the status bar"
}

If running a different version of Gnome-Shell, find out what version you’re running by typing:

gnome-shell --version

Then, create a new file named extension.js (in that same directory). Its contents should be:

// Creates a system status notification icon for Pidgin
const StatusIconDispatcher = imports.ui.statusIconDispatcher;

// Gnome-Shell extension entry point
function main() {
  StatusIconDispatcher.STANDARD_TRAY_ICON_IMPLEMENTATIONS['pidgin'] = 'pidgin';
}

Then simply restart gnome-shell by pressing alt-f2, typing r, and hitting enter. That’s it. When Pidgin is started now its icon should display in the status bar.

Strace and Apache

Adding a new tool to my toolbox: strace. I’m surprised I haven’t used it before as it seems very useful.

So what is it? Strace is a debugging utility for Linux and some other Unix-like systems to monitor the system calls used by a program and all the signals it receives. So, for instance, to monitor Apache on Ubuntu, I can (while the Apache process is running) start strace like:

ps auxw | grep sbin/apache | awk '{print"-p " $2}' | sudo xargs strace

Then perform an action (e.g. request a webpage) and I’ll have some useful output to look over:

[pid  9669] <... accept4 resumed> {sa_family=AF_INET6, sin6_port=htons(36244), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 18
[pid  9669] getsockname(18, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
[pid  9669] fcntl(18, F_GETFL)          = 0x2 (flags O_RDWR)
[pid  9669] fcntl(18, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid  9669] read(18, "GET /archive/square_thumbnails/d"..., 8000) = 523
[pid  9669] stat("/var/www/omeka/archive/square_thumbnails/d1e08e21f0cb91e12f5be093225ec7ac.jpg", 0x7fff2f2cc110) = -1 EACCES (Permission denied)
[pid  9669] lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  9669] lstat("/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  9669] open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  9669] lstat("/var/www/omeka", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  9669] open("/var/www/omeka/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  9669] lstat("/var/www/omeka/archive", {st_mode=S_IFLNK|0777, st_size=38, ...}) = 0
[pid  9669] stat("/var/www/omeka/archive", 0x7fff2f2cc090) = -1 EACCES (Permission denied)
[pid  9669] write(7, "[Mon Aug 04 15:33:25.627750 2014"..., 172) = 172
[pid  9669] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f146758e000
[pid  9669] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f146758c000
[pid  9669] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f146758a000
[pid  9669] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1467588000
[pid  9669] read(18, 0x7f1467590048, 8000) = -1 EAGAIN (Resource temporarily unavailable)
[pid  9669] writev(18, [{"HTTP/1.1 403 Forbidden\r\nDate: Mo"..., 216}, {"<!DOCTYPE HTML PUBLIC \"-//IETF//"..., 334}], 2) = 550
[pid  9669] write(10, "127.0.0.1 - - [04/Aug/2014:15:33"..., 272) = 272
[pid  9669] times({tms_utime=0, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 1736382825

This gives me a lot of detail into what Apache is doing on the system. I can see what files it’s requesting and where it’s having problems.

Simple Apache Configuration for Multiple Dev Sites

I often want to run different apps (Drupal, Omeka, BaseX, etc.) on my dev machine. What follows is a simple Apache configuration that, with the help of a Linux system’s /etc/hosts file, makes this “multiple sites on a single machine” configuration easy.

The first thing to do is to configure each of these “sites” in the system’s /etc/hosts file. The following is the configuration for a dev machine that has a Drupal site, Omeka site, and static site.

127.0.0.1   localhost omeka drupal
127.0.1.1   ksclarke-laptop

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Next, we’ll need a configuration for each of the virtual hosts in the Apache config file. The most basic configuration, would look like:

NameVirtualHost *:80

<VirtualHost *:80>
    ServerName localhost
    ServerAdmin ksclarke@gmail.com
    DocumentRoot /var/www/html

    LogLevel info
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    <Location />
            Order deny,allow
            Allow from all
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ServerName drupal
    ServerAdmin ksclarke@gmail.com
    DocumentRoot /var/www/drupal

    LogLevel info
    ErrorLog ${APACHE_LOG_DIR}/drupal-error.log
    CustomLog ${APACHE_LOG_DIR}/drupal-access.log combined

    <Location />
            Order deny,allow
            Allow from all
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ServerName omeka
    ServerAdmin ksclarke@gmail.com
    DocumentRoot /var/www/omeka

    LogLevel info
    ErrorLog ${APACHE_LOG_DIR}/omeka-error.log
    CustomLog ${APACHE_LOG_DIR}/omeka-access.log combined

    <Location />
            Order deny,allow
            Allow from all
    </Location>
</VirtualHost>

Each of these virtual host configurations could be put into their own config file (for instance, on Ubuntu in /etc/apache2/sites-available (with a link created in /etc/apache2/sites-enabled)), but I tend to reserve individual configuration files for real world hosts: worklog.kevinclarke.info, www.kevinclarke.info, etc. Instead, I put the above in the default virtual host config file.

Once the above are configured, accessing each site is as simple as going to http://drupal/ or http://omeka/ in a Web browser.

For apps that are proxied, individual mod_proxy configurations can be put in the relevant VirtualHost configuration.