Testing

From Dryad wiki
Jump to: navigation, search

Overview

Testing strategies were discussed in Developer Meetings on 2014-07-11. See Google Doc.

Dryad employs several broad testing strategies:

  1. JUnit tests run via Maven within the build system, using the surefire plugin
  2. Selenium tests run externally, connecting to any running Dryad server
  3. JMeter tests to determine how the server acts under high load.

Test Configuration

Tests added to Dryad in 2014 require a postgres database and a minimal DSpace directory. The vagrant-dryad provisioning installs a script install_dryad_test_database.sh inside a VM that prepares such a testing environment.

The script perform the following steps:

  1. Drop and create a postgres database for test data.
  2. Load required database schema/content from sql files in dryad-repo:
    1. (e.g. psql < ~/dryad-repo/dspace/etc/postgres/database_schema.sql)
  3. Remove the /opt/dryad/test directory if it exists
  4. Copy the template dspace dir from dryad-repo/test to /opt/dryad/test
  5. Insert the database connection information (url, username, and password) into the /opt/dryad/test/config/dspace.cfg

Additionally, a test_dryad.sh script is installed, which runs the tests under maven.

If you wish to install a test environment outside the Vagrant VM, See install_dryad_test_database.sh.j2 for details. Additionally, the .travis.yml file provides a similar set of steps to set up a test environment under Travis CI.

Running Unit Tests

Dryad uses the standard set of DSpace unit tests, based on JUnit. These tests are found in the src/test directory of each module. By default, testing is disabled for normal compilation.

Maven tests are run from the dryad-repo directory with:

mvn package -DskipTests=false -Ddefault.dspace.dir=/path/to/dryad-test

Note the option to skip tests was previously -Dmaven.test.skip=false. skipTests is more modern and required to share base classes across maven projects (e.g. ContextUnitTest).

In the Vagrant VM, tests can be run using the script

test_dryad.sh

Issues with Unit Tests

Though surefire supports running a single test (and this works for our selenium tests), there is a DSpace configuration issue that prevents it from working: http://sourceforge.net/p/dspace/mailman/message/30149768/.

It is also documented that the tests should be runnable without building via

mvn test -P env-dev

But this fails with an error that org.dspace:dspace-xmlui-webapp:war:1.7.3-SNAPSHOT could not be found.

Package level tests

Maven tests can be run for a single package rather than the entire suite, e.g.:

mvn -pl dspace/modules/journal-landing/journal-landing-webapp package -DskipTests=false -Dmaven.test.skip=false -Ddefault.dspace.dir=/opt/dryad/test

Debugging unit tests

See command line flags for debugging unit tests here: IDEs_and_Remote_Debugging

Selenium

Selenium tests UI-specific functionality. Dryad maintains a set of Selenium tests in the selenium directory of the repository. These tests are run separately from the normal compilation process, because they require a running instance of Dryad.

Running Selenium Tests

These tests can be checked out and run from any host. Since they work through a web browser, they do not need any Dryad code present on the testing machine.

Within the selenium directory, there is a runtests.sh script that will run all the selenium tests. You must specify the URL of the server to test

./runtests.sh http://localhost:9999

The tests are run by maven and the surefire plugin. You can run an individual test by running a maven command like the following:

mvn package -DseleniumTestURL="http://localhost:9999" -Dtest=HomePageTest

Selenium IDE

  • Firefox plugin for automating tasks within firefox.
  • It internally stores scripts as HTML, but it can export scripts in Java/Ruby/Python code suitable for use with WebDriver/RC.
  • can record script through web page actions and then replay to test
  • can export script in variety of formats (JUnit, Ruby, C#)
  • Exported scripts are relatively brittle and must be modified for general use, but are a great starting point.

Selenium WebDriver

  • Tool that can launch a browser, execute a test script, and report results of the test.
  • Can be used with JUnit or other testing frameworks.
  • Rather than using JavaScript, it works directly with each browser's API for greater control of the browser.
  • Doesn't require running a separate server.
  • The "future" of Selenium.

Configuring Firefox

NOTE: As of 07/2014, Selenium testing using Firefox is possible once the Selenium library version is updated to 2.42.2. Previous versions of Selenium encountered startup issues with the web driver. If web driver issues are encountered running Selenium tests against Firefox, verify that the selenium-java version is specified at or above version 2.42.2 in the dryad-repo/selenium/pom.xml file.

This is no problem on a desktop/laptop with Firefox installed. It will work automatically. On a server, you must:

  • ensure firefox is installed
  • in the root account, run "Xvfb :99"
  • the the account that is running selenium, "export DISPLAY=:99"

Configuring Chrome

NOTE: testing with Chrome is not consistently successful, due to unknown startup issues. The files mentioned below have not been merged into the dryad-repo master repository as of 07/2014.

Chrome may be used as the browser for running Selenium tests, but additional system and testing configuration is required. During testing, Chrome is driven using a binary provided by the Selenium project. See ChromeDriver for downloads and details. The most recent version of the ChromeDriver binary (as of 07/2014) has been added to the Dryad Repository's Selenium directory as "dryad-repo/selenium/bin/chromedriver-osx-2.10". Note that this binary is for OSX 32-bit systems.

A system property must be set to enable browser testing on Chrome, "webdriver.chrome.driver". For example, the command:

mvn package -DseleniumTestURL="http://localhost:9999" -DseleniumTestBrowser="Chrome" -
Dtest=DescribePublicationAJAXTest -Dwebdriver.chrome.driver="./bin/chromedriver-osx-2.10"

when run from the dryad-repo/selenium directory will start a testing run on the tests in file DescribePublicationAJAXTest.java using the provided path to the ChromeDriver binary. The system property "seleniumTestBrowser" is used within the DescribePublicationAJAXTest.java file only in a switch to instantiate the Chrome driver instead of the default driver.

Best practices

  • Create initial tests with IDE, then export to your target language/format and edit.
  • Use the HtmlUnitDriver instead of the default FirefoxDriver if possible -- it is faster, and runs without modification on more systems.
  • When creating a test in the IDE, use "assert" statements for all tests, because "verify" statements don't export properly to the code.
  • When modifying the testing code, use "verify" statements for any non-fatal errors. Use "assert" statements for errors that will cause the rest of the test to be meaningless.

Checklist for exporting a test from the IDE to Webdriver:

  1. Ensure the test has a name ending in Test (for JUnit compatability)
  2. Perform the export.
  3. Make the test extend JUnit's TestCase class.
  4. Modify the test to use HtmlUnitDriver.
  5. Modify all assertions/verifications (these will appear as comments with an error message) to use the existing webdriver api
  6. Ensure that a base URL is set.
  7. Add it to any appropriate test suites.

Headless Selenium tests

Travis-CI

As documented on the Travis-CI site, headless Selenium tests may be as part of a a build on the Travis-CI platform. A Travis VM is pre-configured to support running Firefox headlessly. Writing Selenium tests that run successfully on Travis-CI requires (a) declaring the Selenium library dependencies in the project pom.xml file, (b) writing Selenium tests, and (c) integrating the tests into the Travis-CI build script (.travis.yml). Integration with the build/test process also requires that browser-accessible resources be available during a test run, either by accessing world-available resources or by serving the resources from within the build process.

During development of the Dryad display widget, an issue was encountered attempting to use the conventional separation of unit tests from integrations tests, by configuring the POM to use the `maven-surefire-plugin` for unit tests (executing as `mvn test` or `mvn package -DskipTests=false`) and the `maven-failsafe-plugin` for integration/verification tests (executing as `mvn failsafe:integration-test failsafe:verify`):

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
<configuration>
<excludes>
<exclude>it/**</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>
<include>it/**</include>
</includes>
<excludes>
<exclude>org/**</exclude>
</excludes>
</configuration>
</plugin>

This practice of separation of unit tests from integration tests follows from the fact that integration tests generally require orchestrating a number of services and resources in an attempt to replicate a larger unit of an application than would be tested during other phases.

The Dryad build process runs the Maven `package` phase, and does not perform a local installation (Maven `install` phase) of Dryad .jar files to the build environment's `.m2`, which is necessary for those dependencies to be available to a Dryad module (e.g., the widgets module) during an integration test run declared from within that module.

When run as part of an integration test phase, the headless Selenium tests for the Dryad Display Widget project do not run at all, due to the .m2 repository issue for uninstalled dspace dependencies, and lead to a Maven installation error. When run as part of the unit test phase, the headless Selenium tests for the Dryad Display Widget project do not run successfully, due to errors that were unable to be resolved, for example:

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.032 sec <<< FAILURE!
testBtnDownload(it.BtnDownloadTest)  Time elapsed: 15.02 sec  <<< ERROR!
org.openqa.selenium.WebDriverException: f.QueryInterface is not a function
Command duration or timeout: 10.11 seconds

Vagrant

Headless Selenuim tests may also be run in a virtual machine. The Vagrant configuration of a Dryad dev/build environment is configured to install the dependencies required to run Selenium tests headlessly. As with Travis-CI headless testing, pre-test configuration must be performed to start the servers and services used during tests. For example, the following shell commands (taken from a Dryad Widgets submodule build) would build and test a project configured with unit and integration/verification test phases, as listed above:

export SERVER=99
export SCREEN=0
export ITPORT=2341
export DISPLAY=":${SERVER}.${SCREEN}"
sh -e /etc/init.d/xvfb start
python -m SimpleHTTPServer $ITPORT > /dev/null 2>&1 &
mvn clean package -DskipTests=true
mvn test
mvn failsafe:integration-test failsafe:verify -DseleniumTestURL="http://localhost:$ITPORT/dryad-widgets-webapp/src/test/java/it"
pkill -f SimpleHTTPServer
sh -e /etc/init.d/xvfb stop

(Manual) Smoke Tests

These are the basic tests to ensure all components of Dryad are working. These tests are not covered by the Selenium tests. Once something is moved into Selenium, it should be removed from here.

Users:

  • A new user account can be registered (email works, clicking on the emailed token works)

Pages:

  • DSpace home page
  • About page
  • Partners page

Item search:

  • Simple searches work.
  • browses work
  • faceting works
  • "view more" on the facets works

Item view:

  • Item view pages display correctly.
    • Items can be viewed by pages built from various java classes: ItemViewer.java, ViewItem.java, InternalItemTransformer.java, DryadReviewTransformer.java
  • Bitstreams can be downloaded.
  • Statistics appear correctly.
  • Handles resolve properly.

Journal submit:

  • Does the journal URL work properly?

Journal metadata processing:

  • Create/copy over a metadata xml file from submission/journalMetadata.
  • Submit a test dataset using that manuscript ID.
  • Make sure that the submission is prepopulated with the metadata.
  • Make sure the item enters the journal review workflow.

Basic submit:

  • Submit a test dataset, with at least one file to upload.
  • Make sure a proper Handle is assigned.
  • Make sure the item is searchable.
  • View the item summary page.
  • Remove the item.

"In Review" submit:

  • Submit a test dataset for an item with the status "in review" (need to set up a dummy journal, so real editors don't get the email)
  • Make sure the item is viewable when using the review URL.
  • Make sure the item is not viewable to normal users when not using the review URL.
  • Remove the item.

Item rejection:

  • Submit a test dataset from a non-curating user.
  • As a curator, reject the dataset and return it to the user.
  • As the submitter, verify that the item has returned to your workspace.
  • This should be done for items both in regular workflow and in journal review workflow.

JMeter Load Tests

See documentation for the JMeter test suite.

Relation to DSpace

Dryad uses the standard DSpace unit tests and integration tests. See DSpace Testing.