Difference between revisions of "Testing"

From Dryad wiki
Jump to: navigation, search
(Test Configuration)
Line 1: Line 1:
{{StatusBox|This page is a work-in-progress - --[[User:Dan|Dan]] ([[User talk:Dan|talk]]) 10:12, 14 July 2014 (EDT)dcl9, 2014-07-14}}
 
 
 
= Overview =
 
= Overview =
  
Line 36: Line 34:
 
</pre>
 
</pre>
  
'''Note''' the option to skip tests was previously -Dmaven.test.skip=false. skipTests is more modern and required to share base classes across maven projects (e.g. ContextUnitTest).  
+
'''Note''' the option to skip tests was previously -Dmaven.test.skip=false. skipTests is more modern and required to share base classes across maven projects (e.g. ContextUnitTest).
 
Until [https://github.com/dleehr/dryad-repo/commit/fa54d664abcf55ce2ccbc9fdf9ace7dea475fe81 commit fa54d664abcf55ce2ccbc9fdf9ace7dea475fe81] is merged into dryad-master, you can use -Dmaven.test.skip=false.
 
Until [https://github.com/dleehr/dryad-repo/commit/fa54d664abcf55ce2ccbc9fdf9ace7dea475fe81 commit fa54d664abcf55ce2ccbc9fdf9ace7dea475fe81] is merged into dryad-master, you can use -Dmaven.test.skip=false.
+
 
 
=== Issues with Unit Tests ===
 
=== Issues with Unit Tests ===
  

Revision as of 11:48, 15 July 2014

Overview

Testing strategies were discussed in Developer Meetings on 2014-07-11. See Google Doc.

Dryad employs two broad testing strategies:

  1. JUnit tests run via Maven within the build system, using the surefire plugin
  2. Selenium tests run externally, connecting to any running Dryad server

Test Configuration

Tests added to Dryad in 2014 require a postgres database and a minimal DSpace directory. The vagrant-dryad provisioning installs a script install_dryad_test_database.sh inside a VM that prepares such a testing environment.

The script perform the following steps:

  1. Drop and create a postgres database for test data.
  2. Load required database schema/content from sql files in dryad-repo:
    1. (e.g. psql < ~/dryad-repo/dspace/etc/postgres/database_schema.sql)
  3. Remove the /opt/dryad/test directory if it exists
  4. Copy the template dspace dir from dryad-repo/test to /opt/dryad/test
  5. Insert the database connection information (url, username, and password) into the /opt/dryad/test/config/dspace.cfg

Additionally, a test-dryad.sh script is installed, which runs the tests under maven.

If you wish to install a test environment outside the Vagrant VM, See install_dryad_test_database.sh.j2 for details. Additionally, the .travis.yml file provides a similar set of steps to set up a test environment under Travis CI.

Running Unit Tests

Dryad uses the standard set of DSpace unit tests, based on JUnit. These tests are found in the src/test directory of each module. By default, testing is disabled for normal compilation.

Maven tests are run from the dryad-repo directory with:

mvn package -DskipTests=false -Ddefault.dspace.dir=/path/to/dryad-test

Note the option to skip tests was previously -Dmaven.test.skip=false. skipTests is more modern and required to share base classes across maven projects (e.g. ContextUnitTest). Until commit fa54d664abcf55ce2ccbc9fdf9ace7dea475fe81 is merged into dryad-master, you can use -Dmaven.test.skip=false.

Issues with Unit Tests

Though surefire supports running a single test (and this works for our selenium tests), there is a DSpace configuration issue that prevents it from working: http://sourceforge.net/p/dspace/mailman/message/30149768/.

It is also documented that the tests should be runnable without building via

mvn test -P env-dev

But this fails with an error that org.dspace:dspace-xmlui-webapp:war:1.7.3-SNAPSHOT could not be found.

Selenium

Selenium tests UI-specific functionality. Dryad maintains a set of Selenium tests in the selenium directory of the repository. These tests are run separately from the normal compilation process, because they require a running instance of Dryad.

Running Selenium Tests

These tests can be checked out and run from any host. Since they work through a web browser, they do not need any Dryad code present on the testing machine.

Within the selenium directory, there is a runtests.sh script that will run all the selenium tests. You must specify the URL of the server to test

./runtests.sh http://localhost:9999

The tests are run by maven and the surefire plugin. You can run an individual test by running a maven command like the following:

mvn package -DseleniumTestURL="http://localhost:9999" -Dtest=HomePageTest

Selenium IDE

  • Firefox plugin for automating tasks within firefox.
  • It internally stores scripts as HTML, but it can export scripts in Java/Ruby/Python code suitable for use with WebDriver/RC.
  • can record script through web page actions and then replay to test
  • can export script in variety of formats (JUnit, Ruby, C#)
  • Exported scripts are relatively brittle and must be modified for general use, but are a great starting point.

Selenium WebDriver

  • Tool that can launch a browser, execute a test script, and report results of the test.
  • Can be used with JUnit or other testing frameworks.
  • Rather than using JavaScript, it works directly with each browser's API for greater control of the browser.
  • Doesn't require running a separate server.
  • The "future" of Selenium.

Configuring Firefox

NOTE: As of 07/2014, Selenium testing using Firefox is possible once the Selenium library version is updated to 2.42.2. Previous versions of Selenium encountered startup issues with the web driver. If web driver issues are encountered running Selenium tests against Firefox, verify that the selenium-java version is specified at or above version 2.42.2 in the dryad-repo/selenium/pom.xml file.

This is no problem on a desktop/laptop with Firefox installed. It will work automatically. On a server, you must:

  • ensure firefox is installed
  • in the root account, run "Xvfb :99"
  • the the account that is running selenium, "export DISPLAY=:99"

Configuring Chrome

NOTE: testing with Chrome is not consistently successful, due to unknown startup issues. The files mentioned below have not been merged into the dryad-repo master repository as of 07/2014.

Chrome may be used as the browser for running Selenium tests, but additional system and testing configuration is required. During testing, Chrome is driven using a binary provided by the Selenium project. See ChromeDriver for downloads and details. The most recent version of the ChromeDriver binary (as of 07/2014) has been added to the Dryad Repository's Selenium directory as "dryad-repo/selenium/bin/chromedriver-osx-2.10". Note that this binary is for OSX 32-bit systems.

A system property must be set to enable browser testing on Chrome, "webdriver.chrome.driver". For example, the command:

mvn package -DseleniumTestURL="http://localhost:9999" -DseleniumTestBrowser="Chrome" -
Dtest=DescribePublicationAJAXTest -Dwebdriver.chrome.driver="./bin/chromedriver-osx-2.10"

when run from the dryad-repo/selenium directory will start a testing run on the tests in file DescribePublicationAJAXTest.java using the provided path to the ChromeDriver binary. The system property "seleniumTestBrowser" is used within the DescribePublicationAJAXTest.java file only in a switch to instantiate the Chrome driver instead of the default driver.

Best practices

  • Create initial tests with IDE, then export to your target language/format and edit.
  • Use the HtmlUnitDriver instead of the default FirefoxDriver if possible -- it is faster, and runs without modification on more systems.
  • When creating a test in the IDE, use "assert" statements for all tests, because "verify" statements don't export properly to the code.
  • When modifying the testing code, use "verify" statements for any non-fatal errors. Use "assert" statements for errors that will cause the rest of the test to be meaningless.

Checklist for exporting a test from the IDE to Webdriver:

  1. Ensure the test has a name ending in Test (for JUnit compatability)
  2. Perform the export.
  3. Make the test extend JUnit's TestCase class.
  4. Modify the test to use HtmlUnitDriver.
  5. Modify all assertions/verifications (these will appear as comments with an error message) to use the existing webdriver api
  6. Ensure that a base URL is set.
  7. Add it to any appropriate test suites.

(Manual) Smoke Tests

These are the basic tests to ensure all components of Dryad are working. These tests are not covered by the Selenium tests. Once something is moved into Selenium, it should be removed from here.

Users:

  • A new user account can be registered (email works, clicking on the emailed token works)

Pages:

  • DSpace home page
  • About page
  • Partners page

Item search:

  • Simple searches work.
  • browses work
  • faceting works
  • "view more" on the facets works

Item view:

  • Item view pages display correctly.
  • Bitstreams can be downloaded.
  • Statistics appear correctly.
  • Handles resolve properly.

Journal submit:

  • Does the journal URL work properly?

Basic submit:

  • Submit a test dataset, with at least one file to upload.
  • Make sure a proper Handle is assigned.
  • Make sure the item is searchable.
  • View the item summary page.
  • Remove the item.

"In Review" submit:

  • Submit a test dataset for an item with the status "in review" (need to set up a dummy journal, so real editors don't get the email)
  • Make sure the item is viewable when using the review URL.
  • Make sure the item is not viewable to normal users when not using the review URL.
  • Remove the item.

Relation to DSpace

Dryad uses the standard DSpace unit tests and integration tests. See DSpace Testing.