Dryad REST API Technology

= Overview and Technologies =

The Dryad REST API (API) is implemented as a DSpace module. It provides a RESTful interface to Dryad, leveraging JSON and OAuth standards. It uses the following libraries:


 * Jersey 1.18 - RESTful web services in Java
 * Apache Oltu - OAuth Protocol implementation in Java

The initial version of the API is intended for partner journals to exchange manuscript information with Dryad. More features will be added in the future.

= Installation =

Code
The API webapp code exists within, and will be compiled if the module is enabled.

Database Schema
The API uses database objects for storing resources, OAuth tokens, and access control lists. These objects can be installed via  (and removed with  ).

Metadata Schema
For purposes of recording publication date, a field has been added to the  registry. The metadata fileds should be loaded to ensure this field is present in the database.

Server deployment
The API is implemented as a Java Webapp, and should be added to the Tomcat server's. See  for an example.

= API Endpoints =

Versioning
The API uses versioning within resource paths. This document describes functionality in Version 1, so all of the endpoints begin with

When installed on a Dryad server, the API webapp is designed to service, with   handled by a single servlet.

Transport Encryption
The OAuth2 spec requires HTTPS for all communication. Without transport encryption, tokens in the URL or header could easily be read by third parties. This should be enforced at deployment time, possibly through Apache proxy rules.

Resources
The initial version of the API is intended for partner journals to exchange manuscript information, so the resources are organizations and manuscripts.

See Dryad Manuscript API on Apiary.

Organizations
Organizations represent partner journal organizations - companies, publishers, or entities that will be supplying metadata relating to Dryad submissions. Organizations are identified by a short organizationCode, assigned by Dryad staff. Organizations are stored in the  table, and include a code and a descriptive name.

Organization codes are the same concept as journal codes currently defined in the  file, and must adopt any existing assignments.

Manuscripts
Manuscripts represent documents of metadata, corresponding to a journal article or other work, which may have data submitted to Dryad. Individual manuscripts can be accessed through their manuscriptId, which is an identifier assigned by the organization and unique within the organization.

A `GET` request will list all available manuscripts in an organization. These requests can be modified with query parameters:


 * count limits the number of results returned (default is 1000),
 * search filters the results for the presence of that one word in the metadata.

= Authentication and Authorization =

OAuth2 tokens are used for authentication - there is currently no support for anonymous or unauthenticated access. Tokens are tied to an eperson (user account) record, and access control lists specify which resources (API Paths) can be accessed and how. Since the application is RESTful, these are specified in terms of HTTP verbs that can be performed on a certain path.

OAuth 2 references


 * 1) http://aaronparecki.com/articles/2012/07/29/1/oauth2-simplified
 * 2) http://blogs.steeplesoft.com/posts/2013/07/11/a-simple-oauth2-client-and-server-example-part-i/
 * 3) https://github.com/hasanozgan/apache-oltu-oauth2-provider-demo/blob/master/src/main/java/com/bilyoner/api/endpoints/TokenEndpoint.java

Tokens
OAuth 2 Tokens are stored in the  table. Tokens are 32-character strings generated by MD5. The table row includes the token string, the eperson id, and an expiration date.

Token Generation
Currently, tokens must be created and assigned manually. This can be done by generating a random md5, and inserting it into the token table, with an expiration date and eperson id $ head /dev/random | md5 f36def40ade795ede0401c1f74144852

$ psql dryad_repo dryad_repo=# select eperson_id from eperson where email = 'dan.leehr@nescent.org'; eperson_id

6307 (1 row)

dryad_repo=# INSERT INTO oauth_token (eperson_id, token, expires) VALUES (6307,'f36def40ade795ede0401c1f74144852','2014-12-31'); INSERT 0 1

Token Usage
Tokens can be supplied by the client in a few ways, according to the OAuth specification. The simplest is to include an  query parameter in the URL. This works for both GET and POST requests.

For example: https://datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852 Tokens can also be included as an HTTP request header Authorization: Bearer f36def40ade795ede0401c1f74144852

Authorization & Access Control
A valid OAuth token that corresponds to an eperson account will identify the bearer (authentication), but a token alone does not identify what the bearer is authorized to do.

Resource Access control lists are stored in the  table. Each table row consists of an eperson ID, an HTTP method (GET/POST/PUT/DELETE), and a resource path (e.g. ). These rows determine what a user can access.

Note that granting access to a resource path is recursive - if a user has GET access to, he/she has read access to every organization and manuscript in the system.

As an example, suppose a partner journal user has eperson id 2860 and we've assigned organization code test. Their resource authorizations should probably be eperson_id | http_method |          resource_path 2860 | POST       | organizations/test/manuscripts 2860 | PUT        | organizations/test/manuscripts 2860 | GET        | organizations/test POST allows creation of new manuscripts within the test organization. PUT allows replacement of manuscripts (though the PUT would actually happen at organizations/test/manuscripts/:id), and GET allows them to read the organization and all of its sub-resources.

Note the user is not granted DELETE, and has no access anywhere outside of organizations/test.

= Interaction with other components =

These initial APIs are designed to handle exchange of manuscript metadata with partner journals and processing of changes. This job is currently done by the journal-submit webapp, which parses metadata emails from journals and stores them for the submission system.

Journal-Submit webapp
The journal-submit webapp parses emails and generates JSON documents to store in the database. When authors enter a manuscript in the submission system, Dryad reads their manuscript metadata out of the database, based on the entered Manuscript number.

The API webapp has been built to be compatible with this behavior. As manuscripts are created (POST) or updated (PUT), their JSON structure is stored in the  table.

This integration requires the organization code (in the organizations table) and journal code (in the concepts) to match. This is why the organizations table is constructed as a view. It simplifies the journal concepts into a simpler table with the concept_id (as organization_id) and some other concept metadata.

See


 * dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptXMLConverterHandler.java
 * dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/converters/ManuscriptToLegacyXMLConverter.java

Data Package Metadata
Upon creating (POST) or updating (PUT) a manuscript, the API webapp will attempt to locate a Dryad Data Package (provided the manuscript includes a ) and synchronize the following metadata:


 * 1) Publication DOI
 * 2) Manuscript Number
 * 3) Manuscript Keywords
 * 4) Manuscript Title
 * 5) Manuscript Abstract
 * 6) Publication Date

The values from the manuscript are copied into the data package metadata if the manuscript is submitted or accepted. If rejected, the values are removed from the metadata

See


 * dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptMetadataSynchronizationHandler.java

Review workflow
Upon creating (POST) or updating (PUT) a manuscript, the API webapp will check the status (submitted/accepted/rejected) and transform the status into a review action if a corresponding data package (by ) is currently in review.

The ApproveRejectReviewItem class has been refactored to support this action

See


 * dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptReviewStatusChangeHandler.java

= API usage examples =

Below are some examples using cURL to interact with the API. Remember, these require creating tokens and access control in the database:

Listing organizations - GET /api/v1/organizations curl --insecure \ --request GET \ https://staging.datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852 Creating an organization - POST a JSON object to /api/v1/organizations. curl --insecure \ --header "Content-Type:application/json" \ --request POST \ -d '{"organizationCode":"test","organizationName":"Dryad Test Organization"}' \ https://staging.datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852 Creating a manuscript - POST a JSON object to /api/v1/organizations/{organizationCode}/manuscripts. Object must have manuscriptId, which will be the identifier for future operations curl --insecure \ --header "Content-Type:application/json" \ --request POST \ --data-binary @manuscript.json \ https://staging.datadryad.org/api/v1/organizations/test/manuscripts?access_token=f36def40ade795ede0401c1f74144852 Updating a manuscript - change the data within the JSON and PUT to /api/v1/organizations/{organizationCode}/manuscripts/{manuscriptId} curl --insecure \ --header "Content-Type:application/json" --request PUT --data-binary @manuscript-revised.json \ https://staging.datadryad.org/api/v1/organizations/test/manuscripts/MS12345?access_token=f36def40ade795ede0401c1f74144852 Example JSON documents are in

Listing manuscripts in an organization: can use filter query terms curl --insecure \ --request GET \ https://staging.datadryad.org/api/v1/organizations/test/manuscripts/?access_token=f36def40ade795ede0401c1f74144852&count=10&search=Smith = Implementation of review workflow using REST API and Dryad-first option =

For "Dryad-first":


 * 1) authors submit data packages as “in preparation or in review” and don’t enter a manuscript number (they won’t have one yet)
 * 2) data packages go to review (still have no manuscript number). Dryad sends review emails (including DOI) and review link to author and to journal
 * 3) author submits manuscript to journal, including the Dryad DOI and/or review link
 * 4) Journal sends API calls to Dryad on acceptance/rejection, including the Dryad DOI and/or review URL. Generally works like the email system - they specify a journal code (organization code) and a structured JSON object containing manuscript metadata
 * 5) *Examples here: Metadata Submission via API
 * 6) API finds the data package that matches the incoming manuscript metadata (it looks it up by review URL, DOI, or manuscript number).
 * 7) *In submit-to-dryad first, this needs to be review URL or DOI, which means this information must be included in the API call from the journal on acceptance/rejection
 * 8) API approves the submission from review to curation (when the status is “accepted”) and synchronizes the following metadata fields
 * 9) Publication DOI
 * 10) Manuscript Number
 * 11) Keywords
 * 12) Title (takes the title and prefixes it with “Data from: “
 * 13) Abstract
 * 14) Blackout until date aka publication date

Some other things to note:


 * 1) When we receive API calls we will probably not be receiving metadata emails. The JSON data is protected with OAuth, and we can grant access to Dryad personnel.
 * 2) The API will store the received JSON data in the database, but it will also make XML files just like the email system does - so these can be checked too. This supports the submit-to-journal first case
 * 3) We’re recording the publication date in the dc.date.blackoutUntil field (new field!). This date will be used to automatically publish things from blackout but this part is not yet functional.
 * 4) If the metadata sent by the journal is invalid or doesn’t match a data package in review, the API won’t be able to find the data package and it will stay in review. Something to be aware of.