Latest revision as of 13:36, 3 April 2019

Overview and Technologies

The Dryad REST API (API) is implemented as a DSpace module. It provides a RESTful interface to Dryad, leveraging JSON and OAuth standards. It uses the following libraries:

• Jersey 1.18 - RESTful web services in Java
• Apache Oltu - OAuth Protocol implementation in Java

The initial version of the API is intended for partner journals to exchange manuscript information with Dryad. More features will be added in the future.

Installation

Code

The API webapp code exists within dspace/modules/dryad-rest-webapp, and will be compiled if the module is enabled.

Database Schema

The API uses database objects for storing resources, OAuth tokens, and access control lists. These objects can be installed via dspace/etc/postgres/dryad-rest-webapp.sql (and removed with clean-dryad-rest-webapp.sql).

Metadata Schema

For purposes of recording publication date, a field has been added to the dspace/config/registries/dublin-core-types.xml registry. The metadata fileds should be loaded to ensure this field is present in the database.

/opt/dryad/bin/dsrun org.dspace.administer.MetadataImporter -u /opt/dryad/config/registries/dublin-core-types.xml

Server deployment

The API is implemented as a Java Webapp, and should be added to the Tomcat server's server.xml. See dspace/etc/server/server.xml-fragment for an example.

API Endpoints

Versioning

The API uses versioning within resource paths. This document describes functionality in Version 1, so all of the endpoints begin with /api/v1/

When installed on a Dryad server, the API webapp is designed to service /api, with /api/v1/ handled by a single servlet ().

Transport Encryption

The OAuth2 spec requires HTTPS for all communication. Without transport encryption, tokens in the URL or header could easily be read by third parties. This should be enforced at deployment time, possibly through Apache proxy rules.

Resources

The initial version of the API is intended for partner journals to exchange manuscript information, so the resources are organizations and manuscripts.

See Dryad Manuscript API on Apiary.

Organizations

Organizations represent partner journal organizations - companies, publishers, or entities that will be supplying metadata relating to Dryad submissions. Organizations are identified by a short organizationCode, assigned by Dryad staff. Organizations are stored in the organization table, and include a code and a descriptive name.

Organization codes are the same concept as journal codes currently defined in the DryadJournalSubmission.properties file, and must adopt any existing assignments.

Manuscripts

Manuscripts represent documents of metadata, corresponding to a journal article or other work, which may have data submitted to Dryad. Individual manuscripts can be accessed through their manuscriptId, which is an identifier assigned by the organization and unique within the organization.

A `GET` request will list all available manuscripts in an organization. These requests can be modified with query parameters:

• count limits the number of results returned (default is 1000),
• search filters the results for the presence of that one word in the metadata.

Authentication and Authorization

OAuth2 tokens are used for authentication - there is currently no support for anonymous or unauthenticated access. Tokens are tied to an eperson (user account) record, and access control lists specify which resources (API Paths) can be accessed and how. Since the application is RESTful, these are specified in terms of HTTP verbs that can be performed on a certain path.

OAuth 2 references

1. http://aaronparecki.com/articles/2012/07/29/1/oauth2-simplified
2. http://blogs.steeplesoft.com/posts/2013/07/11/a-simple-oauth2-client-and-server-example-part-i/
3. https://github.com/hasanozgan/apache-oltu-oauth2-provider-demo/blob/master/src/main/java/com/bilyoner/api/endpoints/TokenEndpoint.java

Tokens

OAuth 2 Tokens are stored in the oauth_token table. Tokens are 32-character strings generated by MD5. The table row includes the token string, the eperson id, and an expiration date.

Token Generation

Currently, tokens must be created and assigned manually. This can be done by generating a random md5, and inserting it into the token table, with an expiration date and eperson id

$ head /dev/random | md5
f36def40ade795ede0401c1f74144852

$ psql dryad_repo
dryad_repo=# select eperson_id from eperson where email = 'dan.leehr@nescent.org';
eperson_id
------------
6307
(1 row)

dryad_repo=# INSERT INTO oauth_token (eperson_id, token, expires) VALUES
(6307,'f36def40ade795ede0401c1f74144852','2014-12-31');
INSERT 0 1

Token Usage

Tokens can be supplied by the client in a few ways, according to the OAuth specification. The simplest is to include an access_token query parameter in the URL. This works for both GET and POST requests.

For example:

https://datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852

Tokens can also be included as an HTTP request header

Authorization: Bearer f36def40ade795ede0401c1f74144852

Authorization & Access Control

A valid OAuth token that corresponds to an eperson account will identify the bearer (authentication), but a token alone does not identify what the bearer is authorized to do.

Resource Access control lists are stored in the rest_resource_authz table. Each table row consists of an eperson ID, an HTTP method (GET/POST/PUT/DELETE), and a resource path (e.g. organizations/org1/manuscripts). These rows determine what a user can access.

Note that granting access to a resource path is recursive - if a user has GET access to organizations, he/she has read access to every organization and manuscript in the system.

As an example, suppose a partner journal user has eperson id 2860 and we've assigned organization code test. Their resource authorizations should probably be

eperson_id | http_method |           resource_path
2860 | POST        | organizations/test/manuscripts
2860 | PUT         | organizations/test/manuscripts
2860 | GET         | organizations/test

POST allows creation of new manuscripts within the test organization. PUT allows replacement of manuscripts (though the PUT would actually happen at organizations/test/manuscripts/:id), and GET allows them to read the organization and all of its sub-resources.

Note the user is not granted DELETE, and has no access anywhere outside of organizations/test.

Interaction with other components

These initial APIs are designed to handle exchange of manuscript metadata with partner journals and processing of changes. This job is currently done by the journal-submit webapp, which parses metadata emails from journals and stores them for the submission system.

Journal-Submit webapp

The journal-submit webapp parses emails and generates JSON documents to store in the database. When authors enter a manuscript in the submission system, Dryad reads their manuscript metadata out of the database, based on the entered Manuscript number.

The API webapp has been built to be compatible with this behavior. As manuscripts are created (POST) or updated (PUT), their JSON structure is stored in the manuscript table.

This integration requires the organization code (in the organizations table) and journal code (in the concepts) to match. This is why the organizations table is constructed as a view. It simplifies the journal concepts into a simpler table with the concept_id (as organization_id) and some other concept metadata.

See

• dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptXMLConverterHandler.java
• dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/converters/ManuscriptToLegacyXMLConverter.java

Data Package Metadata

Upon creating (POST) or updating (PUT) a manuscript, the API webapp will attempt to locate a Dryad Data Package (provided the manuscript includes a dryadDataDOI) and synchronize the following metadata:

1. Publication DOI
2. Manuscript Number
3. Manuscript Keywords
4. Manuscript Title
5. Manuscript Abstract
6. Publication Date

The values from the manuscript are copied into the data package metadata if the manuscript is submitted or accepted. If rejected, the values are removed from the metadata

See

• dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptMetadataSynchronizationHandler.java

Review workflow

Upon creating (POST) or updating (PUT) a manuscript, the API webapp will check the status (submitted/accepted/rejected) and transform the status into a review action if a corresponding data package (by dryadDataDOI) is currently in review.

The ApproveRejectReviewItem class has been refactored to support this action

See

• dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptReviewStatusChangeHandler.java

API usage examples

Below are some examples using cURL to interact with the API. Remember, these require creating tokens and access control in the database:

Listing organizations - GET /api/v1/organizations

curl --insecure \
--request GET \
https://staging.datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852 

Creating an organization - POST a JSON object to /api/v1/organizations.

curl --insecure \
--header "Content-Type:application/json" \
--request POST \
-d '{"organizationCode":"test","organizationName":"Dryad Test Organization"}' \
https://staging.datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852 

Creating a manuscript - POST a JSON object to /api/v1/organizations/{organizationCode}/manuscripts. Object must have manuscriptId, which will be the identifier for future operations

curl --insecure \
--header "Content-Type:application/json" \
--request POST \
--data-binary @manuscript.json \
https://staging.datadryad.org/api/v1/organizations/test/manuscripts?access_token=f36def40ade795ede0401c1f74144852

Updating a manuscript - change the data within the JSON and PUT to /api/v1/organizations/{organizationCode}/manuscripts/{manuscriptId}

curl --insecure \
--header "Content-Type:application/json"
--request PUT
--data-binary @manuscript-revised.json \
https://staging.datadryad.org/api/v1/organizations/test/manuscripts/MS12345?access_token=f36def40ade795ede0401c1f74144852

Example JSON documents are in dspace/modules/dryad-rest-webapp/src/main/resources

Listing manuscripts in an organization: can use filter query terms

curl --insecure \
--request GET \
https://staging.datadryad.org/api/v1/organizations/test/manuscripts/?access_token=f36def40ade795ede0401c1f74144852&count=10&search=Smith

Implementation of review workflow using REST API and Dryad-first option

For "Dryad-first":

1. authors submit data packages as “in preparation or in review” and don’t enter a manuscript number (they won’t have one yet)
2. data packages go to review (still have no manuscript number). Dryad sends review emails (including DOI) and review link to author and to journal
3. author submits manuscript to journal, including the Dryad DOI and/or review link
4. Journal sends API calls to Dryad on acceptance/rejection, including the Dryad DOI and/or review URL. Generally works like the email system - they specify a journal code (organization code) and a structured JSON object containing manuscript metadata
   • Examples here: Metadata Submission via API
5. API finds the data package that matches the incoming manuscript metadata (it looks it up by review URL, DOI, or manuscript number).
   • In submit-to-dryad first, this needs to be review URL or DOI, which means this information must be included in the API call from the journal on acceptance/rejection
6. API approves the submission from review to curation (when the status is “accepted”) and synchronizes the following metadata fields
   1. Publication DOI
   2. Manuscript Number
   3. Keywords
   4. Title (takes the title and prefixes it with “Data from: “
   5. Abstract
   6. Blackout until date aka publication date

Some other things to note:

1. When we receive API calls we will probably not be receiving metadata emails. The JSON data is protected with OAuth, and we can grant access to Dryad personnel.
2. The API will store the received JSON data in the database, but it will also make XML files just like the email system does - so these can be checked too. This supports the submit-to-journal first case
3. We’re recording the publication date in the dc.date.blackoutUntil field (new field!). This date will be used to automatically publish things from blackout but this part is not yet functional.
4. If the metadata sent by the journal is invalid or doesn’t match a data package in review, the API won’t be able to find the data package and it will stay in review. Something to be aware of.