Difference between revisions of "Dryad REST API Technology"
Ryan Scherle (talk | contribs) (→Journal-Submit webapp) |
|||
(4 intermediate revisions by 2 users not shown) | |||
Line 122: | Line 122: | ||
== Journal-Submit webapp == | == Journal-Submit webapp == | ||
− | The journal-submit webapp parses emails and generates | + | The journal-submit webapp parses emails and generates JSON documents to store in the database. When authors enter a manuscript in the submission system, Dryad reads their manuscript metadata out of the database, based on the entered Manuscript number. |
− | The API webapp has been built to be compatible with this behavior. As manuscripts are created (POST) or updated (PUT), their JSON structure is stored in the <code>manuscript</code> table | + | The API webapp has been built to be compatible with this behavior. As manuscripts are created (POST) or updated (PUT), their JSON structure is stored in the <code>manuscript</code> table. |
− | This integration requires the organization code (in the organizations table) and journal code (in the | + | This integration requires the organization code (in the organizations table) and journal code (in the concepts) to match. This is why the organizations table is constructed as a view. It simplifies the journal concepts into a simpler table with the concept_id (as organization_id) and some other concept metadata. |
See | See | ||
Line 191: | Line 191: | ||
<pre>curl --insecure \ | <pre>curl --insecure \ | ||
--request GET \ | --request GET \ | ||
− | https://staging.datadryad.org/api/v1/organizations/test/manuscripts/?access_token=f36def40ade795ede0401c1f74144852&count=10&search=Smith</pre> | + | https://staging.datadryad.org/api/v1/organizations/test/manuscripts/?access_token=f36def40ade795ede0401c1f74144852&count=10&search=Smith</pre> |
+ | = Implementation of review workflow using REST API and Dryad-first option = | ||
+ | |||
+ | For "Dryad-first": | ||
+ | |||
+ | #authors submit data packages as “in preparation or in review” and don’t enter a manuscript number (they won’t have one yet) | ||
+ | #data packages go to review (still have no manuscript number). Dryad sends review emails (including DOI) and review link to author and to journal | ||
+ | #author submits manuscript to journal, including the Dryad DOI and/or review link | ||
+ | #Journal sends API calls to Dryad on acceptance/rejection, including the Dryad DOI and/or review URL. Generally works like the email system - they specify a journal code (organization code) and a structured JSON object containing manuscript metadata | ||
+ | #*Examples here: [[Journal Metadata#Metadata Submission via API|Metadata Submission via API]] | ||
+ | #API finds the data package that matches the incoming manuscript metadata (it looks it up by review URL, DOI, or manuscript number). | ||
+ | #*In submit-to-dryad first, this needs to be review URL or DOI, which means '''this information must be included in the API call from the journal on acceptance/rejection''' | ||
+ | #API approves the submission from review to curation (when the status is “accepted”) and synchronizes the following metadata fields | ||
+ | ##Publication DOI | ||
+ | ##Manuscript Number | ||
+ | ##Keywords | ||
+ | ##Title (takes the title and prefixes it with “Data from: “ | ||
+ | ##Abstract | ||
+ | ##Blackout until date aka publication date | ||
+ | |||
+ | Some other things to note: | ||
+ | |||
+ | #When we receive API calls we will probably not be receiving metadata emails. The JSON data is protected with OAuth, and we can grant access to Dryad personnel. | ||
+ | #The API will store the received JSON data in the database, but it will also make XML files just like the email system does - so these can be checked too. This supports the submit-to-journal first case | ||
+ | #We’re recording the publication date in the dc.date.blackoutUntil field (new field!). This date will be used to automatically publish things from blackout but this part is not yet functional. | ||
+ | #If the metadata sent by the journal is invalid or doesn’t match a data package in review, the API won’t be able to find the data package and it will stay in review. Something to be aware of.<br/><br/> | ||
+ | [[Category:Technical Documentation|REST API Technology]]<br/>[[Category:API]]<br/>[[Category:Software]]<br/>[[Category:Submission Integration|Submission_Integration]] |
Latest revision as of 13:36, 3 April 2019
Contents
Overview and Technologies
The Dryad REST API (API) is implemented as a DSpace module. It provides a RESTful interface to Dryad, leveraging JSON and OAuth standards. It uses the following libraries:
- Jersey 1.18 - RESTful web services in Java
- Apache Oltu - OAuth Protocol implementation in Java
The initial version of the API is intended for partner journals to exchange manuscript information with Dryad. More features will be added in the future.
Installation
Code
The API webapp code exists within dspace/modules/dryad-rest-webapp
, and will be compiled if the module is enabled.
Database Schema
The API uses database objects for storing resources, OAuth tokens, and access control lists. These objects can be installed via dspace/etc/postgres/dryad-rest-webapp.sql
(and removed with clean-dryad-rest-webapp.sql
).
Metadata Schema
For purposes of recording publication date, a field has been added to the dspace/config/registries/dublin-core-types.xml
registry. The metadata fileds should be loaded to ensure this field is present in the database.
/opt/dryad/bin/dsrun org.dspace.administer.MetadataImporter -u /opt/dryad/config/registries/dublin-core-types.xml
Server deployment
The API is implemented as a Java Webapp, and should be added to the Tomcat server's server.xml
. See dspace/etc/server/server.xml-fragment
for an example.
API Endpoints
Versioning
The API uses versioning within resource paths. This document describes functionality in Version 1, so all of the endpoints begin with /api/v1/
When installed on a Dryad server, the API webapp is designed to service /api
, with /api/v1/
handled by a single servlet ().
Transport Encryption
The OAuth2 spec requires HTTPS for all communication. Without transport encryption, tokens in the URL or header could easily be read by third parties. This should be enforced at deployment time, possibly through Apache proxy rules.
Resources
The initial version of the API is intended for partner journals to exchange manuscript information, so the resources are organizations and manuscripts.
See Dryad Manuscript API on Apiary.
Organizations
Organizations represent partner journal organizations - companies, publishers, or entities that will be supplying metadata relating to Dryad submissions. Organizations are identified by a short organizationCode, assigned by Dryad staff. Organizations are stored in the organization
table, and include a code and a descriptive name.
Organization codes are the same concept as journal codes currently defined in the DryadJournalSubmission.properties
file, and must adopt any existing assignments.
Manuscripts
Manuscripts represent documents of metadata, corresponding to a journal article or other work, which may have data submitted to Dryad. Individual manuscripts can be accessed through their manuscriptId, which is an identifier assigned by the organization and unique within the organization.
A `GET` request will list all available manuscripts in an organization. These requests can be modified with query parameters:
- count limits the number of results returned (default is 1000),
- search filters the results for the presence of that one word in the metadata.
Authentication and Authorization
OAuth2 tokens are used for authentication - there is currently no support for anonymous or unauthenticated access. Tokens are tied to an eperson (user account) record, and access control lists specify which resources (API Paths) can be accessed and how. Since the application is RESTful, these are specified in terms of HTTP verbs that can be performed on a certain path.
OAuth 2 references
- http://aaronparecki.com/articles/2012/07/29/1/oauth2-simplified
- http://blogs.steeplesoft.com/posts/2013/07/11/a-simple-oauth2-client-and-server-example-part-i/
- https://github.com/hasanozgan/apache-oltu-oauth2-provider-demo/blob/master/src/main/java/com/bilyoner/api/endpoints/TokenEndpoint.java
Tokens
OAuth 2 Tokens are stored in the oauth_token
table. Tokens are 32-character strings generated by MD5. The table row includes the token string, the eperson id, and an expiration date.
Token Generation
Currently, tokens must be created and assigned manually. This can be done by generating a random md5, and inserting it into the token table, with an expiration date and eperson id
$ head /dev/random | md5 f36def40ade795ede0401c1f74144852 $ psql dryad_repo dryad_repo=# select eperson_id from eperson where email = 'dan.leehr@nescent.org'; eperson_id ------------ 6307 (1 row) dryad_repo=# INSERT INTO oauth_token (eperson_id, token, expires) VALUES (6307,'f36def40ade795ede0401c1f74144852','2014-12-31'); INSERT 0 1
Token Usage
Tokens can be supplied by the client in a few ways, according to the OAuth specification. The simplest is to include an access_token
query parameter in the URL. This works for both GET and POST requests.
For example:
https://datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852
Tokens can also be included as an HTTP request header
Authorization: Bearer f36def40ade795ede0401c1f74144852
Authorization & Access Control
A valid OAuth token that corresponds to an eperson account will identify the bearer (authentication), but a token alone does not identify what the bearer is authorized to do.
Resource Access control lists are stored in the rest_resource_authz
table. Each table row consists of an eperson ID, an HTTP method (GET/POST/PUT/DELETE), and a resource path (e.g. organizations/org1/manuscripts
). These rows determine what a user can access.
Note that granting access to a resource path is recursive - if a user has GET access to organizations
, he/she has read access to every organization and manuscript in the system.
As an example, suppose a partner journal user has eperson id 2860 and we've assigned organization code test. Their resource authorizations should probably be
eperson_id | http_method | resource_path 2860 | POST | organizations/test/manuscripts 2860 | PUT | organizations/test/manuscripts 2860 | GET | organizations/test
POST allows creation of new manuscripts within the test organization. PUT allows replacement of manuscripts (though the PUT would actually happen at organizations/test/manuscripts/:id), and GET allows them to read the organization and all of its sub-resources.
Note the user is not granted DELETE, and has no access anywhere outside of organizations/test.
Interaction with other components
These initial APIs are designed to handle exchange of manuscript metadata with partner journals and processing of changes. This job is currently done by the journal-submit webapp, which parses metadata emails from journals and stores them for the submission system.
Journal-Submit webapp
The journal-submit webapp parses emails and generates JSON documents to store in the database. When authors enter a manuscript in the submission system, Dryad reads their manuscript metadata out of the database, based on the entered Manuscript number.
The API webapp has been built to be compatible with this behavior. As manuscripts are created (POST) or updated (PUT), their JSON structure is stored in the manuscript
table.
This integration requires the organization code (in the organizations table) and journal code (in the concepts) to match. This is why the organizations table is constructed as a view. It simplifies the journal concepts into a simpler table with the concept_id (as organization_id) and some other concept metadata.
See
- dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptXMLConverterHandler.java
- dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/converters/ManuscriptToLegacyXMLConverter.java
Data Package Metadata
Upon creating (POST) or updating (PUT) a manuscript, the API webapp will attempt to locate a Dryad Data Package (provided the manuscript includes a dryadDataDOI
) and synchronize the following metadata:
- Publication DOI
- Manuscript Number
- Manuscript Keywords
- Manuscript Title
- Manuscript Abstract
- Publication Date
The values from the manuscript are copied into the data package metadata if the manuscript is submitted or accepted. If rejected, the values are removed from the metadata
See
- dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptMetadataSynchronizationHandler.java
Review workflow
Upon creating (POST) or updating (PUT) a manuscript, the API webapp will check the status (submitted/accepted/rejected) and transform the status into a review action if a corresponding data package (by dryadDataDOI
) is currently in review.
The ApproveRejectReviewItem class has been refactored to support this action
See
- dspace/modules/dryad-rest-webapp/src/main/java/org/datadryad/rest/handler/ManuscriptReviewStatusChangeHandler.java
API usage examples
Below are some examples using cURL to interact with the API. Remember, these require creating tokens and access control in the database:
Listing organizations - GET /api/v1/organizations
curl --insecure \ --request GET \ https://staging.datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852
Creating an organization - POST a JSON object to /api/v1/organizations.
curl --insecure \ --header "Content-Type:application/json" \ --request POST \ -d '{"organizationCode":"test","organizationName":"Dryad Test Organization"}' \ https://staging.datadryad.org/api/v1/organizations/?access_token=f36def40ade795ede0401c1f74144852
Creating a manuscript - POST a JSON object to /api/v1/organizations/{organizationCode}/manuscripts. Object must have manuscriptId, which will be the identifier for future operations
curl --insecure \ --header "Content-Type:application/json" \ --request POST \ --data-binary @manuscript.json \ https://staging.datadryad.org/api/v1/organizations/test/manuscripts?access_token=f36def40ade795ede0401c1f74144852
Updating a manuscript - change the data within the JSON and PUT to /api/v1/organizations/{organizationCode}/manuscripts/{manuscriptId}
curl --insecure \ --header "Content-Type:application/json" --request PUT --data-binary @manuscript-revised.json \ https://staging.datadryad.org/api/v1/organizations/test/manuscripts/MS12345?access_token=f36def40ade795ede0401c1f74144852
Example JSON documents are in dspace/modules/dryad-rest-webapp/src/main/resources
Listing manuscripts in an organization: can use filter query terms
curl --insecure \ --request GET \ https://staging.datadryad.org/api/v1/organizations/test/manuscripts/?access_token=f36def40ade795ede0401c1f74144852&count=10&search=Smith
Implementation of review workflow using REST API and Dryad-first option
For "Dryad-first":
- authors submit data packages as “in preparation or in review” and don’t enter a manuscript number (they won’t have one yet)
- data packages go to review (still have no manuscript number). Dryad sends review emails (including DOI) and review link to author and to journal
- author submits manuscript to journal, including the Dryad DOI and/or review link
- Journal sends API calls to Dryad on acceptance/rejection, including the Dryad DOI and/or review URL. Generally works like the email system - they specify a journal code (organization code) and a structured JSON object containing manuscript metadata
- Examples here: Metadata Submission via API
- API finds the data package that matches the incoming manuscript metadata (it looks it up by review URL, DOI, or manuscript number).
- In submit-to-dryad first, this needs to be review URL or DOI, which means this information must be included in the API call from the journal on acceptance/rejection
- API approves the submission from review to curation (when the status is “accepted”) and synchronizes the following metadata fields
- Publication DOI
- Manuscript Number
- Keywords
- Title (takes the title and prefixes it with “Data from: “
- Abstract
- Blackout until date aka publication date
Some other things to note:
- When we receive API calls we will probably not be receiving metadata emails. The JSON data is protected with OAuth, and we can grant access to Dryad personnel.
- The API will store the received JSON data in the database, but it will also make XML files just like the email system does - so these can be checked too. This supports the submit-to-journal first case
- We’re recording the publication date in the dc.date.blackoutUntil field (new field!). This date will be used to automatically publish things from blackout but this part is not yet functional.
- If the metadata sent by the journal is invalid or doesn’t match a data package in review, the API won’t be able to find the data package and it will stay in review. Something to be aware of.