Difference between revisions of "TreeBASE Submission Integration"

From Dryad wiki
Jump to: navigation, search
(saving so I don't lose anything)
Line 1: Line 1:
−
  '''Status:''' Initial implementation is complete,  
+
  '''Status:''' Initial implementation is complete, but it will continue to be enhanced.
−
but it will continue to be enhanced. (This page
 
−
needs to be updated with current implementation details)
 
  
 
== Overview  ==
 
== Overview  ==
Line 9: Line 7:
 
TreeBASE content will be searchable through Dryad, even if the author has not explicitly included the content in a Dryad data package.
 
TreeBASE content will be searchable through Dryad, even if the author has not explicitly included the content in a Dryad data package.
  
−
This integration will be based on the following technologies:
+
== Workflow ==
−
 
 
−
*[https://wiki.ucop.edu/display/Curation/BagIt BagIt] -- A lightweight format for packaging digital content and ensuring that it is transferred intact.
 
−
*[http://www.openarchives.org/pmh/ OAI-PMH] -- A protocol developed by the digital library community to allow harvesting of metadata from remote repositories.
 
−
 
 
−
We are evaluating the [http://purl.org/net/sword/ SWORD] protocol to manage the transfer of BagIt packages, but we have not yet determined whether SWORD will be lightweight enough to justify its use.
 
−
 
 
−
== Relevant Text from the Grant Proposal  ==
 
−
 
 
−
*"[handshaking] so that, where required by the journal or requested by the author, data will simultaneously be deposited in Dryad and... TreeBASE."
 
−
*"Dryad will collect any metadata required by the target database that has not already been captured, submit the pertinent data to the target database using a non-interactive programmatic gateway, and obtain the submission status, accession numbers, or possible error messages from the target database."
 
−
*"For TreeBASE, we will design and implement a robust, web-service based submission Application Programming Interface (API). An extensive redesign of TreeBASE by the CIPRES project (www.phylo.org) is scheduled for release in 2007. However, it currently lacks a submission API. The software to be added will include the automated data validation steps that are part of the new TreeBASE submission process (e.g. validating the NEXUS format, matching terminal taxa against the uBio NameBank). When TreeBASE rejects a submission, the depositor will be notified, advised how to correct the problem, and asked to resubmit. "
 
−
 
 
−
== Hanshaking from Dryad Perspective ==
 
−
 
 
−
There are three possible ways that Dryad may integrate with TreeBASE.
 
  
−
=== User submits to Dryad first  ===
+
[todo: user description and pictures here]
−
 
 
−
#User submits Nexus to Dryad and selects to send the file to TreeBASE in the final stage of submission.
 
−
#Dryad pushes object to TreeBASE. (This is before the object is curated in Dryad)
 
−
##Metadata (converted to Dryad Application Profile) and all uploaded nexus files are packed into a BagIt package and pushed to TreeBASE
 
−
##TreeBASE has a PUT RESTful service for receiving data
 
−
##TreeBASE only accepts the PUT if the sender's IP is within the Dryad range
 
−
##TreeBASE responds by returning an URL
 
−
#Dryad emails the user to inform that file(s) have been uploaded to TreeBASE and includes a URL to start submission at TreeBASE
 
−
#User goes to TreeBASE and completes record.
 
−
##Upon the user logging in, TreeBASE is triggered to unpack the BagIt package and create a submission based on the contents
 
−
#Dryad harvests TreeBASE content and picks up any additional relevant metadata or updated files.
 
−
 
 
−
=== User submits to TreeBASE first, links from Dryad record  ===
 
−
 
 
−
#User submits/edits package in Dryad and includes a TB ID
 
−
#Dryad adds TB ID to the record and creates a link to the HTML view of the TB record
 
−
#Dryad curator checks link to confirm that TB ID looks valid (we decided to leave ID in record if the link isn't active yet but looks like a valid TB ID).
 
−
#General link checking software will be run to catch dead links that have been dead for awhile.
 
−
 
 
−
=== User submits to TreeBASE, Dryad harvests only  ===
 
−
 
 
−
*Treat as any other harvested content (second-class)
 
  
 
== Process for completing a submission within TreeBASE ==
 
== Process for completing a submission within TreeBASE ==
Line 93: Line 54:
 
** user can leave items as "in progress" as long as they want -- this is a "poor man's embargo" system
 
** user can leave items as "in progress" as long as they want -- this is a "poor man's embargo" system
  
−
== Handshaking from the TreeBASE Perspective<br>  ==
+
== Technical Documentation ==
  
−
#Dryad offers an option to upload files&nbsp;to TreeBASE for Dryad submissions that have phylogenetic data
+
This integration will be based on the following technologies:
−
#Choosing this causes Dryad to send a BagIt package to TreeBASE via rest service: "http://www.treebase.org/treebase-web//handshaking/dryadImport"
 
−
#The REST service stores the BagIt package and returns a unique URL as a string, resembling: "~/treebase-web/login.jsp?importKey=123456789"
 
−
#Dryad emails this URL to the user and asks the user to follow it so as to finish the submission on the TreeBASE website
 
−
#Whoever follows the URL will have the ownership of the data. The user is confronted with a login page, including the option of creating a new account. If a new account is required, so long as the user remains within the same session (i.e. does not start a new browser), the BagIt is unpacked and the contents become parked under a new submission that is created in the user's account.
 
−
#The user can open the new submission and continue filling in information. The new submission should have citation information already filled in and any NEXUS files in the BagIt will have been parsed and stored.
 
−
#TreeBASE does not make content available until the associated article is published.
 
  
−
== Dryad Script to Initiate Handshaking from the Command Line  ==
+
*[https://wiki.ucop.edu/display/Curation/BagIt BagIt] -- A lightweight format for packaging digital content and ensuring that it is transferred intact.
 +
*[http://www.openarchives.org/pmh/ OAI-PMH] -- A protocol developed by the digital library community to allow harvesting of metadata from remote repositories.
  
−
There is a script that exports Dryad data packages and requested data files into the BagIt format and sends the BagIt package to the TreeBASE web service. The script is implemented as DSpace package disseminator and can be run through the standard DSpace disseminator script. For example:
+
We are evaluating the [http://purl.org/net/sword/ SWORD] protocol to manage the transfer of BagIt packages, but we have not yet determined whether SWORD will be lightweight enough to justify its use.
−
 
 
−
/opt/dryad/bin/dspace packager -d -i 10255/dryad.630 -e submitter@email.org -o xwalk=DRYAD-V3 -o repo=TREEBASE -o 'files=10255/dryad.631;10255/dryad.632;10255/dryad.633;10255/dryad.634' -t BAGIT -
 
−
 
 
−
'''/opt/dryad/bin/dspace''' = the standard DSpace interface to calling scripts from the command line; this must be run as the same user as the Tomcat serving DSpace is run as.
 
−
 
 
−
'''packager -d''' = the specific DSpace packager script; the -d tells it to run the packager to disseminate (there isn't a bagit ingester yet, but -i would be used if there were).
 
−
 
 
−
'''-i 10255/dryad.630''' = the Dryad data package's pseudo-handle (that DSpace uses as an internal identifier); this must be the pseudo-handle of a data package, not a data file.
 
−
 
 
−
'''-e submitter@email.org''' = the email address of the person who is making the submission to Dryad; we have this because someone must be logged in in order to submit to Dryad.
 
−
 
 
−
'''-o xwalk=DRYAD-V3''' = this is a hard-coded value at the moment (will always be the same); it says to export the metadata stored in DSpace in the Dryad Application Metadata Profile format (version 3 is the latest version).
 
−
 
 
−
'''-o repo=TREEBASE''' = this is also hard-coded for TreeBASE; repo just indicates the remote repository to which the bagit file should be uploaded to (in the future, Dryad may send BagIt files to other repositories as well).
 
−
 
 
−
'''-o files=10255/dryad.631;10255/dryad.632;10255/dryad.633;10255/dryad.634''' = the list of data files, associated with the data package, that the submitter wants exported to TreeBASE; it may not be all the data files, but all the files of a particular type (for instance, in the case of TreeBASE, Nexus files). For calling the script from the command line, this argument will probably have to be enclosed in single quotes (see the example above).
 
  
−
'''-t BAGIT''' = the package disseminator that should be used to export the files; in the case of TreeBASE, this is always hard-coded to BAGIT because that is the format that the TreeBASE importer supports.
+
More detail on the TreeBASE/BagIt handshaking can be found on the Dryad [http://code.google.com/p/dryad/wiki/BagItHandshaking technical documentation wiki]
  
−
'''-''' = This indicates that output should be sent to the command line; currently, the script follows the UNIX philosophy that 'no response' is an indicator of success (optionally, the code can be uncommented to allow the output of 'success' on a successful export). If there is a problem, an exception will be thrown and an email with details will be sent to the address configured as the DSpace admin.
 
  
 
See also: [[TreeBASE OAI Provider]]
 
See also: [[TreeBASE OAI Provider]]
  
 
[[Category:Other Repositories]]
 
[[Category:Other Repositories]]

Revision as of 09:59, 19 April 2011

Status: Initial implementation is complete, but it will continue to be enhanced.

Overview

Authors who submit content to TreeBASE or Dryad will be have the option to make their content appear in both systems.

TreeBASE content will be searchable through Dryad, even if the author has not explicitly included the content in a Dryad data package.

Workflow

[todo: user description and pictures here]

Process for completing a submission within TreeBASE

Minimum Requirements

  • nexus file
    • at least one tree OR at least one matrix
    • if there is a tree and a matrix, the taxon labels must match up.
    • must be "understood" by Mesquite
  • citation
  • analysis info linking matrices and trees

Detailed Process

  • create account
  • login
  • create new submission
  • type title
    • the submission gets a PURL at this point
    • the PURL can have a code added for reviewer access
  • fill in citation
    • minimum: year, title, journal name (or book/section title)
    • journal names auto-suggest as you type
  • add authors
    • minimum: at least one author (with first name and last name)
    • must always search for an existing author first, even if you know they're not in the system
    • allows reordering or deleting authors while you're in the process
  • upload file(s)
    • minimum: must be nexus, as described above
  • (optional) add notes
    • this is a textarea, with a reasonable character limit (not enough for a readme file)
  • (optional) edit details for matrices
  • (optional) edit row segment template
    • minimum: row ID, start index, end index
  • (optional) provide more details for trees
  • (optional) taxa
    • match all named taxa against ubio or ncbi
    • although the cleanup is optional, the TB editor may reject it if it's not cleaned up
  • analysis
    • minimum: create an analysis with at least one step. Typically, this will be a matrix that is processed to create one or more trees.
    • minimum: otu labels must match in the analysis steps
  • when initial submission complete, user clicks "change to ready state"
    • this triggers the curator to look at it
    • user can leave items as "in progress" as long as they want -- this is a "poor man's embargo" system

Technical Documentation

This integration will be based on the following technologies:

  • BagIt -- A lightweight format for packaging digital content and ensuring that it is transferred intact.
  • OAI-PMH -- A protocol developed by the digital library community to allow harvesting of metadata from remote repositories.

We are evaluating the SWORD protocol to manage the transfer of BagIt packages, but we have not yet determined whether SWORD will be lightweight enough to justify its use.

More detail on the TreeBASE/BagIt handshaking can be found on the Dryad technical documentation wiki


See also: TreeBASE OAI Provider

Retrieved from "http://wiki.datadryad.org/index.php?title=TreeBASE_Submission_Integration&oldid=12151"