Journal Integration Process

= Integration checklist =

Steps that Dryad must complete for a journal to be integrated:

Initial documentation
 send basic info (wiki documentation, Integration Overview slides) alert management and tech add journal to Salesforce and Trello (formerly Partner spreadsheet) (liaison/mgt) determine strategy; what issues are there, who can approve moving ahead etc. arrange conference call, ask them to look at/fill out Pre-Integration Questionnaire (PIQ), to walk through integration implementation steps and answer questions 

Get-to-know-you phone conversation (Executive Director and Operations Manager, occasionally Repository Architect)
 answer questions from journal about workflows, optional customizations, etc. get journal timeline for testing and rollout</li> obtain/confirm PIQ</li> goes to “in discussion” on Trello board</li> </ul>

Implementation phase (from PIQ --> successful completed test) - Technical Specialist

 * Set up Journal Concept on the production server
 * journal.integrated should be set to false.
 * journal.subscriptionPaid should be set to false or "SHOULD BE TRUE" (even if the journal will sponsor payments, this field must not be set to true until the journal is ready to assume payment from individual submitters).
 * Liaise with journal to verify that metadata emails are received and processed correctly (see Testing).
 * Finalize integration:
 * Set journal.subscriptionPaid to true if the journal will sponsor payments for their authors
 * Set journal.integrated to true in the Concept on production
 * Add journal to list of integrated titles on website
 * Obtain cover image/logo; add to folder on Dropbox
 * Ensure cover image is added to Dryad, both on the data package pages and in the "recently integrated" box on the homepage
 * Notify curation team (staff email list)
 * Add journal to list of completed journals.&lt;/li&gt;

Completion phase (after completed test --> rollout and ongoing maintenance)
<ol style="margin-top:0pt;margin-bottom:0pt;"> (Meredith) Concluding email with journal contact<ul style="margin-top:0pt;margin-bottom:0pt;"> tell journal they are officially integrated and we now expect to receive notices from them</li> tell journal about weekly curator reports; confirm recipients and add to Trello any updates since the PIQ was completed</li> check on policy for data references & updated journal instructions (Journal Instructions)</li> ask journal to inform us of any forthcoming changes to their manuscript system that will affect integration</li> invite journal representative to subscribe to the Announce and/or Users mailing lists</li> suggest journal use help@datadryad.org address for all urgent questions</li> offer statistics</li> explain Dryad membership, pricing plans, offer more information if not already done</li> ask permission to promote if applicable</li> </ul></li> (Meredith) Follow-up<ul style="margin-top:0pt;margin-bottom:0pt;"> PR: tweet &/or blog post</li> (liaison/dir) maintain at least annual contact to confirm that notices conform to specifications, discuss journal options, update contact information, solicit feedback, and encourage participation in the Dryad organization</li> </ul></li> </ol>

= Testing overview = <ul style="margin-top:0pt;margin-bottom:0pt;"> Set up a new journal concept on the Production server, with journal.integrated set to false. Journal settings come from the PIQ.<ul style="margin-top:0pt;margin-bottom:0pt;"> <li>NOTE: if a journal is integrating, wants to sponsor submissions once the integration is complete, and does not have an existing sponsorship in place, do not set journal.subscriptionPaid to true on Production until the integration is launched. (Journals can be sponsored without being integrated, so this will cause them to appear as a sponsored journal in the Shopping Cart report when they are not—yet).</li> </ul></li> <li>Dryad staff sends test metadata emails to journal-submit@datadryad.org and verifies success<ul style="margin-top:0pt;margin-bottom:0pt;"> <li>Format specifications for automated email notices to Dryad: Journal Metadata (structure differs from different MS processing systems)</li> </ul></li> <li>Journal sends test metadata emails to journal-submit@datadryad.org.
 * Format specifications for automated email notices to Dryad: Journal Metadata (structure differs from different MS processing systems)
 * sometimes journals use real articles in testing, sometimes they do fake ones</li>

</ul></ul>
 * The Dryad production server runs a webapp, journal-submit, that checks the Gmail account every minute for a specific label (“journal-submit”) and processes any messages with the label (see Journal Metadata Processing Technology for specifics)<ul style="margin-top:0pt;margin-bottom:0pt;">


 * currently, all new emails are automatically tagged with "journal-submit".


 * The journal-submit webapp processes each message:
 * It looks for the Journal Code and looks up the matching Concept. If no Journal Code is provided, it attempts to look up the Concept by Journal Name.
 * It parses the email based on the parser provided in the Concept metadata field journal.parsingScheme.
 * If the email fails to parse properly, the message is tagged with "journal-submit-error" in Gmail.
 * Parsed emails are written out as metadata into the manuscript table in the database.


 * Dryad staff checks the server for the new metadata
 * Manuscript metadata can be looked up either directly in the database or via the REST API.
 * Pro tip: set up a Gmail filter on journal-submit-app to notify integration staff of emails received from the new journal.
 * Once metadata looks good, staff turns integration “on” on Production

= Sample data =

Sample journal concept
When a new journal goes into integration testing, create a new journal concept with the following skeleton metadata:

ID   Field Name    Value 121    journal.journalID    {Dryad internal journal code} 122    journal.fullname    {Full journal name} 123    journal.metadataDir    /opt/dryad/submission/journalMetadata/{journalcode} 124    journal.parsingScheme    manuscriptCentral 125    journal.integrated    false 126    journal.allowReviewWorkflow    false 127    journal.embargoAllowed    128    journal.publicationBlackout    false 129    journal.subscriptionPaid    false 130    journal.sponsorName    131    journal.notifyOnReview    automated-messages@datadryad.org 132    journal.notifyOnArchive    automated-messages@datadryad.org 133    journal.notifyWeekly    automated-messages@datadryad.org 134    journal.paymentPlanType    136    journal.canonicalManuscriptNumberPattern    137    journal.customerID

Sample email record
Submit this message from your email account in plain text, addressed to journal-submit@datadryad.org. Indicate in the Subject line that Dryad is ready to receive test messages from this journal.

Make sure the Journal Code field has the proper code and the MS Reference Number/MS Dryad ID have unique names/numbers. This is saved as JSON data in the database: For these emails, what you're mostly looking for is odd formatting issues. Our system reads most emails and converts them appropriately, but sometimes the odd journal will format their author names weirdly.

Once you are satisfied the emails are okay, then you can send an email to the client letting them know our system is ready to receive their test emails.

Here’s a sample of an email I just sent to Jennifer at Physiological and Biochemical Zoology (you should send it to all contacts listed for journal integration questions on the PIQ) Hello Jennifer, Meredith Morovati asked me to reach out to let you know that I am handling setting up the test messages for Physiological and Biochemical Zoology. The Dryad system is now ready to test messages from Physiological and Biochemical Zoology so when you are able, feel free to send a few test messages to journal-submit@datadryad.org from the journal. As a reminder, the format specifications for the notices to Dryad are here: [http://wiki.datadryad.org/Journal_Metadata http://wiki.datadryad.org/Journal_Metadata

]The 6 mandatory fields are described there. The journal code specified in the Pre-Integration Questionnaire is PBZ, so we would expect to see your notices contain this field as follows: Journal Code: PBZ  Because these notices are processed automatically by our submission system and are not read by a human, these format requirements are important. Please let me know if you have any questions. Also, if possible, let me know when you send a test so I can be sure to check on its status in our system and let you know the results. It would be great if you could provide a variety of emails—some with a single author, others with multiple authors, all with abstract and MS number. Please let me know if you have any questions or concerns. Best regards, Debra Once you’ve received actual test messages from journals, repeat the steps 5-10 above.

Resolve issues if there are any. If not, you can contact journal and let them know we’re ready to go live with their integration. (Also ask for journal cover image if that hasn’t been handled before.)

Related Documentation / resources
<ul style="margin-top:0pt;margin-bottom:0pt;"> <li>Overview</li> <li>Submission Integration: Implementation</li> <li>Current status(includes checklist)</li> <li>PIQ</li> <li>Format specifications for automated email notices to Dryad: Journal Metadata</li> <li>inviting them to deposit data in Dryad</li> <li>Submission process from a journal's perspective: Integration Overview Slides (PDF).</li> <li>Blog post describing one journal's workflow for maximizing data archiving: How to decide what data should be archived at publication</li> <li>Journal Integration Trello Board</li> <li>Sample emails during various stages of integration</li> <li>Journal instructions: This page collects examples of journal instructions to authors that refer to Dryad, as well as journal policies about data availability, and suggests the critical elements for citations to data in Dryad.</li> <li>“All journals” spreadsheet (essential info about journals - add to Trello)<ul style="margin-top:0pt;margin-bottom:0pt;"> <li>Weekly curation report from Erin has “pink” titles. Add those new card on Prospects list of the Journal Integration Trello board. These titles were previously being placed on the all.journals spreadsheet. If *any* of those journals are Elsevier titles, you’ll need to send an email to Helena at Elsevier. Sample below.</li> </ul></li> <li>Journal check-in notesNotes from the 2015 Integrated journal check in</li> <li>Sample monthly emails to journals who pay for submissions</li> </ul>