Difference between revisions of "Monthly Reports"

From Dryad wiki
Jump to: navigation, search
(Monthly Updates of Dryad-Data)
m (Monthly Shopping Cart Report)
Line 30: Line 30:
 
== Monthly Shopping Cart Report ==
 
== Monthly Shopping Cart Report ==
  
'''This report is partially automated.''' The current script is in https://github.com/datadryad/dryad-data/tree/master/monthlyReporting/monthly-report.sh, but it is incomplete.
+
'''This report is automated.'' The Python script is located in https://github.com/datadryad/dryad-data/tree/master/monthlyReporting/monthlyReport.py.
  
 
On production, export CSVs using commands:
 
On production, export CSVs using commands:
Line 36: Line 36:
 
cd dryad-data/monthlyReporting
 
cd dryad-data/monthlyReporting
 
git pull
 
git pull
./monthly-report.sh
+
./monthlyReport.py
 
</pre>
 
</pre>
  
 
Then:
 
Then:
# Import filtered file (shoppingChargedFiltered.csv) into google spreadsheet.
+
# Create new blank spreadsheet in Google Drive (https://drive.google.com/open?id=0BzWuiqo-UGbqTUFlck1PamMxcDg).
# Copy/paste headings from previous report.
+
# Import report CSV file (name is monthlyReport followed by date in YYYYMMDD format, for example monthlyReport20170101.csv) and verify that column headers and content match up as expected.
# Remove three columns before the journal name -- journal names should line up with the headings.
 
# Move the last_mod_date and archive_date to the right, above the columns of dates.
 
# Remove columns that do not have headings.
 
# Filter spreadsheet to remove items that have a payment date beyond the correct month. (In the future, this should be accomplished by restricting the Postgres queries, as long as all carts now have dates.)
 
# After filtering is complete, copy the first column (cart ID) into https://github.com/datadryad/dryad-data/tree/master/monthlyReporting/shoppingCartIdsSeenInReports.txt. Sort this file numerically (sort -n). Push the changes to github.
 
 
# Select the entire spreadsheet. sort by journal, subsort by archive_date.
 
# Select the entire spreadsheet. sort by journal, subsort by archive_date.
 
# Share the spreadsheet with the appropriate people.
 
# Share the spreadsheet with the appropriate people.

Revision as of 22:40, 30 April 2017

Monthly Update of Sitemaps

To update the sitemaps used for CLOCKSS harvesting, login to the production server and run

sudo /opt/dryad/bin/generate-sitemaps

This process should be automated, but we had a problem with it producing incorrect output when running from a cron job, so we currently run it manually.

Monthly Updates of Dryad-Data

First, get all of the deposit dates. On the production server:

postgres-client.sh -c "select text_value from metadatavalue where metadata_field_id=12 and item_id in (select item_id from item where owning_collection=2 and in_archive='t');" > dryadSubmitDates.txt

(collection 2 is "Data Packages" and metadata field 12 is "Date Available")

Edit the file to remove the timestamps. Leave only the date portion. This can be done using the sed command as follows:

sed 's/T.*//' dryadSubmitDates.txt

Sort the dates.

In the dryad-data GitHub repository:

  • Add the updated dryadSubmitDates.txt
  • Count the number of new packages for the given month (grep for the year-month combination and pipe to wc)
  • Manually edit the dataPackagesInDryad.csv file to include the new month.

Monthly Shopping Cart Report

'This report is automated. The Python script is located in https://github.com/datadryad/dryad-data/tree/master/monthlyReporting/monthlyReport.py.

On production, export CSVs using commands:

cd dryad-data/monthlyReporting
git pull
./monthlyReport.py

Then:

  1. Create new blank spreadsheet in Google Drive (https://drive.google.com/open?id=0BzWuiqo-UGbqTUFlck1PamMxcDg).
  2. Import report CSV file (name is monthlyReport followed by date in YYYYMMDD format, for example monthlyReport20170101.csv) and verify that column headers and content match up as expected.
  3. Select the entire spreadsheet. sort by journal, subsort by archive_date.
  4. Share the spreadsheet with the appropriate people.

See Also