User:Chrisftaylor/Miscellany/MappingAndRelevance

Requirements

 * Wholesale processing of all data sets’ metadata through HIVE (or an equivalent?) to obtain controlled vocabulary tags (noting that typographical errors will remain an issue).
 * Construction (weekly) and updating (nightly) of a graph whose nodes are the set of all terms assigned in the HIVE-based analysis; each node listing all data sets assigned that tag.
 * Querying of the graph by data set(s) to find related data sets (i.e., those sharing one or more nodes with the query data set, ranked by number of nodes shared).
 * Finding related data by selecting HIVE-assigned terms from the results set in a Wordle to contruct a new tag-based query.

Back-end processes
Phase three requires significant additions to the back-end functionality supporting Dryad Search. Arrow #3 in the diagram for the Phase One back-end implementation now has an additional component: as metadata on data sets are collected during (overnight) indexing, all metadata from each data set are sent in parallel to a HIVE-based service that assigns all the tags all that it can (based on a fixed set of vocabularies in specific versions). These tags are then used to build a graph of all HIVE-assigned terms, each term linking to a list of all the data sets to which it has been assigned. That graph will then permit searches for related data sets, where a selected data set’s HIVE-assigned terms (ideally listed unobtrusively below the main search result, perhaps made individually clickable) are matched against all other data sets; the ‘hits’ being ordered by the number of tags they share with the query data set and returned to the user.

User interface
The image below shows an example of a Wordle. In the scenario considered, these terms would be those assigned by HIVE to all the data sets in the query results set. Selecting terms from the Wordle (then clicking ‘Go’) would allow users to formulate tag-based queries they may not otherwise have conceived. "http://biology2.sciencecommunity.wikispaces.net/file/view/climate%20change%20wordle.JPG/354121198/800x349/climate%20change%20wordle.JPG"