MRD Hack Day Ideas
Thanks to all of those who contributed ideas for the MRD Hack Day before the event using the comments on this post. We summarised these during the event and used them as the basis of our brainstorming session.
Here is a complete record of all the ideas shared on the ideas wall during the brainstorming session:
SWORD related
- Implementing SWORD server
- Create experiment with SWORD (Tim Brody)
- Post real time data using HTTP Range (Tim Brody)
- Finish experiment with SWORD (Tim Brody)
- RSS and XML Time Sequences
- ATOM Pub publishes to feeds
- Sending Data by reference
- Allowing other protocols for transferring data to the server.
- Server a file retrieval co-ordinates
- DSapce SWORD large data ingest
Large data
- DSapce SWORD large data ingest
- Ingesting large data sets (Ian Wellaway and Dan Small)
- Moving big data – any ideas? (Jon Besson)
- Need a walk through of options to store files (5Gb) in a native format with a link accessible via a network share (Alex Henderson)
Metadata
- Aggregator of research data metadata across institutions (OER Xpert for research) – (Lindsay Wood)
- Managing ‘anecdotal’ data vs quantitative data…? Metadata explanations: User metadata is different to actual metadata…explore (John Nicholls)
- How to ease the burden of adding metadata to data in subject areas where adding metadata is not the cultural norm? Policies? Tools? (Aaron Turner)
- Online metadata designer tool to work with data deposit.
- Propose a common metadata schema for repositories (e.g. institutional and other repositories share metadata, data held in one place only) (Brian Hole)
- Exploring DataCite compliant metadata from Eprints, DSpace, CRISes etc (Alex Ball)
- How best to deal with granular metadata, displaying for researchers (Alexis Wolton)
- Metadata gets mapped to Data. This metadata is corrected and produces a second version this is mapped to the original data source and a corrected data source (Gerard Devine)
- Devise effective peudonomisation algorithm to identify patient data
Identifiers
- Tracking impact of data through identifiers
- ORCID and DataCite Interoperability (Researcher and dataset identifiers)
- Devise really simple unique ID generator
- DOIs – quickest and best method to mint DOIs from eprints (Alexis Wolton)
- Minting URIs to locate remote files (also just minting URIs) (Jon Besson)
Other
- Strategies for displaying groups of datafiles in e-prints (Alexis Wolton)
- Examining Automatic Data Capture from instruments (Kathleen Haigh-Hutchinson)
- Concept mapping / visualisation of researchers, research projects ad outputs (papers / data) (Lindsay Wood)
- Look at ways to remotely access / interrogate data in a repository (e.g. .csv, .txt, .doc, .docx etc), prototpye (Brian Hole)
- Breadcrumbs data search
- Online tools for documenting
- Differences between arts and sciences data (Tom Parsons)
- Neat and rapid way to get different software packages to use shared ontologies (i.e. code sets)
- Integrate heterogeneous applications on Cloud like CiViCRM, CA Tissues, Onyx and I2B2 (University of Leicester)
- Electronic signatures / proof issues / validation of data submitted in a repository (Tom Parsons)
- Workflow for depositing final research data (Metadata requirements, mint dois (at which stage), documentation) (Tom Parsons)
- Object Stores – what is the minimum necessary to turn an about store (Amazon S3) into a data repository (Jez Cope)
- Digital Object Stores, Mongo Db, Couch Db (Jon Besson)
The Orbital Project
By Joss Winn
The Orbital project is developing a pilot system for Engineering researchers to manage their data. Already, researchers are asking us where can they put their data so that it can be immediately cited (the urgent motivation for this is for funding bids and preparation for the REF).
How should we quickly mint URIs for public data that will be permanently citable, but flexible enough to change with our developing system? Can the DataCite API help us right now?
If so, we’d like to use the MRD hackdays for implementing this feature.
I’m interested in finding out more about allocating identifiers via DataCite.
If not, the Handle system could be used, as is the case with DSpace.
For data citation purposes, it is also useful to provide an automatic suggested citation out of the metadata fields (preferably required ones) of the deposited item. Edinburgh DataShare does this, based on the fields:
Data Creator(s): Family Name, Given Name/Initials; Family Name, Given Name/Initials. (Date Accessioned – YEAR ONLY). Title, Time Period [Item Type]. Data Publisher / Funder.
A DOI or handle can be added as well. This makes it easy for a user to do the right thing and include it with the rest of their citations.
By Joss Winn
The Orbital project is developing a pilot system for Engineering researchers to manage their data. Part of our work so far has been on aggregating staff profile data so that Orbital knows about a user when they log in.
We’d like to assign a unique ID for each researcher who uses Orbital and have this in place for the first release in May.
Is ORCID the way to go for this?
What other people IDs should we be building into Orbital? Is ORCID ready for use now?
What can we implement over the two days of the MRD hack event?
Dealing with real-time instrument data
Who is interested in working on hacks in this area? Please reply to this post with your ideas.
We are working in this area at RAL, STFC (see the above link), so we are interested in getting to know others who also work in this area, i.e. dealing with real-time instrument data in laboratories!
A few areas we are working on: 1) characterisation and visualisation of real time data; 2) continuously monitoring and tracking of experiments; 3) sample registration and tracking; 4) automated data reduction and analysis; 5) data provenance, data publication, and semantic data repository; 6) scalability testing of laboratory data infrastructure; 7) data citation for aggregated datasets
If you are working in these areas, please get in touch (erica.yang@stfc.ac.uk). I hope we can form a working group in one of these areas!
We are working in this area at RAL, STFC (see the link), so we are very interested in getting to know others who also work in this area, i.e. dealing with real-time data in laboratories!
A few areas we are working on: 1) characterisation and visualisation of real time data; 2) continuously monitoring and tracking of experiments; 3) sample registration and tracking; 4) automated data reduction and analysis; 5) data provenance, data publication, and semantic data repository; 6) scalability testing of laboratory data infrastructure; 7) data citation for aggregated datasets
If you are working in these areas, please get in touch (erica.yang@stfc.ac.uk). I hope we can form a working group in one of these areas!
Are there any real-time instrument data feeds that we can use for the hack day?
I would like to tie this into SWORD 2, for auto-feeding data into (data) repositories.
Translating OAIS into user stories.
Idea by Joss Winn
Identify people who have investigated OAIS as a data model for datasets.
Mapping to a top level generic DC based description of data sets.
Possibly get Julie Allinson to speak about her work?
As for OAIS and datasets, the UKDA did a thorough report on how their processes mapped to OAIS (and METS) in 2004 (Hilary Beedham). See
http://www.data-archive.ac.uk/media/1692/OAISMETS_report.pdf
The DCC’s Data Asset Framework has some relevance here as well.
http://www.dcc.ac.uk/resources/tools-and-applications/data-asset-framework
Using generic level DC metadata to describe data sets
Use the DC scoping study developed by Alex Ball, particularly chapter 5, which outlines a generic DC description, see http://www.ukoln.ac.uk/projects/sdapss/
Chapter 5, Existing metadata standards and profiles, http://www.ukoln.ac.uk/projects/sdapss/papers/ball2009sda-v11.pdf
Is there any scope to develop a tool which would map to generic dataset metadata?
I’m interested in finding out more about metadata standards and tools.
Other examples include the Edinburgh DataShare metadata set,
https://www.wiki.ed.ac.uk/display/datashare/metadata
and the DataCite Metadata Kernel http://schema.datacite.org/meta/
and the Dryad metadata application profile,
http://wiki.datadryad.org/Metadata_Profile
(apols if these are included in your links above)
rifCS
Are there any people interested in doing things around this standard, please reply to this posting.
CERIF compatibility
Are people who are working with CERIF or are interested in working with it like to identify some possible hacks in this space?
I’m interested in finding out more about CERIF.
Implementing SWORD2
Who is implementing SWORD 2 for datasets, are there people interested in working around this area? Please reply to this posting.
We’re (University of Bath) interested in using SWORD2 to provide a straightforward deposit interface to multiple repositories from our Sakai-based VRE.
I’m interested in finding out more about SWORD2.
OSS Watch is part of the DataFlow project, which involves two software projects:
DataStage is a secure personalized ‘local’ file management environment for use at the research group level, appearing as a mapped drive on the end-user’s computer.
DataBank is a scalable data repository designed for institutional deployment.
You can push datasets from DataStage to DataBank using Sword.
I am not a developer on DataFlow myself, but I would be interested to see if we can connect DataStage and DataBank to other repositories.
We’ve succesfully sent packages to DSpace from DataStage (we have a custom bagit ingester which may be required for this). The tweak we’d like to see is the ability to select a DSpace Collection (~ DataBank silo ?) for the target.
We also have a packaged Python SWORDv2 client:
https://code.soundsoftware.ac.uk/projects/sworduploader
[NB: I'm not the developer for these!]
[EDIT: So far we've managed to upload metadata to DSpace from DataStage, but not data!]
In EPrints we have gone beyond SWORD by implementing full HTTP content negotiation. This allows items to be added/updated/removed using AtomPub, using formats supported via plugins. Happy to share what we’ve been up to!
Identifiers: ORCID API, Datacite/DOIs (BL DataCite API)
Are there people interested in working with identifiers? Please reply to this posting.
I’m interested in finding out more about identifiers.
We, in the PIMMS project, are interested in identifiers and how they may possibly be used to map metadata to data.
DataCite resources include:
OAI-PMH Data Provider which exposes the metadata store
http://oai.datacite.org/
DataCite Content Service which exposes the metadata store in various formats
http://data.datacite.org/
DataCite Search API
http://search.datacite.org/ui We use Solr Search Handler for our API calls, the endpoint is: http://search.datacite.org/api.
Please check Solr’s common query parameters documentation in order to understand how to use API
Examples
http://search.datacite.org/api?q=wind simple search for wind
http://search.datacite.org/api?q=wind&fl=doi,title&rows=5 search for wind, retrieve only doi and title, and return (at max.) 5 results
http://search.datacite.org/api?q=wind&fl=doi,title&wt=csv csv output
http://search.datacite.org/api?q=wind&fl=doi,title&wt=json&indent=true json output
DMPOnline APIs.
Is anyone interested in working on these?
Object stores
Is there anyone interested in this?
We’re (University of Bath) interested in how an object store such as the Hitachi Content Platform can be used to store both live research data (via a NAS interface – CIFS/NFS/etc) and archived data (via e.g. a web-based repository-type interface).
NoSQL
Is there anyone interested in this?
We are interested in use of NoSQL – specifically MongoDB. We’re interested in what others think of the technologies, seeing it in action, how it might be used in conjunction with a traditional RDBMS.
We’re using MongoDB at Lincoln to do all kinds of voodoo, and I’d be happy to share our experiences with it etc.
OAuth
Is there anyone interested in this?
Yes, I’m interested in oauth. Are their any apps that need oauth adding to them?
ODIN Project: ORCID and DataCite Interopability Network
This is an EU funded project starting in Sept 2012 with DataCite, ORCID, British Library, CERN and non-EU partners. British Library have been charged with completing a proof of concept for Social Science data and we will be recruiting to a project post in Summer 2012.
We have chosen to look at Birth Cohort Studies research data (as they go back to 1946) and to track the impact of that the data and their authors by tracking publications, authors, researchers, secondary analysis, derived data and policy impact.
I’m interested in seeing if there are any possible technical development solutions to this proof of concept as I work up our approach.
I am interestyed in all things data management. More specifically, I would like to explore ideas relating to documentation, documenting, technical authoring and online interactive resource developmnent around data management training, teaching and planning. Does anyone have an intersecting line of interest?