DevCSI teamed up with the Open Knowledge Foundation for two days of hacking and sharing ideas about open bibliographic metadata at Queen Mary University of London.

The event provided an opportunity to hack with open bibliographic datasets, to experiment with new tools, and to help improve existing systems to provide new ways for institutions to benefit from bibliographic data.  It attracted a range of developers from the DevCSI community, bibliographic data specialists and librarians with a keen interest in making better use of open bibliographic data.

Workshops

Day one of the event saw a parallel stream of workshop sessions which provided a context for the practical work in the main hack room.

These sessions addressed the technical aspects of opening up cultural heritage data, examined some of the best open source tools available for doing that, and explored the best standards for preparing and exposing data for reuse.

Introduction to APIs and Linked Data

Click here to view the embedded video.

Preparing your Data for a Hackathon

Click here to view the embedded video.

Diverse Metatdata Standards and the Europeana Solution

Click here to view the embedded video.

Case Study: Cambridge University Library

Click here to view the embedded video.

Reuse of Open Cultural Heritage Data

Click here to view the embedded video.

Case study: The British Library

Click here to view the embedded video.

Lightning Talks

Participants in the hack element of the event shared their ideas and provided some background to their own work in the field in a series of lightning talks. These explored:

• Content and Data Mining and PDF extractor by Peter Murray Rust and Ross Mounce
• m-biblio Project by Mike Jones
• ORI/RJB by Ian Stuart
• Making a BibServer Parser by Etienne Posthumus
• IDFind: Identifying Identifiers by Emanuil Tolev
• BibServer: What we have been doing recently, how that ties into the open access index idea by Mark MacGillivray
• TEXTUS by Tom Oinn
• Pundit: Collaborative semantic annotations of texts by Simone Fonda
• Linked Data by Ian Stuart

Themes

Further discussion of these issues and the ideas shared via the event etherpad led to the realisation that many of the tools and solutions people wanted to work on throughout the event could be broadly brought under the umbrella of a Bibliographic Toolkit.  Under this umbrella, a number of distinct groups formed to investigate specific issues.

These included…

BibServer Group

This group began exploring how to unite tools several tools, including PubCrawler, BibServer and BibSoup, to create a tool for abstracting and displaying bibliographic metadata.  They planned to connect PubCrawler, which has collected 12 million bibliographic metadata records and will allow you to find out if papers are Open Access, with BibServer, which collects and displays the data, and BibSoup, which allows the creation of local community groups of references for specific interest areas. The group also planned to investigate how easy it was to get BibServer deployed on a variety of operating systems and document that process.  They hoped to test the tool using data from the German National Bibliography data, with a subgroup working on how to parse this data ready for use.

In this short video interview, Mike Jones from the University of Bristol describes his side project associated with this group, which involved getting the m.biblio app he is currently developing to connect to BibServer…

Click here to view the embedded video.

This video is also available on Vimeo.

TEXTUS Group

This group focussed on discussing potential roles and developments for TEXTUS, an open source platform for working with collections of texts and with collections of texts and associated metadata, including annotations.  The goal was to create useful documentation about how the tool can be developed and extended, rather than working code.

In this short video interview, Simone Fonda from Net7 outlines the progress the group made as a result of these discussions and explained how he sees TEXTUS developing in the future…

Click here to view the embedded video.

This video is also available on Vimeo.

Open Access Index Group

This group set out to build a list of all the journals in the world, including their access policies, where available. Their intention was to create a search facility to query this data, a form to help crowd source further data and updates, and an API to access this data. The intention is to create a tool that will enable them to get a better idea of what is available to support other work.

In this short video interview, Ian Stuart from EDINA describes the work of the group in more detail, and explains the value he feels developers can bring to the process of creating open access services…

Click here to view the embedded video.

This video is also available on Vimeo.

Useful Tools

Throughout the event, participants shared links and information about a number of useful tools.  Here is a quick reference guide to some of the tools mentioned:

BibServer

Github: https://github.com/okfn/bibserver

BibSoup

Link: http://bibsoup.net

TEXTUS

URL: http://textusproject.org

Github: https://github.com/okfn/textus

IDFind

Live here: http://idfind.cottagelabs.com/

Code here: https://github.com/CottageLabs/idfind/

Microsoft Academic Search

API documentation: http://academic.research.microsoft.com/

PubCrawler

Link: https://bitbucket.org/wwmm/pub-crawler

wget

Link: http://www.gnu.org/software/wget/manual/wget.html

PDFBox

Link: http://pdfbox.apache.org/

Final Outcomes

As the event drew to a close, each of the groups provided a short presentation to describe their progress:

Open Access Index Group

The group created an academic catalogue of journal titles, which they called ACat.  This was the first step of an ambition plan to create an index that will allow you to browse open access resources.  During the course of the event, they collected 55,000 journal titles in an elastic database, then created a front end based on Facetview to provide a searchable interface.  Whilst the journal title data was difficult to get, now they have it the group can query various APIs to determine licensing information to develop the project further.

Annotation Tools

Tom Oinn from TEXTUS explained how the event allowed him to gain a much better idea of where to take the TEXTUS project next and how to get it to play with other projects, including BibServer.

At the moment you can annotate texts using comments, but Oinn hopes to move towards any annotation being able to contain references. Notably, he hopes to move towards creating personal reading lists, which begin to look like BibSoup instances, and to start using TEXTUS as an annotation tool in its own right.

BibServer Group

The group spent a lot of time getting BibServers up and running on different architectures, noting that getting a high quality, distributable BibServer out is a high priority.  Their work to identify the issues with different set ups will help with this.  The group engaged in a number of discussions to explore possible connections with other tools. The big unpredicted outcome of this for this group was realising the value of linking BibServer with TEXTUS.

There were also several splinter projects from this core group, including work to connect BibServer and m.biblio, and efforts to add national bibliographic data from the UK, Germany, Spain and Sweden. The latter helped to identify the problem of character encoding, which is likely to crop up as BibServer gets used more outside the UK.  The developers will be insisting on UTF 8 for ingest from now on, as a result of this work.

You can watch the final group presentations in full:

Click here to view the embedded video.

This video is also available on Vimeo.

Participant Responses

There have been a number of blog posts documenting the technical details of the event if you want to know more:

Bringing the Open German National Bibliography to a BibServer by Etienne Posthumus and Adrian Pohl.

Software Cooperative News: Bibliohack London by MJ Ray

Bibliographic References in Textus by Tom Oinn

If you have blogged about this event, please leave a comment with a link to your post.

The Open Knowledge Foundation’s Open Biblio group, and Working Group on Open Data in Cultural Heritage, along with DevCSI, present BiblioHack: an open Hackathon to kick-start the summer months. From Wednesday 13th – Thursday 14th June, 2012.

We’ll be meeting at Queen Mary, University of London, East London, and any budding hackers are welcome, along with anyone interested in opening up metadata and the open cause – this free event aims to bring together software developers, project managers, librarians and experts in the area of Open Bibliographic Data. A workshop will run alongside the coding on the 13th, and a meet-up on the evening of the 12th is open to all whether you’re attending the Hackathon or not.

What is BiblioHack?

BiblioHack will be two days of hacking and sharing ideas about open bibliographic metadata.

There will be opportunities to hack on open bibliographic datasets and experiment with new prototypes and tools. The focus will be on building things and improving existing systems that enable people and institutions to get the most of bibliographic data.

If you’re a non-coder there are sessions for you too. We will be running a hands-on workshop addressing the technical aspects to opening up cultural heritage data looking at best of breed open source tools for doing that, preparing your data for a hackathon and the best standards for storing and exposing your data to make it more easily re-used.

When and where?

• The main hackathon will take place over two days between 13th and 14th June at Queen Mary University of London
• On the morning of the 13th June we’ll be running the workshop addressed at the technical challenges to opening up metadata. So for those unable to participate in the hack due to time constraints or lack of coding know how – this is for you!
• On the 12th June – Tuesday evening (details TBC but will be a pub in central / east London!) – we’ll also be hosting a meet-up for anyone attending the hack and open data more generally. Whether it’s open bibliographic data, spending or government data that floats your boat all tribes are welcome!

Who is organising the event?

• Open Biblio – emphasising the utilisation of tools we’ve developed. We will be focussing on our existing tools and software, the most recent updates of which are available here: front-end BibServer / BibSoup, BibJSON, back-end BibServer
• DevCSI – encouraging diverse thinking around software development and technical innovation
• Open Knowledge Foundation’s Working Group on Open Data in Cultural Heritage and Open GLAM
• DM2E – a new EU funded project devoted to enabling more cultural institutions integrate their collections into Europeana

Who else is involved?

We’ve already lined up a whole host of speakers and groups who’ll be attending both the hack and the workshop. The list so far includes UK Discovery, CKAN, Europeana, Total Impact, Neontribe, The British Library with many more to be added in the coming days…

You’re giving your time and expertise – what do you get if you attend the whole hack?

• Accommodation at QMUL overnight on the 13th
• Food and refreshments across the 3 days
• The chance to work with experts in their fields
• Admiration and respect from your peers
• We could expound at length, but… go on, you know you want to (it’s free!)

How can I sign up?

• Register here for the 2 day hack
• Register here for workshop only
• Register here for Meet-up only

Please note, if you wish to attend all 3 events you should sign up for each, and the Workshop will run in parallel with the hacking on the morning of the 13th.

More questions?

Contact Naomi Lillie on admin [@] okfn.org