Open Data Benchmarking: 3 Approaches

Open Data Barometer

License: CC-BY-SA / Data by Tim Davies / Visualisation by David Tarrant

In some areas of the world Open Data has already entered the public and political agenda several years ago. Ubiquitous examples are the United States of America and the United Kingdom. In other countries the move towards Open Data has been – gently speaking – cautious. Germany is a good example for the latter, as its national data portal finally soft-launched in early 2013. However, current negotiations for an upcoming Great Coalition give hope for a more profound integration of Open Data policies within the next legislative period, as the recently published draft of a position paper on digital topics revealed (from line 209 onwards).

The pathways for Open Data to enter one or another political agenda are without a doubt diverse. One of the rather established tools used by civil activists, academics and other interested parties is transnational and trans-sectoral benchmarking. Within this blog post I will present three actual benchmarking projects, which differ in scope and methodology, but share a normative impetus.

Open Data Barometer

The Open Data Barometer is a transnational index assembled by the World Wide Web Foundation and the Open Data Institute under the supervision of Tim Davies. It was published for the first time in October 2013, accompanying the more established and much broader Web Index. The Open Data Barometer monitors Open Government Data in 77 countries and is structured in three sections: Readiness, implementation and impact.

The index is based on three data categories: The first one consists of peer-reviewed expert survey responses on the “context” and “impact” of Open Government Data within the relative countries. Each question asked for a response on a 0-10 scale backed by a textual justification. The second part of the data is a detailed assessment of 15 particular datasets – like census, transport or budget data. For the purpose of data triangulation secondary data from five sources was used in a third and final part of the data collection.

Along with publishing the final report, the Open Data Barometer makes clear that not only Open Data initiatives are new and kind of “beta” but the associated assessment methods , too. Therefore, as the team states, the 2013 Open Data Barometer is focused on piloting methods, building towards further iterations of the methodology and survey in 2014 and beyond. Dave Tarrant at the Open Data Institute has drawn on the open data published from the Barometer to put together an interactive tool for looking at it across all the countries included in the study.

I like the tool and already crave for the visualizations that will be possible when future data allows longitudinal comparisons.

Open Data Index / Census

The Open Data Index is an Open Data benchmarking tool run be the Open Knowledge Foundation and was – like the Open Data Barometer – launched in October 2013 just before the Open Government Partnership summit in London. The Open Data Index provides a peer-reviewed assessment of openness in a range of ten key datasets, reflecting the realities of public information release and practice in 70 countries. The results of the Open Data Index are based on the Open Data Census which in turn was launched in April 2012 to coincide with the OGP meeting in Brasilia. Whilst the Open Data Census is designed as an ongoing collection, the Open Data Index is more like a snapshot at a time.

The Open Data Index is compiled in a crowd-sourced manner using contributions from civil society members and Open Data practitioners around the world. Contributors (who may be citizens, experts, officials, journalists, or anyone else) submit information about the availability of key datasets in their country. These submissions are reviewed by a community of country editors around the world. Country editors either approve (with or without amendments) or reject the submission. If approved, these submissions become an official entry and are displayed on the website.

The questions for each dataset have different weighting factors ranging from 5 (e.g. Does the data exist?) to 30 (e.g. Is the data openly licensed?). The country score is found by adding up all the dataset scores per country. The underlying data for the country index is available in bulk in CSV and JSON formats from datahub.


The Digital Openness Index [do:index] is a multi-dimensional measure of the contribution of public bodies to the digital commons, which is headed by Prof. Leonhard Dobusch at Free University Berlin, who at the same time supervises my PhD research. In contrast to the Open Data Barometer and Index, the [do:index] has a smaller geographical scope (only Germany, Austria and Switzerland) but targets more administrative layers (city level, regional and nationwide level) and is designed across different sectors of digital openness. The first results will be published in early 2014 and are planned to be updated annually.

The index consists of five sub-rankings to tackle different types of initiatives: Open Data, Open Education, Open Policies, Open Source Software, and Open Infrastructure. After a townhall-like open planning phase, a core team of 15 experts – mainly from NGO’s associated with the digital commons like FSF, CCC or OKF – shaped the scope and design of the index. In late summer 2013 the data collection in 48 local and regional authorities started and is still ongoing. The sample comprises of the five largest cities in Austria and Switzerland, the 16 German federal states and there capitals, as well as six other cities with more than half a million inhabitants.

In comparison to the Open Data Barometer and the Open Data Index, the [do:index] does not use domain experts in the first place but developed a web survey form, which allows the local authorities to answer the questions themselves. Only if the authority is not responding, not willing to cooperate or the person in charge can’t be identified, the [do:index]-team engages in web research to ranks the authority from outside.

[Disclaimer: I was working as a reviewer for the Open Data Barometer, did some minor edits within the Open Data Census and I will probably be engaged in the 2014 version of the do:index.]