Category: Open Government

Background

As part of its Open Government Initiative, the Ontario Government has issued an Open Data Directive that requires provincial Ministries and the Treasury Board Secretariat establish and publish an Inventory and Catalogue of government-wide data.

Application Program Interface (API)

The Ontario Government describes the Data Catalogue as a “one-window central platform” that provides faster and easier access to Government Data, thereby facilitating evidence-based policy, informs service delivery and promotes greater transparency and accountability.

The Data Catalogue’s Application Program Interface (API) helps users identify and access potentially relevant datasets with features that support:

• searching (free-text);
• filtering (Status, Publisher, File Type, Topics); and
• sorting (A-Z, Z-A, Date Added, Date Modified, Update Frequency);
• browsing; and
• downloading datafiles

Metadata Elements

A sampling of web pages served by the API shows that the Data Catalogue and Data Inventory use many of the same – but also some different – metadata elements – to describe the Ontario Government’s datasets. Table 1 presents

• the twenty metadata elements used to describe datasets in the Data Inventory;
• the labeling and/or display of metadata elements on web pages served for these datasets by the Data Catalogue’s API; and
• the data elements of the dynamic HTML serving these web pages to users of the Data Catalogue.

Many metadata elements associated with datasets in Ontario’s Data Inventory (e.g. Public Title, Short Description, Long Description, etc.) have clear counterparts in Ontario’s Data Catalogue. Other metadata elements associated with these datasets are provided only in the Data Inventory (e.g. Data Custodian Branch, Date Created) or only in the Data Catalogue (e.g. Data last modified). A few pairs of related metadata elements are worth brief comments.

DI: Contains Geographic Markers versus DC: Geographic Coverage

The Data Inventory associates TRUE or FALSE with the value of Contains Geographic Markers; the Data Catalogue associates Ontario or Canada with the value of Geographic Coverage.

DI: Tags versus DC: Topics

The Data Inventory associates the Ontario Government’s datasets with many dozens of different values of Tag – in what appears to be an uncontrolled process; the Data Catalogue, on the other hand, uses controlled input to associate these datasets with values of Topic selected from a list of official Ontario.ca topics, that resides on the Government’s intranet.

DI: Dataset URL and DC: Download data

The assignment of values to the Dataset URL metadata element in Ontario’s Data Inventory is incomplete and inconsistent; by contrast, the values of Download data associated with datasets in the Data Catalogue reflect the three options offered to the Ministries:

1. You can have your data downloadable from the catalogue itself (we will upload the file and it will appear under “Download data”);
2. You can link to the data for download (and it will appear under “Download data”; or
3. If you can’t link to a direct file, you can put the link under the “Access data” section.

Harvesting Ontario’s Data Catalogue

Accessing the metadata associated with the Ontario Government’s datasets as they are served to users by the Data Catalogue’s Application Programming Interface (API) involves scraping and parsing 2,352+ dynamic HTML web pages – a complex and time-consuming process – one that is unlikely to scale and prone to error. Nonetheless, we have harvested the Data Catalogue (circa September 1, 2017) and want to share four ZIP files that contain the dynamic HTML of the web pages associated with the Ontario Government’s datasets that were Open, To Be Opened, Restricted, and Under Review.

Merging the Data Inventory and Data Catalogue

Finally, we are sharing (in both  .xlsx and .ods  format files) our preliminary alignment and merge of the two sets of metadata elements that were used to describe the Ontario’s Government’s datasets in the Data Inventory and Data Catalogue.

Background

As part of its Open Government Initiative, the Ontario Government has issued an Open Data Directive that requires provincial Ministries and the Treasury Board Secretariat to establish and publish an Inventory and Catalogue of government-wide data.

Metadata Elements of the Data Inventory

On September 1, 2017, the Ontario Government’s Data Inventory was available for download as a Comma-Separated Values (CSV) file at https://files.ontario.ca/opendata/ontariodatainventory_31.csv.

Table 1 presents the twenty metadata elements used to describe datasets in the Data Inventory:

The Evolution of the Data Inventory

Table 2 presents the dates of publication and download links to every release of the Ontario Government’s Data Inventory between July 2016 and August 2017. ¹ Notice that, on different days (column B), the Ontario Government published two (08/05/2016, 09/20/2016, 10/21/2016, 11/18/2016, 01/25/2017, 02/27/2017, 03/01/2017, 04/04/2017, 05/19/2017, and 08/03/2017), three (01/09/2017), four (02/13/2017), and sometimes even five (10/06/2016) releases of its Data Inventory.

We’re going to ignore any discrepancies among same-day releases of the Data Inventory, and focus instead on the sole or final release of the day (column C). Upon inspection, we determine that these fourteen releases all differ from one another, and so may be regarded as distinct versions of the Data Inventory.

For ease of reference and access, Table 2 (column D) provides and designates download links for the fourteen distinct versions of the Data Inventory  OntGDI_00.csv, OntGDI_01.csv, OntGDI_02.csv,… OntGDI_13.csv.

Cleaning Up

The Ontario Government’s Data Inventory confronts the sort of data quality issues (e.g. typos, variant spellings, inconsistent use of punctuation, abbreviations, acronyms, etc.) that are expected with Open Data initiatives – especially in their beginnings.

A few discrepancies in the terminology/orthography used to designate a dataset’s Access Level also appear in different versions of the Data Inventory. These discrepancies are potentially quite troublesome for data analysis – though they are easily remedied (e.g. we replace designations, like “Will be made open/public,” “To be opened,” etc. with the most common designation, “To Be Opened”).

A more vexing problem arises when the same Public Title is assigned mistakenly to more than one dataset (e.g. the Public Title “Tax collections: client clearances” is assigned to datasets #350 and #371 in ontariodatainventory_8.csv). Resolving this problem requires:

• identifying every duplicate use of any Public Title across different versions of the Data Inventory;
• generating a Unique Title for every dataset, including those with the same Public Title (e.g. generating “Tax collections: client clearances_01” for dataset #350 and “Tax collections: client clearances_02” for dataset #371 in ontariodatainventory_8.csv,); and
• assigning the same Unique Title to every dataset across different versions of the Data Inventory.

Both of these data quality issues with the original ontarioinventory_nn.csv files (column A) are addressed in the OntGDI_nn_EN.csv files (column D) provided in Table 2.

Next:Ontario\’s Government-wide Data Catalogue