Install OpenProject

https://www.openproject.org/download-and-installation/

1. Add the OpenProject package source

sudo wget -O /etc/yum.repos.d/openproject-ce.repo \
  https://dl.packager.io/srv/opf/openproject-ce/stable/7/installer/el/7.repo

2. Install the OpenProject Community Edition package

Using the following command, yum will install the package and all required dependencies.

sudo yum install openproject

Frequently asked questions – FAQ

How can I install an OpenProject plugin?

Our official installation page has instructions on how to customize your OpenProject installation.

How to backup and restore my OpenProject installation?

Please refer to the documentation at https://www.openproject.org/operations/backup/

How can I install a free SSL certificate using let’s encrypt?

You can get an SSL certificate for free via Let’s Encrypt. Here is how you do it using certbot:

curl https://dl.eff.org/certbot-auto > /usr/local/bin/certbot-auto
chmod a+x /usr/local/bin/certbot-auto

certbot-auto certonly --webroot --webroot-path /opt/openproject/public -d openprojecct.mydomain.com

This requires your OpenProject server to be available from the Internet on port 443 or 80. If this works the certificate (cert.pem) and private key (privkey.pem) will be created under /etc/letsencrypt/live/openproject.mydomain.com/. Configure these for OpenProject to use by running openproject reconfigure and choosing yes when the wizard asks for SSL.

Now this Let’s Encryt certificate is only valid for 90 days. To renew it automatically all you have to do is to add the following entry to your crontab (run crontab -e):

0 1 * * * certbot-auto renew --quiet --post-hook "service apache2 restart"

This will execute certbot renew every day at 1am. The command checks if the certificate is expired and renews it if that is the case. The web server is restarted in a post hook in order for it to pick up the new certificate.


https://www.openproject.org/operations/configuration/

OpenProject configuration

This file describes a part of the OpenProject configuration. You can find general installation instructions here. OpenProject also allows configuring many aspects via its admin interface. The config/settings.yml file should not be used for changing these settings.

OpenProject can be configured either via a configuration.yml file, environment variables or a mix of both. While the latter is probably a bad idea, the environment variable option is often helpful for automatically deploying production systems. Using the configuration file is probably the simplest way of configuration.

You can find a list of options below and an example file in config/configuration.yml.example.

Environment variables

When using environment variables, you can set the options by setting environment variables with the name of the options below in uppercase. So for example, to configure email delivery via an SMTP server, you can set the following environment variables:

EMAIL_DELIVERY_METHOD="smtp"
SMTP_ADDRESS="smtp.example.net"
SMTP_PORT="587"
SMTP_DOMAIN="example.net"
SMTP_AUTHENTICATION="plain"
SMTP_USER_NAME="user"
SMTP_PASSWORD="password"
SMTP_ENABLE_STARTTLS_AUTO="true"

In case you want to use environment variables, but you have no easy way to set them on a specific systme, you can use the dotenv gem. It automatically sets environment variables written to a .env file for a Rails application.

Nested values

You can override nested configuration values as well by joining the respective hash keys with underscores. Underscores within keys have to be escaped by doubling them. For example, given the following configuration:

storage:
  tmp_path: tmp

You can override it by defining the following environment variable:

OPENPROJECT_STORAGE_TMP__PATH=/some/other/path

You can also add new values this way. For instance you could add another field ‘type’ to the storage config above like this:

OPENPROJECT_STORAGE_TYPE=nfs

List of options

Setting session options

Use session_store to define where session information is stored. In order to store sessions in the database and use the following options, set that configuration to :active_record_store.

Delete old sessions for the same user when logging in (Disabled by default)

To enable, set the configuration option drop_old_sessions_on_login to true.

Delete old sessions for the same user when logging out (Enabled by default)

To disable, set the configuration option drop_old_sessions_on_logout to false.

Passing data structures

The configuration uses YAML to parse overrides from ENV. Using YAML inline syntax, you can:

  1. Pass a symbol as an override using OPENPROJECT_SESSION_STORE=":active_record_store"
  2. Pass arrays by wrapping values in brackets (e.g., [val1, val2, val3]).
  3. Pass hashes with {key: foo, key2: bar}.

To pass symbol arrays or hashes with symbol keys, use the YAML !ruby/symbol notiation. Example: {!ruby/symbol key: !ruby/symbol value} will be parsed as { key: :value }.

Please note: The Configuration is a HashWithIndifferentAccess and thus it should be irrelevant for hashes to use symbol keys.

disable password login

default: false

If you enable this option you have to configure at least one omniauth authentication provider to take care of authentication instead of the password login.

All username/password forms will be removed and only a list of omniauth providers presented to the users.

auth source sso

default: nil

Example:

auth_source_sso:
  header: X-Remote-User
  secret: s3cr3t

Can be used to automatically login a user defined through a custom header sent by a load balancer or reverse proxy in front of OpenProject, for instance in a Kerberos Single Sign-On (SSO) setup via apache. The header with the given name has to be passed to OpenProject containing the logged in user and the defined global secret as in $login:$secret.

omniauth direct login provider

default: nil

Example:

omniauth_direct_login_provider: google

Per default the user may choose the usual password login as well as several omniauth providers on the login page and in the login drop down menu. With his configuration option you can set a specific omniauth provider to be used for direct login. Meaning that the login provider selection is skipped and the configured provider is used directly instead.

If this option is active /login will lead directly to the configured omniauth provider and so will a click on ‘Sign in’ (as opposed to opening the drop down menu).

Note that this does not stop a user from manually navigating to any other omniauth provider if additional ones are configured.

attachments storage

default: file

Attachments can be stored using fog as well. You will have to add further configuration through fog, e.g. for Amazon S3:

attachments_storage: fog
fog:
  directory: bucket-name
  credentials:
    provider: 'AWS'
    aws_access_key_id: 'AKIAJ23HC4KNPWHPG3UA'
    aws_secret_access_key: 'PYZO9phvL5IgyjjcI2wJdkiy6UyxPK87wP/yxPxS'
    region: 'eu-west-1'

In order to set these values through ENV variables, use:

<pre>
OPENPROJECT_ATTACHMENTS__STORAGE=fog
OPENPROJECT_FOG_CREDENTIALS_AWS__ACCESS__KEY__ID="AKIAJ23HC4KNPWHPG3UA"
OPENPROJECT_FOG_CREDENTIALS_AWS__SECRET__ACCESS__KEY="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"
OPENPROJECT_FOG_CREDENTIALS_PROVIDER=AWS
OPENPROJECT_FOG_CREDENTIALS_REGION="eu-west-1"
OPENPROJECT_FOG_DIRECTORY=uploads
</pre>

backend migration

You can migrate attachments between the available backends. One example would be that you change the configuration from the file storage to the fog storage. If you want to put all the present file-based attachments into the cloud, you will have to use the following rake task:

rake attachments:copy_to[fog]

It works the other way around too:

rake attachments:copy_to[file]

Note that you have to configure the respective storage (i.e. fog) beforehand as described in the previous section. In the case of fog you only have to configure everything under fog, however. Don’t change attachments_storage to fog just yet. Instead leave it as file. This is because the current attachments storage is used as the source for the migration.

Overriding the help link

You can override the default help menu of OpenProject by specifying a force_help_link option to the configuration. This value is used for the href of the help link, and the default dropdown is removed.

hidden menu items

default: {}

You can disable specific menu items in the menu sidebar for each main menu (such as Administration and Projects). The following example disables all menu items except ‘Users’, ‘Groups’ and ‘Custom fields’ under ‘Administration’:

hidden_menu_items:
  admin_menu:
    - roles
    - types
    - statuses
    - workflows
    - enumerations
    - settings
    - ldap_authentication
    - colors
    - project_types
    - export_card_configurations
    - plugins
    - info

The configuration can be overridden through environment variables. You have to define one variable for each menu. For instance ‘Roles’ and ‘Types’ under ‘Administration’ can be disabled by defining the following variable:

OPENPROJECT_HIDDEN__MENU__ITEMS_ADMIN__MENU='roles types'

blacklisted routes

default: []

You can blacklist specific routes The following example forbid all routes for above disabled menu:

blacklisted_routes:
  - 'admin/info'
  - 'admin/plugins'
  - 'export_card_configurations'
  - 'project_types'
  - 'colors'
  - 'settings'
  - 'admin/enumerations'
  - 'workflows/*'
  - 'statuses'
  - 'types'
  - 'admin/roles'

The configuration can be overridden through environment variables.

OPENPROJECT_BLACKLISTED__ROUTES='admin/info admin/plugins'

disabled modules

default: []

Modules may be disabled through the configuration. Just give a list of the module names either as an array or as a string with values separated by spaces.

Array example:

disabled_modules:
  - backlogs
  - meetings

String example:

disabled_modules: backlogs meetings

The option to use a string is mostly relevant for when you want to override the disabled modules via ENV variables:

OPENPROJECT_DISABLED__MODULES='backlogs meetings'

local checkout path

default: “repositories”

Remote git repositories will be checked out here.

APIv3 basic auth control

default: true

You can enable basic auth access to the APIv3 with the following configuration option:

apiv3_enable_basic_auth: true

global basic auth

default: none

You can define a global set of credentials used to authenticate towards API v3. Example section for configuration.yml:

default:
  authentication:
    global_basic_auth:
      user: admin
      password: admin

Email configuration

  • email_delivery_method: The way emails should be delivered. Possible values: smtp or sendmail

SMTP Options:

  • smtp_address: SMTP server hostname, e.g. smtp.example.net
  • smtp_port: SMTP server port. Common options are 25 and 587.
  • smtp_domain: The domain told to the SMTP server, probably the hostname of your OpenProject instance (sent in the HELO domain command). Example: example.net
  • smtp_authentication: Authentication method, possible values: plainlogincram_md5 (optional, only when authentication is required)
  • smtp_user_name: Username for authentication against the SMTP server (optional, only when authentication is required)
  • smtp_password (optional, only when authentication is required)
  • smtp_enable_starttls_auto: You can disable STARTTLS here in case it doesn’t work. Make sure you don’t login to a SMTP server over a public network when using this. This setting can’t currently be used via environment variables, since setting options to false is only possible via a YAML file. (default: true, optional)
  • smtp_openssl_verify_mode: Define how the SMTP server certificate is validated. Make sure you don’t just disable verification here unless both, OpenProject and SMTP servers are on a private network. Possible values: nonepeerclient_once or fail_if_no_peer_cert

Cache options:

  • rails_cache_storememcache for memcached or memory_store (default: file_store)
  • cache_memcache_server: The memcache server host and IP (default: 127.0.0.1:11211)
  • cache_expires_in: Expiration time for memcache entries (default: 0, no expiry)
  • cache_namespace: Namespace for cache keys, useful when multiple applications use a single memcache server (default: none)

Asset options:

  • rails_asset_host: A custom host to use to serve static assets such as javascript, CSS, images, etc. (default: nil)

Onboarding variables:

  • ‘onboarding_video_url’: An URL for the video displayed on the onboarding modal. This is only shown when the user logs in for the first time.

Edit on GitHub


https://www.openproject.org/plugins/

Plugins

Install plugins

You can install plugins for the following installation methods:

Develop plugins

We encourage you to extend OpenProject yourself by writing a plugin. Please, read the plugin-contributions guide for more information.

If you are a plugin author and want your plugins to be listed here, leave us a note in our forums, via twitter or any other communication channel. We’re looking forward to your contribution.

Please note that the plugin author is responsible to keep his plugin up to date with OpenProject development. In case a plugin causes errors, please contact the plugin author and/or write in our support forums. We will remove plugins from this page if we notice that they are not maintained.

Supported plugins

Supported plugins are compatible with the current OpenProject version. When a plugin has a dependency to another plugin it is necessary that you list both plugins in the Gemfile.plugins.

Backlogs

gem "openproject-backlogs", git: "https://github.com/finnlabs/openproject-backlogs.git", :branch => 'stable/5'
gem "openproject-pdf_export", git: "https://github.com/finnlabs/openproject-pdf_export.git", :branch => 'stable/5'

Cost

gem 'openproject-costs', git: 'https://github.com/finnlabs/openproject-costs.git', :branch => 'stable/5'

Dark-Theme

  • Description: Allows users to switch to the dark theme instead of the OpenProject theme
  • Author: OpenProject GmbH
  • Link: Source
gem 'openproject-themes-dark', git: 'https://github.com/finnlabs/openproject-themes-dark.git', :branch => 'stable/5'

Documents

gem 'openproject-documents', git: 'https://github.com/opf/openproject-documents.git', :branch => 'stable/5'

Global Roles

gem 'openproject-global_roles', git: 'https://github.com/finnlabs/openproject-global_roles.git', :branch => 'stable/5'

Local Avatars

gem 'openproject-local_avatars', git: 'https://github.com/finnlabs/openproject-local_avatars', :branch => 'stable/5'

Meeting

gem 'openproject-meeting', git: 'https://github.com/finnlabs/openproject-meeting.git', :branch => 'stable/5'

My Project Page

gem 'openproject-my_project_page', git: 'https://github.com/finnlabs/openproject-my_project_page.git', :branch => 'stable/5'

OmniAuth OpenID-Connect-Providers

  • Description: Offers a convenient way to configure different OmniAuth OpenIDConnect providers (preconfigured for Google and Heroku).
  • Author: OpenProject GmbH
  • Link: Source
gem 'omniauth-openid_connect-providers', git: 'https://github.com/finnlabs/omniauth-openid_connect-providers.git', :branch => 'dev'

OpenID-Connect

  • Description: Adds support for OmniAuth OpenID Connect strategy providers, most importantly Google.
  • Author: OpenProject GmbH
  • Link: Source
gem 'omniauth-openid-connect', git: 'https://github.com/finnlabs/omniauth-openid-connect.git', :branch => 'dev'

PDF Export

  • Description: Enables story card export for OpenProject Backlogs
  • Author: OpenProject GmbH
  • Link: Source
  • Dependency: Backlogs
gem 'openproject-pdf_export', git: 'https://github.com/finnlabs/openproject-pdf_export.git', :branch => 'stable/5'
gem 'openproject-backlogs', git: 'https://github.com/finnlabs/openproject-backlogs.git', :branch => 'stable/5'

Reporting

  • Description: Customized cost reporting features
  • Author: OpenProject GmbH
  • Link: Source
  • Dependency: Costs, Reporting Engine
gem 'reporting_engine', git: 'https://github.com/finnlabs/reporting_engine.git', :branch => 'dev'
gem 'openproject-costs', git: 'https://github.com/finnlabs/openproject-costs.git', :branch => 'stable/5'
gem 'openproject-reporting', git: 'https://github.com/finnlabs/openproject-reporting.git', :branch => 'stable/5'

XLS-Export

gem 'openproject-xls_export', git: 'https://github.com/finnlabs/openproject-xls_export.git', :branch => 'stable/5'

Community plugins

There are several plugins developed by dedicated community members. We really appreciate their hard work!

Please keep in mind that it can not always be guaranteed that they are compatible with the current OpenProject version. Authors are responsible for maintaining their plugins and are doing the best they can to keep the plugins up to date. Please contact the authors directly if you have any questions or feedback, or even want to help them out.

Auto Project

gem "openproject-auto_project", git: "https://github.com/oliverguenther/openproject-auto_project.git", :branch => 'stable'

Ensure Project Hierarchy

  • Description: Ensures subproject identifiers are prefixed with their parent project’s identifier
  • Author: Oliver Günther
  • Link: Source
gem "openproject-ensure_project_hierarchy", git: "https://github.com/oliverguenther/openproject-ensure_project_hierarchy.git", :branch => 'stable'

Plugins in development (potentially unstable)

OpenProject Git Hosting

  • Description: Provides extensive features for managing Git repositories within OpenProject
  • Author: Oliver Günther
  • Link: Source
gem "openproject-git_hosting", git: "https://github.com/oliverguenther/openproject-git_hosting.git", :branch => "dev"

Note:

You can find the master document in GitHub. You can propose a change to this guide by creating a pull request on GitHub.


https://www.openproject.org/plugins/install-plugins-packaged/

Install plugins – packaged installation

Note: this guide only applies if you’ve installed OpenProject using our DEB/RPM packages.

A number of plugins exist for use with OpenProject. Most plugins that are maintained by us are shipping with OpenProject, however there are several plugins contributed by the community.

Previously, using them in a packaged installation was not possible without losing your changes on every upgrade. With the following steps, you can now use third party plugins.

Note: We cannot guarantee upgrade compatibility for third party plugins nor do we provide support for them. Please carefully check whether the plugins you use are available in newer versions before upgrading your installation.

1. Add a custom Gemfile

If you have a plugin you wish to add to your packaged OpenProject installation, create a separate Gemfile with the Gem dependencies, such as the following:

gem 'openproject-emoji', git: 'https://github.com/tessi/openproject-emoji.git', :branch => 'op-5-stable'

We suggest to store the Gemfile under /etc/openproject/Gemfile.custom, but the choice is up to you, just make sure the openproject user is able to read it.

2. Propagate the Gemfile to the package

You have to tell your installation to use the custom gemfile via a config setting:

openproject config:set CUSTOM_PLUGIN_GEMFILE=/etc/openproject/Gemfile.custom

3. Re-run the installer

To re-bundle the application including the new plugins, as well as running migrations and precompiling their assets, simply re-run the installer while using the same configuration as before.

openproject configure

Using configure will take your previous decisions in the installer and simply re-apply them, which is an idempotent operation. It will detect the Gemfile config option being set and re-bundle the application with the additional plugins.

Note:

You can find the master document in GitHub. You can propose a change to this guide by creating a pull request on GitHub.

Visualizing MCYS Service Regions and CYMH Service Areas in Ontario

MCYS Service Regions and Service Areas

In 2014 – 2015, the Ministry of Children and Youth Services (MCYS) defined five Service Regions and thirty-three Children and Youth Mental Health (CYMH) Service Areas in Ontario.

We have used free and open source software (FOSS) to convert the Ontario government’s Shapefile archives of the geographic boundaries of these MCYS Service Regions and CYMH Service Areas into TopoJSON files that are convenient for visualizing these geographic bodies and their various characteristics (e.g. population, population density, rate of population growth, etc).

D3-GEO

Data-Driven Documents (D3) is a free and open source (Javascript) software library for manipulating documents based on data. D3-GEO is a powerful component of D3 that is designed specifically to visualize geographic data using web standards like HTML, CSS and SVG.

Free and Open Service for Us

We have used D3-GEO to transform our TopoJSON files into (a sample of) area, regional, and provincial views of the MCYS Service Regions and CYMH Service Areas in Ontario:

Area Views
Algoma Algoma CYMH Service Area
Kenora/Rainy River Kenora/Rainy River CYMH Service Area
Regional Views
Central
East CYMH Service Areas in East Region
North CYMH Service Areas in North Region
Toronto Toronto Service Area in Toronto Region
West CYMH Service Areas in West Region
Provincial Views
Service Regions MCYS Service Regions in Ontario
Service Areas
(Interactive)
CYMH Service Areas in Ontario

Previous: Geographic Boundaries of MCYS Service Regions and CYMH Service Areas in Ontario

Next: Population Projections for MCYS Service Regions and CYMH Service Areas in Ontario (2016-2041)

License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License. Readers are free to access, modify, and re-use the TopoJSON files and Javascript source code that generate these views of the MCYS Service Regions and CYMH Service Areas in Ontario.

Geographic Boundaries of MCYS Service Regions and CYMH Service Areas in Ontario

MCYS Service Regions and Service Areas

In 2014 – 2015, the Ministry of Children and Youth Services (MCYS) defined five Service Regions and thirty-threeChildren and Youth Mental Health (CYMH) Service Areas in Ontario.

The Service Regions replaced nine Service Delivery Divisions Regions and four Youth Justice Services Division Regions. The CYMH Service Areas were defined on the basis of the projected population of children and youth in Statistics Canada’s census divisions.

Ontario’s Open Data

As part of its Open Data Directive, the Ontario government has published two Shapefile archives that define the geographic boundaries of the Service Regions and Service Areas within the province.

Free and Open Service for Us

We have used free and open source software (FOSS) like the Geospatial Data Abstraction Library (GDAL) and TopoJSON to convert the Ontario government’s Shapefile archives into a set of TopoJSON files. Shortly we will use these TopoJSON files to visualize of geographic boundaries and socio-demographic characteristics of the MCYS Service Regions and CYMH Service Areas.

MCYS Service Region/CYMH Service Area
Central
Dufferin/Wellington
Halton
Peel
Simcoe
Waterloo
York
East
Durham
Frontenac/Lennox and Addington
Haliburton/Kawartha Lakes/Peterborough
Hastings/Prince Edward/Northumberland
Lanark/Leeds and Grenville
Ottawa
Prescott and Russell
Renfrew
Stormont, Dundas and Glengarry
North
Algoma
TopoJSON
Kenora/Rainy River
Nipissing/Parry Sound/Muskoka
Thunder Bay
Cochrane/Timiskaming (including James Bay Coast)
Toronto
Toronto
West
Brant
Bruce/Grey
Chatham-Kent
Elgin/Oxford
Essex
Haldimand-Norfolk
Hamilton
Huron/Perth
Lambton
Middlesex
Niagara

Next: Visualizing MCYS Service Regions and CYMH Service Areas in Ontario

License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License. Readers are free to access, modify, and re-use our TopoJSON files for the MCYS Service Regions and CYMH Service Areas.

Population Projections for MCYS Service Regions and CYMH Service Areas in Ontario (2016-2041)

MCYS Service Regions and CYMH Service Areas

In 2014 – 2015, the Ministry of Children and Youth Services (MCYS) defined five Service Regions and thirty-three Children and Youth Mental Health (CYMH) Service Areas in Ontario.

The Service Regions replaced nine Service Delivery Divisions Regions and four Youth Justice Services Division Regions. The CYMH Service Areas were defined on the basis of the projected population of children and youth in Statistics Canada’s census divisions.

Population Projections

In June 2017, the Ministry of Finance published its population projections (2016 – 2041) for each census division in Ontario. From these data, we have derived the following projections for children and youth 0 to 17 years old in each Service Region and CYMH Service Area:

  • projected total population
  • projected population of 0 to 17 year old children and youth
  • projected share of the total population represented by 0 to 17 year old children and youth
  • projected annual growth in the population of 0 to 17 year old children and youth
  • projected annual rate of growth (per cent) in the population of 0 to 17 year old children and youth

We have also calculated the land area of each MCYS Service Region and CYMH Service Area in Ontario to derive:

  • projected population densities (person per sq. km.) for 0 to 17 year old children and youth

Free and Open Service for Us

We are happy to share our population projections for each Service Region and CYMH Service Area in Ontario in a set of Comma-Separated Values (CSV-UTF-8) files:

Both sets of population projections are also available in a single Open Data Format (ODF) file. The ODF format is supported by Libre Office, a free and open source alternative to Microsoft Office, that we encourage everyone to consider using.

Previous: Visualizing MCYS Service Regions and CYMH Service Areas in Ontario

Next:

License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License. Readers are free to access, modify, and re-use our population projections for MCYS Service Regions and CYMH Service Areas in Ontario.

Ontario’s Open Data Initiative May Be Getting Bogged Down

Ontario’s Open Government Initiative presents a great opportunity to study the impact of an Open Data Directive in a sizable, relatively well-resourced, democratic sub-national government.

In a series of posts on FOS4US: Free, Open Services for Us, we laid the groundwork for this sort of study by:

  • Identifying fourteen distinct iterations of Ontario’s Data Inventory since its inception in July 2016;
  • Visualizing the evolution of Ontario’s Data Inventory over its first year of development;
  • Harvesting over 2,300 dynamic HTML web pages served by the Application Program Interface (API) of Ontario’s Data Catalogue;
  • Merging Ontario’s Data Inventory and Data Catalogue to provide a single authoritative source of information about Ontario’s government-wide datasets; and
  • Presenting a novel graphical interface for users to explore and access the rich content served by Ontario’s Data Catalogue

In this update, we examine more closely the evolution of Data Inventory over its first year of development – and raise a concern that, after making a promising start, the Ontario Government’s Open Data initiative may be stalled.

Figure 1 presents the number of new datasets that have been added to the Ontario Government’s Data Inventory in two month intervals, beginning in July-August 2016, as well as the increase in the total size of the Data Inventory over time. We see that the Data Inventory has tended to experience “growth spurts” rather than steady growth – with no substantial increase in the total size of the Data Inventory since April 2017.

Figure 1. Growth of Ontario Governments Data Inventory July 2016 – August 2017.

Figure 2 presents the distribution of Access Levels that had been assigned to datasets when they first entered the Ontario Government’s Data Inventory and the distribution of Access Levels as of August 2017. We see that the two distributions are very similar. Indeed, 2,321/2,352 (98.7%) datasets that were listed in the Ontario Government’s Data Inventory in August 2017 had seen no change in their Access Level since first entering the Data Inventory.

Figure 2. Change in Access Level in Ontario Governments Data Inventory – July 2016 – August 2017.

Figure 3 presents the cumulative number of datasets that were “To Be Opened” or “Under Review” when they first entered the Ontario Government’s Data Inventory and remained so in August 2017.

Figure 3. Cumulative Growth of TBO, UR datasets in Ontario Governments Data Inventory July 2016 – August 2017.

Taken together, these Figures suggest that the Ontario Government has taken the reasonable decision to hold off on adding new datasets to its Data Inventory, while it figures out how to deal with the increasing backlog of datasets that are To Be Opened or Under Review.

Visualizing the Ontario Government’s Open Data

Visualizing the Data Inventory

In previous posts, we compiled and shared a wealth of data and metadata connected with Ontario’s government-wide Data Inventory and Data Catalogue. These materials lay the groundwork for – and hopefully inspire – the study of the early development of an open data initiative undertaken by a sizable, democratic sub-national government.

Our final contribution (for now) to kick-starting  this study is to share a few ways of visualizing the Ontario Government’s Data Inventory.

First, we use D3 to convert any version of the Data Inventory to a force-layout graph, where

  • source nodes represent Ministries;
  • target nodes represent Datasets;
  • edges connecting source nodes and target nodes represent the relationship hasPublished; and
  • a target node’s colour (e.g. lime, yellowgreen, red, white) represents the Access Level of the corresponding Dataset (Open, To Be Opened, Restricted, Under Review).

Figures 1 and 2 represent Ontario’s Government-wide Data Inventory on July 29,2016 (its inception) and September 1, 2017, respectively.

Figure 1. Ontario’s Government-wide Data Inventory, July 29, 2016.
ontdi_00
 

 

Figure 2. Ontario’s Government-wide Data Inventory, August 3, 2017.
 

We may also animate the Data Inventory’s evolution over time:

  • ontdi_00
  • ontdi_01
  • ontdi_02
  • ontdi_03
  • ontdi_04
  • ontdi_05
  • ontdi_06
  • ontdi_07
  • ontdi_08
  • ontdi_09
  • ontdi_10
  • ontdi_11
  • ontdi_12
  • ontdi_13

 

Finally, let’s add some functionality (tooltips and hyperlinks) to our visualization of the Data Inventory, so that users may explore and click on nodes to access the rich content about datasets on the web pages served by the Data Catalogue’s API.

Previous: Ontario’s  Government-wide Data Inventory and Data Catalogue

Ontario’s Government-wide Data Catalogue

Background

As part of its Open Government Initiative, the Ontario Government has issued an Open Data Directive that requires provincial Ministries and the Treasury Board Secretariat establish and publish an Inventory and Catalogue of government-wide data.

Application Program Interface (API)

The Ontario Government describes the Data Catalogue as a “one-window central platform” that provides faster and easier access to Government Data, thereby facilitating evidence-based policy, informs service delivery and promotes greater transparency and accountability.

The Data Catalogue’s Application Program Interface (API) helps users identify and access potentially relevant datasets with features that support:

  • searching (free-text);
  • filtering (Status, Publisher, File Type, Topics); and
  • sorting (A-Z, Z-A, Date Added, Date Modified, Update Frequency);
  • browsing; and
  • downloading datafiles

Metadata Elements

A sampling of web pages served by the API shows that the Data Catalogue and Data Inventory use many of the same – but also some different – metadata elements – to describe the Ontario Government’s datasets. Table 1 presents

  • the twenty metadata elements used to describe datasets in the Data Inventory;
  • the labeling and/or display of metadata elements on web pages served for these datasets by the Data Catalogue’s API; and
  • the data elements of the dynamic HTML serving these web pages to users of the Data Catalogue.
Table 3. Metadata associated with the Ontario Government’s datasets in the Data Inventory and Data Catalogue. * denotes metadata elements associated with Open datasets only; ** denotes metadata elements associated with Restricted datasets only.
Data Inventory Data Catalogue
Metadata Property Label/Display Dynamic HTML Data Element
Public Title Display only content.title
Short Description Display only content.intro
Long Description Display only content.intro <div class=”main-content row”>
Other Title
Data Custodian Branch
Tags
Topics content.tags.tagList
Date Range – Start Time captured* content.dataset.dateStart
Date Range – End Time captured* content.dataset.dateEnd
Date Created
Date Published Date added content.datePublished
Contains Geographic Markers
Geographical coverage content.dataset.geographicalCoverage
Publisher Publisher content.dataset.publisherList
Update Frequency Update frequency content.dataset.updateFrequency
Data last modified* content.dataset.dateDataUpdated
Access Level Data status content.dataset.status
Exemption Exemption** content.dataset.exemptionList
Rationale Rationale not to release** content.dataset.exemptionRationale
Dataset URL
Download data* content.dataset.filelist, or content.datasetexternalDataLink
License Type Terms of Use* content.dataset.license
hidden link to Technical documentation content.dataset.technicalDocument.url
File Types (extensions) (link text)* fileGroup.list
Additional Comments

Many metadata elements associated with datasets in Ontario’s Data Inventory (e.g. Public Title, Short Description, Long Description, etc.) have clear counterparts in Ontario’s Data Catalogue. Other metadata elements associated with these datasets are provided only in the Data Inventory (e.g. Data Custodian Branch, Date Created) or only in the Data Catalogue (e.g. Data last modified). A few pairs of related metadata elements are worth brief comments.

DI: Contains Geographic Markers versus DC: Geographic Coverage

The Data Inventory associates TRUE or FALSE with the value of Contains Geographic Markers; the Data Catalogue associates Ontario or Canada with the value of Geographic Coverage.

DI: Tags versus DC: Topics

The Data Inventory associates the Ontario Government’s datasets with many dozens of different values of Tag – in what appears to be an uncontrolled process; the Data Catalogue, on the other hand, uses controlled input to associate these datasets with values of Topic selected from a list of official Ontario.ca topics, that resides on the Government’s intranet.

DI: Dataset URL and DC: Download data

The assignment of values to the Dataset URL metadata element in Ontario’s Data Inventory is incomplete and inconsistent; by contrast, the values of Download data associated with datasets in the Data Catalogue reflect the three options offered to the Ministries:

  1. You can have your data downloadable from the catalogue itself (we will upload the file and it will appear under “Download data”);
  2. You can link to the data for download (and it will appear under “Download data”; or
  3. If you can’t link to a direct file, you can put the link under the “Access data” section.

Harvesting Ontario’s Data Catalogue

Accessing the metadata associated with the Ontario Government’s datasets as they are served to users by the Data Catalogue’s Application Programming Interface (API) involves scraping and parsing 2,352+ dynamic HTML web pages – a complex and time-consuming process – one that is unlikely to scale and prone to error. Nonetheless, we have harvested the Data Catalogue (circa September 1, 2017) and want to share four ZIP files that contain the dynamic HTML of the web pages associated with the Ontario Government’s datasets that were Open, To Be Opened, Restricted, and Under Review.

Merging the Data Inventory and Data Catalogue

Finally, we are sharing (in both  .xlsx and .ods  format files) our preliminary alignment and merge of the two sets of metadata elements that were used to describe the Ontario’s Government’s datasets in the Data Inventory and Data Catalogue.

Previous:Ontario\’s Government-wide Data Inventory Next:Visualizing the Ontario Government\’s Open Data

Ontario’s Government-wide Data Inventory

Background

As part of its Open Government Initiative, the Ontario Government has issued an Open Data Directive that requires provincial Ministries and the Treasury Board Secretariat to establish and publish an Inventory and Catalogue of government-wide data.

Metadata Elements of the Data Inventory

On September 1, 2017, the Ontario Government’s Data Inventory was available for download as a Comma-Separated Values (CSV) file at https://files.ontario.ca/opendata/ontariodatainventory_31.csv.

Table 1 presents the twenty metadata elements used to describe datasets in the Data Inventory:

Table 1. Metadata elements used to describe datasets in the Ontario Government’s Data Inventory. Definitions/examples are sourced from the Ontario Government’s Open Data Guidebook, 2015.
Metadata Element Definition
Public Title The name given to a dataset. This must be unique and restricted to 200 characters or less. E.g. Consular offices in Ontario.
Short Description Limited description to 200 characters or less. E.g. List of consular offices operating in Ontario.
Long Description Limited description to 600 characters or less (approximately 100 words). E.g. The data contained includes the name of the Consul General or Honourary Consul General, office address, telephone and fax numbers, email and web addresses.
Other Title Internal name of the dataset. Must be unique. Limited to 200 characters.
Data Custodian Branch E.g. Office of International Relations and Protocol.
Tags
Date Range – Start Date format yyyy-mm-dd.
Date Range – End Date format yyyy-mm-dd.
Date Created Date format yyyy-mm-dd.
Date Published Date format yyyy-mm-dd. E.g. 2016-04-20.
Contains Geographic Markers True if this dataset contains geographic markers, false otherwise. E.g. TRUE.
Publisher Ministry or agency. Selected from a drop-down menu. E.g. Intergovernmental Affairs.
Update Frequency The frequency of update of the dataset. Selected from a drop-down menu. E.g. On demand.
Access Level Whether this data can be targeted for public release, is restricted from public release, or needs more assessment. Selected from a drop-down menu. E.g. Under review.
Exemption Choose the specific exemption that applies to the restricted dataset. Selected from a drop-down menu.
Rationale
Dataset URL (Optional) A web page that can be navigated to to gain access to the dataset. E.g. https://www.ontario.ca/page/consular-offices.
License Type Choose the license under which the distribution is made available. Selected from a drop-down menu. E.g. Ontario.ca Terms of Use.
File Types (extensions) Comma separated list of file types. Selected from a drop-down menu. E.g. DOCX.
Additional Comments Please provide any other information about your dataset that would help us to assess its suitability for open data.

The Evolution of the Data Inventory

Table 2 presents the dates of publication and download links to every release of the Ontario Government’s Data Inventory between July 2016 and August 2017. 1 Notice that, on different days (column B), the Ontario Government published two (08/05/2016, 09/20/2016, 10/21/2016, 11/18/2016, 01/25/2017, 02/27/2017, 03/01/2017, 04/04/2017, 05/19/2017, and 08/03/2017), three (01/09/2017), four (02/13/2017), and sometimes even five (10/06/2016) releases of its Data Inventory.

We’re going to ignore any discrepancies among same-day releases of the Data Inventory, and focus instead on the sole or final release of the day (column C). Upon inspection, we determine that these fourteen releases all differ from one another, and so may be regarded as distinct versions of the Data Inventory.

For ease of reference and access, Table 2 (column D) provides and designates download links for the fourteen distinct versions of the Data Inventory  OntGDI_00.csvOntGDI_01.csv, OntGDI_02.csv,OntGDI_13.csv.

Cleaning Up

The Ontario Government’s Data Inventory confronts the sort of data quality issues (e.g. typos, variant spellings, inconsistent use of punctuation, abbreviations, acronyms, etc.) that are expected with Open Data initiatives – especially in their beginnings.

A few discrepancies in the terminology/orthography used to designate a dataset’s Access Level also appear in different versions of the Data Inventory. These discrepancies are potentially quite troublesome for data analysis – though they are easily remedied (e.g. we replace designations, like “Will be made open/public,” “To be opened,” etc. with the most common designation, “To Be Opened”).

A more vexing problem arises when the same Public Title is assigned mistakenly to more than one dataset (e.g. the Public Title “Tax collections: client clearances” is assigned to datasets #350 and #371 in ontariodatainventory_8.csv). Resolving this problem requires:

  • identifying every duplicate use of any Public Title across different versions of the Data Inventory;
  • generating a Unique Title for every dataset, including those with the same Public Title (e.g. generating “Tax collections: client clearances_01” for dataset #350 and “Tax collections: client clearances_02” for dataset #371 in ontariodatainventory_8.csv,); and
  • assigning the same Unique Title to every dataset across different versions of the Data Inventory.

Both of these data quality issues with the original ontarioinventory_nn.csv files (column A) are addressed in the OntGDI_nn_EN.csv files (column D) provided in Table 2.

Table 2. Publication of the Ontario Government’s Data Inventory, July 2016 to September 2017.
A
Metadata Files
B
Release Date/Time
C
Sole/Final Release of the Day
D
Distinct Versions
ontariodatainventory.csv 07/29/2016 – 14:23 07/29/2016 – 14:23 OntGDI_00_EN.csv
ontariodatainventory_0.csv 08/05/2016 – 10:18
ontariodatainventory_1.csv 08/05/2016 – 10:19 08/05/2016 – 10:19 OntGDI_01_EN.csv
ontariodatainventory_2.csv 09/20/2016 – 10:29
ontariodatainventory_3.csv 09/20/2016 – 10:32 09/20/2016 – 10:32 OntGDI_02_EN.csv
ontariodatainventory_4.csv 10/06/2016 – 09:51
ontariodatainventory_5.csv 10/06/2016 – 10:02
ontariodatainventory_6.csv 10/06/2016 – 10:09
ontariodatainventory_7.csv 10/06/2016 – 10:28
ontariodatainventory_8.csv 10/06/2016 – 10:30 10/06/2016 – 10:30 OntGDI_03_EN.csv
ontariodatainventory_9.csv 10/21/2016 – 15:49
ontariodatainventory_10.csv 10/21/2016 – 15:50 10/21/2016 – 15:50 OntGDI_04_EN.csv
ontariodatainventory_11.csv 11/18/2016 – 10:21
ontariodatainventory_12.csv 11/18/2016 – 10:24 11/18/2016 – 10:24 OntGDI_05_EN.csv
ontariodatainventory_13.csv 01/09/2017 – 13:19
ontariodatainventory_14.csv 01/09/2017 – 13:22
ontariodatainventory_15.csv 01/09/2017 – 13:27 01/09/2017 – 13:27 OntGDI_06_EN.csv
ontariodatainventory_16.csv 01/25/2017 – 11:06
ontariodatainventory_17.csv 01/25/2017 – 11:08 01/25/2017 – 11:08 OntGDI_07_EN.csv
ontariodatainventory_18.csv 02/13/2017 – 13:04
ontariodatainventory_19.csv 02/13/2017 – 13:06
ontariodatainventory_20.csv 02/13/2017 – 13:56
ontariodatainventory_21.csv 02/13/2017 – 13:57 02/13/2017 – 13:57 OntGDI_08_EN.csv
ontariodatainventory_22.csv 02/27/2017 – 15:25
ontariodatainventory_23.csv 02/27/2017 – 15:28 02/27/2017 – 15:28 OntGDI_09_EN.csv
ontariodatainventory_24.csv 03/01/2017 – 15:22
ontariodatainventory_25.csv 03/01/2017 – 15:24 03/01/2017 – 15:24 OntGDI_10_EN.csv
ontariodatainventory_26.csv 04/04/2017 – 13:25
ontariodatainventory_27.csv 04/04/2017 – 13:26 04/04/2017 – 13:26 OntGDI_11_EN.csv
ontariodatainventory_28.csv 05/19/2017 – 11:38
ontariodatainventory_29.csv 05/19/2017 – 11:39 05/19/2017 – 11:39 OntGDI_12_EN.csv
ontariodatainventory_30.csv 08/03/2017 – 12:47
ontariodatainventory_31.csv 08/03/2017 – 12:48 08/03/2017 – 12:48 OntGDI_13_EN.csv

Next:Ontario\’s Government-wide Data Catalogue

  1. Personal correspondence, Dawn Edmonds, Team Lead, Policy and Partnerships, Open Government Office, Treasury Board Secretariat, October 11, 2017.

Milestones in Ontario’s Open Government Initiative

October 2013 Open Government initiative launched.
April 2014 Open data voting tool launched.
175+ open data sets published.
May 2015 Public consulted on the Open Data Directive.
400+ open data sets published.
November 2015 Final Open Data Directive released.
April 2016 Ontario is selected to be part of the Open Government Partnership’s new pilot program. Ontario’s Open Data Directive came into force.
500+ open data sets published
August 2016 Public invited to submit open government ideas as part of the Open Government Partnership pilot program. Ministries start to add their data inventories to the Data Catalogue.
May 2017 We adopted the International Open Data Charter.
We published more directives as part of our commitment to Open Government.
June 2017 Our mid-term self-assessment report and progress update is posted.

Available at https://www.ontario.ca/page/open-government [Accessed 2017-09-01]

WordPress

FOS4US.CA is hosted on our own server and is published using the 2016 theme of WordPress. A few plug-ins (also free and open-source) provide some additional functionality:

  • Askimet: to check readers’ comments against a global spam database to prevent us from publishing garbage or malicious content
  • TinyMCE Advanced: to add, remove or arrange buttons on the Visual Editor toolbar related to 15 TinyMCE plug-ins
  • Footnotation: to add footnotes to our pages and posts

While we’re most familiar with WordPress, you may prefer one of these other free and open-source Content Management Systems:

FYI

WordPress is a free and open-source content management system (CMS) based on PHP and MySQL. To function, WordPress has to be installed on a web server, which would either be part of an Internet hosting service or a network host in its own right. An example of the first scenario may be a service like WordPress.com, for example, and the second case could be a computer running the software package WordPress.org. A local computer may be used for single-user testing and learning purposes. Features include a plugin architecture and a template system. WordPress was used by more than 27.5% of the top 10 million websites as of February 2017. WordPress is reportedly the most popular website management or blogging system in use on the Web, supporting more than 60 million websites. WordPress is released under the GPLv2 (or later) license.