Skip to content
Alpha

This site is an alpha version and is still being developed in the open - your feedback will help us improve it. See the roadmap. Last updated: 14 October 2026.

Version in force: 0.3.0 (6 October 2026).

Publishing open data with metadata

Publish data once, in open formats, with metadata that lets people find it, understand it and trust it - and an automated pipeline that keeps it current.

Context

Defra holds a great deal of data that is useful outside government: environmental monitoring, flood risk, land use, species records. Data published as a one-off spreadsheet with no description is hard to find, quickly out of date and easy to misuse. Data published by hand can also leak information that should not be public.

Solution

Publish through an automated pipeline from the data's source. The pipeline checks the data against the rules for what can be published, produces open formats, and publishes the data with metadata: what it is, who owns it, how it was collected, how good it is and how often it changes.

flowchart LR
    accTitle: Publishing open data with metadata
    accDescr: Source data owned by an information asset owner flows into a publishing pipeline. The pipeline checks the data contains nothing personal or sensitive, validates it against data standards and measures its quality. It produces open formats and metadata in UK GEMINI or DCAT, then publishes them to the Defra Data Services Platform or data.gov.uk under the Open Government Licence. Users and other services download the data or call an API.
    SRC[("Source data<br/>named owner")] --> P["Publishing pipeline"]
    P --> CHK{"Safe to publish?<br/>no personal or<br/>sensitive data"}
    CHK -->|"no"| STOP["Stop and alert<br/>the owner"]
    CHK -->|"yes"| VAL["Validate against data<br/>standards, measure quality"]
    VAL --> OUT["Open formats<br/>and metadata"]
    OUT --> PUB["Defra Data Services Platform<br/>or data.gov.uk"]
    PUB --> USERS(["Users, researchers,<br/>other services"])

How it works:

  1. Confirm the data set has a named owner and is recorded in the information asset register.
  2. Check, automatically and on every run, that nothing personal, commercially sensitive or security-relevant is included. If the check fails, publish nothing and alert the owner.
  3. Validate against the data standards, so the data can be joined with other data sets.
  4. Publish in open formats (such as CSV, GeoJSON or GeoPackage) under the Open Government Licence.
  5. Publish metadata in UK GEMINI for geospatial data or DCAT for other data, including quality, lineage, update frequency and contact.
  6. Keep stable URLs for each data set and version, so people can cite them.

What users see

To be confirmed

TODO: what users of the published data see, such as the dataset page, licence and update dates, the content for them, and what to test with users. Write these three sections with a designer and a user researcher.

Content to design

To be written - see the box above.

What to test with users

To be written - see the box above.

Guardrails it helps you meet

Guardrail Level What the guardrail asks
GR-DATA-07 Open by default Should Publish non-personal, non-sensitive data as open data under the Open Government Licence, through the Defra Data Services Platform or data.gov.uk.
GR-DATA-05 Describe your data Should Publish metadata for data sets so they can be found and understood - UK GEMINI for geospatial data and DCAT for other data sets.
GR-DATA-03 Use agreed data standards and identifiers Should Use the data standards for dates, addresses, locations, identifiers and code lists, so data can be joined across services.
GR-DATA-08 Manage data quality Should Define, measure and report data quality using the Government Data Quality Framework, especially for data that feeds payments, regulatory decisions or official statistics.
GR-DATA-01 Every data set has an owner Should Each data set a service creates or holds has a named business owner (information asset owner) and is recorded in the information asset register.
GR-SEC-03 Classify information Must Identify the government security classification and data types the service handles, and design controls to match.

When not to use it

  • The data includes personal data that cannot be safely aggregated or anonymised. Share it under a data sharing agreement instead (GR-DATA-04).
  • Official statistics. Follow the Code of Practice for Statistics and your statistics producers' release process, which this pattern does not replace.