Data and analytics

An exploratory starting shape for collecting, managing, analysing and publishing evidence and environmental data.

Draft - to be confirmed

Exploratory, early work. Service patterns are starting points for discussion, not agreed designs, and have not been reviewed by the Technical Design Authority. Talk to the architecture team before building on one.

Typical business capabilities: 01 Act as a custodian of the environment and ecosystems, 02 Conduct research and analyse evidence, and the insight behind every other capability.

Context

Defra is one of government's most data-intensive departments: environmental monitoring, scientific research, earth observation, spatial data, official statistics and the operational data generated by services. This service pattern describes how data moves from source to insight to publication, with governance throughout.

Architecture

flowchart LR
    accTitle: Data and analytics service pattern
    accDescr: Data from services, sensors, research and external sources lands in raw storage on the data platform, is curated and quality checked, and becomes data products used for analysis and AI, dashboards and statistics, open data and APIs, and geospatial services. Cataloguing, lineage, access control and quality apply across the platform.
    subgraph Sources
        S1["Services and casework"]
        S2["Sensors, gauges and IoT"]
        S3["Survey, sampling<br/>and research"]
        S4["External and<br/>earth observation data"]
    end

    subgraph Platform["Data platform"]
        RAW[("Raw / landing")]
        CUR[("Curated, quality-checked")]
        PROD[("Data products")]
        RAW --> CUR --> PROD
    end

    S1 -->|"events, APIs"| RAW
    S2 -->|"streaming"| RAW
    S3 --> RAW
    S4 --> RAW

    PROD --> AN["Analysis, models and AI"]
    PROD --> BI["Dashboards and<br/>official statistics"]
    PROD --> PUB["Open data and APIs"]
    PROD --> GEO["Geospatial services"]

    GOV["Catalogue, lineage,<br/>access control, quality"] -.-> Platform

Principles for this architecture

1. Data products, not data dumps. Curated data sets have an owner, a contract (schema, quality, refresh frequency) and documentation.
2. Govern once, at the platform. Access control, lineage, cataloguing and quality checks are provided by the platform, not reinvented per team.
3. Keep raw data immutable so analysis can be reproduced.
4. Spatial is a first-class concern. Use British National Grid (EPSG:27700) for storage and analysis of GB data and publish in standard formats - see data standards.
5. Publish openly by default through the Defra Data Services Platform.

Key decisions to record

• Classification and access model for each data product.
• Retention and archiving of raw data, particularly high-volume sensor data.
• Which data products are authoritative for which Defra on a page entities.