Observability and operations¶
Services live for years. These guardrails make sure they can be run, supported and improved by people who did not build them.
Doctrine: 1. Platforms before projects. Built as products, 2. Standards and guardrails before exceptions
Principles: 1. Delivery-focused architecture
Who these guardrails apply to
The core department. Whether the observability and operations guardrails also apply to Defra's arm's length bodies is still to be confirmed.
Technology capability Operations: observability and security monitoring.
GR-OPS-01 Use the platform's observability tooling¶
Should Send structured logs, metrics and traces to the platform's observability tooling, and security events to the security operations centre.
GR-OPS-02 Log in a structured, safe way¶
Should Use structured (JSON) logs with correlation identifiers across service boundaries. Never log secrets, tokens or unnecessary personal data.
GR-OPS-03 Define and measure service levels¶
Should Agree service level objectives for availability, latency and correctness with the service owner, monitor them, and alert on what matters to users rather than on every metric.
GR-OPS-04 Health checks and graceful degradation¶
Should Expose health endpoints, set timeouts and retries on dependencies, and degrade gracefully (for example save progress and tell the user) when a dependency fails.
GR-OPS-05 Be ready for live before you go live¶
Must Before public beta, agree the support model, on-call arrangements, runbooks, incident process and who owns the service in live. Register the service in the service catalogue.
GR-OPS-06 Learn from incidents¶
Should Hold blameless post-incident reviews for significant incidents and share the lessons in the open where appropriate.
GR-OPS-07 Measure performance and cost¶
Should Publish the mandatory key performance indicators and tag cloud resources so cost can be attributed to the service and business capability.