Testing
This section outlines how to write automated tests.
Unit Testing
Just as with writing source code, the generation of good unit tests requires clear requirements, rules for AI, a suitable model and good prompting. Read more about this in the four pillars
You may actually find that you need to put more time and effort into prompting and reviewing test generation than source code.
You must review and refine all tests generated by AI to ensure that the functionality is being tested as per your expectations.
The prompt library contains example prompts for writing unit test prompts. They advocate a style of testing that focuses on test clarity and readability, testing functionality over implementation, ensuring tests are not brittle and enable future refactoring or changes to be made confidently.
Unit testing is covered in both the development workflow and the test driven development workflow.
Journey Tests
In Defra, the term journey tests is used for testing the entire application and its dependencies via its primary interface — whether that be a human interface or a computer interface, such as an API. They test complete user workflows from start to finish.
Prerequisites
- Clear requirements: Write detailed feature requirements with defined scope, as outlined in Product Requirements
- AI rules: Set up project Rules for AI to define apporach and patterns for writing journey tests
- Capable model: Use the most capable large language model (LLM) available for your task to get quality results
1. Prompt the Coding Assistant
- Use an AI Coding Assistant (AICA) in Agent mode
- Use prompt-add-journey-test and reference your requirements files directly with the Coding Assistant Agent
- Generate the initial tests
- Accept changes, then review them in the git diff viewer
2. Run the Tests and Refine
- Execute the tests
- If issues arise with the tests themselves, feed the results back into the chat to address them, providing sufficiently detailed prompting to guide the best course of action
- If there are opportunities for better test coverage, prompt the Coding Assistant with precise guidance and include the output from the test coverage report
3. User Review
- Review and understand tests thoroughly
- Refine appropriately
- Do not allow agents to generate tests unchecked
Guidelines
Let the requirements drive the tests: It is crucial that the requirements drive the tests, not the source code. This is achieved by referring to the requirements file when prompting, as per the workflow above.
Let the requirements drive the outcomes: When bugs or issues are found running the journey tests, ask the model to evaluate the issue against the original requirement and determine whether it is an issue with the test or an issue with the source code.
Be very clear on test terminology: Software testing often suffers from unclear terminology, with similar terms used interchangeably or different terms referring to the same concept. Just as this lack of standardisation can lead to miscommunication among team members, it can also affect how models interpret prompts. Be very clear in both your rules for AI and your prompts, as per the examples in this playbook.
