Responsible AI
GOV.UK
Please refer to: GovUK Responsible AI Toolkit
Microsoft
Please refer to Azure AI Responsible AI
General
-
Resources and information: Evaluating LLM systems: Metrics, challenges, and best practices
-
Checklist for AI deployment: USAID Artificial Intelligence Ethics Checklist.pdf
In addition to examining the model from various perspectives, such as data source, model design, and production environment, the best practice is to evaluate the LLM application using pre-designed questions in different RAI categories, including:
| Potential harm categories | Harm description with sample evaluation datasets |
|---|---|
| Harmful content | Self-harm Hate Sexual Violence Fairness Attacks Jailbreaks: System breaks out of instruction, leading to harmful content |
| Regulation | Copyright Privacy and security Third-party content regulation Advice related to highly regulated domains, such as medical, financial and legal Generation of malware Jeopardizing the security system |
| Hallucination | Ungrounded content: non-factual Ungrounded content: conflicts Hallucination based on common world knowledge |
| Other categories | Transparency Accountability: Lack of provenance for generated content (origin and changes of generated content may not be traceable) Quality of Service (QoS) disparities Inclusiveness: Stereotyping, demeaning, or over- and underrepresenting social groups Reliability and safety |