Safety & Alignment

Safety asks whether a system can cause harm or fail in ways that matter. Alignment asks whether its behaviour remains within the intent, constraints and oversight that people have set for it. The two concerns overlap, but neither can be reduced to a single benchmark score.

NIST’s AI Risk Management Framework treats risk management as an ongoing organisational activity: teams govern, map, measure and manage risks rather than assuming that a model’s initial release settles them.¹

From intent to evidence

An intended use should make clear who may use a system, for which tasks, and which outcomes require escalation or review. Tests can then examine the failure modes that are relevant to that context: unsafe outputs, unreliable operation, misuse, or a loss of human control. This is an editorial inference from the framework’s risk-management approach, not a claim that one universal test can establish safety.

Generative AI can introduce additional risks through its ability to produce novel text, images or other outputs. NIST’s Generative AI Profile identifies actions that organisations can use alongside the AI RMF to address those risks.²

What this topic covers

This hub follows the practical work around intended use, red-teaming, safeguards, human oversight, incident response and monitoring. It distinguishes evidence of a specific control from a broad assurance claim, and keeps the conditions under which a safety result was obtained visible.

Footnotes

1. NIST, AI Risk Management Framework — full source details. ↩

2. NIST, Generative AI Profile — full source details. ↩