Category: Trustworthy AI Principles

Secure and Resilient

Also known as: Security and Resilience

Simply put

"Secure and resilient" describes a system's ability to both protect against adverse events, such as attacks or accidents, and to keep functioning or quickly recover when those events occur. Security emphasizes preventing and defending against threats, while resilience emphasizes withstanding, adapting to, and recovering from disruptions. The two goals are related but distinct, and a system can be relatively strong in one while weaker in the other.

Formal definition

As commonly used in cybersecurity and critical infrastructure contexts, security refers to controls and measures intended to protect systems and data against threats, while resilience refers to the ability to prepare for, adapt to, withstand, and recover from deliberate attacks, accidents, or naturally occurring threats or incidents (per NIST's glossary treatment of resilience and CISA's usage). In AI system contexts these terms are frequently paired to indicate both the protection of a system against compromise and its capacity to maintain or restore intended function under adverse conditions. The precise scope varies by framework and sector, and the evidence provided does not supply an AI-specific or standardized combined definition; this entry therefore generalizes from cybersecurity and infrastructure sources rather than from a single authoritative AI governance instrument. Practitioners should not treat "secure" and "resilient" as synonymous: strong preventive security does not guarantee resilience, and controls described here reduce and manage risk rather than eliminate it.

Why it matters

In AI governance and model risk contexts, treating "secure" and "resilient" as a single objective is a common and consequential error. Security concentrates on preventing and defending against threats, while resilience concentrates on continuing to function or recovering after a disruptive event occurs. A system can be relatively strong in one dimension and weak in the other: robust preventive controls do not guarantee that a system will withstand or rapidly recover from an incident, and a highly recoverable system may still be poorly defended against compromise. Distinguishing the two goals helps organizations avoid a false sense of assurance from investing heavily in prevention alone.

For AI systems specifically, the pairing signals two related concerns: protecting the system against compromise and maintaining or restoring its intended function under adverse conditions. The evidence available here draws on cybersecurity and critical infrastructure sources (such as NIST's glossary treatment of resilience and CISA's usage) rather than a single standardized AI-specific definition, so practitioners should recognize that the precise scope varies by framework and sector. This matters for governance because the applicable expectations for an AI system may be shaped by whichever cybersecurity or infrastructure framework an organization has adopted, and those frameworks do not use fully interchangeable definitions.

The practical stake is that controls described under these terms reduce and manage risk rather than eliminate it. Framing security and resilience as distinct but complementary objectives allows compliance officers, risk managers, and auditors to test each capability separately: whether threats are being prevented and defended against, and, independently, whether the organization can continue to deliver intended outcomes and recover when adverse events nonetheless occur.

Who it's relevant to

Model Risk Managers

Model risk managers use the distinction to avoid conflating preventive security controls with the ability of an AI system to keep functioning or recover after an incident. Assessing security and resilience separately supports more accurate risk measurement and monitoring, since strong defenses do not by themselves establish that a system can withstand or recover from disruption.

Compliance Officers and Auditors

Compliance and audit professionals benefit from testing prevention and recovery as separate capabilities. Because the applicable definitions vary by cybersecurity or critical infrastructure framework, these professionals should confirm which framework governs a given system rather than assuming a single universal standard, and should document that controls reduce and manage risk rather than eliminate it.

Security and Infrastructure Teams

Teams responsible for protecting AI systems apply security controls to defend against threats and resilience measures to maintain or restore intended function under adverse conditions. This audience is closest to the source cybersecurity and infrastructure framing (such as NIST and CISA usage) from which the paired term is generalized.

Policy and Governance Specialists

Policy specialists translating cybersecurity and infrastructure concepts into AI governance should note that no single authoritative AI-specific combined definition is supplied by the available evidence. They should scope requirements to the relevant framework and sector and use qualified language when describing expectations, rather than presenting a standardized definition as settled.

Inside Secure and Resilient

Security

In the context of AI systems, security typically refers to the protection of a system and its components—including training data, model artifacts, and inference endpoints—against unauthorized access, tampering, and adversarial manipulation. As commonly framed in the NIST AI Risk Management Framework (issued by the U.S. National Institute of Standards and Technology, a voluntary framework rather than binding law), security is one dimension among several characteristics of trustworthy AI.

Resilience

Resilience typically refers to a system's ability to withstand, adapt to, and recover from adverse events, disruptions, or unexpected conditions while maintaining acceptable function. It is often distinguished from security in that resilience concerns continuity and graceful degradation under stress, whereas security concerns protection against threats. The two overlap but are not interchangeable.

Adversarial threat considerations

AI systems can be subject to threats that differ from those affecting conventional software, such as attempts to manipulate model behavior through crafted inputs or to compromise data used in training. Addressing these is commonly treated as part of securing an AI system, though the specific taxonomy of such threats varies across frameworks and is still evolving.

Recovery and continuity measures

Measures intended to restore acceptable operation after a disruption, such as fallback procedures, backups, or the ability to safely take a system offline. These support the resilience dimension and are generally described as risk-reducing rather than risk-eliminating.

Relationship to governance and model risk management

Secure and resilient practices sit at the intersection of AI governance (the organizational structures, policies, and accountability for oversight) and model risk management (the identification, measurement, monitoring, and control of risks from model use). Governance may assign responsibility for security and resilience, while model risk management may treat security or resilience failures as sources of model risk; the two functions overlap here without being the same.

Common questions

Answers to the questions practitioners most commonly ask about Secure and Resilient.

Is being 'secure and resilient' the same as ordinary IT cybersecurity for AI systems?

Not exactly. While secure and resilient AI draws heavily on established cybersecurity practices, the property as commonly framed also addresses risks specific to AI systems—such as adversarial manipulation of inputs, data poisoning, and model extraction—that fall outside the scope of conventional IT security controls. Professionals frequently err by assuming that a mature enterprise cybersecurity program automatically satisfies this characteristic; AI-specific threat surfaces typically require additional consideration. The precise boundaries vary by framework and are still evolving.

Does making an AI system 'secure and resilient' mean it can withstand any attack or failure?

No. Security and resilience are risk-reduction properties, not guarantees. As commonly defined, resilience refers to a system's ability to withstand adverse events and to recover or degrade gracefully rather than fail catastrophically, while security concerns protection against unauthorized access and manipulation. Neither eliminates risk; they manage and reduce it. Describing a system as secure and resilient should not be read as implying immunity from compromise or disruption.

What is the difference between the 'secure' and the 'resilient' aspects when implementing controls?

These are related but distinct. Security controls typically focus on preventing and detecting unauthorized access, tampering, and misuse—for example, access management, input validation, and protection against adversarial inputs. Resilience controls focus on maintaining function under stress and recovering afterward—for example, redundancy, graceful degradation, monitoring, and incident recovery procedures. Treating them as a single undifferentiated objective is a common implementation pitfall; teams generally address them through overlapping but separate measures.

Which line of defense is responsible for security and resilience of AI systems?

Responsibility is typically distributed rather than assigned to a single function. In a common three-lines model, the first line (system owners and developers) generally builds and operates controls, the second line (risk and compliance functions) sets policy and provides independent oversight and challenge, and the third line (internal audit) provides independent assurance. The specific allocation depends on an organization's governance structure and is not standardized across all frameworks.

How might an organization test or evaluate whether an AI system is secure and resilient?

Approaches commonly discussed include adversarial testing (sometimes called red-teaming) to probe for manipulation and evasion, stress or scenario testing to observe behavior under degraded or unexpected conditions, and monitoring for anomalous inputs or outputs in production. These evaluation methods relate to but are distinct from validation of model performance; assessing security and resilience concerns how the system holds up under adverse conditions rather than whether it is accurate under normal ones. Testing methodologies are not uniform across frameworks or sectors.

How does this characteristic relate to ongoing monitoring rather than one-time assessment?

Security and resilience are generally treated as ongoing properties rather than a state established once at deployment. Because threat environments, data, and system behavior change over time, organizations typically implement continuous monitoring, periodic re-testing, and incident response and recovery processes. A common error is to treat a pre-deployment security review as sufficient; sustaining these properties usually requires controls that operate throughout the system lifecycle. The appropriate frequency and depth of such activities vary by risk level and context.

Common misconceptions

Secure and resilient mean the same thing.

As commonly defined, security concerns protection against unauthorized access, tampering, and adversarial manipulation, while resilience concerns the ability to withstand and recover from disruptions. A system can be secure yet not resilient, or resilient yet insufficiently secure; experts generally keep these distinct.

Implementing security and resilience controls eliminates the risk of AI system failure or compromise.

Controls typically reduce or manage risk rather than eliminate it. Residual risk generally remains after controls are applied, and characterizing controls as guarantees overstates what they achieve.

A single framework universally defines what secure and resilient AI requires.

Different instruments treat these characteristics differently and have different scope and legal status. For example, the NIST AI Risk Management Framework is a voluntary framework, not binding law, and its treatment of these characteristics is not automatically interchangeable with requirements found in other regimes. Definitions and expectations remain jurisdiction- and framework-dependent.

Best practices

Treat security and resilience as separate objectives with distinct controls, and document which measures address protection against threats versus which address continuity and recovery.

Clarify, through governance structures, who is accountable for security and resilience of each AI system, and connect these responsibilities to model risk management activities such as monitoring and control.

Assess the specific adversarial and disruption scenarios relevant to a given system rather than assuming conventional software controls are sufficient, given that AI systems can face threats that differ from traditional software.

Frame security and resilience measures as steps that reduce or manage risk, and explicitly track and document residual risk that remains after controls are in place.

Identify which framework or regulatory regime applies to your context, note its scope and legal status, and avoid treating voluntary frameworks and binding requirements as interchangeable.

Maintain and periodically test recovery and continuity measures, such as fallback procedures and the ability to safely take a system offline, so resilience is verified rather than assumed.