Category: Trustworthy AI Principles

Generative AI

Also known as: GenAI, gen AI, generative artificial intelligence

Simply put

Generative AI refers to a class of artificial intelligence systems that create new content—such as text, images, audio, video, or software code—based on patterns learned from existing data. Rather than only classifying or predicting from inputs, these systems produce original outputs, often in response to a user prompt. Large language models (LLMs) are a widely cited example of this type of AI.

Formal definition

Generative AI is commonly defined as a subfield of AI that uses generative models to produce new content across modalities including text, images, video, and audio, based on statistical patterns learned from training data. In practice, these systems generate outputs conditioned on inputs such as prompts, and large language models represent one prominent implementation. Note that the evidence provided describes GenAI in general functional terms only; it does not specify model architectures, training methods, or governance and risk-management treatment. From a model risk perspective, generative systems raise distinct considerations (for example, output variability and content generation risk) that differ from those of traditional predictive models, but the evidence packet does not address these, so such matters are out of scope for this definition.

Why it matters

Generative AI has moved quickly from a research topic to a technology embedded in a wide range of enterprise workflows, which makes it a growing focus for governance and risk functions. Because these systems produce new content rather than only classifying or predicting from inputs, they introduce considerations that differ in character from those associated with traditional predictive models. Compliance officers, model risk managers, and auditors increasingly encounter GenAI in contexts where organizations must decide how existing oversight structures apply to a system type that behaves differently from what many frameworks were originally designed to address.

The practical significance lies in the gap between how GenAI is commonly described and how it is governed. The evidence available here defines GenAI in general functional terms—as a class of systems that create text, images, audio, video, or code based on patterns in existing data—but does not address model architectures, training methods, or risk-management treatment. For governance professionals, this means that classifying a system as GenAI is only a starting point; the specific risk profile, applicable controls, and regulatory treatment depend on details that a functional definition does not supply and that must be assessed case by case.

Because GenAI is an evolving area, definitions and regulatory approaches remain in flux across jurisdictions and sectors. Professionals should treat the term as a broad category rather than a precise indicator of any single risk posture or compliance obligation. Where an organization's policies, or an applicable framework, attach specific requirements to generative systems, those requirements—not the general definition—determine what is actually expected.

Who it's relevant to

AI governance specialists and policy teams

Those responsible for organizational oversight of AI need a clear working definition of what qualifies as a generative system, since governance policies increasingly distinguish generative from predictive AI. This entry provides the functional definition but does not prescribe governance structures; teams should map the term to their own policies and any applicable frameworks to determine accountability and oversight expectations.

Model risk managers

Model risk functions may be asked to bring generative systems within their remit even where existing practices were designed around traditional predictive models. This definition establishes what GenAI is at a functional level but does not address validation, monitoring, or the specific risk considerations generative systems raise; those must be assessed separately against the relevant risk-management approach.

Compliance officers and legal professionals

Professionals evaluating regulatory exposure need to recognize when a system falls into the generative category, as some emerging and sector-specific requirements treat such systems distinctly. Because regulatory treatment of GenAI is evolving and varies by jurisdiction, this general definition should not be read as indicating any particular legal obligation; specific requirements depend on the applicable law or guidance.

Auditors and second- and third-line reviewers

Reviewers assessing controls over AI systems benefit from a consistent understanding of the term when scoping engagements. This entry supports classification of a system as generative but does not describe the controls, testing, or evidence expectations appropriate to such systems, which fall outside its scope.

Inside GenAI

Generative models

The underlying machine learning models—commonly large language models, diffusion models, or other generative architectures—that produce new content (text, images, code, audio, or other outputs) rather than solely classifying or predicting from fixed labels. The specific architecture matters for how outputs and risks are characterized.

Training data and provenance

The corpora used to train or fine-tune generative models. Provenance, licensing, and representativeness of this data bear on intellectual property, privacy, and bias concerns, and are frequently a focus of governance review.

Generated outputs and their variability

The content produced in response to inputs or prompts. Outputs are typically probabilistic and can vary across runs for the same input, which distinguishes GenAI behavior from deterministic model outputs and complicates reproducibility and testing.

Prompting and input interfaces

The mechanisms—such as user prompts, retrieval-augmented context, or system instructions—through which users direct model behavior. These interfaces are a common point of control and a common source of unintended or adversarial behavior.

Foundation vs. fine-tuned or third-party components

GenAI deployments often combine a base (foundation) model, possibly supplied by a third party, with organization-specific fine-tuning, adapters, or integration layers. The split of responsibility across these components is relevant to accountability and third-party risk considerations.

Human oversight and review points

The points at which humans review, approve, or intervene in generated outputs. The degree of human involvement typically influences how the associated risk is assessed and controlled.

Common questions

Answers to the questions practitioners most commonly ask about GenAI.

Is generative AI just a more advanced form of the predictive models covered by traditional model risk management?

Not exactly. Generative AI produces new content (such as text, images, or code) rather than only producing a score, classification, or forecast, which is the typical output of the predictive or statistical models historically addressed by model risk management guidance such as SR 11-7. That said, generative systems can still fall within the scope of model risk management where an organization treats them as models, and many of the same principles—validation, monitoring, and controls—are often applied. The pitfall is assuming that frameworks designed around well-defined inputs and quantifiable outputs map cleanly onto generative systems, whose outputs are open-ended and harder to evaluate against a single ground truth.

Does deploying generative AI responsibly mean the same thing as having AI governance in place?

These are related but distinct. AI governance refers to the organizational structures, policies, accountability, and oversight that guide how AI systems are developed and used. Responsible deployment of a generative system draws on that governance but also depends on operational risk controls, testing, and monitoring specific to the system. Governance provides the framework within which generative AI is managed; it does not by itself constitute the technical risk measurement and control activities. Professionals sometimes conflate the two, treating a governance policy as sufficient evidence that a generative system's risks have been assessed and controlled.

How might an organization approach validating a generative AI system when there is no single correct output to compare against?

Because generative outputs are open-ended, validation typically shifts from comparing predictions against known outcomes toward evaluating output quality, consistency, and appropriateness across a range of scenarios. In many frameworks this may involve structured testing for accuracy, relevance, and undesired behaviors, human review, and defined acceptance criteria. Approaches vary by use case and remain an evolving area of practice, so organizations often document the limitations of their validation methods rather than treating validation as complete or definitive.

What controls do organizations commonly consider to manage risks specific to generative AI outputs?

Commonly discussed measures include human oversight of outputs, restrictions on use cases, testing for undesired or misleading content, monitoring of system behavior over time, and clear documentation of intended use and limitations. Such controls are typically framed as measures that reduce or manage risk rather than eliminate it. The appropriate set of controls generally depends on the use case, the sensitivity of the context, and applicable regulatory or organizational requirements.

How does the three lines of defense model apply to generative AI systems?

As commonly defined, the first line owns and manages the system and its risks, the second line provides independent oversight and challenge, and the third line offers independent assurance. Applying this model to generative AI raises practical questions about where responsibility sits, particularly when systems are procured from third parties rather than built in-house. Organizations often need to clarify roles across these lines, but the specific allocation varies by organization and is not standardized across all sectors.

What monitoring considerations apply once a generative AI system is in production?

Ongoing monitoring often addresses whether the system continues to behave as intended, whether output quality changes over time, and whether use extends beyond the intended scope. This may include tracking undesired outputs and reviewing changes to underlying models or data. Monitoring generative systems can be more challenging than monitoring predictive models because there is often no single quantifiable performance metric, so organizations typically define what they will observe and acknowledge the limits of those measures.

Common misconceptions

Generative AI is simply a type of model that fits neatly within existing model risk management practices designed for predictive or statistical models.

GenAI shares some concerns with traditional model risk management, but its probabilistic, variable outputs, open-ended input space, and frequent reliance on third-party foundation models can complicate conventional validation and monitoring approaches. Whether and how existing frameworks apply is often context-specific and, in many settings, still evolving rather than settled.

Governance controls and human review make generative AI outputs reliable or eliminate the risk of incorrect or harmful content.

Oversight measures reduce and help manage risk but do not eliminate it. Generative outputs can be inaccurate or unintended even where review points exist, so controls are best described as risk-reducing rather than risk-removing.

A single definition of generative AI applies uniformly across regulatory frameworks and sectors.

Definitions and treatment of GenAI vary by jurisdiction, by instrument, and by sector, and in several regimes the regulatory approach is emerging rather than fixed. Practitioners should scope the applicable definition to their specific framework and use case rather than assume interchangeability.

Best practices

Document the composition of any GenAI deployment—foundation model, fine-tuning, integration layers, and third-party components—so accountability and third-party dependencies are clearly mapped.

Record and assess training or fine-tuning data provenance and licensing where available, treating gaps in provenance as a limitation to be noted rather than assumed away.

Establish defined human oversight and review points for generated outputs, and calibrate their intensity to the assessed risk of the use case.

Account for output variability by testing behavior across repeated and varied inputs rather than relying on single-run reproducibility, and document known limitations of such testing.

Scope any applicable regulatory or standards obligations to the specific jurisdiction, instrument, and sector, using qualified language where treatment is still evolving.

Frame governance and monitoring controls as measures that reduce and manage risk, and communicate residual risk explicitly to relevant stakeholders.