When your model risk committee asks you to explain what's happening inside a 7-billion-parameter model, performance metrics won't be enough. You need structured methods to characterize behavior in spaces too complex to visualize.
This checklist provides a framework for documenting and validating model behavior when traditional inspection methods fail. It's built around geometric and topological concepts that mathematicians use to understand high-dimensional spaces, the same spaces where your model weights and hidden activations live.
Purpose of the Checklist
Use this template to:
- Document model behavior beyond accuracy scores for validation evidence.
- Explain representation learning to auditors who ask, "What is the model actually doing?"
- Compare models with similar performance but different internal characteristics.
- Assess generalization risk, calibration, or robustness properties.
- Respond to regulatory requirements for model transparency, such as the EU AI Act Technical Documentation and SR 11-7 validation standards.
The checklist applies to any neural network where you have access to hidden activations and weights but cannot meaningfully visualize the decision space.
Prerequisites
Before using this checklist, ensure you have:
- Model access: Ability to extract hidden activations at each layer.
- Evaluation infrastructure: Test sets that represent your deployment distribution and known out-of-distribution scenarios.
- Computational resources: Intrinsic dimension estimation and curvature analysis can be memory-intensive for large models.
- Baseline metrics: Standard performance statistics (accuracy, calibration error, etc.) already calculated.
- Technical stakeholders: Someone who can interpret geometric properties and translate them into risk language.
The Checklist Template
Copy this into your model Validation Evidence. Customize the thresholds and specific analyses based on your risk tier and regulatory requirements.
MODEL CHARACTERIZATION CHECKLIST
Model ID: _______________
Validation Date: _______________
Risk Tier: _______________
Section 1: Intrinsic Dimension Analysis
Purpose: Estimate the true degrees of freedom in your data and representations, independent of the ambient dimension.
Training data intrinsic dimension calculated
Method used: _______________
Estimated dimension: _______________
Interpretation: Does this align with known data complexity? Lower intrinsic dimension relative to input dimension suggests data lives on a lower-dimensional manifold.Test set intrinsic dimension calculated
Estimated dimension: _______________
Deviation from training set: _______________
Risk flag: If test set dimension exceeds training set dimension by >20%, document distribution shift risk.Layer-wise intrinsic dimension tracked
Input layer: _______________
Middle layers: _______________
Final hidden layer: _______________
Pattern observed: (e.g., compression then expansion, monotonic decrease)
Expected behavior: Document whether pattern matches architecture intent (e.g., autoencoders should compress, classifiers often expand near decision boundary).Intrinsic dimension correlated with performance
Domains with higher intrinsic dimension: _______________
Corresponding error rates: _______________
Conclusion: Use this to explain why the model struggles in certain domains.
Section 2: Representation Geometry
Purpose: Characterize how the model organizes information spatially.
Curvature of representation manifold estimated
Layers analyzed: _______________
Curvature properties: (positive/negative/flat, magnitude)
Interpretation: High curvature regions may indicate decision boundaries or areas where small input changes cause large representation changes.Local neighborhoods examined
Sample points selected: _______________
Neighborhood radius: _______________
Homogeneity within neighborhoods: _______________
Risk flag: Heterogeneous neighborhoods suggest unstable predictions.Class separation measured
Distance metric used: _______________
Separation scores by class pair: _______________
Poorly separated classes: _______________
Mitigation: Document whether poor separation explains confusion matrix patterns.
Section 3: Weight Space Properties
Purpose: Understand model capacity and potential for overfitting or mode collapse.
Effective parameter count estimated
Total parameters: _______________
Effective rank or intrinsic dimension of weight matrices: _______________
Interpretation: Large gap between nominal and effective parameters may indicate redundancy or that the model hasn't fully utilized its capacity.Weight distribution characterized
Layers with high variance: _______________
Layers with low variance: _______________
Dead neurons identified: _______________
Action: If >10% of neurons are dead, document impact on model capacity.Symmetry properties documented
Known data symmetries: (e.g., translation, rotation)
Architectural symmetry enforcement: (e.g., convolutional structure)
Empirical symmetry in learned weights: _______________
Alignment: Does the model respect expected symmetries?
Section 4: Robustness Indicators
Purpose: Use geometric properties to predict and explain robustness failures.
Adversarial example detection via intrinsic dimension
Intrinsic dimension of adversarial examples: _______________
Deviation from clean examples: _______________
Detection threshold: _______________
Validation: Test on known adversarial set.Out-of-distribution detection via representation geometry
OOD samples tested: _______________
Geometric anomaly score: _______________
Comparison to in-distribution baseline: _______________
Threshold for flagging: _______________Local Lipschitz constants estimated
Regions analyzed: _______________
Maximum local Lipschitz constant: _______________
Interpretation: High values indicate sensitivity to small input perturbations.
Section 5: Documentation for Validation Evidence
Findings summarized in non-technical language
Key insight 1: _______________
Key insight 2: _______________
Key insight 3: _______________Limitations documented
What these analyses cannot tell you: _______________
Complementary analyses needed: _______________Risk implications stated
Elevated risks identified: _______________
Recommended controls: _______________Comparison to similar models
Reference models: _______________
Geometric properties comparison: _______________
Relative risk assessment: _______________
Customizing the Checklist
For lower-risk models: Skip Section 3 (weight space properties) and focus on Sections 1, 2, and 4. Use coarser estimation methods for intrinsic dimension.
For high-risk models (EU AI Act Annex III, SR 11-7 Tier 1): Add quantitative thresholds for each check. Require sign-off from a senior validator. Include temporal tracking, repeat the checklist quarterly and document drift in geometric properties.
For generative models: Add Section 6 to track intrinsic dimension evolution during the generation process (e.g., across diffusion timesteps). Document where in the process the model is most sensitive to perturbations.
For models with known data symmetries: Expand Section 3.3 to include quantitative tests of symmetry preservation. For example, if your data has rotational symmetry, verify that rotating an input and then passing it through the model yields the same result as passing it through and then rotating the representation.
For federated or privacy-preserving models: Adapt Section 1 to estimate intrinsic dimension on encrypted representations if using Homomorphic Encryption, or on aggregated statistics if using Federated Learning. Document limitations of estimation under these constraints.
Validation Steps
Pilot the checklist on a reference model where you already understand the failure modes. Verify that geometric properties align with known behavior.
Establish baselines: Run the checklist on 3-5 models in your inventory to calibrate what "normal" looks like for your domain and architecture family.
Set thresholds: Use your baseline data to define red flags. For instance, if your baseline models have training/test intrinsic dimension ratios between 0.9 and 1.1, flag any new model outside that range.
Integrate with existing validation workflows: Map each checklist section to a specific validation requirement. For example, Section 4 (robustness indicators) provides validation evidence for EU AI Act Article 15 (accuracy, robustness, and cybersecurity requirements).
Train your validators: Geometric concepts are unfamiliar to many model risk teams. Run a workshop where you walk through a completed checklist and explain what each finding means in risk terms.
Review and refine quarterly: As you accumulate data, you'll discover which checks provide the most signal. Retire low-value checks and add new ones based on incidents or near-misses.
When regulators or auditors ask how you know what your model is doing beyond test accuracy, hand them a completed checklist. You're not guessing, you're characterizing behavior in the only language that works when visualization fails.
