The question at hand
Your AI governance framework likely covers model validation, data lineage, and bias testing. But does it address why your data scientists might disable security controls to meet a sprint deadline? Or why your compliance team skips documentation steps when the workflow tool is cumbersome?
The launch of NIST's Human-Centered Cybersecurity Community of Interest (COI) raises a broader question for AI governance leaders: should human factors research be a formal part of your governance controls, or does that complicate compliance unnecessarily?
This isn't just theoretical. The gap between documented procedures and actual human behavior creates vulnerabilities. The debate is whether closing that gap requires embedding human-centered design principles into your governance framework or simply enforcing existing controls more rigorously.
The case for embedding human factors
Advocates for human-centered governance argue that technical controls often fail when they conflict with how people work.
Consider your model approval workflow. If the process requires uploading Technical Documentation (Annex IV) through a system that times out on large files, your team will find workarounds. They'll split documents, email attachments outside the audit trail, or skip optional sections. Your governance framework says one thing; human behavior produces another.
The human-centered approach sees this as a design problem, not just a compliance issue. Under ISO/IEC 42001, your AI Management System must include controls that are both effective and practical. If Annex A controls create friction that people bypass, you don't have effective controls. You have documented theater.
Embedding human factors means:
- Testing governance workflows with actual users before rollout
- Measuring time-to-complete for required tasks and treating excessive duration as a control failure
- Conducting Root Cause Analysis on policy violations that trace back to interface design, not just user error
- Building Stakeholder Engagement into control design, not just control communication
The NIST COI's focus on connecting researchers and practitioners offers a structured way to bring this expertise in-house. Researchers study why security behaviors fail; practitioners know which controls get ignored. The COI aims to facilitate the exchange of ideas and feedback between these groups, creating a feedback loop that could inform more realistic governance requirements.
For AI systems, the stakes are higher. Your Model Cards might document Model Limitations and Use Restrictions perfectly, but if the interface makes those restrictions invisible to end users, you haven't met your Disclosure of AI Interaction obligations. The EU AI Act requires Instructions for Use that people can actually follow. That's a human factors requirement disguised as a documentation requirement.
The case for enforcement over redesign
The opposing view: your governance framework already works if you enforce it consistently. Adding human factors research creates new dependencies, delays control implementation, and introduces subjective criteria into processes that need objective audit trails.
This camp argues that treating policy violations as design problems excuses non-compliance. If your data scientists disable logging to speed up model training, that's not a usability issue. That's a violation of your Post-Market Monitoring requirements that needs disciplinary action and better training.
The practical concerns are real:
- Human factors research takes time you don't have during incident response
- Usability testing delays control deployment when regulators expect immediate remediation
- "User-friendly" often means fewer verification steps, which conflicts with Validation Evidence requirements
- Your audit trail needs to show controls were followed, not that controls were comfortable
Under SR 11-7, your model risk management framework must demonstrate effective challenge. If you redesign controls every time users complain they're burdensome, you're not challenging model risk. You're negotiating it down.
The enforcement-first approach focuses on:
- Clear accountability for control violations
- Training that emphasizes why controls exist, not just how to complete them
- Monitoring that detects workarounds before they become normalized
- Consequences that make non-compliance more painful than compliance friction
For regulated entities, this matters. Your examiner doesn't care that your feature store interface is clunky. They care whether your Feature Store maintains lineage for all production features. If your team bypasses the feature store because it's slow, you have a compliance gap regardless of the UX justification.
Where practitioners actually land
Most governance teams end up in the middle, though they rarely formalize it.
You'll redesign controls when the friction is severe enough to create systematic violations. You'll enforce controls when violations are isolated or when regulatory deadlines don't allow redesign time. The decision is usually reactive and political rather than principled.
What you don't see often: a formal process for evaluating when human factors research should inform control design. The NIST COI provides a venue for that conversation, but most organizations lack an internal mechanism to act on it.
The practical middle ground looks like:
- Triaging control violations by root cause (design flaw vs. willful bypass)
- Setting a threshold (if more than X% of users report a control as impractical, trigger a redesign review)
- Building usability requirements into your control validation process from the start
- Maintaining enforcement for critical controls while accepting iteration on operational controls
Our take
Embed human factors analysis into your control design process, but don't let it become an excuse for weak controls.
The EU AI Act's emphasis on Instructions for Use and the NIST AI RMF's focus on Contextual Risk Factors both point toward the same reality: governance controls that ignore human behavior don't survive contact with production environments. If your Responsible Disclosure process is so cumbersome that security researchers give up and post vulnerabilities publicly instead, your process failed regardless of what your documentation says.
But human-centered design doesn't mean eliminating verification steps or making controls optional. It means making required actions the path of least resistance. Your Data Protection Impact Assessment template should be easier to complete correctly than to skip. Your Model Recalibration workflow should make it harder to deploy an outdated model than to deploy the current one.
Start here: the next time you investigate a governance control violation, ask whether the documented process was technically possible to complete within normal working constraints. If it wasn't, you have a design problem. If it was and someone chose not to, you have an enforcement problem. Different root causes require different remediation.
The NIST COI offers a practical resource for this work. Researchers who study security behavior can help you identify which friction points predict systematic violations. Practitioners can validate whether proposed controls will survive real-world implementation. That exchange of ideas and feedback between researchers and practitioners creates governance controls that are both rigorous and sustainable.
Your governance framework should be hard to bypass accidentally and easy to follow correctly. That's not a contradiction. That's human-centered design applied to compliance.
