Amazon's verified semantic cache architecture shows how your team can reduce hallucinations in production LLM deployments while cutting costs and latency. This approach uses similarity scoring to route queries between curated answers, few-shot prompting, and full LLM processing.
How the Architecture Works
The solution implements a three-tier response system based on semantic similarity thresholds. When a user submits a query, the system calculates similarity against verified question-answer pairs stored in Amazon Bedrock Knowledge Bases. Queries scoring above 80% similarity return curated answers directly, bypassing the LLM entirely. Queries scoring between 60-80% trigger few-shot prompting, where verified answers guide the LLM's response. Queries below 60% similarity default to standard LLM processing.
This isn't theoretical. The architecture processes a sample query about AWS re:Invent dates in under one second by matching it at 85% similarity to a verified answer, eliminating both the hallucination risk and the inference cost of an LLM call.
Key Findings for Model Risk Teams
Similarity thresholds determine control strength. The 80% cutoff for deterministic answers represents a policy decision, not a technical constraint. Your team must calibrate this threshold based on domain risk. In regulated environments where factual precision matters, like pricing disclosures or clinical protocols, you might raise the threshold to 90% or implement human review for borderline matches. In lower-stakes contexts, you might accept 70% similarity for cached responses.
The cache becomes a validation artifact. Every curated question-answer pair in the semantic cache represents a documented control decision. You're explicitly defining correct outputs for specific inputs. This creates validation evidence under SR 11-7's ongoing monitoring requirements. When you update the cache, stored in Amazon S3 and synchronized to the knowledge base, you're performing model maintenance that should trigger change control procedures.
Few-shot prompting creates a middle ground. The 60-80% similarity band uses verified answers as examples rather than direct responses. This approach reduces hallucination risk compared to unconstrained generation, but it doesn't eliminate it. Your validation framework must treat few-shot responses differently than cached responses. You can't claim deterministic output, but you can document reduced variance.
Cache coverage becomes a performance metric. Track what percentage of production queries hit each tier: cached (>80%), guided (60-80%), or uncontrolled (<60%). If most queries fall below 60%, your cache isn't covering actual user needs. If everything hits above 80%, you might be over-indexing on FAQs and missing edge cases where the LLM adds value.
Offline updates prevent drift. The architecture separates cache updates from real-time operations. You add verified pairs to S3, then synchronize them to the knowledge base in batch. This design choice supports change management: you can review, approve, and version-control cache updates before they affect production responses. It also prevents the cache from learning incorrect information through user interactions.
Implications for Your Team
You can't validate an LLM the way you validate a logistic regression model, but you can validate the control architecture around it. The semantic cache creates three distinct validation scopes:
Scope 1: Cached responses (>80% similarity). These are deterministic. You validate the curated answer itself, the similarity calculation, and the threshold policy. Test whether inappropriate queries incorrectly match verified answers. Document what happens when a verified answer becomes outdated.
Scope 2: Few-shot responses (60-80% similarity). These are probabilistic but constrained. You validate whether the examples actually reduce hallucination rates compared to unconstrained generation. Measure response variance for queries in this band. Determine whether few-shot prompting introduces new failure modes.
Scope 3: Unconstrained responses (<60% similarity). These carry full LLM risk. Your existing RAG validation approach applies here. The cache doesn't help you; it just routes low-similarity queries to standard processing.
The architecture also shifts your monitoring strategy. Instead of trying to detect hallucinations in LLM outputs after the fact, you measure cache hit rates and similarity score distributions. If similarity scores drift lower over time, your cache is becoming stale relative to user needs. If scores cluster just below your 80% threshold, you're forcing too many queries into few-shot processing instead of adding them to the verified cache.
Action Items by Priority
Immediate (this sprint):
- Identify 20-50 questions where hallucinations would create Materiality or regulatory exposure. These become your initial cache entries.
- Define similarity thresholds for your risk context. Don't default to 80/60 without testing whether those cutoffs match your accuracy requirements.
- Establish a process for adding verified pairs to the cache. Who drafts them? Who approves them? How do you version-control the S3 bucket?
Near-term (this quarter):
- Instrument cache hit rates and similarity distributions in production. You need visibility into which tier handles each query.
- Test boundary conditions: queries that score 79% or 61% similarity. Verify that responses meet your accuracy standards at threshold edges.
- Document the cache architecture in your model inventory. The semantic cache is a control, and controls require documentation under ISO/IEC 42001 Annex A.
Ongoing:
- Review cache coverage monthly. If hit rates drop, either user needs have shifted or your cache needs expansion.
- Audit few-shot responses quarterly. Measure whether guided generation actually reduces hallucinations compared to unconstrained processing.
- Update verified answers when source material changes. Stale cache entries create a different kind of hallucination: confidently wrong answers that bypass the LLM entirely.
