Public evals
My post Evals as Product Feedback argues for small, durable eval sets you actually read. This page holds Ask my notes to that standard: 47 hand-checked cases, run against the exact pipeline that serves visitors, results committed to the repo. Failures stay on the board.
Refusal correctness
40/4785%
Retrieval accuracy
24/3177%
Last run Mode: retrieval-onlyNo model call, retrieval & gate only
Retrieval-only runs grade the grounding layer (the gate and the ranking) without spending tokens; full runs additionally grade generated answers.
What fails and why
12 of 47 cases fail. The retriever was not tuned to pass them, so each one stays in the set with its cause.
Wrong post ranked firstParaphrase
When the search step comes back empty-handed, should the chatbot still improvise a reply?
Top match: “The feature flag I always stage behind” in decomposing-custom-apps-without-breaking-prod, score 5.86
Shares no keyword with the section that answers it; ‘step’ and ‘back’ land on a ServiceNow migration section instead.
Wrong post ranked firstParaphrase
My assistant keeps hammering one costly API over and over. How do I cap it?
Top match: “Cite the chunks, not the documents” in rag-that-doesnt-lie, score 4.68
Shares no keyword with the tool budget section; ‘one’ and ‘over’ land on a RAG section instead.
Wrong post ranked firstParaphrase
Is a tiny handpicked test suite better than a huge synthetic collection?
Top match: “What felt like solutions looking for problems” in now-assist-rollout-notes, score 4.71
Shares no keyword with the section that answers it; ‘better’ and ‘than’ alone land on a Now Assist section.
Refused, should answerParaphrase
After reviewing configuration data, how do I keep it from rotting again?
Top match: “Introduction” in now-assist-rollout-notes, score 4.05
Shares no keyword with the section that answers it; its best match, the Now Assist intro, shares only ‘again’, so the term check refuses it.
Refused, should answerParaphrase
Should machine-written replies that reach clients be labeled from launch?
Top match: “Hold out a query set you didn't build the system for” in rag-that-doesnt-lie, score 3.20
Shares no keyword with the guardrail section; its best match shares only ‘written’, so it scores under the 3.6 gate and fails the term check too.
Wrong post ranked firstParaphrase
How do I make sure a retried outbound call never files the same ticket twice?
Top match: “Retrieval has to fail loudly” in rag-that-doesnt-lie, score 6.46
Shares no keyword with the idempotency section; ‘call’ and ‘never’ land on the RAG post instead.
Answered, should refuseNear-miss
How do I fine-tune an embedding model for better retrieval?
Top match: “Introduction” in rag-that-doesnt-lie, score 9.20
‘Fine’ (from ‘reads fine’), ‘model’ and ‘retrieval’ match the RAG intro, but no post covers fine-tuning embeddings.
Answered, should refuseNear-miss
How much does a Now Assist license cost per user?
Top match: “Introduction” in now-assist-rollout-notes, score 6.50
The product name alone gives two matching terms, enough for the gate; the rollout post never mentions pricing.
Answered, should refuseNear-miss
How do I write a business rule script in ServiceNow?
Top match: “Introduction” in decomposing-custom-apps-without-breaking-prod, score 6.56
‘Business’ and ‘ServiceNow’ match the decomposition post, which mentions business rules but never shows how to write one.
Answered, should refuseNear-miss
Which GPU should I buy to train a neural network?
Top match: “Sigmoid vs. ReLU” in neural-networks-101, score 5.39
‘Neural’ from the post title and ‘train’ from the activation table pass the gate; nothing says which GPU to buy.
Answered, should refuseNear-miss
How do I configure Discovery credentials for the CMDB?
Top match: “The five queries I trust more than any dashboard” in cmdb-health-honest-audit-playbook, score 7.19
‘Discovery’ and ‘CMDB’ match the audit playbook, which never covers Discovery credentials.
Wrong post ranked firstTwo-post
How do I keep retries from doing the same work twice, in flows and in agent loops?
Top match: “Introduction” in agent-loops-that-dont-sprawl, score 9.13
‘Agent’ and ‘loops’ sit in the agent post’s title on every chunk; the idempotency section shares no keyword with the question.
Gate explorer
Ask answers only when its best section scores at or above the gate and passes a term check. Slide the gate to see what a stricter or looser setting gets wrong. At 3.6, the live setting, this is the committed run. Below 3.6 nothing changes on this set: every case that scores under it also fails the term check. The gate only decides whether Ask answers, so cases that rank the wrong post first are a ranking problem no gate setting fixes.
In scope
- Should machine-written replies that reach clients be labeled from launch? Top score 3.20, refused at any gate by the term check.
- What is RAG? Top score 3.69, answered.
- What are evals? Top score 3.70, answered.
- What is a CMDB? Top score 3.84, answered.
- After reviewing configuration data, how do I keep it from rotting again? Top score 4.05, refused at any gate by the term check.
- My assistant keeps hammering one costly API over and over. How do I cap it? Top score 4.68, answered.
- Is a tiny handpicked test suite better than a huge synthetic collection? Top score 4.71, answered.
- When should a RAG system refuse to answer? Top score 5.81, answered.
- When the search step comes back empty-handed, should the chatbot still improvise a reply? Top score 5.86, answered.
- How do I make sure a retried outbound call never files the same ticket twice? Top score 6.46, answered.
- How do I get people to trust AI-generated answers at work? Top score 6.49, answered.
- How do I roll out a risky platform change one step at a time? Top score 7.35, answered.
- Who should I interview during a CMDB audit? Top score 7.38, answered.
- Should evals block releases when they regress? Top score 7.43, answered.
- Which Now Assist capabilities earned user trust fastest? Top score 8.90, answered.
- How do I keep retries from doing the same work twice, in flows and in agent loops? Top score 9.13, answered.
- How many days without an update makes a CI stale? Top score 9.13, answered.
- What is a neuron in a neural network, actually? Top score 10.22, answered.
- How do I stop an agent loop from sprawling? Top score 10.66, answered.
- What should I log every turn to debug an agent? Top score 12.48, answered.
- How many examples should an eval set start with? Top score 12.68, answered.
- Who should write the questions I use to test a RAG system, and how many? Top score 12.78, answered.
- What should I extract first when splitting a sprawling custom app? Top score 12.99, answered.
- What queries should I run to audit CMDB health? Top score 13.27, answered.
- How should a subflow's error handler classify failures? Top score 14.07, answered.
- Why should citations point to chunks instead of documents? Top score 14.96, answered.
- How do I migrate tables to a new scope without breaking production? Top score 15.32, answered.
- How should I name Flow Designer flows so they survive refactors? Top score 16.92, answered.
- What values does the read_target migration property take? Top score 18.03, answered.
- Why is stuffing more context into the prompt an anti-pattern? Top score 18.86, answered.
- Why was auto-routing low-volume queues a poor fit for Now Assist? Top score 19.95, answered.
Out of scope
- How much does a WSD license cost? No section matched, refused.
- What's your favorite pizza topping? No section matched, refused.
- How do I configure a Kubernetes ingress controller? No section matched, refused.
- What's the best hotel in Paris? No section matched, refused.
- Write me a poem about the ocean. Top score 3.15, refused at any gate by the term check.
- Is my company liable for this safety incident? Top score 3.54, refused at any gate by the term check.
- Which agent framework is best, LangChain or CrewAI? Top score 3.65, refused at any gate by the term check.
- Should I train? Top score 3.79, refused at any gate by the term check.
- How should I train for a marathon? Top score 3.79, refused at any gate by the term check.
- What is magic? Top score 4.34, refused at any gate by the term check.
- What is the stock price of ServiceNow today? Top score 4.55, refused at any gate by the term check.
- Which GPU should I buy to train a neural network? Top score 5.39, answered.
- How much does a Now Assist license cost per user? Top score 6.50, answered.
- How do I write a business rule script in ServiceNow? Top score 6.56, answered.
- How do I configure Discovery credentials for the CMDB? Top score 7.19, answered.
- How do I fine-tune an embedding model for better retrieval? Top score 9.20, answered.
- Answered
- Refused
- Gate
- Live gate 3.6
A ring right of the solid line failed the term check: its best section matched fewer than two of the question's keywords, or a one-word question named a topic the notes rarely mention. A score of 8 or more skips the check.
- Wrong answers
- 5
- Out-of-scope questions Ask would answer.
- How do I fine-tune an embedding model for better retrieval?9.20
- How much does a Now Assist license cost per user?6.50
- How do I write a business rule script in ServiceNow?6.56
- Which GPU should I buy to train a neural network?5.39
- How do I configure Discovery credentials for the CMDB?7.19
- Wrong refusals
- 2
- In-scope questions Ask would refuse.
- After reviewing configuration data, how do I keep it from rotting again?4.05
- Should machine-written replies that reach clients be labeled from launch?3.20
All 47 cases, 35 passed
- Passed
How do I stop an agent loop from sprawling?
in-scopetop: agent-loops-that-dont-sprawl (10.66)
- Passed
What should I log every turn to debug an agent?
in-scopetop: agent-loops-that-dont-sprawl (12.48)
- Passed
What queries should I run to audit CMDB health?
in-scopetop: cmdb-health-honest-audit-playbook (13.27)
- Passed
Who should I interview during a CMDB audit?
in-scopetop: cmdb-health-honest-audit-playbook (7.38)
- Passed
When should a RAG system refuse to answer?
in-scopetop: rag-that-doesnt-lie (5.81)
- Passed
Why should citations point to chunks instead of documents?
in-scopetop: rag-that-doesnt-lie (14.96)
- Passed
Why is stuffing more context into the prompt an anti-pattern?
in-scopetop: rag-that-doesnt-lie (18.86)
- Passed
How many examples should an eval set start with?
in-scopetop: evals-as-product-feedback (12.68)
- Passed
Should evals block releases when they regress?
in-scopetop: evals-as-product-feedback (7.43)
- Passed
What should I extract first when splitting a sprawling custom app?
in-scopetop: decomposing-custom-apps-without-breaking-prod (12.99)
- Passed
How do I migrate tables to a new scope without breaking production?
in-scopetop: decomposing-custom-apps-without-breaking-prod (15.32)
- Passed
How should I name Flow Designer flows so they survive refactors?
in-scopetop: flow-designer-patterns-that-survive-refactors (16.92)
- Passed
Which Now Assist capabilities earned user trust fastest?
in-scopetop: now-assist-rollout-notes (8.90)
- Passed
What is a neuron in a neural network, actually?
in-scopetop: neural-networks-101 (10.22)
- Passed
What is RAG?
in-scopetop: rag-that-doesnt-lie (3.69)
- Passed
What are evals?
in-scopetop: evals-as-product-feedback (3.70)
- Passed
What is a CMDB?
in-scopetop: cmdb-health-honest-audit-playbook (3.84)
- Passed
What is magic?
out-of-scoperefused
- Passed
Should I train?
out-of-scoperefused
- Passed
How much does a WSD license cost?
out-of-scoperefused
- Passed
Is my company liable for this safety incident?
out-of-scoperefused
- Passed
What's your favorite pizza topping?
out-of-scoperefused
- Passed
How do I configure a Kubernetes ingress controller?
out-of-scoperefused
- Passed
What's the best hotel in Paris?
out-of-scoperefused
- Passed
Write me a poem about the ocean.
out-of-scoperefused
- Passed
What is the stock price of ServiceNow today?
out-of-scoperefused
Adversarial: lexically close to in-scope content. Lexical retrieval may pass the gate here, kept deliberately as an honest hard case.
- Passed
How should I train for a marathon?
out-of-scoperefused
- Failed
When the search step comes back empty-handed, should the chatbot still improvise a reply?
in-scopeParaphrasetop: decomposing-custom-apps-without-breaking-prod (5.86)
- Failed
My assistant keeps hammering one costly API over and over. How do I cap it?
in-scopeParaphrasetop: rag-that-doesnt-lie (4.68)
- Failed
Is a tiny handpicked test suite better than a huge synthetic collection?
in-scopeParaphrasetop: now-assist-rollout-notes (4.71)
- Failed
After reviewing configuration data, how do I keep it from rotting again?
in-scopeParaphraserefused
- Failed
Should machine-written replies that reach clients be labeled from launch?
in-scopeParaphraserefused
- Failed
How do I make sure a retried outbound call never files the same ticket twice?
in-scopeParaphrasetop: rag-that-doesnt-lie (6.46)
- Failed
How do I fine-tune an embedding model for better retrieval?
out-of-scopeNear-misstop: rag-that-doesnt-lie (9.20)
- Failed
How much does a Now Assist license cost per user?
out-of-scopeNear-misstop: now-assist-rollout-notes (6.50)
- Failed
How do I write a business rule script in ServiceNow?
out-of-scopeNear-misstop: decomposing-custom-apps-without-breaking-prod (6.56)
- Failed
Which GPU should I buy to train a neural network?
out-of-scopeNear-misstop: neural-networks-101 (5.39)
- Passed
Which agent framework is best, LangChain or CrewAI?
out-of-scopeNear-missrefused
- Failed
How do I configure Discovery credentials for the CMDB?
out-of-scopeNear-misstop: cmdb-health-honest-audit-playbook (7.19)
- Passed
Who should write the questions I use to test a RAG system, and how many?
in-scopeTwo-posttop: rag-that-doesnt-lie (12.78)
- Passed
How do I roll out a risky platform change one step at a time?
in-scopeTwo-posttop: decomposing-custom-apps-without-breaking-prod (7.35)
- Failed
How do I keep retries from doing the same work twice, in flows and in agent loops?
in-scopeTwo-posttop: agent-loops-that-dont-sprawl (9.13)
- Passed
How do I get people to trust AI-generated answers at work?
in-scopeTwo-posttop: now-assist-rollout-notes (6.49)
- Passed
What values does the read_target migration property take?
in-scopeServiceNowtop: decomposing-custom-apps-without-breaking-prod (18.03)
- Passed
How many days without an update makes a CI stale?
in-scopeServiceNowtop: cmdb-health-honest-audit-playbook (9.13)
- Passed
How should a subflow's error handler classify failures?
in-scopeServiceNowtop: flow-designer-patterns-that-survive-refactors (14.07)
- Passed
Why was auto-routing low-volume queues a poor fit for Now Assist?
in-scopeServiceNowtop: now-assist-rollout-notes (19.95)
Run history
- 2026-11-0535/47retrieval-only
- 2026-11-0435/47retrieval-only
- 2026-10-3127/27retrieval-only
- 2026-08-2125/25retrieval-only
- 2026-08-2123/23retrieval-only
- 2026-07-0320/20retrieval-only
Methodology
- Refusal correctness
- Out-of-scope questions must be refused before any model call; in-scope questions must pass the gate.
- Retrieval accuracy
- For in-scope questions, the expected post must be the #1 retrieved section.
- Groundedness
- Generated answers must carry at least one citation to a retrieved section.
- Citation accuracy
- At least one citation must point at the post the question is actually about.
The hard cases come in four kinds: paraphrases that share no keyword with the section that answers them, near misses that sound like a post but aren't covered, questions whose answer spans two posts, and ServiceNow detail. They stay in even when they fail. A failing eval you can see is worth more than a green dashboard you can't trust.