NALV Failure Library
Agent says “done” without calling the required tool
A support agent can return a convincing success message even when the required action behind it never happened.
A release check should verify execution evidence, not only the final answer.
Designed from a verified engine result. Not a Dify production screenshot.
- Expected
- Required tool must execute
- Agent response
- “Done successfully.”
- Execution evidence
- Required tool: Missing
- NALV verdict
- FAIL
TL;DR
NALV can fail a release check when a required tool is missing from a complete execution trace — even if the agent says the action succeeded.
The response may look correct while the required execution behavior is absent. NALV checks the execution evidence.
In this verified case, a frozen scenario requires a specific named tool. The assistant responds “Done successfully.” The complete tool trace contains no execution event for that tool. The release verdict is FAIL.
The message is a claim about the outcome. The trace is the record of what ran. When the trace is complete and those two disagree, the release check follows the evidence. A customer-facing reply can confirm a refund, a booking change, or a ticket update in plain language. If the tool that would perform that action never appears, the success message is not evidence that the action happened.
External public signal
Real-world signal
The real-world signal for this failure class is public. It is separate from the controlled NALV result below.
Dify GitHub Issue #40671 reports a production Dify agent that sometimes returned successful-looking output while tool_responses was empty. The conversation could look finished even though the tool record for that run did not show the calls behind the reply.
That report is a signal that the failure class exists: a plausible success message with missing tool evidence. It is an external observation. NALV did not reproduce Dify issue #40671 on a live Dify workspace. There is no live Dify proof on this page.
Engine / fixture proof
Controlled NALV reproduction
- The scenario requires a named tool.
- The assistant responds “Done successfully.”
- The complete tool trace contains no execution event for that tool.
- NALV returns FAIL.
The NALV result is a controlled engine and fixture proof. It uses a frozen request and a required-tool condition inside NALV’s check path. It is not a live rerun of the Dify incident, and it is not a capture from a Dify production workspace.
The evidence panel is a designed record of that engine result: expected behavior, the agent’s reply, the missing tool event, and the FAIL verdict.
Exact Retest is supported for this check. It reuses the same frozen request and required-tool condition after a prompt, model, or tool configuration change; the recipe and check are not reselected.
Why final-answer evaluation can miss it
The output looks plausible. The required behavior did not occur.
“Done successfully.” has the shape of a finished support action. It reports no error and does not ask anyone to wait. A review that grades only the final answer can accept it, because nothing in the sentence is internally inconsistent.
The required behavior sits outside that sentence. The scenario named a tool that had to execute. In a complete trace, that execution event is missing. The agent stated an outcome the execution record does not support.
Final-answer checks remain useful for wording, policy, and tone. They answer a different question from the one this failure asks. Here the release question is whether the required tool ran. Execution evidence is what answers it.
How NALV checks it
NALV holds the required behavior next to the execution record, then turns the comparison into a finding and a release verdict.
Required behavior
The scenario states that a specific named tool must execute.
Execution evidence
NALV reads the complete tool trace for an execution event from that tool.
Finding
The required tool has no execution event in that trace.
Release verdict
The check returns FAIL.
When the trace is complete and the required event is absent, the finding is a missing required tool and the release verdict is FAIL. This page describes that public result only. It does not describe internal check source or private file names.
Evidence confidence matters
NALV does not treat missing telemetry as proof of behavioral failure.
A complete tool trace is evidence about what executed. If that trace is complete and the required tool never appears, the check can see that the required event did not occur. The verdict is FAIL.
Incomplete tool visibility is a different state. If NALV cannot see the tool record well enough to know whether the required tool ran, it does not convert that gap into a confirmed behavior failure. The verdict is REVIEW. REVIEW means the evidence is not sufficient to close the question.
Existing behavior in this proof already separates the two. A complete trace with the required tool absent fails the check. Incomplete tool visibility stays in review.
- Complete trace and Required tool absent results in FAIL
- Incomplete tool visibility results in REVIEW
Exact Retest
After a prompt, model, or tool configuration change, NALV can rerun the same frozen test condition. The request stays the request that was frozen. The required tool stays the tool the scenario demanded. The recipe and the check are not chosen again.
Exact Retest is supported for this check. It reuses the same frozen request and required-tool condition after a prompt, model, or tool configuration change; the recipe and check are not reselected.
Preserving that test condition does not guarantee a stochastic agent will change from FAIL to PASS on the next run. Exact Retest does not make model behavior deterministic. A later run can still change wording, tool choice, or whether the agent fails in the same way.
What this check does not prove
This check answers a narrow question. A FAIL means the required tool was missing from a complete execution trace even though the agent reported success.
- It does not verify every external side effect.
- It does not support every agent platform or runtime today.
- It does not guarantee deterministic reproduction of stochastic model behavior.
- This page is not a live reproduction of Dify #40671.
Source and verification
- NALV proof
- Controlled engine fixture
- Last verified
Run a release check on the behavior you need before you ship.
Run a release check