tech-news · 29 October 2026
Cisco’s FAITH Puts Security Agents Through the Tool Loop
Cisco’s open-source FAITH harness now evaluates multi-step security agents, recording their tool use and testing whether they can find vulnerable files without flagging patched code.
Cisco has extended its open-source FAITH evaluation framework from question-and-answer tests into the messier world of agents that inspect code, choose tools and make a sequence of decisions. The new Agentic FAITH workflow matters because a security agent can produce a plausible final answer while taking a poor route to it. A useful evaluation needs to show not only whether the answer was right, but what the agent inspected, which commands it ran and why the run stopped.
This is a practical shift in how agent capability can be measured. Instead of grading a single response in isolation, the harness records the whole trajectory through a task. That gives developers something closer to a test bench for autonomous work, rather than another leaderboard built around polished answers.
What changed this week
FAITH already provided a common workflow for cybersecurity knowledge benchmarks. Its query , eval and summarize stages run tests, grade results and aggregate reports. Agentic FAITH keeps that structure, but adds a managed loop in which the model requests a tool action, receives the result and decides what to do next. The loop ends when the model submits an answer or reaches a benchmark-defined limit.
The first integrated agentic workload is VLoc, Cisco Foundation AI’s vulnerability-localisation benchmark. It asks an agent to explore a repository through terminal tools and identify files affected by a specified class of vulnerability. The agent then faces the corresponding patched snapshot and must recognise that the vulnerability is no longer present.
Those two phases are important. Phase A scores the submitted file set against the known affected files using File F1, so both missed files and bad guesses count. Phase B measures the true-negative rate on patched code. An agent that flags everything may appear energetic, but it is not useful. An agent that always declares the code clean is no better. Pairing discovery with restraint exposes both failure modes.
The execution environment is deliberately constrained. Cisco says repositories are mounted read-only, with local Docker intended for trusted inputs and Kubernetes supported through Agent Sandbox. Benchmark authors can set prompts, available tools, turn limits, tool-call budgets and timeouts. Deployment settings can change the environment without rewriting the task itself.
The run record is equally significant. FAITH stores the conversation, tool calls and responses, final submission, execution errors and termination reason. It also checkpoints completed tasks, allowing a long evaluation to resume at task boundaries instead of paying to repeat finished work after an interruption.
This week’s release is the integration of that agent loop into FAITH, not a fresh claim that security agents have suddenly become dependable. Independent technical coverage notes that the October announcement does not add a new set of model scores . It packages the existing style of vulnerability-localisation task into a reusable evaluation framework.
Why it matters
Agent evaluation often collapses a complicated process into one pass-or-fail number. That can hide the difference between sound investigation, lucky guessing, tool failure and a run that simply exhausted its budget. Agentic FAITH makes those distinctions inspectable.
For software teams experimenting with coding or security agents, that is more useful than a broad model ranking. A team can ask whether an agent searched relevant directories, followed imports, opened the changed files or wasted most of its allowance on unhelpful commands. If two models reach the same score, their trajectories may still reveal very different operational risks and costs.
The framework also supports a sensible separation between the benchmark and the place where it runs. A small trusted trial can execute in Docker, while a larger programme can use a Kubernetes sandbox. Read-only repository access narrows the blast radius, although it does not remove the need to treat model output and unfamiliar repository content as untrusted.
The published VLoc results provide a useful reality check. The benchmark covers 500 tasks from 290 repositories across six package ecosystems and 147 CWE categories. As summarised in the independent analysis, the best reported system reached only 0.229 File F1, while 38.4 percent of tasks were unsolved by every evaluated model. Those figures come from the earlier VLoc work, not from a new October rerun inside FAITH, but they show why trajectory-level diagnosis is needed. Current systems are still missing a great deal of relevant code.
There are trade-offs. Results depend on the prompt, tool set, command budget, timeout and sandbox configuration, so a score without its run conditions is weak evidence. The benchmark distribution is also uneven across ecosystems, and labels derived from security patches are reproducible without necessarily capturing every vulnerable file. Provider refusals, errors and timeouts can further complicate the clean-code phase.
What to watch next
The strongest next signal would be reproducible Agentic FAITH runs published with complete configurations and comparable trajectory data. That would show whether the shared harness actually reduces evaluation drift between teams.
It will also be worth watching for additional agentic benchmarks beyond vulnerability localisation, especially tasks that test remediation, evidence gathering and safe tool use without granting write access by default. More diverse repositories and clearer treatment of refusals and failed runs would make comparisons stronger.
For now, the valuable part is not a headline score. It is the move toward evaluating the work an agent actually performs. If the framework attracts outside benchmark authors and repeatable result submissions, Agentic FAITH could help turn vague claims about security agents into evidence that can be inspected one tool call at a time.