September 8, 2026
Compliant Is Not Correct: The AI Gave Us the Wrong Hashcat Mode
Compliant Is Not Correct: The AI Gave Us the Wrong Hashcat Mode
Omair
New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 14+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
Your agent estate comes up in a governance review. Someone who does not work in security asks four questions.
Did the agent do what we intended? Was the data it acted on authentic? Can we prove what it did? Was the transaction untampered from end to end?
These are ordinary assurance questions, the kind anyone would ask about a payment system. What makes the room go quiet is that the honest answer to all four is usually "we believe so."
Those four are the plain-language version of the four properties we set out in AI integrity: an autonomous action worth trusting is genuine, authorized, untampered, and reconstructable. This post maps each to an assessment domain — what gets tested, what the tester does, and what evidence comes back. You can buy them separately, and most teams should.
"Did it do what we intended?" and "was its data authentic?" are one test seen from two ends: in an agentic system the data channel is the instruction channel, so a retrieved document carrying instructions is not a data quality problem that becomes a security problem later. It is the security problem. And the list is missing authorization — not "does the agent have a role," but "was it allowed to take this action, at this moment, as itself." That is the one that fails most often in our experience, and it is rarely asked, because a role assignment exists on paper and looks settled.
A composite of what we see, illustrative rather than any one client.
A company commissions AI agent security testing and gets back a model evaluation: jailbreak attempts on the chat interface, refusal rates, a system prompt that leaks under pressure. Competent work, and none of it touches the agent — which has eleven tools, a shared service account, persistent memory, and the ability to move money. The deliverable answered questions about the model. The board asked about the system.
What gets tested. Every path by which content the agent did not author enters its context: retrieval corpus, ticket and email bodies, web fetches, uploaded files, tool results, memory, messages from other agents. The OWASP Top 10 for Agentic Applications, published in December 2025 for 2026, names the outcomes — goal hijack, memory and context poisoning — and MITRE ATLAS supplies the technique language.
What the tester does. Plants instruction-bearing content in each path with a benign but unmistakable objective attached — a one-dollar refund to a controlled account, a write to a seeded record — then instruments the tool layer instead of reading the chat window. Twice: in-session, which is prompt injection as most people picture it, and delayed, seeding the retrieval corpus or memory and returning days later in a clean session.
What comes back. Per path: the payload, the retrieved content as it appeared in context, the tool call it produced, and whether it executed. "The model repeated the injected text" is not the finding. "The refund tool fired" is.
What gets tested. What the agent authenticates as at each hop, what it inherits, what it can delegate, and whether any approval step in its path is real human oversight or decoration. OWASP files these failures under identity and privilege abuse.
What the tester does. Enumerates the credentials actually presented at runtime, which is often not what the design document says, then attempts the crossings. Can agent A reach agent B's tools through a shared service account or gateway. Can a token be replayed outside the window it was issued for. Can the approval be satisfied without a person — by the bot that posts the request, by replying as the agent, or through an API that skips the interface the button lives on. Then an action inside the role but outside the mandate: the role permits refunds, so what stops forty thousand dollars of them in an hour?
What comes back. The observed identity graph beside the intended one, each escalation attempt with its request and response, and the approval step exercised, showing what satisfied it.
What gets tested. Everything between the agent's decision and its effect in the world: tool definitions, MCP servers, gateways, and the return path. Treat it as supply chain. The NSA's design considerations for MCP-based automation, released in May 2026, is a useful primer on implicit trust and dynamic tool invocation.
What the tester does. Pins and re-verifies. Captures each tool definition and server identity as approved, then compares it against what is live now, because a tool description can change after approval and nothing in the transcript looks different when it does. Diffs three hops: the instruction issued, the call as it reached the server, the result as it returned. The return path gets its own attempt, since a tool result is untrusted content the agent will act on. All of it holds only for the day it was checked — the clearest case in AI assurance for continuous testing.
What comes back. A manifest of tool and server versions with drift called out, the per-hop diffs, and whether hostile return content reached a later action.
What gets tested. The evidence the system leaves, judged against a real question rather than a retention policy.
What the tester does. Not a log review. Picks a consequential action from about thirty days ago and answers "why did it do that" from what was retained: the instruction, the retrieved content as it existed then, memory state, tool versions, identity, approval. Time-boxed, the way an investigation is. Check whether the record is tamper-evident too, since an audit trail the agent's own identity can write to is not one.
What comes back. What was and was not recoverable, each gap tied to the field or store that is missing, and the retention window as observed rather than as configured. Teams routinely find the tool call recorded and the cause absent, which is a finding on its own: an incident you have already had cannot be reconstructed.
You cannot run four domains across an estate. Pick the agent whose worst plausible action would hurt most, threat model that one, and start there. Whether a domain needs adversarial testing or configuration validation is a separate call, worth settling before a proposal arrives.
Then hold what you buy to one bar: every finding carries the attempt, the observed result, the artifact, and the conditions it ran under. Frameworks help you organize and share vocabulary — OWASP and ATLAS for technique language, CSA's AI Controls Matrix for control coverage, NIST's control overlays for securing AI systems, which has single-agent and multi-agent use cases on its roadmap and is still in draft. They are maps, and a map is not a test. Alignment is not certification, and even where certification exists, as it does for ISO/IEC 42001, it certifies a management system rather than proving a control held under attack.
None of this eliminates risk. Independent adversarial testing reduces it and reveals it, and revealing it is usually the more valuable half. The first two domains are moving research areas, so a result there describes what held on one day against one build.
For a self-check, take one agent that is live today and answer four questions with evidence rather than intent. Which of its ingestion paths can reach a consequential tool call? What identity does it present, and can any approval in its path be satisfied without a person? Do its tool servers still behave the way they did the day you approved them? Can you reconstruct, from stored evidence alone, why it did something last month?
If the answers come back as "probably," that is your scope, and a normal place to be right now. If you want a second set of eyes, we are happy to have that conversation.
We are a CREST-accredited, ISO/IEC 27001 certified offensive security firm, and we test agentic systems by the four domains above rather than by questionnaire. Our AI and ML penetration testing and red teaming work attempts the control and hands you the attempt, the result, the artifact, and the conditions. Where the system changes faster than an annual cycle, we run it continuously. We are services-first, so we assure the stack you already chose.
