Agnostics vs garak

garak is an open-source probe lab for LLM vulnerabilities. Agnostics is a release workflow that turns adversarial results into a ship decision.

garak

For: Security researchers and engineers who want a Python probe suite they can run locally.

Does well

• Broad libraries of known jailbreak, toxicity, and leakage probes.
• Good for exploring a model or endpoint as a lab exercise.
• Open source, scriptable, and easy to point at a single completion API.

Breaks down

• You assemble the report, the severity call, and the retest yourself.
• Less oriented to product surfaces with retrieval, tools, and auth.
• No built-in ship recommendation or release report for a launch meeting.

Agnostics

For: Teams shipping an AI product who need a repeatable pre-release pass.

Does well

• Targets the app users hit: chat, RAG, agents, MCP, APIs.
• Grouped findings and a Release Gate instead of a raw probe dump.
• Retest after fixes so the gate can move.

Breaks down

• Not a research probe workbench. You will not get every garak generator here.
• Coverage follows the packs you run, not an exhaustive academic catalog.

What Agnostics adds

• A ship call tied to policy, not a folder of probe output.
• Packs aimed at product failure modes, including retrieval and tool actions.
• A path from first scan to Launch Audit or a paid plan.

Honest limitations

• Agnostics is not a substitute for a dedicated research scanner.
• If your goal is to publish or extend probes, garak is the better lab.
• No scan, including ours, can promise it found every jailbreak.

When not to choose Agnostics

• You want a probe lab you run from Python and do not need a product UI.
• You are testing a bare model endpoint, not an application with users.
• You already have a red-team program and only need extra generators.