Agnostics vs writing your own evals

Unit tests and homegrown eval scripts give you control. They also leave you to invent the attack catalog, severity, and ship rule.

Homegrown evals and unit tests

For: Teams with a strong test culture who want zero extra vendors.

Does well

• Full control over fixtures, assertions, and CI.
• Cheap to start for a handful of known cases.
• Fits existing pytest, Jest, or eval-harness habits.

Breaks down

• You must invent and maintain the hostile inputs users will try.
• Findings live in logs and tickets unless you build a second product around them.
• The ship rule is a comment in CI, not a policy you can show in a meeting.

Agnostics

For: Teams who want that workflow without building it first.

Does well

• Attack packs you did not have to author from scratch.
• Grouped findings, severity, and a Release Gate.
• Retest and scan history so coverage does not rot.

Breaks down

• You give up some control. Packs and scoring are ours, not your internal DSL.
• A vendor will never match a platform you have staffed for years.

What Agnostics adds

• A maintained attack catalog instead of a growing pile of one-off prompts.
• A ship recommendation you can defend in a launch review.
• A path to Launch Audit if you only need one serious pass.

Honest limitations

• If you already have a mature internal eval platform, keep it.
• Agnostics will not express every custom assertion your suite already has.
• You still own the final launch call.

When not to choose Agnostics

• You have a staffed platform team and a catalog you already trust.
• You only need a few assertions on deterministic functions, not an AI surface.
• You want to build the muscle internally and are not in a launch crunch.