Agnostics vs LangSmith evals

LangSmith evals watch quality and traces in the LangChain world. Agnostics pressure-tests adversarial failures before you ship.

LangSmith evals

For: Teams already tracing LangChain or LangGraph apps who want dataset evals and production observability.

Does well

• Traces, datasets, and LLM-as-judge quality checks in one place.
• Strong when you already live in the LangSmith workflow.
• Helps catch regressions on known examples after a prompt or model change.

Breaks down

• Quality evals on golden sets are not the same as hostile user input.
• Observability shows what happened. It does not decide whether to launch.
• Adversarial coverage is not the core job of a tracing platform.

Agnostics

For: Teams who need a pre-release adversarial pass and a ship recommendation.

Does well

• Attack packs for injection, leakage, tool misuse, and retrieval drift.
• Findings grouped for engineering, plus a Release Gate.
• Works on the product surface, not only on traced LangChain spans.

Breaks down

• Not a production tracing or observability suite.
• Will not replace LangSmith if you need dataset evals and run history there.

What Agnostics adds

• Adversarial coverage aimed at launch risk, not only quality scores.
• A Ready, Monitor, Fix, or Blocked call you can take to a meeting.
• Retest so a fix is proven before the gate moves.

Honest limitations

• Agnostics and LangSmith can sit side by side. They are not the same job.
• If you only need traces and golden-set quality, stay where you already are.
• Agnostics does not ingest LangSmith traces or replace production monitoring.

When not to choose Agnostics

• Your question is "did quality regress on our dataset?", not "what happens when users attack this?".
• You are standardized on LangSmith and do not have an adversarial launch risk to test.
• You need observability more than a pre-release gate.