The test case
LLMTestCaseLoading recorded case…
DEEPEVAL × JEV
A passing score
needs a good explanation.
Learn by testing a fictional support assistant. Experiment with Jev judgments, release gates, and drift alerts.
PUBLIC DEMO
This site replays saved provider results. Controls recalculate decisions; they do not call a model. The live Python app is included in the repository.
Download evidence ↓FROM ANSWER TO VERDICT
What did the judge actually see?
The decision rule
Local replayThese controls reuse the selected call’s probabilities. Changing an answer or rubric requires a fresh Jev call in the live lab.
Inside the judgment
JevEvalFollow the evidence
Exact selected fields, raw TypeSafe payload, and a reproducible native metric.
SAME CASES. SAME JUDGE. DIFFERENT POLICY.
Would you release this change?
One recorded, five-pair DeepEval dataset run. The generator received current or obsolete policy; Jev always saw the authoritative reference.
Every pair, visible
Per-case pass threshold: 80%| Case | Reference · current policy | Candidate · obsolete policy | Change | Candidate verdict | Inspect |
|---|
This small synthetic set teaches regression testing. It does not estimate production accuracy, isolate model drift, or provide a statistical confidence interval.
MAKE THE ALERT RULE VISIBLE
When does a change become a signal?
Replay real recorded scores in a constructed sequence. Explore warm-up, detection, and recovery.
Event 0 / 10
Rolling mean quality
Quality is a Jev judgment, not human-measured accuracy.
Rolling meanFrozen referenceAlert boundary
What entered the current window?
The operational rules
Application logicQuality Reference mean minus rolling mean. Watch at 8 percentage points; alert at 15.
Category mix Base-2 Jensen–Shannon divergence. Watch at 0.10; alert at 0.20.
Warm-up Five reference records, then a complete selected window. No early pass.
Interpretation KS is a descriptive distance. These rules do not compute significance or establish the cause of a change.
FROM THE PUBLIC DEMO TO YOUR WORKLOAD
Make the next evaluation yours.
The repository includes the full live monitor, rubric workbench, paired dataset runner, audit scripts, and this public demo.
1. Start the live lab
Python 3.12git clone https://github.com/shyamsridhar123/JudgeJev.git
cd JudgeJev
python -m venv .venv
# Activate .venv for your shell, then:
python -m pip install -r requirements.txt
# Copy .env.example to .env and add your key.
python serve.pyOpen http://127.0.0.1:8780/lab. A TypeSafe key is enough to evaluate a fixed answer. Configure your own Responses-compatible generator to create fresh answers and run paired experiments.
2. Understand the boundaries
AYour generatorProduces the answer from the selected policy. Bring your endpoint and model ID.
BJev through TypeSafeReceives the selected evidence and typed questions; returns structured judgments.
CNative DeepEval JevEvalNormalizes values, applies weights, handles applicability, and returns a metric verdict.
DThe applicationStores traces locally, streams completed results, applies release rules, and measures drift.
Data you control
The local app sends generation inputs to your chosen provider and selected evaluation fields to TypeSafe. Journals and keys are excluded from Git. Optional DeepEval telemetry is disabled.
Enterprise adoption guide ↗Proof you can rerun
Audit the saved evidence without any API key. Tests check native semantics, missing data, gate boundaries, and corrupted evidence. Fresh judge results can vary.
python scripts/verify_demo.py
python -m pytest -q
node --test tests/demo.test.mjsExplore with your AI
Ask your preferred assistant to read AGENTS.md. It can guide hands-on experiments, help you inspect the evidence, or present the demo to your team.
AI learning guide ↗Native JevEval documentation ↗
The live app binds to loopback. Enterprise hosting needs your organization’s authentication, authorization, secret management, and approved provider/data policies. This repository is an exploration lab.