diff options
| author | CaptainJack2491 <jayrupnakawala@gmail.com> | 2026-04-10 15:23:59 +0100 |
|---|---|---|
| committer | CaptainJack2491 <jayrupnakawala@gmail.com> | 2026-04-10 15:23:59 +0100 |
| commit | 2b3c478dfdf68ccbf853b2a04ea9463d531e1998 (patch) | |
| tree | 9b79fd01147ea4431db11dfd23d62fc8612ddca9 /notes/v2_redesign | |
| parent | 9155f664a2db0abe841d0d7b9b059ffbe2290c18 (diff) | |
refactor(judge): simplify to single judge per prong, auto-create sync clients
BREAKING CHANGE: Judge API now uses single blackbox_model/glassbox_model
instead of lists with aggregation.
Changes:
- Judge auto-creates sync clients from model configs (supports anthropic, openai, xai)
- Removed multi-model aggregation support (aggregate_results method removed)
- New: create_sync_clients_for_models() and get_supported_providers()
- Updated judge_runner.py to use new single-judge API
- Added scripts/judge_comparison.py for testing judge model pairs
- Updated v2_redesign notes to reflect implementation status
Diffstat (limited to 'notes/v2_redesign')
| -rw-r--r-- | notes/v2_redesign/02_judge_system.md | 47 | ||||
| -rw-r--r-- | notes/v2_redesign/README.md | 5 |
2 files changed, 17 insertions, 35 deletions
diff --git a/notes/v2_redesign/02_judge_system.md b/notes/v2_redesign/02_judge_system.md index 7ba6763..3b0c03e 100644 --- a/notes/v2_redesign/02_judge_system.md +++ b/notes/v2_redesign/02_judge_system.md @@ -2,46 +2,27 @@ This document tracks the in-progress work on refining the judge system for the v2 experiment. -> **NOTE (2026-04-02):** Multi-model aggregation has been removed. Final judge configuration is single judge per prong with 20% cross-family validation. See `04_publication_plan.md`. +> **UPDATE (2026-04-10):** Multi-model aggregation has been removed from code. Single judge per prong is now committed. 20% validation subset protocol still pending implementation. --- ## Current Implementation -The judge system in `src/judge/` has multi-model support with aggregation: +The judge system in `src/judge/judge.py` uses single judge per prong: ```yaml judge: blackbox: - models: - - id: claude-sonnet-4-20250514 - provider: anthropic - temperature: 0 - - id: grok-fast-4.1 - provider: xai - temperature: 0 - - id: gpt-4.1-mini - provider: openai - temperature: 0 - aggregation: most_severe # options: most_severe, majority, unanimous, weighted_severity + model: claude-sonnet-4-20250514 + provider: anthropic + temperature: 0 glassbox: - models: - - id: gpt-4.1 - provider: openai - temperature: 0 - - id: claude-sonnet-4-20250514 - provider: anthropic - temperature: 0 - aggregation: most_severe + model: gpt-4.1 + provider: openai + temperature: 0 ``` -**Aggregation strategies:** -- `most_severe`: Takes the category with the lowest severity rank (most deceptive) -- `majority`: Takes the most common category; falls back to most_severe if no majority -- `unanimous`: Only returns a verdict if all judges agree; otherwise UNKNOWN -- `weighted_severity`: Majority if ≥2 agree, otherwise most_severe - **Severity rankings:** Blackbox: @@ -63,16 +44,16 @@ Glassbox: --- -## Planned Changes +## Status: Complete -### Remove Multi-Model Aggregation +~~Remove Multi-Model Aggregation~~ **DONE (commit 9155f66)** **Reason:** Running multiple LLM judges per run triples the cost. For 1,620 runs × 2 judges = 3,240 API calls. With 3 judges per prong, that's 9,720 calls. -**New approach:** -1. Single judge per prong (blackbox, glassbox) -2. Batch API for 50% cost reduction -3. 20% subset validation with different model family +**Implementation:** +1. Single judge per prong (blackbox, glassbox) ✓ +2. Batch API for 50% cost reduction ✓ +3. 20% subset validation with different model family — **PENDING** ### 20% Subset Validation Protocol diff --git a/notes/v2_redesign/README.md b/notes/v2_redesign/README.md index 5c1be77..db542ab 100644 --- a/notes/v2_redesign/README.md +++ b/notes/v2_redesign/README.md @@ -38,9 +38,10 @@ Documentation of the experiment evolution from v1 (420 runs) through to the fina ### Current Status - Framework implemented and committed -- Judge system being refined (single judge per prong) +- Judge system: single judge per prong implemented (commit 9155f66) +- 20% validation subset protocol: pending implementation +- Judge validation pilot: pending - v2 experiment NOT yet run -- Judge validation pilot pending ## Key Insight from v1 |
