From 2b3c478dfdf68ccbf853b2a04ea9463d531e1998 Mon Sep 17 00:00:00 2001 From: CaptainJack2491 Date: Fri, 10 Apr 2026 15:23:59 +0100 Subject: refactor(judge): simplify to single judge per prong, auto-create sync clients BREAKING CHANGE: Judge API now uses single blackbox_model/glassbox_model instead of lists with aggregation. Changes: - Judge auto-creates sync clients from model configs (supports anthropic, openai, xai) - Removed multi-model aggregation support (aggregate_results method removed) - New: create_sync_clients_for_models() and get_supported_providers() - Updated judge_runner.py to use new single-judge API - Added scripts/judge_comparison.py for testing judge model pairs - Updated v2_redesign notes to reflect implementation status --- notes/v2_redesign/02_judge_system.md | 47 +++++++++++------------------------- 1 file changed, 14 insertions(+), 33 deletions(-) (limited to 'notes/v2_redesign/02_judge_system.md') diff --git a/notes/v2_redesign/02_judge_system.md b/notes/v2_redesign/02_judge_system.md index 7ba6763..3b0c03e 100644 --- a/notes/v2_redesign/02_judge_system.md +++ b/notes/v2_redesign/02_judge_system.md @@ -2,46 +2,27 @@ This document tracks the in-progress work on refining the judge system for the v2 experiment. -> **NOTE (2026-04-02):** Multi-model aggregation has been removed. Final judge configuration is single judge per prong with 20% cross-family validation. See `04_publication_plan.md`. +> **UPDATE (2026-04-10):** Multi-model aggregation has been removed from code. Single judge per prong is now committed. 20% validation subset protocol still pending implementation. --- ## Current Implementation -The judge system in `src/judge/` has multi-model support with aggregation: +The judge system in `src/judge/judge.py` uses single judge per prong: ```yaml judge: blackbox: - models: - - id: claude-sonnet-4-20250514 - provider: anthropic - temperature: 0 - - id: grok-fast-4.1 - provider: xai - temperature: 0 - - id: gpt-4.1-mini - provider: openai - temperature: 0 - aggregation: most_severe # options: most_severe, majority, unanimous, weighted_severity + model: claude-sonnet-4-20250514 + provider: anthropic + temperature: 0 glassbox: - models: - - id: gpt-4.1 - provider: openai - temperature: 0 - - id: claude-sonnet-4-20250514 - provider: anthropic - temperature: 0 - aggregation: most_severe + model: gpt-4.1 + provider: openai + temperature: 0 ``` -**Aggregation strategies:** -- `most_severe`: Takes the category with the lowest severity rank (most deceptive) -- `majority`: Takes the most common category; falls back to most_severe if no majority -- `unanimous`: Only returns a verdict if all judges agree; otherwise UNKNOWN -- `weighted_severity`: Majority if ≥2 agree, otherwise most_severe - **Severity rankings:** Blackbox: @@ -63,16 +44,16 @@ Glassbox: --- -## Planned Changes +## Status: Complete -### Remove Multi-Model Aggregation +~~Remove Multi-Model Aggregation~~ **DONE (commit 9155f66)** **Reason:** Running multiple LLM judges per run triples the cost. For 1,620 runs × 2 judges = 3,240 API calls. With 3 judges per prong, that's 9,720 calls. -**New approach:** -1. Single judge per prong (blackbox, glassbox) -2. Batch API for 50% cost reduction -3. 20% subset validation with different model family +**Implementation:** +1. Single judge per prong (blackbox, glassbox) ✓ +2. Batch API for 50% cost reduction ✓ +3. 20% subset validation with different model family — **PENDING** ### 20% Subset Validation Protocol -- cgit v1.2.3