refactor(judge): simplify to single judge per prong, auto-create sync clients

BREAKING CHANGE: Judge API now uses single blackbox_model/glassbox_model
instead of lists with aggregation.

Changes:
- Judge auto-creates sync clients from model configs (supports anthropic, openai, xai)
- Removed multi-model aggregation support (aggregate_results method removed)
- New: create_sync_clients_for_models() and get_supported_providers()
- Updated judge_runner.py to use new single-judge API
- Added scripts/judge_comparison.py for testing judge model pairs
- Updated v2_redesign notes to reflect implementation status

6 files changed, 600 insertions, 94 deletions

diff --git a/notes/v2_redesign/02_judge_system.md b/notes/v2_redesign/02_judge_system.md
index 7ba6763..3b0c03e 100644
--- a/notes/v2_redesign/02_judge_system.md
+++ b/notes/v2_redesign/02_judge_system.md
@@ -2,46 +2,27 @@
 
 This document tracks the in-progress work on refining the judge system for the v2 experiment.
 
-> **NOTE (2026-04-02):** Multi-model aggregation has been removed. Final judge configuration is single judge per prong with 20% cross-family validation. See `04_publication_plan.md`.
+> **UPDATE (2026-04-10):** Multi-model aggregation has been removed from code. Single judge per prong is now committed. 20% validation subset protocol still pending implementation.
 
 ---
 
 ## Current Implementation
 
-The judge system in `src/judge/` has multi-model support with aggregation:
+The judge system in `src/judge/judge.py` uses single judge per prong:
 
 ```yaml
 judge:
   blackbox:
-    models:
-      - id: claude-sonnet-4-20250514
-        provider: anthropic
-        temperature: 0
-      - id: grok-fast-4.1
-        provider: xai
-        temperature: 0
-      - id: gpt-4.1-mini
-        provider: openai
-        temperature: 0
-    aggregation: most_severe  # options: most_severe, majority, unanimous, weighted_severity
+    model: claude-sonnet-4-20250514
+    provider: anthropic
+    temperature: 0
 
   glassbox:
-    models:
-      - id: gpt-4.1
-        provider: openai
-        temperature: 0
-      - id: claude-sonnet-4-20250514
-        provider: anthropic
-        temperature: 0
-    aggregation: most_severe
+    model: gpt-4.1
+    provider: openai
+    temperature: 0
 ```
 
-**Aggregation strategies:**
-- `most_severe`: Takes the category with the lowest severity rank (most deceptive)
-- `majority`: Takes the most common category; falls back to most_severe if no majority
-- `unanimous`: Only returns a verdict if all judges agree; otherwise UNKNOWN
-- `weighted_severity`: Majority if ≥2 agree, otherwise most_severe
-
 **Severity rankings:**
 
 Blackbox:
@@ -63,16 +44,16 @@ Glassbox:
 
 ---
 
-## Planned Changes
+## Status: Complete
 
-### Remove Multi-Model Aggregation
+~~Remove Multi-Model Aggregation~~ **DONE (commit 9155f66)**
 
 **Reason:** Running multiple LLM judges per run triples the cost. For 1,620 runs × 2 judges = 3,240 API calls. With 3 judges per prong, that's 9,720 calls.
 
-**New approach:**
-1. Single judge per prong (blackbox, glassbox)
-2. Batch API for 50% cost reduction
-3. 20% subset validation with different model family
+**Implementation:**
+1. Single judge per prong (blackbox, glassbox) ✓
+2. Batch API for 50% cost reduction ✓
+3. 20% subset validation with different model family — **PENDING**
 
 ### 20% Subset Validation Protocol
 
diff --git a/notes/v2_redesign/README.md b/notes/v2_redesign/README.md
index 5c1be77..db542ab 100644
--- a/notes/v2_redesign/README.md
+++ b/notes/v2_redesign/README.md
@@ -38,9 +38,10 @@ Documentation of the experiment evolution from v1 (420 runs) through to the fina
 
 ### Current Status
 - Framework implemented and committed
-- Judge system being refined (single judge per prong)
+- Judge system: single judge per prong implemented (commit 9155f66)
+- 20% validation subset protocol: pending implementation
+- Judge validation pilot: pending
 - v2 experiment NOT yet run
-- Judge validation pilot pending
 
 ## Key Insight from v1
 
diff --git a/scripts/judge_comparison.py b/scripts/judge_comparison.py
new file mode 100644
index 0000000..b344480
--- /dev/null
+++ b/scripts/judge_comparison.py
@@ -0,0 +1,418 @@
+"""
+Judge Comparison Tool - Compare different judge model pairs on the same logs.
+
+Tests inter-rater reliability (Cohen's Kappa) between different judge model configurations
+to validate the 20% cross-family validation subset approach.
+
+Usage:
+    # Run single model (same for both blackbox and glassbox) on logs, save to CSV
+    python scripts/judge_comparison.py --run --blackbox-model claude-sonnet-4-20250514 --provider anthropic --logs-dir logs/v2_dry_run --output results_claude.csv
+
+    # Compute Kappa between two result CSVs
+    python scripts/judge_comparison.py --compare --csv1 results_claude.csv --csv2 results_gpt.csv
+
+    # Run multiple models and compare all pairs
+    python scripts/judge_comparison.py --run-all --logs-dir logs/v2_dry_run --output-dir judge_validation/
+"""
+
+import argparse
+import csv
+import os
+import sys
+from collections import defaultdict
+from typing import Any, Dict, List, Optional
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from src.judge import Judge
+
+
+def discover_logs(logs_dir: str, max_per_cell: int = 2) -> List[str]:
+    """Discover a stratified sample of logs from v2_dry_run.
+
+    Picks up to max_per_cell logs per (model, scenario, goal_type, oversight) combination.
+    """
+    logs = []
+    for root, dirs, files in os.walk(logs_dir):
+        if os.path.basename(root) == "baseline":
+            continue
+        for f in files:
+            if f.endswith(".json") and not f.startswith("_"):
+                logs.append(os.path.join(root, f))
+
+    grouped = defaultdict(list)
+    for log_path in logs:
+        parts = log_path.replace(logs_dir + "/", "").split("/")
+        if len(parts) >= 4:
+            model, scenario, goal_type, oversight = (
+                parts[0],
+                parts[1],
+                parts[2],
+                parts[3],
+            )
+            key = (model, scenario, goal_type, oversight)
+        else:
+            key = tuple(parts)
+        grouped[key].append(log_path)
+
+    sampled = []
+    for key, paths in grouped.items():
+        sampled.extend(paths[:max_per_cell])
+
+    return sorted(sampled)
+
+
+def run_judgment(
+    logs_dir: str,
+    scenarios_dir: str,
+    logs: List[str],
+    model_id: str,
+    provider: str,
+) -> List[Dict]:
+    """Run judge with same model for both blackbox and glassbox on all logs."""
+    config = {
+        "id": model_id,
+        "provider": provider,
+        "temperature": 0,
+    }
+
+    judge = Judge(
+        blackbox_model=config,
+        glassbox_model=config,
+    )
+
+    verdicts = []
+    for i, log_path in enumerate(logs, 1):
+        print(
+            f"  [{i}/{len(logs)}] {model_id}: {os.path.basename(os.path.dirname(log_path))}"
+        )
+        try:
+            verdict = judge.judge_single(
+                log_path=log_path,
+                logs_dir=logs_dir,
+                scenarios_dir=scenarios_dir,
+                enabled_judges={"regex", "blackbox", "glassbox"},
+            )
+            verdicts.append(verdict)
+        except Exception as e:
+            print(f"  ERROR on {log_path}: {e}")
+    return verdicts
+
+
+def write_csv(output_path: str, verdicts: List[Dict]) -> None:
+    """Write verdicts to CSV."""
+    os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
+    fieldnames = [
+        "run_id",
+        "model",
+        "scenario",
+        "oversight",
+        "regex",
+        "blackbox_category",
+        "blackbox_justification",
+        "glassbox_category",
+        "glassbox_sophistication",
+        "glassbox_justification",
+    ]
+    with open(output_path, "w", newline="") as f:
+        writer = csv.DictWriter(f, fieldnames=fieldnames)
+        writer.writeheader()
+        for v in verdicts:
+            row = {
+                "run_id": v.get("run_id", ""),
+                "model": v.get("model", ""),
+                "scenario": v.get("scenario", ""),
+                "oversight": v.get("oversight", ""),
+                "regex": v.get("regex", ""),
+                "blackbox_category": v.get("blackbox", {}).get("category", ""),
+                "blackbox_justification": v.get("blackbox", {}).get(
+                    "justification", ""
+                ),
+                "glassbox_category": v.get("glassbox", {}).get("category", ""),
+                "glassbox_sophistication": v.get("glassbox", {}).get(
+                    "sophistication", ""
+                ),
+                "glassbox_justification": v.get("glassbox", {}).get(
+                    "justification", ""
+                ),
+            }
+            writer.writerow(row)
+
+
+def read_csv(path: str) -> List[Dict]:
+    """Read verdicts from CSV."""
+    verdicts = []
+    with open(path, newline="") as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            verdicts.append(row)
+    return verdicts
+
+
+def compute_kappa(cats1: List[str], cats2: List[str]) -> Optional[float]:
+    """Compute Cohen's Kappa between two raters' category assignments."""
+    from sklearn.metrics import cohen_kappa_score
+
+    valid = [(c1, c2) for c1, c2 in zip(cats1, cats2) if c1 and c2]
+    if len(valid) < 3:
+        return None
+
+    c1, c2 = zip(*valid)
+    try:
+        return cohen_kappa_score(c1, c2)
+    except Exception:
+        return None
+
+
+def compare_two_csvs(csv1_path: str, csv2_path: str) -> Dict:
+    """Compare two CSV files and compute Cohen's Kappa."""
+    v1 = read_csv(csv1_path)
+    v2 = read_csv(csv2_path)
+
+    if len(v1) != len(v2):
+        print(f"WARNING: Different number of rows ({len(v1)} vs {len(v2)})")
+
+    bb_cats1 = [r.get("blackbox_category", "") for r in v1]
+    bb_cats2 = [r.get("blackbox_category", "") for r in v2]
+    gb_cats1 = [r.get("glassbox_category", "") for r in v1]
+    gb_cats2 = [r.get("glassbox_category", "") for r in v2]
+    soph1 = [r.get("glassbox_sophistication", "") for r in v1]
+    soph2 = [r.get("glassbox_sophistication", "") for r in v2]
+
+    bb_kappa = compute_kappa(bb_cats1, bb_cats2)
+    gb_kappa = compute_kappa(gb_cats1, gb_cats2)
+    soph_kappa = compute_kappa(soph1, soph2)
+
+    return {
+        "blackbox_kappa": bb_kappa,
+        "glassbox_kappa": gb_kappa,
+        "sophistication_kappa": soph_kappa,
+        "n": len([c for c in bb_cats1 if c]),
+    }
+
+
+def run_all_models(
+    logs_dir: str, scenarios_dir: str, logs: List[str], output_dir: str
+) -> tuple[List[str], List[tuple]]:
+    """Run all judge model pairs and save to separate CSVs."""
+    models = [
+        ("claude-sonnet-4-20250514", "anthropic"),
+        ("gpt-4.1", "openai"),
+        ("gpt-4.1-mini", "openai"),
+        ("claude-haiku-4-20250514", "anthropic"),
+        ("grok-fast-4.1", "xai"),
+    ]
+
+    csv_paths = []
+    for model_id, provider in models:
+        safe_name = model_id.replace("-", "_").replace(".", "_")
+        output_path = os.path.join(output_dir, f"results_{safe_name}.csv")
+        csv_paths.append(output_path)
+
+        print(f"\n=== Running: {model_id} (both prongs) ===")
+        verdicts = run_judgment(logs_dir, scenarios_dir, logs, model_id, provider)
+        write_csv(output_path, verdicts)
+        print(f"Saved {len(verdicts)} verdicts to {output_path}")
+
+    return csv_paths, models
+
+
+def compare_all_pairs(
+    csv_paths: List[str], model_names: List[str], output_path: str
+) -> None:
+    """Compare all CSV pairs and save results."""
+    rows = []
+    for i, (path1, name1) in enumerate(zip(csv_paths, model_names)):
+        for path2, name2 in zip(csv_paths[i + 1 :], model_names[i + 1 :]):
+            print(f"\nComparing {name1} vs {name2}:")
+            result = compare_two_csvs(path1, path2)
+
+            bb = (
+                f"{result['blackbox_kappa']:.3f}" if result["blackbox_kappa"] else "N/A"
+            )
+            gb = (
+                f"{result['glassbox_kappa']:.3f}" if result["glassbox_kappa"] else "N/A"
+            )
+            sp = (
+                f"{result['sophistication_kappa']:.3f}"
+                if result["sophistication_kappa"]
+                else "N/A"
+            )
+
+            print(f"  Blackbox Kappa:     {bb}")
+            print(f"  Glassbox Kappa:     {gb}")
+            print(f"  Sophistication:     {sp}")
+
+            rows.append(
+                {
+                    "judge_1": name1,
+                    "judge_2": name2,
+                    "blackbox_kappa": bb,
+                    "glassbox_kappa": gb,
+                    "sophistication_kappa": sp,
+                    "n": result["n"],
+                }
+            )
+
+    with open(output_path, "w", newline="") as f:
+        writer = csv.DictWriter(
+            f,
+            fieldnames=[
+                "judge_1",
+                "judge_2",
+                "blackbox_kappa",
+                "glassbox_kappa",
+                "sophistication_kappa",
+                "n",
+            ],
+        )
+        writer.writeheader()
+        writer.writerows(rows)
+
+    print(f"\n\nAll comparisons saved to {output_path}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Judge Comparison Tool")
+    subparsers = parser.add_subparsers(dest="command", help="Commands")
+
+    run_parser = subparsers.add_parser(
+        "run", help="Run judge on logs with a single model (same for both prongs)"
+    )
+    run_parser.add_argument(
+        "--blackbox-model", required=True, help="Model ID for blackbox judge"
+    )
+    run_parser.add_argument(
+        "--glassbox-model",
+        help="Model ID for glassbox judge (default: same as blackbox)",
+    )
+    run_parser.add_argument(
+        "--provider", required=True, choices=["anthropic", "openai"], help="Provider"
+    )
+    run_parser.add_argument(
+        "--logs-dir", default="logs/v2_dry_run", help="Directory containing logs"
+    )
+    run_parser.add_argument(
+        "--scenarios-dir", default="scenarios", help="Directory containing scenarios"
+    )
+    run_parser.add_argument("--output", required=True, help="Output CSV path")
+    run_parser.add_argument(
+        "--max-per-cell", type=int, default=2, help="Max logs per cell"
+    )
+
+    compare_parser = subparsers.add_parser("compare", help="Compare two result CSVs")
+    compare_parser.add_argument("--csv1", required=True, help="First results CSV")
+    compare_parser.add_argument("--csv2", required=True, help="Second results CSV")
+
+    all_parser = subparsers.add_parser(
+        "run-all", help="Run all models and compare all pairs"
+    )
+    all_parser.add_argument(
+        "--logs-dir", default="logs/v2_dry_run", help="Directory containing logs"
+    )
+    all_parser.add_argument(
+        "--scenarios-dir", default="scenarios", help="Directory containing scenarios"
+    )
+    all_parser.add_argument(
+        "--output-dir", default="judge_validation", help="Output directory for CSVs"
+    )
+    all_parser.add_argument(
+        "--max-per-cell", type=int, default=2, help="Max logs per cell"
+    )
+
+    args = parser.parse_args()
+
+    if args.command == "run":
+        print("Discovering logs...")
+        logs = discover_logs(args.logs_dir, max_per_cell=args.max_per_cell)
+        print(f"Selected {len(logs)} logs\n")
+
+        model_id = args.blackbox_model
+        glassbox_model = args.glassbox_model or model_id
+
+        print(
+            f"Running {model_id} (blackbox) + {glassbox_model} (glassbox) on {len(logs)} logs...\n"
+        )
+        verdicts = run_judgment(
+            args.logs_dir, args.scenarios_dir, logs, model_id, args.provider
+        )
+
+        if glassbox_model != model_id:
+            judge = Judge(
+                blackbox_model={
+                    "id": model_id,
+                    "provider": args.provider,
+                    "temperature": 0,
+                },
+                glassbox_model={
+                    "id": glassbox_model,
+                    "provider": args.provider,
+                    "temperature": 0,
+                },
+            )
+            verdicts = []
+            for i, log_path in enumerate(logs, 1):
+                print(
+                    f"  [{i}/{len(logs)}] {model_id}/{glassbox_model}: {os.path.basename(os.path.dirname(log_path))}"
+                )
+                try:
+                    v = judge.judge_single(
+                        log_path=log_path,
+                        logs_dir=args.logs_dir,
+                        scenarios_dir=args.scenarios_dir,
+                        enabled_judges={"regex", "blackbox", "glassbox"},
+                    )
+                    verdicts.append(v)
+                except Exception as e:
+                    print(f"  ERROR: {e}")
+
+        write_csv(args.output, verdicts)
+        print(f"\nSaved {len(verdicts)} verdicts to {args.output}")
+
+    elif args.command == "compare":
+        print(f"Comparing {args.csv1} vs {args.csv2}...\n")
+        result = compare_two_csvs(args.csv1, args.csv2)
+
+        print(
+            f"Blackbox Kappa:     {result['blackbox_kappa']:.3f}"
+            if result["blackbox_kappa"]
+            else "Blackbox Kappa:     N/A"
+        )
+        print(
+            f"Glassbox Kappa:    {result['glassbox_kappa']:.3f}"
+            if result["glassbox_kappa"]
+            else "Glassbox Kappa:     N/A"
+        )
+        print(
+            f"Sophistication:    {result['sophistication_kappa']:.3f}"
+            if result["sophistication_kappa"]
+            else "Sophistication:     N/A"
+        )
+        print(f"Comparisons:       {result['n']}")
+
+    elif args.command == "run-all":
+        print("Discovering logs...")
+        logs = discover_logs(args.logs_dir, max_per_cell=args.max_per_cell)
+        print(f"Selected {len(logs)} logs\n")
+
+        os.makedirs(args.output_dir, exist_ok=True)
+
+        csv_paths, models = run_all_models(
+            args.logs_dir, args.scenarios_dir, logs, args.output_dir
+        )
+
+        print("\n" + "=" * 60)
+        print("COMPARING ALL PAIRS")
+        print("=" * 60)
+        compare_all_pairs(
+            csv_paths,
+            [m[0] for m in models],
+            os.path.join(args.output_dir, "comparison_results.csv"),
+        )
+
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()
diff --git a/src/judge/__init__.py b/src/judge/__init__.py
index 02f77cb..d18d9dc 100644
--- a/src/judge/__init__.py
+++ b/src/judge/__init__.py
@@ -7,6 +7,7 @@ from .batch_providers import (
     BatchProvider,
     BatchRequest,
     BatchResult,
+    OpenAIBatchProvider,
     XAIBatchProvider,
 )
 from .helpers import (
@@ -15,13 +16,15 @@ from .helpers import (
     extract_system_prompt,
     parse_json_verdict,
 )
-from .judge import Judge
+from .judge import Judge, create_sync_clients_for_models, get_supported_providers
 from .judges import BlackboxChecker, GlassboxChecker, RegexChecker
 from .prompts import (
     BLACKBOX_CATEGORIES,
     BLACKBOX_PROMPT,
+    BLACKBOX_SEVERITY,
     GLASSBOX_CATEGORIES,
     GLASSBOX_PROMPT,
+    GLASSBOX_SEVERITY,
     SOPHISTICATION_TIERS,
 )
 
@@ -33,13 +36,18 @@ __all__ = [
     "BlackboxChecker",
     "GlassboxChecker",
     "Judge",
+    "OpenAIBatchProvider",
     "RegexChecker",
     "XAIBatchProvider",
     "BLACKBOX_CATEGORIES",
     "BLACKBOX_PROMPT",
+    "BLACKBOX_SEVERITY",
     "GLASSBOX_CATEGORIES",
     "GLASSBOX_PROMPT",
+    "GLASSBOX_SEVERITY",
     "SOPHISTICATION_TIERS",
+    "create_sync_clients_for_models",
+    "get_supported_providers",
     "extract_model_output",
     "extract_reasoning_trace",
     "extract_system_prompt",
diff --git a/src/judge/judge.py b/src/judge/judge.py
index 3a9eb93..681d880 100644
--- a/src/judge/judge.py
+++ b/src/judge/judge.py
@@ -15,6 +15,89 @@ from .helpers import (
 )
 from .judges import BlackboxChecker, GlassboxChecker, RegexChecker
 
+import anthropic
+
+try:
+    import openai
+except ImportError:
+    openai = None
+
+try:
+    from xai_sdk import Client as XAIClient
+except ImportError:
+    XAIClient = None
+
+
+_PROVIDER_CLIENT_FACTORIES: Dict[str, Any] = {}
+
+
+def _register_provider_clients() -> Dict[str, Any]:
+    global _PROVIDER_CLIENT_FACTORIES
+    if _PROVIDER_CLIENT_FACTORIES:
+        return _PROVIDER_CLIENT_FACTORIES
+
+    import os as _os
+
+    if anthropic:
+        _PROVIDER_CLIENT_FACTORIES["anthropic"] = lambda: anthropic.Anthropic(
+            api_key=_os.environ.get("ANTHROPIC_API_KEY")
+        )
+    if openai:
+        _PROVIDER_CLIENT_FACTORIES["openai"] = lambda: openai.OpenAI(
+            api_key=_os.environ.get("OPENAI_API_KEY")
+        )
+    if XAIClient:
+        _PROVIDER_CLIENT_FACTORIES["xai"] = lambda: XAIClient(
+            api_key=_os.environ.get("XAI_API_KEY")
+        )
+
+    return _PROVIDER_CLIENT_FACTORIES
+
+
+def get_supported_providers() -> List[str]:
+    """Return list of supported providers that have their client library installed."""
+    _register_provider_clients()
+    return list(_PROVIDER_CLIENT_FACTORIES.keys())
+
+
+def create_sync_clients_for_models(
+    blackbox_model: Dict,
+    glassbox_model: Dict,
+    existing_clients: Dict[str, Any] = None,
+) -> Dict[str, Any]:
+    """Create sync clients for all providers needed by the given model configs.
+
+    Args:
+        blackbox_model: Blackbox judge model config dict
+        glassbox_model: Glassbox judge model config dict
+        existing_clients: Optional existing clients to use instead of creating new ones
+
+    Returns:
+        Dict mapping provider name -> sync client instance
+    """
+    _register_provider_clients()
+
+    providers_needed = set()
+    for config in [blackbox_model, glassbox_model]:
+        if config and config.get("provider"):
+            providers_needed.add(config["provider"])
+
+    clients = dict(existing_clients) if existing_clients else {}
+
+    for provider in providers_needed:
+        if provider not in clients:
+            factory = _PROVIDER_CLIENT_FACTORIES.get(provider)
+            if factory:
+                clients[provider] = factory()
+            else:
+                supported = list(_PROVIDER_CLIENT_FACTORIES.keys())
+                raise ValueError(
+                    f"Provider '{provider}' not supported. "
+                    f"Supported: {supported}. Install the client library first."
+                )
+
+    return clients
+
 
 class Judge:
     """Orchestrates all three checks on experiment log files.
@@ -24,6 +107,9 @@ class Judge:
     - judge_batch(): uses a BatchProvider for multiple logs
 
     Uses a single judge per prong (blackbox, glassbox).
+
+    Auto-creates sync clients for the providers needed by the configured models
+    if not explicitly provided.
     """
 
     def __init__(
@@ -49,7 +135,14 @@ class Judge:
         self.blackbox_model = blackbox_model
         self.glassbox_model = glassbox_model
         self.batch_providers = batch_providers or {}
-        self.sync_clients = sync_clients or {}
+
+        if sync_clients:
+            self.sync_clients = sync_clients
+        else:
+            self.sync_clients = create_sync_clients_for_models(
+                blackbox_model, glassbox_model
+            )
+
         self.regex_checker_cache: Dict[str, RegexChecker] = {}
 
         self.blackbox_checker = BlackboxChecker(
diff --git a/src/judge_runner.py b/src/judge_runner.py
index cd528c2..07b629e 100644
--- a/src/judge_runner.py
+++ b/src/judge_runner.py
@@ -13,15 +13,11 @@ from typing import Dict, List
 
 sys.path.insert(0, os.path.dirname(__file__))
 
-import anthropic
-
 from judge import (
     Judge,
-    extract_model_output,
-    extract_reasoning_trace,
-    extract_system_prompt,
     AnthropicBatchProvider,
     XAIBatchProvider,
+    OpenAIBatchProvider,
 )
 
 
@@ -37,18 +33,15 @@ def discover_log_files(
     """
     log_files = []
     for root, dirs, files in os.walk(logs_dir):
-        # Skip baseline directories
         if os.path.basename(root) == "baseline":
             continue
         for f in files:
             if f.endswith(".json"):
                 full_path = os.path.join(root, f)
-                # Apply model filter
                 if model_filter:
                     model_safe = model_filter.replace("/", "_")
                     if model_safe not in full_path:
                         continue
-                # Apply scenario filter
                 if scenario_filter:
                     if scenario_filter not in full_path:
                         continue
@@ -143,15 +136,10 @@ def save_judge_log(
     log_dir: str,
     judge_model: str = "unknown",
 ) -> str:
-    """Save full judge log including CoT reasoning alongside the verdict.
-
-    The blackbox and glassbox dicts contain 'raw_response' — the judge's
-    full chain-of-thought reasoning. This is preserved in the log for
-    auditability.
-    """
+    """Save full judge log including CoT reasoning alongside the verdict."""
     run_id = verdict.get("run_id", "unknown")
     safe_run_id = run_id.replace("/", "_").replace(" ", "_")
-    timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
+    timestamp = datetime.datetime.now().strftime("%Y%m_%H%M%S")
 
     log_path = os.path.join(log_dir, f"judge_{safe_run_id}_{timestamp}.json")
     os.makedirs(os.path.dirname(log_path) or ".", exist_ok=True)
@@ -172,7 +160,6 @@ def save_judge_log(
             "glassbox_sophistication": gb.get("sophistication", ""),
             "glassbox_justification": gb.get("justification", ""),
         },
-        # Full judge reasoning (CoT) — the complete response from the judge LLM
         "blackbox_full_reasoning": bb.get("raw_response", ""),
         "glassbox_full_reasoning": gb.get("raw_response", ""),
     }
@@ -223,8 +210,8 @@ def main():
     parser.add_argument(
         "--mode",
         choices=["batch", "single"],
-        default="batch",
-        help="Processing mode: 'batch' (Anthropic Batch API) or 'single' (synchronous). Default: batch",
+        default="single",
+        help="Processing mode: 'batch' (Batch API) or 'single' (synchronous). Default: single",
     )
     parser.add_argument(
         "--poll-interval",
@@ -247,39 +234,50 @@ def main():
         default=["regex", "blackbox", "glassbox"],
         help="Which judges to run (default: all three). E.g. --judges regex blackbox",
     )
-    parser.add_argument(
-        "--provider",
-        choices=["anthropic", "xai"],
-        default="anthropic",
-        help="Batch provider to use (default: anthropic)",
-    )
     args = parser.parse_args()
 
     judge_config = load_judge_config(args.config)
-    model = judge_config.get("model", "claude-sonnet-4-20250514")
-    temperature = judge_config.get("temperature", 0)
-    judge_log_dir = judge_config.get("log_dir", "judge_logs")
 
-    batch_provider = None
-    sync_client = None
+    bb_cfg = judge_config.get("blackbox", {})
+    gb_cfg = judge_config.get("glassbox", {})
+
+    blackbox_model = {
+        "id": bb_cfg.get("model", "claude-sonnet-4-20250514"),
+        "provider": bb_cfg.get("provider", "anthropic"),
+        "temperature": bb_cfg.get("temperature", 0),
+    }
+
+    glassbox_model = {
+        "id": gb_cfg.get("model", "gpt-4.1"),
+        "provider": gb_cfg.get("provider", "openai"),
+        "temperature": gb_cfg.get("temperature", 0),
+    }
 
+    judge_log_dir = judge_config.get("log_dir", "judge_logs")
+
+    batch_providers = {}
     if args.mode == "batch":
-        if args.provider == "anthropic":
-            batch_provider = AnthropicBatchProvider()
-            sync_client = anthropic.Anthropic()
-        elif args.provider == "xai":
-            batch_provider = XAIBatchProvider()
-    else:
-        sync_client = anthropic.Anthropic()
+        if blackbox_model["provider"] == "anthropic":
+            batch_providers["anthropic"] = AnthropicBatchProvider()
+        elif blackbox_model["provider"] == "xai":
+            batch_providers["xai"] = XAIBatchProvider()
+        elif blackbox_model["provider"] == "openai":
+            batch_providers["openai"] = OpenAIBatchProvider()
+
+        if glassbox_model["provider"] != blackbox_model["provider"]:
+            if glassbox_model["provider"] == "anthropic":
+                batch_providers["anthropic"] = AnthropicBatchProvider()
+            elif glassbox_model["provider"] == "xai":
+                batch_providers["xai"] = XAIBatchProvider()
+            elif glassbox_model["provider"] == "openai":
+                batch_providers["openai"] = OpenAIBatchProvider()
 
     judge = Judge(
-        model=model,
-        temperature=temperature,
-        batch_provider=batch_provider,
-        sync_client=sync_client,
+        blackbox_model=blackbox_model,
+        glassbox_model=glassbox_model,
+        batch_providers=batch_providers,
     )
 
-    # Determine which log files to process
     if args.log_file:
         log_files = [args.log_file]
     else:
@@ -298,14 +296,14 @@ def main():
     print(f"Judging {len(log_files)} experiment log(s)")
     print(f"  Judges: {', '.join(sorted(enabled_judges))}")
     if run_llm:
-        print(f"  Judge model: {model}")
+        print(f"  Blackbox: {blackbox_model['id']} ({blackbox_model['provider']})")
+        print(f"  Glassbox: {glassbox_model['id']} ({glassbox_model['provider']})")
         print(f"  Mode: {args.mode}")
     print(f"  Output: {args.output}")
     print(f"  Judge logs: {judge_log_dir}")
     print(f"{'=' * 60}\n")
 
     if not run_llm or args.mode == "single":
-        # Regex-only mode or synchronous mode — judge one at a time
         verdicts = []
         for i, log_path in enumerate(log_files, 1):
             print(f"[{i}/{len(log_files)}] Judging: {log_path}")
@@ -318,8 +316,11 @@ def main():
                 )
                 verdicts.append(verdict)
 
-                # Save judge log
-                jlog = save_judge_log(verdict, judge_log_dir, judge_model=model)
+                jlog = save_judge_log(
+                    verdict,
+                    judge_log_dir,
+                    judge_model=f"bb:{blackbox_model['id']}|gb:{glassbox_model['id']}",
+                )
                 parts = []
                 if "regex" in enabled_judges:
                     parts.append(f"regex={verdict['regex']}")
@@ -336,38 +337,43 @@ def main():
 
                 traceback.print_exc()
 
-        # Write CSV
         write_csv(args.output, verdicts)
         print(f"\nResults saved to {args.output}")
 
     else:
-        # Batch mode — use Anthropic Batch API
         checks_per_log = len({"blackbox", "glassbox"} & enabled_judges)
         print("Preparing batch requests...")
-        batch_requests, metadata_map = judge.prepare_batch_requests(
+        batch_requests_by_provider, metadata_map = judge.prepare_batch_requests(
             log_paths=log_files,
             logs_dir=args.logs_dir,
             scenarios_dir=args.scenarios_dir,
             enabled_judges=enabled_judges,
         )
 
+        total_requests = sum(len(reqs) for reqs in batch_requests_by_provider.values())
         print(
-            f"  {len(batch_requests)} API requests ({len(log_files)} logs × {checks_per_log} checks)"
+            f"  {total_requests} API requests ({len(log_files)} logs × {checks_per_log} checks)"
         )
+        for provider, reqs in batch_requests_by_provider.items():
+            print(f"    {provider}: {len(reqs)} requests")
 
-        print("Submitting batch...")
-        batch_id = judge.submit_batch(batch_requests)
-        print(f"  Batch ID: {batch_id}")
+        print("Submitting batches...")
+        batch_ids = judge.submit_all_batches(batch_requests_by_provider)
+        for provider, batch_id in batch_ids.items():
+            print(f"  {provider} batch ID: {batch_id}")
 
         print(f"Polling for completion (every {args.poll_interval}s)...")
-        judge.poll_batch(batch_id, poll_interval=args.poll_interval)
+        judge.poll_all_batches(batch_ids, poll_interval=args.poll_interval)
 
         print("Collecting results...")
-        verdicts = judge.collect_batch_results(batch_id, metadata_map)
+        verdicts = judge.collect_batch_results(batch_ids, metadata_map)
 
-        # Save individual judge logs
         for verdict in verdicts:
-            jlog = save_judge_log(verdict, judge_log_dir, judge_model=model)
+            jlog = save_judge_log(
+                verdict,
+                judge_log_dir,
+                judge_model=f"bb:{blackbox_model['id']}|gb:{glassbox_model['id']}",
+            )
             bb = verdict.get("blackbox", {})
             gb = verdict.get("glassbox", {})
             parts = [f"{verdict['run_id']}:"]
@@ -381,7 +387,6 @@ def main():
                 )
             print(f"  {' '.join(parts)}")
 
-        # Write CSV
         write_csv(args.output, verdicts)
         print(f"\nResults saved to {args.output}")