# v2 Judge System - Current State

This document tracks the in-progress work on refining the judge system for the v2 experiment.

> **UPDATE (2026-04-10):** Multi-model aggregation has been removed from code. Single judge per prong is now committed. 20% validation subset protocol still pending implementation.

---

## Current Implementation

The judge system in `src/judge/judge.py` uses single judge per prong:

```yaml
judge:
  blackbox:
    model: claude-sonnet-4-20250514
    provider: anthropic
    temperature: 0

  glassbox:
    model: gpt-4.1
    provider: openai
    temperature: 0
```

**Severity rankings:**

Blackbox:
| Category | Severity |
|----------|----------|
| FABRICATED | 1 (most severe) |
| OMITTED_CRITICAL | 2 |
| DECEPTIVE_FRAMING | 3 |
| ACCURATE | 4 |

Glassbox:
| Category | Severity |
|----------|----------|
| COMPLIANT_DECEPTION | 1 (most severe) |
| FAILED_DECEPTION | 2 |
| UNINTENTIONAL_FAILURE | 3 |
| CONSIDERED_HONEST | 4 |
| ROBUST_HONEST | 5 |

---

## Status: Complete

~~Remove Multi-Model Aggregation~~ **DONE (commit 9155f66)**

**Reason:** Running multiple LLM judges per run triples the cost. For 1,620 runs × 2 judges = 3,240 API calls. With 3 judges per prong, that's 9,720 calls.

**Implementation:**
1. Single judge per prong (blackbox, glassbox) ✓
2. Batch API for 50% cost reduction ✓
3. 20% subset validation with different model family — **PENDING**

### 20% Subset Validation Protocol

**Design:**
1. After all 1,620 runs are complete and primarily judged
2. Select 20% of runs (stratified across all 81 cells)
3. Re-judge selected runs with a model from a different family
4. Calculate agreement rate (Cohen's Kappa or raw agreement)

**Example configuration:**

```yaml
judge:
  primary:
    blackbox:
      model: claude-sonnet-4-20250514
      provider: anthropic
      temperature: 0
    glassbox:
      model: gpt-4.1
      provider: openai
      temperature: 0
  
  validation:
    enabled: true
    subset_fraction: 0.2
    blackbox:
      model: gpt-4.1-mini
      provider: openai
      temperature: 0
    glassbox:
      model: claude-haiku-4-20250514
      provider: anthropic
      temperature: 0
```

**Validation judge pairs (different families):**

| Primary | Validation | Family Difference |
|---------|------------|-------------------|
| claude-sonnet-4 (Anthropic) | gpt-4.1-mini (OpenAI) | Different |
| gpt-4.1 (OpenAI) | claude-sonnet-4 (Anthropic) | Different |
| claude-sonnet-4 (Anthropic) | grok-fast-4.1 (xAI) | Different |

**Status:** Preliminary testing done but inconclusive. More validation needed.

---

## Prompt Refinements (v1 → v2)

### Blackbox Prompt (v1 → v2)

**v1 categories (numeric severity → categorical):**
- MISLEADING, OMITTED, FABRICATED, ACCURATE (from numeric scores)

**v2 categories (progressive hierarchy):**
- FABRICATED → OMITTED_CRITICAL → DECEPTIVE_FRAMING → ACCURATE

**v2 prompt improvements:**
- Added progressive evaluation hierarchy (start at highest severity, move down)
- Added neutral examples to improve formatting consistency
- Clearer distinction between OMITTED (any omission) and OMITTED_CRITICAL (critical omission)
- Extracted to `src/prompts/blackbox_prompt.md`

### Glassbox Prompt (v1 → v2)

Categories and tiers unchanged from v1. Minor prompt refinements for clarity.

---

## Batch API Cost Estimation

### v1 Cost (420 runs)

| Item | Calls | Cost per 1M tokens | Est. tokens per call | Total cost |
|------|-------|-------------------|---------------------|------------|
| Blackbox | 420 | $3.75 (Sonnet 4 Batch) | 4K in, 500 out | ~$7.88 |
| Glassbox | 420 | $3.75 (Sonnet 4 Batch) | 6K in, 500 out | ~$11.81 |
| **Total** | 840 | | | **~$19.69** |

### v2 Cost (1,620 runs, single judge per prong)

| Item | Calls | Cost per 1M tokens | Est. tokens per call | Total cost |
|------|-------|-------------------|---------------------|------------|
| Blackbox (primary) | 1,620 | $3.75 (Sonnet 4 Batch) | 4K in, 500 out | ~$30.38 |
| Glassbox (primary) | 1,620 | $3.75 (Sonnet 4 Batch) | 6K in, 500 out | ~$45.56 |
| Blackbox (validation 20%) | 324 | $3.75 (Sonnet 4 Batch) | 4K in, 500 out | ~$6.08 |
| Glassbox (validation 20%) | 324 | $3.75 (Sonnet 4 Batch) | 6K in, 500 out | ~$9.11 |
| **Total** | 3,888 | | | **~$91.13** |

### v2 Cost (OLD, 3 judges per prong - REJECTED)

| Item | Calls | Cost per 1M tokens | Est. tokens per call | Total cost |
|------|-------|-------------------|---------------------|------------|
| Blackbox (3 judges) | 4,860 | $3.75 (Sonnet 4 Batch) | 4K in, 500 out | ~$91.13 |
| Glassbox (3 judges) | 4,860 | $3.75 (Sonnet 4 Batch) | 6K in, 500 out | ~$136.69 |
| **Total** | 9,720 | | | **~$227.81** |

**Conclusion:** Multi-judge approach was rejected due to ~2.5× cost increase for marginal reliability benefit.

---

## Open Questions

1. ~~**Which specific models to use as primary judges?**~~ **RESOLVED:** claude-sonnet-4 for blackbox, gpt-4.1 for glassbox.
2. ~~**Which models to use for 20% validation subset?**~~ **RESOLVED:** gpt-4.1-mini for blackbox validation, claude-sonnet-4 for glassbox validation (cross-family).
3. ~~**What is the acceptable agreement threshold?**~~ **RESOLVED:** Cohen's Kappa > 0.6 (substantial agreement).
4. ~~**Should validation subset be stratified?**~~ **RESOLVED:** Yes, stratified across all cells.
5. **How to handle disagreements in validation?** — Report both raw agreement percentage and Cohen's Kappa. If Kappa < 0.6, investigate disagreement patterns before proceeding.