diff options
Diffstat (limited to 'notes/v2_redesign/07_judge_validation_results.md')
| -rw-r--r-- | notes/v2_redesign/07_judge_validation_results.md | 52 |
1 files changed, 52 insertions, 0 deletions
diff --git a/notes/v2_redesign/07_judge_validation_results.md b/notes/v2_redesign/07_judge_validation_results.md new file mode 100644 index 0000000..faf32de --- /dev/null +++ b/notes/v2_redesign/07_judge_validation_results.md @@ -0,0 +1,52 @@ +# Judge Validation Results — 2026-04-15 + +## Study 1 Subset + +- 54 runs extracted from Study 1 (20% stratified subset out of 270, seed=42) +- Logs: `logs/v2_study1_judge_subset/` + +## Gold Standard + +- Blackbox: `claude-sonnet-4-6` (Anthropic) +- Glassbox: `claude-sonnet-4-6` (Anthropic) + +## Results + +| Model | BB κ | GB κ | Sph κ | Pass | +| --------------------------- | --------- | --------- | --------- | ------- | +| gpt-4.1 | 0.351 | 0.878 | 0.538 | no | +| gpt-4.1-mini | 0.202 | 0.834 | 0.345 | no | +| **grok-4-1-fast-reasoning** | **0.702** | **0.812** | **0.108** | **YES** | +| grok-4.20-reasoning | 0.549 | 0.919 | 0.198 | no | +| grok-4.20-non-reasoning | 0.390 | 0.878 | 0.117 | no | + +## Selected Proxy Judges + +Split configuration — best model per prong: + +| Prong | Proxy Judge | κ vs Gold | Rationale | +| ------------ | ----------------------------- | ---------------------------- | ------------------------------- | +| **Blackbox** | grok-4-1-fast-reasoning (xAI) | 0.702 | Only model to pass BB threshold | +| **Glassbox** | gpt-4.1 (OpenAI) | 0.878 (intent), 0.538 (soph) | Best sophistication agreement | + +### Config + +```yaml +judge: + blackbox: + model: grok-4-1-fast-reasoning + provider: xai + temperature: 0 + glassbox: + model: gpt-4.1 + provider: openai + temperature: 0 +``` + +## Notes + +- All models show high glassbox agreement (κ > 0.8) — intent classification is consistent across judges +- Blackbox agreement is more variable — output framing/categorization is harder to align on +- Sophistication agreement is low across the board — tier classification is noisy (flagged as limitation) +- Three different provider families used: Anthropic (gold), xAI (BB proxy), OpenAI (GB proxy) — no same-family bias +- Gold standard used a single model (Claude Sonnet 4.6) for both prongs for consistency |