blob: faf32def52b31b680d2a984c12feb961151d2c19 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
|
# Judge Validation Results — 2026-04-15
## Study 1 Subset
- 54 runs extracted from Study 1 (20% stratified subset out of 270, seed=42)
- Logs: `logs/v2_study1_judge_subset/`
## Gold Standard
- Blackbox: `claude-sonnet-4-6` (Anthropic)
- Glassbox: `claude-sonnet-4-6` (Anthropic)
## Results
| Model | BB κ | GB κ | Sph κ | Pass |
| --------------------------- | --------- | --------- | --------- | ------- |
| gpt-4.1 | 0.351 | 0.878 | 0.538 | no |
| gpt-4.1-mini | 0.202 | 0.834 | 0.345 | no |
| **grok-4-1-fast-reasoning** | **0.702** | **0.812** | **0.108** | **YES** |
| grok-4.20-reasoning | 0.549 | 0.919 | 0.198 | no |
| grok-4.20-non-reasoning | 0.390 | 0.878 | 0.117 | no |
## Selected Proxy Judges
Split configuration — best model per prong:
| Prong | Proxy Judge | κ vs Gold | Rationale |
| ------------ | ----------------------------- | ---------------------------- | ------------------------------- |
| **Blackbox** | grok-4-1-fast-reasoning (xAI) | 0.702 | Only model to pass BB threshold |
| **Glassbox** | gpt-4.1 (OpenAI) | 0.878 (intent), 0.538 (soph) | Best sophistication agreement |
### Config
```yaml
judge:
blackbox:
model: grok-4-1-fast-reasoning
provider: xai
temperature: 0
glassbox:
model: gpt-4.1
provider: openai
temperature: 0
```
## Notes
- All models show high glassbox agreement (κ > 0.8) — intent classification is consistent across judges
- Blackbox agreement is more variable — output framing/categorization is harder to align on
- Sophistication agreement is low across the board — tier classification is noisy (flagged as limitation)
- Three different provider families used: Anthropic (gold), xAI (BB proxy), OpenAI (GB proxy) — no same-family bias
- Gold standard used a single model (Claude Sonnet 4.6) for both prongs for consistency
|
