| Age | Commit message (Collapse) | Author |
|---|
|
|
|
- config_study1.yaml: 3 models × 3 oversight × bare framing × n=30
- Judge validation: 54-run subset, gold=Sonnet 4.6, proxy=Grok 4.1 Fast (BB κ=0.702) + GPT-4.1 (GB κ=0.878)
- Fix OpenAI batch provider: BytesIO, method/url fields, response.body parsing
- Scripts: extract_subset.py, judge_validation.py
- Dissertation chapters updated (intro, methodology, results, conclusion)
|
|
|
|
BREAKING CHANGE: Judge API now uses single blackbox_model/glassbox_model
instead of lists with aggregation.
Changes:
- Judge auto-creates sync clients from model configs (supports anthropic, openai, xai)
- Removed multi-model aggregation support (aggregate_results method removed)
- New: create_sync_clients_for_models() and get_supported_providers()
- Updated judge_runner.py to use new single-judge API
- Added scripts/judge_comparison.py for testing judge model pairs
- Updated v2_redesign notes to reflect implementation status
|
|
- 01: v1→v2 evolution history, results, limitations, rationalization hypothesis
- 02: judge system state, single judge per prong, 20% validation protocol
- 03: research plan analysis, Option B (two-phase) selected
- 04: publication plan — paper structure, three-layer analysis strategy,
Gemini ceiling effect handling, budget, venue targets, timeline
Key decisions:
- Two-phase design (540 runs) over full factorial (1,620)
- Single paper: v1 exploratory → Study 1 (oversight) → Study 2 (framing)
- Three-layer analysis: pooled → per-model → sensitivity excluding ceiling models
|
|
|
|
|
|
- Add src/judge.py: RegexChecker, BlackboxChecker, GlassboxChecker, Judge orchestrator
- Add src/judge_runner.py: CLI for batch/single judging with CSV output
- Add regex_rules.yaml for corporate_sabotage_v2 and child_protection scenarios
- Add judge config section to config.yaml (model, temperature, log_dir)
- Add tests/test_judge.py: 34 unit tests covering all components
- Uses Anthropic Batch API for half-cost LLM judge calls
- Judge logs include full CoT reasoning and judge model for auditability
|
|
|
|
|
|
|
