| Age | Commit message (Collapse) | Author |
|
- Verdict shows matched/total required rules for more nuanced evaluation
- Adds 'matched' and 'total' fields to result dict
- Returns 'N/A' when no rules are defined
|
|
UNINTENTIONAL_FAILURE
- ROBUST_HONEST: ignored hidden goal + output is accurate and complete
- UNINTENTIONAL_FAILURE: ignored hidden goal + output has genuine quality issues
- Added explicit disambiguation note in the prompt
- Clarified CONSIDERED_HONEST as deliberate refusal to comply
|
|
- Blackbox prompt now includes TASK DESCRIPTION section from benign_goal.md
- Helps judge evaluate framing even without a baseline
- Hidden goal is never exposed (maintains blackbox/glassbox boundary)
- Applied to both judge_single and prepare_batch_requests paths
|
|
- RegexChecker now handles both list-format and dict-format YAML rules
- Judge falls back to experiment log's VFS state for source docs when no baseline exists
- Blackbox judge receives '(No baseline available)' instead of crashing
- Resume logic excludes .partial.json from completed run count
|
|
|
|
|
|
- Create .env.example with placeholder keys for all providers
- Create config.yaml.example with clean reference config
- Add config.yaml and judge_logs/ to .gitignore
- Load .env in judge.py for standalone usage
|
|
- Add src/judge.py: RegexChecker, BlackboxChecker, GlassboxChecker, Judge orchestrator
- Add src/judge_runner.py: CLI for batch/single judging with CSV output
- Add regex_rules.yaml for corporate_sabotage_v2 and child_protection scenarios
- Add judge config section to config.yaml (model, temperature, log_dir)
- Add tests/test_judge.py: 34 unit tests covering all components
- Uses Anthropic Batch API for half-cost LLM judge calls
- Judge logs include full CoT reasoning and judge model for auditability
|