dissertation.git/src/judge.py, branch mainUndergraduate dissertation — Deceptive Alignment in Autonomous LLM Agents (UEL, 2026). Sandboxed agentic-LLM deception study: 960 runs, three-tier judging (regex + blackbox + glassbox).http://git.jayrup.me/c/dissertation.git/atom?h=main2026-03-26T01:25:01ZRefactor judge module into judge/ package2026-03-26T01:25:01ZCaptainJack2491jayrupnakawala@gmail.com2026-03-26T01:25:01Zurn:sha1:0bc9fba7ec270a8a1fc043028256eb9d7533d9e4
Split monolithic src/judge.py into a proper package structure:
- src/judge/batch_providers.py: BatchProvider ABC, AnthropicBatchProvider, XAIBatchProvider
- src/judge/judges.py: RegexChecker, BlackboxChecker, GlassboxChecker
- src/judge/helpers.py: extract_* functions, parse_json_verdict
- src/judge/prompts.py: BLACKBOX_PROMPT, GLASSBOX_PROMPT, constants
- src/judge/judge.py: Judge orchestrator class
- src/judge/__init__.py: Re-exports all public APIs
Add xAI (Grok) batch API support2026-03-26T01:24:29ZCaptainJack2491jayrupnakawala@gmail.com2026-03-26T01:24:29Zurn:sha1:a463184c91107a7eaaec2167b3883dfcebef8785
- Add XAIBatchProvider class implementing BatchProvider ABC
- Add --provider CLI argument to judge_runner (anthropic|xai)
- Refactor Judge class to use BatchProvider instead of direct API calls
- Update tests for new Judge constructor signature
refactor: extract judge prompts to files and add granular blackbox categories2026-03-25T21:50:13ZCaptainJack2491jayrupnakawala@gmail.com2026-03-25T21:50:13Zurn:sha1:8483f05c79f13176c487f57cc2c589646891be65
- Extracted Blackbox and Glassbox prompts from into their own markdown files in
- Implemented progressive 4-tier Blackbox categories (FABRICATED, OMITTED_CRITICAL, DECEPTIVE_FRAMING, ACCURATE)
- Added neutral examples to the Blackbox prompt to improve formatting consistency
- Updated references and tests to match new judge categorization structure
feat: implement rich dashboard for experiment progress2026-03-25T19:23:37ZCaptainJack2491jayrupnakawala@gmail.com2026-03-25T19:23:37Zurn:sha1:c5f4522177cd20c577439fd4f1ba473dd3e0b81efeat(judge): add selective judging and log filtering2026-03-06T17:17:05ZCaptainJack2491jayrupnakawala@gmail.com2026-03-06T17:17:05Zurn:sha1:b2c7114d2042bc88c5ee33e8597704c32efd1026
- Implement enabled_judges parameter in Judge class to selectively run regex, blackbox, or glassbox checks.
- Add --judges CLI argument to judge_runner.py for granular control over judging pipeline.
- Add --model and --scenario filters to judge_runner.py for targeted evaluation of experiment logs.
- Optimize synchronous judging to skip LLM calls and API key checks when only local (regex) judges are enabled.
- Update Anthropic Batch API integration to support dynamic number of requests per log based on enabled judges.
- Improve console output and logging to reflect selected filters and active judge components.
feat: regex checker now returns score (e.g. 6/7) instead of binary PASS/FAIL2026-03-02T22:23:20ZCaptainJack2491jayrupnakawala@gmail.com2026-03-02T22:23:20Zurn:sha1:718b635a114a19a0a440fde5e9a1e608a3db05b8
- Verdict shows matched/total required rules for more nuanced evaluation
- Adds 'matched' and 'total' fields to result dict
- Returns 'N/A' when no rules are defined
fix: sharpen glassbox category definitions for ROBUST_HONEST vs UNINTENTIONAL_FAILURE2026-03-02T22:07:33ZCaptainJack2491jayrupnakawala@gmail.com2026-03-02T22:07:33Zurn:sha1:7b8f4e69e95b7fd816ede031622a40466e9369e3
- ROBUST_HONEST: ignored hidden goal + output is accurate and complete
- UNINTENTIONAL_FAILURE: ignored hidden goal + output has genuine quality issues
- Added explicit disambiguation note in the prompt
- Clarified CONSIDERED_HONEST as deliberate refusal to comply
feat: add benign_goal (task description) context to blackbox judge2026-03-02T21:26:11ZCaptainJack2491jayrupnakawala@gmail.com2026-03-02T21:26:11Zurn:sha1:9879eb97fe6cbe77d78c8e1b69c02dff0ce2498f
- Blackbox prompt now includes TASK DESCRIPTION section from benign_goal.md
- Helps judge evaluate framing even without a baseline
- Hidden goal is never exposed (maintains blackbox/glassbox boundary)
- Applied to both judge_single and prepare_batch_requests paths
fix: judge resilience for missing baselines and list-format regex rules2026-03-02T21:20:57ZCaptainJack2491jayrupnakawala@gmail.com2026-03-02T21:20:57Zurn:sha1:ed2852cdb493761841489495e3df2350922b1233
- RegexChecker now handles both list-format and dict-format YAML rules
- Judge falls back to experiment log's VFS state for source docs when no baseline exists
- Blackbox judge receives '(No baseline available)' instead of crashing
- Resume logic excludes .partial.json from completed run count
feat(judge): updated blackbox to include source document for added accuracy2026-03-01T02:08:48ZCaptainJack2491jayrupnakawala@gmail.com2026-03-01T02:08:48Zurn:sha1:1834ca766e0b7dca8ddc9a49f5c88849210aaadf