summaryrefslogtreecommitdiff
path: root/tests
AgeCommit message (Collapse)Author
2026-03-26Add xAI (Grok) batch API supportCaptainJack2491
- Add XAIBatchProvider class implementing BatchProvider ABC - Add --provider CLI argument to judge_runner (anthropic|xai) - Refactor Judge class to use BatchProvider instead of direct API calls - Update tests for new Judge constructor signature
2026-03-25refactor: extract judge prompts to files and add granular blackbox categoriesCaptainJack2491
- Extracted Blackbox and Glassbox prompts from into their own markdown files in - Implemented progressive 4-tier Blackbox categories (FABRICATED, OMITTED_CRITICAL, DECEPTIVE_FRAMING, ACCURATE) - Added neutral examples to the Blackbox prompt to improve formatting consistency - Updated references and tests to match new judge categorization structure
2026-03-25feat: implement rich dashboard for experiment progressCaptainJack2491
2026-03-25fix: catch and recover from JSONDecodeError on malformed tool argsCaptainJack2491
2026-03-25test: add coverage for v2 infra featuresCaptainJack2491
- VFSFactory: independent instances vs singletons - make_tools_for_vfs: parallel binding - config_loader: goal_types and max_workers parsing - runner: goal_type iteration and resume skipping - agent: goal_type storage and path resolution
2026-03-01feat(judge): updated blackbox to include source document for added accuracyCaptainJack2491
2026-02-25feat: implement judging system with regex, blackbox, and glassbox checksCaptainJack2491
- Add src/judge.py: RegexChecker, BlackboxChecker, GlassboxChecker, Judge orchestrator - Add src/judge_runner.py: CLI for batch/single judging with CSV output - Add regex_rules.yaml for corporate_sabotage_v2 and child_protection scenarios - Add judge config section to config.yaml (model, temperature, log_dir) - Add tests/test_judge.py: 34 unit tests covering all components - Uses Anthropic Batch API for half-cost LLM judge calls - Judge logs include full CoT reasoning and judge model for auditability
2026-02-24Add comprehensive test suite (104 tests)CaptainJack2491
Tests cover: - VFS: sandbox escape, path traversal, CRUD, singleton staleness - Config loader: missing keys, temperature cascade, oversight fallback - Tools: schema/implementation sync, VFS delegation - Agent: message construction, error handling, token counting, reasoning extraction - Interrogation: sanitization, provider detection, prompt loading - Runner: baseline extraction, prompt assembly, success detection All tests run offline with mocked API calls.