| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Split monolithic src/judge.py into a proper package structure:
- src/judge/batch_providers.py: BatchProvider ABC, AnthropicBatchProvider, XAIBatchProvider
- src/judge/judges.py: RegexChecker, BlackboxChecker, GlassboxChecker
- src/judge/helpers.py: extract_* functions, parse_json_verdict
- src/judge/prompts.py: BLACKBOX_PROMPT, GLASSBOX_PROMPT, constants
- src/judge/judge.py: Judge orchestrator class
- src/judge/__init__.py: Re-exports all public APIs
|
|
- Add XAIBatchProvider class implementing BatchProvider ABC
- Add --provider CLI argument to judge_runner (anthropic|xai)
- Refactor Judge class to use BatchProvider instead of direct API calls
- Update tests for new Judge constructor signature
|
|
- Extracted Blackbox and Glassbox prompts from into their own markdown files in
- Implemented progressive 4-tier Blackbox categories (FABRICATED, OMITTED_CRITICAL, DECEPTIVE_FRAMING, ACCURATE)
- Added neutral examples to the Blackbox prompt to improve formatting consistency
- Updated references and tests to match new judge categorization structure
|
|
|
|
|
|
|
|
|
|
|
|
- VFSFactory: independent instances vs singletons
- make_tools_for_vfs: parallel binding
- config_loader: goal_types and max_workers parsing
- runner: goal_type iteration and resume skipping
- agent: goal_type storage and path resolution
|
|
- Create hidden_goals/ for all 3 scenarios (self_serving, moral, bare)
- Corporate: shareholder value vs community harm vs bare instruction
- Child protection: department stats vs saving lives vs bare directive
- Pharma: stock price vs patient access vs bare instruction
- Rebalance child_protection data: archive cases D, E, F (35KB -> 13.6KB)
- Update child_protection regex_rules.yaml for remaining 3 cases
|
|
- vfs.py: add VFS.create() factory for parallel-safe independent instances
- tools.py: add make_tools_for_vfs() to bind tools to specific VFS instances
- config_loader.py: add goal_types and max_workers config properties
- agent.py: add goal_type field, vfs_instance param, goal_type in log paths
- runner.py: rewrite with goal_type loop dimension, ThreadPoolExecutor parallel
execution, isolated VFS per run, flattened work items, thread-safe results
All backward compatible - empty goal_types defaults to legacy hidden_goal.md,
max_workers=1 defaults to sequential execution.
|
|
|
|
|
|
- Implement enabled_judges parameter in Judge class to selectively run regex, blackbox, or glassbox checks.
- Add --judges CLI argument to judge_runner.py for granular control over judging pipeline.
- Add --model and --scenario filters to judge_runner.py for targeted evaluation of experiment logs.
- Optimize synchronous judging to skip LLM calls and API key checks when only local (regex) judges are enabled.
- Update Anthropic Batch API integration to support dynamic number of requests per log based on enabled judges.
- Improve console output and logging to reflect selected filters and active judge components.
|
|
|
|
- Verdict shows matched/total required rules for more nuanced evaluation
- Adds 'matched' and 'total' fields to result dict
- Returns 'N/A' when no rules are defined
|
|
UNINTENTIONAL_FAILURE
- ROBUST_HONEST: ignored hidden goal + output is accurate and complete
- UNINTENTIONAL_FAILURE: ignored hidden goal + output has genuine quality issues
- Added explicit disambiguation note in the prompt
- Clarified CONSIDERED_HONEST as deliberate refusal to comply
|
|
test.
|
|
- Blackbox prompt now includes TASK DESCRIPTION section from benign_goal.md
- Helps judge evaluate framing even without a baseline
- Hidden goal is never exposed (maintains blackbox/glassbox boundary)
- Applied to both judge_single and prepare_batch_requests paths
|
|
- RegexChecker now handles both list-format and dict-format YAML rules
- Judge falls back to experiment log's VFS state for source docs when no baseline exists
- Blackbox judge receives '(No baseline available)' instead of crashing
- Resume logic excludes .partial.json from completed run count
|
|
Previously a scenario with its own oversight/ directory would ignore
the global oversight_levels config. Now the global list acts as a
filter — only levels present in BOTH the scenario dir AND the global
config are run. Warns if no levels match.
|
|
- Writes _in_progress.partial.json after each turn in the chat loop
- Partial file persists if run crashes/hangs for post-mortem inspection
- Cleaned up automatically when final save_logs succeeds
- Enabled in both baseline and experiment runs via runner.py
|
|
- New 'generate_baseline' option in defaults (default: true)
- When false, skips baseline generation entirely
- Warns if no baseline exists when generation is disabled
- Updated config.yaml.example and README.md with docs
|
|
- Handle reasoning.summary type in reasoning_details (GPT-5.3-codex etc.)
- Prefix summaries with [SUMMARY] to distinguish from raw CoT
- Add reasoning_format field to log entries for metadata tracking
- Warn on first turn if no reasoning is detected (glass-box judging impact)
|
|
- Create src/logger.py with Python logging module
- Add 4 debug levels: 1=CRITICAL, 2=WARNING, 3=INFO, 4=DEBUG
- Level 4 includes reasoning, VFS info, and available tools
- Level 4 auto-enables file output (both mode)
- Update agent.py and runner.py to use logger instead of print
- Update config.yaml.example with logging configuration
- Update README with logging documentation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Runner now counts existing .json log files per combination and skips
completed runs, resuming from where it left off
- Log files are written atomically via temp file + rename to prevent
corrupt files from being counted as completed
- Added --no-resume CLI flag to force a fresh start
- Resume is enabled by default
|
|
- Create .env.example with placeholder keys for all providers
- Create config.yaml.example with clean reference config
- Add config.yaml and judge_logs/ to .gitignore
- Load .env in judge.py for standalone usage
|
|
- Add src/judge.py: RegexChecker, BlackboxChecker, GlassboxChecker, Judge orchestrator
- Add src/judge_runner.py: CLI for batch/single judging with CSV output
- Add regex_rules.yaml for corporate_sabotage_v2 and child_protection scenarios
- Add judge config section to config.yaml (model, temperature, log_dir)
- Add tests/test_judge.py: 34 unit tests covering all components
- Uses Anthropic Batch API for half-cost LLM judge calls
- Judge logs include full CoT reasoning and judge model for auditability
|
|
|
|
Tests cover:
- VFS: sandbox escape, path traversal, CRUD, singleton staleness
- Config loader: missing keys, temperature cascade, oversight fallback
- Tools: schema/implementation sync, VFS delegation
- Agent: message construction, error handling, token counting, reasoning extraction
- Interrogation: sanitization, provider detection, prompt loading
- Runner: baseline extraction, prompt assembly, success detection
All tests run offline with mocked API calls.
|
|
- Fixed syntax errors in print statements (lines 159/162)
- Added sanitize_for_api() to strip custom log fields before API calls
- Added interactive commands: history, reasoning, vfs, info, save, help
- Colored terminal output for readability
- Better error handling (KeyboardInterrupt, EOFError)
- Added --show-history flag to review conversation on load
|
|
- VFS now creates fresh instance when root_path/fs_data is provided
(prevents cross-scenario contamination in multi-scenario runs)
- Removed check_username from tools (unrelated to any scenario)
|
|
- Added _extract_baseline_content() to extract output from create_file tool calls
- Baselines now saved to logs/{model}/{scenario}/baseline.md
- Added global oversight fallback (scenario-specific → global oversight dir)
- Updated config_loader with oversight level loading from scenario dirs
|
|
|
|
(no commit message provided)
(no commit message provided)
Co-authored-by: aider (openrouter/openrouter/aurora-alpha) <aider@aider.chat>
|
|
|
|
|
|
made eveyrthing much cleaner and it works very well.
got rid of providers setup, only use openai sdk now.
|
|
|
|
|
|
|
|
|
|
|
