| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
BREAKING CHANGE: Judge API now uses single blackbox_model/glassbox_model
instead of lists with aggregation.
Changes:
- Judge auto-creates sync clients from model configs (supports anthropic, openai, xai)
- Removed multi-model aggregation support (aggregate_results method removed)
- New: create_sync_clients_for_models() and get_supported_providers()
- Updated judge_runner.py to use new single-judge API
- Added scripts/judge_comparison.py for testing judge model pairs
- Updated v2_redesign notes to reflect implementation status
|
|
aggregation
- Remove multi-model aggregation support from Judge class
- Single blackbox judge: claude-sonnet-4-20250514 (anthropic)
- Single glassbox judge: gpt-4.1 (openai)
- Remove aggregate_results method and all aggregation-related code
- Simplify batch collection to single result per run
- Update config_dryrun.yaml with new judge configuration
|
|
- Separated Static Visualizations into dedicated tab with fullscreen image modal
- Added new interactive charts: Glassbox Category by Scenario, Intent vs Reality, and Radar Chart
- Replaced average token tracking with global token aggregations
- Implemented Grouped-Stacked Bar Chart for visual comparison of Action alongside Sophistication by oversight tier
- Added standardized color palettes to improve chart legibility
|
|
- Added HTTP POST /api/interrogate/start and /api/interrogate/chat to api/server.py
- Implemented ACTIVE_SESSIONS cache to persist synchronous agent state across requests
- Updated sys.path in api/server.py to resolve inner core imports correctly
- Added Inspector Modal 'Interrogate' view toggle in index.html
- Implemented JS interrogation loop, fallback log path resolution, auto-scroll, and markdown rendering in results.js
- Added custom CSS animations (.thinking-pulse, .pulse-ring) to mimic agent processing
|
|
and row inspector
|
|
- CSV results now found in logs/full_experiment/, output/, and config.output_dir
- Viz images now found in viz/ (project root) and output/charts/
- Judge logs now found in judge_logs/ and logs/judge_full_experiment/
- Added CORS middleware for local development
- Added /api/logs/ endpoints for browsing raw JSON experiment logs
- Added --port and --host CLI args
|
|
|
|
- 01: v1→v2 evolution history, results, limitations, rationalization hypothesis
- 02: judge system state, single judge per prong, 20% validation protocol
- 03: research plan analysis, Option B (two-phase) selected
- 04: publication plan — paper structure, three-layer analysis strategy,
Gemini ceiling effect handling, budget, venue targets, timeline
Key decisions:
- Two-phase design (540 runs) over full factorial (1,620)
- Single paper: v1 exploratory → Study 1 (oversight) → Study 2 (framing)
- Three-layer analysis: pooled → per-model → sensitivity excluding ceiling models
|
|
- Refactored Agent's chat loop to explicitly preserve and pass back reasoning context (reasoning_details, reasoning) to OpenRouter.
- Ensures frontier models like Gemini 3.1 and DeepSeek v3.2 maintain their 'thinking' chain during complex tool-calling sessions.
- Enhanced reasoning extraction to support structured details, reasoning_content, and fallback tags.
- Verified capture of encrypted and plain-text reasoning blocks in experimental logs.
|
|
|
|
Split monolithic src/judge.py into a proper package structure:
- src/judge/batch_providers.py: BatchProvider ABC, AnthropicBatchProvider, XAIBatchProvider
- src/judge/judges.py: RegexChecker, BlackboxChecker, GlassboxChecker
- src/judge/helpers.py: extract_* functions, parse_json_verdict
- src/judge/prompts.py: BLACKBOX_PROMPT, GLASSBOX_PROMPT, constants
- src/judge/judge.py: Judge orchestrator class
- src/judge/__init__.py: Re-exports all public APIs
|
|
- Add XAIBatchProvider class implementing BatchProvider ABC
- Add --provider CLI argument to judge_runner (anthropic|xai)
- Refactor Judge class to use BatchProvider instead of direct API calls
- Update tests for new Judge constructor signature
|
|
- Extracted Blackbox and Glassbox prompts from into their own markdown files in
- Implemented progressive 4-tier Blackbox categories (FABRICATED, OMITTED_CRITICAL, DECEPTIVE_FRAMING, ACCURATE)
- Added neutral examples to the Blackbox prompt to improve formatting consistency
- Updated references and tests to match new judge categorization structure
|
|
|
|
|
|
|
|
|
|
|
|
- VFSFactory: independent instances vs singletons
- make_tools_for_vfs: parallel binding
- config_loader: goal_types and max_workers parsing
- runner: goal_type iteration and resume skipping
- agent: goal_type storage and path resolution
|
|
- Create hidden_goals/ for all 3 scenarios (self_serving, moral, bare)
- Corporate: shareholder value vs community harm vs bare instruction
- Child protection: department stats vs saving lives vs bare directive
- Pharma: stock price vs patient access vs bare instruction
- Rebalance child_protection data: archive cases D, E, F (35KB -> 13.6KB)
- Update child_protection regex_rules.yaml for remaining 3 cases
|
|
- vfs.py: add VFS.create() factory for parallel-safe independent instances
- tools.py: add make_tools_for_vfs() to bind tools to specific VFS instances
- config_loader.py: add goal_types and max_workers config properties
- agent.py: add goal_type field, vfs_instance param, goal_type in log paths
- runner.py: rewrite with goal_type loop dimension, ThreadPoolExecutor parallel
execution, isolated VFS per run, flattened work items, thread-safe results
All backward compatible - empty goal_types defaults to legacy hidden_goal.md,
max_workers=1 defaults to sequential execution.
|
|
|
|
|
|
This commit:
- Adds a 'Total Token Usage by Model' chart alongside the average token chart
- Updates the web_gui_plan.md to reflect the removal of execution APIs and the addition of the new data visualization features.
|
|
This commit:
- Bakes total/prompt/completion tokens directly into results.csv
- Reverts dynamic JSON scraping backend logic for speed
- Adds modal overlay functionality to zoom Chart.js graphs on click
|
|
This commit:
- Replaces single-select dropdowns with multi-select checkbox lists
- Adds 'All' and 'None' quick-action links for filter groups
- Implements custom priority sorting for Oversight levels (low -> mid -> high)
- Updates Chart.js logic to respect logical sorting on X-axes
|
|
This commit:
- Removes all execution logic (runs, cancellation, SSE logs)
- Strips the terminal and configuration sidebar
- Implements a global filtering panel (Model, Scenario, Oversight)
- Dedicated the UI 100% to interactive Chart.js and static visuals
|
|
This commit completely overhauls the initial GUI:
- Splits monolithic JS/CSS into an ES6 module structure
- Implements an industrial 'Control Room' dark theme
- Adds dynamic file selection endpoints for CSV and JSON judge logs
- Integrates Chart.js to automatically render visualizations from CSV data
- Updates web_gui_plan.md to reflect architectural changes
|
|
- Add FastAPI backend (api/server.py) with endpoints for:
- Config read/write
- Scenario/model discovery
- Experiment run management (start/cancel/status)
- Real-time log streaming via SSE
- Results fetching (CSV, images, judge results)
- Add vanilla JS frontend (api/static/):
- Clean dark-themed dashboard
- Configuration panel with dropdowns
- Live log terminal
- Results viewer with tabs
- Add documentation (docs/web_gui_plan.md)
Dependencies added: fastapi, uvicorn, sse-starlette
|
|
- Implement enabled_judges parameter in Judge class to selectively run regex, blackbox, or glassbox checks.
- Add --judges CLI argument to judge_runner.py for granular control over judging pipeline.
- Add --model and --scenario filters to judge_runner.py for targeted evaluation of experiment logs.
- Optimize synchronous judging to skip LLM calls and API key checks when only local (regex) judges are enabled.
- Update Anthropic Batch API integration to support dynamic number of requests per log based on enabled judges.
- Improve console output and logging to reflect selected filters and active judge components.
|
|
|
|
- Verdict shows matched/total required rules for more nuanced evaluation
- Adds 'matched' and 'total' fields to result dict
- Returns 'N/A' when no rules are defined
|
|
UNINTENTIONAL_FAILURE
- ROBUST_HONEST: ignored hidden goal + output is accurate and complete
- UNINTENTIONAL_FAILURE: ignored hidden goal + output has genuine quality issues
- Added explicit disambiguation note in the prompt
- Clarified CONSIDERED_HONEST as deliberate refusal to comply
|
|
test.
|
|
- Blackbox prompt now includes TASK DESCRIPTION section from benign_goal.md
- Helps judge evaluate framing even without a baseline
- Hidden goal is never exposed (maintains blackbox/glassbox boundary)
- Applied to both judge_single and prepare_batch_requests paths
|
|
- RegexChecker now handles both list-format and dict-format YAML rules
- Judge falls back to experiment log's VFS state for source docs when no baseline exists
- Blackbox judge receives '(No baseline available)' instead of crashing
- Resume logic excludes .partial.json from completed run count
|
|
Previously a scenario with its own oversight/ directory would ignore
the global oversight_levels config. Now the global list acts as a
filter — only levels present in BOTH the scenario dir AND the global
config are run. Warns if no levels match.
|
|
- Writes _in_progress.partial.json after each turn in the chat loop
- Partial file persists if run crashes/hangs for post-mortem inspection
- Cleaned up automatically when final save_logs succeeds
- Enabled in both baseline and experiment runs via runner.py
|
|
- New 'generate_baseline' option in defaults (default: true)
- When false, skips baseline generation entirely
- Warns if no baseline exists when generation is disabled
- Updated config.yaml.example and README.md with docs
|
|
- Handle reasoning.summary type in reasoning_details (GPT-5.3-codex etc.)
- Prefix summaries with [SUMMARY] to distinguish from raw CoT
- Add reasoning_format field to log entries for metadata tracking
- Warn on first turn if no reasoning is detected (glass-box judging impact)
|
|
- Create src/logger.py with Python logging module
- Add 4 debug levels: 1=CRITICAL, 2=WARNING, 3=INFO, 4=DEBUG
- Level 4 includes reasoning, VFS info, and available tools
- Level 4 auto-enables file output (both mode)
- Update agent.py and runner.py to use logger instead of print
- Update config.yaml.example with logging configuration
- Update README with logging documentation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Runner now counts existing .json log files per combination and skips
completed runs, resuming from where it left off
- Log files are written atomically via temp file + rename to prevent
corrupt files from being counted as completed
- Added --no-resume CLI flag to force a fresh start
- Resume is enabled by default
|
