| Age | Commit message (Collapse) | Author |
|---|
|
- config_study1.yaml: 3 models × 3 oversight × bare framing × n=30
- Judge validation: 54-run subset, gold=Sonnet 4.6, proxy=Grok 4.1 Fast (BB κ=0.702) + GPT-4.1 (GB κ=0.878)
- Fix OpenAI batch provider: BytesIO, method/url fields, response.body parsing
- Scripts: extract_subset.py, judge_validation.py
- Dissertation chapters updated (intro, methodology, results, conclusion)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
BREAKING CHANGE: Judge API now uses single blackbox_model/glassbox_model
instead of lists with aggregation.
Changes:
- Judge auto-creates sync clients from model configs (supports anthropic, openai, xai)
- Removed multi-model aggregation support (aggregate_results method removed)
- New: create_sync_clients_for_models() and get_supported_providers()
- Updated judge_runner.py to use new single-judge API
- Added scripts/judge_comparison.py for testing judge model pairs
- Updated v2_redesign notes to reflect implementation status
|
|
aggregation
- Remove multi-model aggregation support from Judge class
- Single blackbox judge: claude-sonnet-4-20250514 (anthropic)
- Single glassbox judge: gpt-4.1 (openai)
- Remove aggregate_results method and all aggregation-related code
- Simplify batch collection to single result per run
- Update config_dryrun.yaml with new judge configuration
|
|
- Separated Static Visualizations into dedicated tab with fullscreen image modal
- Added new interactive charts: Glassbox Category by Scenario, Intent vs Reality, and Radar Chart
- Replaced average token tracking with global token aggregations
- Implemented Grouped-Stacked Bar Chart for visual comparison of Action alongside Sophistication by oversight tier
- Added standardized color palettes to improve chart legibility
|
|
- Added HTTP POST /api/interrogate/start and /api/interrogate/chat to api/server.py
- Implemented ACTIVE_SESSIONS cache to persist synchronous agent state across requests
- Updated sys.path in api/server.py to resolve inner core imports correctly
- Added Inspector Modal 'Interrogate' view toggle in index.html
- Implemented JS interrogation loop, fallback log path resolution, auto-scroll, and markdown rendering in results.js
- Added custom CSS animations (.thinking-pulse, .pulse-ring) to mimic agent processing
|
|
and row inspector
|
|
- CSV results now found in logs/full_experiment/, output/, and config.output_dir
- Viz images now found in viz/ (project root) and output/charts/
- Judge logs now found in judge_logs/ and logs/judge_full_experiment/
- Added CORS middleware for local development
- Added /api/logs/ endpoints for browsing raw JSON experiment logs
- Added --port and --host CLI args
|
|
|
|
- 01: v1→v2 evolution history, results, limitations, rationalization hypothesis
- 02: judge system state, single judge per prong, 20% validation protocol
- 03: research plan analysis, Option B (two-phase) selected
- 04: publication plan — paper structure, three-layer analysis strategy,
Gemini ceiling effect handling, budget, venue targets, timeline
Key decisions:
- Two-phase design (540 runs) over full factorial (1,620)
- Single paper: v1 exploratory → Study 1 (oversight) → Study 2 (framing)
- Three-layer analysis: pooled → per-model → sensitivity excluding ceiling models
|
|
- Refactored Agent's chat loop to explicitly preserve and pass back reasoning context (reasoning_details, reasoning) to OpenRouter.
- Ensures frontier models like Gemini 3.1 and DeepSeek v3.2 maintain their 'thinking' chain during complex tool-calling sessions.
- Enhanced reasoning extraction to support structured details, reasoning_content, and fallback tags.
- Verified capture of encrypted and plain-text reasoning blocks in experimental logs.
|
|
|
|
Split monolithic src/judge.py into a proper package structure:
- src/judge/batch_providers.py: BatchProvider ABC, AnthropicBatchProvider, XAIBatchProvider
- src/judge/judges.py: RegexChecker, BlackboxChecker, GlassboxChecker
- src/judge/helpers.py: extract_* functions, parse_json_verdict
- src/judge/prompts.py: BLACKBOX_PROMPT, GLASSBOX_PROMPT, constants
- src/judge/judge.py: Judge orchestrator class
- src/judge/__init__.py: Re-exports all public APIs
|
|
- Add XAIBatchProvider class implementing BatchProvider ABC
- Add --provider CLI argument to judge_runner (anthropic|xai)
- Refactor Judge class to use BatchProvider instead of direct API calls
- Update tests for new Judge constructor signature
|
|
- Extracted Blackbox and Glassbox prompts from into their own markdown files in
- Implemented progressive 4-tier Blackbox categories (FABRICATED, OMITTED_CRITICAL, DECEPTIVE_FRAMING, ACCURATE)
- Added neutral examples to the Blackbox prompt to improve formatting consistency
- Updated references and tests to match new judge categorization structure
|
|
|
|
|
|
|
|
|
|
|
|
- VFSFactory: independent instances vs singletons
- make_tools_for_vfs: parallel binding
- config_loader: goal_types and max_workers parsing
- runner: goal_type iteration and resume skipping
- agent: goal_type storage and path resolution
|
|
- Create hidden_goals/ for all 3 scenarios (self_serving, moral, bare)
- Corporate: shareholder value vs community harm vs bare instruction
- Child protection: department stats vs saving lives vs bare directive
- Pharma: stock price vs patient access vs bare instruction
- Rebalance child_protection data: archive cases D, E, F (35KB -> 13.6KB)
- Update child_protection regex_rules.yaml for remaining 3 cases
|
|
- vfs.py: add VFS.create() factory for parallel-safe independent instances
- tools.py: add make_tools_for_vfs() to bind tools to specific VFS instances
- config_loader.py: add goal_types and max_workers config properties
- agent.py: add goal_type field, vfs_instance param, goal_type in log paths
- runner.py: rewrite with goal_type loop dimension, ThreadPoolExecutor parallel
execution, isolated VFS per run, flattened work items, thread-safe results
All backward compatible - empty goal_types defaults to legacy hidden_goal.md,
max_workers=1 defaults to sequential execution.
|
|
|
|
|
|
This commit:
- Adds a 'Total Token Usage by Model' chart alongside the average token chart
- Updates the web_gui_plan.md to reflect the removal of execution APIs and the addition of the new data visualization features.
|
|
This commit:
- Bakes total/prompt/completion tokens directly into results.csv
- Reverts dynamic JSON scraping backend logic for speed
- Adds modal overlay functionality to zoom Chart.js graphs on click
|
|
This commit:
- Replaces single-select dropdowns with multi-select checkbox lists
- Adds 'All' and 'None' quick-action links for filter groups
- Implements custom priority sorting for Oversight levels (low -> mid -> high)
- Updates Chart.js logic to respect logical sorting on X-axes
|
|
This commit:
- Removes all execution logic (runs, cancellation, SSE logs)
- Strips the terminal and configuration sidebar
- Implements a global filtering panel (Model, Scenario, Oversight)
- Dedicated the UI 100% to interactive Chart.js and static visuals
|
|
This commit completely overhauls the initial GUI:
- Splits monolithic JS/CSS into an ES6 module structure
- Implements an industrial 'Control Room' dark theme
- Adds dynamic file selection endpoints for CSV and JSON judge logs
- Integrates Chart.js to automatically render visualizations from CSV data
- Updates web_gui_plan.md to reflect architectural changes
|
|
- Add FastAPI backend (api/server.py) with endpoints for:
- Config read/write
- Scenario/model discovery
- Experiment run management (start/cancel/status)
- Real-time log streaming via SSE
- Results fetching (CSV, images, judge results)
- Add vanilla JS frontend (api/static/):
- Clean dark-themed dashboard
- Configuration panel with dropdowns
- Live log terminal
- Results viewer with tabs
- Add documentation (docs/web_gui_plan.md)
Dependencies added: fastapi, uvicorn, sse-starlette
|
|
- Implement enabled_judges parameter in Judge class to selectively run regex, blackbox, or glassbox checks.
- Add --judges CLI argument to judge_runner.py for granular control over judging pipeline.
- Add --model and --scenario filters to judge_runner.py for targeted evaluation of experiment logs.
- Optimize synchronous judging to skip LLM calls and API key checks when only local (regex) judges are enabled.
- Update Anthropic Batch API integration to support dynamic number of requests per log based on enabled judges.
- Improve console output and logging to reflect selected filters and active judge components.
|
|
|
|
- Verdict shows matched/total required rules for more nuanced evaluation
- Adds 'matched' and 'total' fields to result dict
- Returns 'N/A' when no rules are defined
|
|
UNINTENTIONAL_FAILURE
- ROBUST_HONEST: ignored hidden goal + output is accurate and complete
- UNINTENTIONAL_FAILURE: ignored hidden goal + output has genuine quality issues
- Added explicit disambiguation note in the prompt
- Clarified CONSIDERED_HONEST as deliberate refusal to comply
|
|
test.
|
|
- Blackbox prompt now includes TASK DESCRIPTION section from benign_goal.md
- Helps judge evaluate framing even without a baseline
- Hidden goal is never exposed (maintains blackbox/glassbox boundary)
- Applied to both judge_single and prepare_batch_requests paths
|
|
- RegexChecker now handles both list-format and dict-format YAML rules
- Judge falls back to experiment log's VFS state for source docs when no baseline exists
- Blackbox judge receives '(No baseline available)' instead of crashing
- Resume logic excludes .partial.json from completed run count
|
|
Previously a scenario with its own oversight/ directory would ignore
the global oversight_levels config. Now the global list acts as a
filter — only levels present in BOTH the scenario dir AND the global
config are run. Warns if no levels match.
|
|
- Writes _in_progress.partial.json after each turn in the chat loop
- Partial file persists if run crashes/hangs for post-mortem inspection
- Cleaned up automatically when final save_logs succeeds
- Enabled in both baseline and experiment runs via runner.py
|
