summaryrefslogtreecommitdiff
path: root/src/runner.py
AgeCommit message (Collapse)Author
2026-04-01Fix error runs not appearing in final summaryCaptainJack2491
2026-03-25feat: implement rich dashboard for experiment progressCaptainJack2491
2026-03-25feat: add tqdm progress bar and fix main config flag loadingCaptainJack2491
2026-03-25feat: add goal_type variation system and parallel processingCaptainJack2491
- vfs.py: add VFS.create() factory for parallel-safe independent instances - tools.py: add make_tools_for_vfs() to bind tools to specific VFS instances - config_loader.py: add goal_types and max_workers config properties - agent.py: add goal_type field, vfs_instance param, goal_type in log paths - runner.py: rewrite with goal_type loop dimension, ThreadPoolExecutor parallel execution, isolated VFS per run, flattened work items, thread-safe results All backward compatible - empty goal_types defaults to legacy hidden_goal.md, max_workers=1 defaults to sequential execution.
2026-03-02fix: judge resilience for missing baselines and list-format regex rulesCaptainJack2491
- RegexChecker now handles both list-format and dict-format YAML rules - Judge falls back to experiment log's VFS state for source docs when no baseline exists - Blackbox judge receives '(No baseline available)' instead of crashing - Resume logic excludes .partial.json from completed run count
2026-03-02fix: global oversight_levels now filters scenario-available levelsCaptainJack2491
Previously a scenario with its own oversight/ directory would ignore the global oversight_levels config. Now the global list acts as a filter — only levels present in BOTH the scenario dir AND the global config are run. Warns if no levels match.
2026-03-02feat: incremental log saving via .partial.json filesCaptainJack2491
- Writes _in_progress.partial.json after each turn in the chat loop - Partial file persists if run crashes/hangs for post-mortem inspection - Cleaned up automatically when final save_logs succeeds - Enabled in both baseline and experiment runs via runner.py
2026-03-02feat: add generate_baseline toggle to configCaptainJack2491
- New 'generate_baseline' option in defaults (default: true) - When false, skips baseline generation entirely - Warns if no baseline exists when generation is disabled - Updated config.yaml.example and README.md with docs
2026-03-01feat: add centralized logging with configurable debug levelsCaptainJack2491
- Create src/logger.py with Python logging module - Add 4 debug levels: 1=CRITICAL, 2=WARNING, 3=INFO, 4=DEBUG - Level 4 includes reasoning, VFS info, and available tools - Level 4 auto-enables file output (both mode) - Update agent.py and runner.py to use logger instead of print - Update config.yaml.example with logging configuration - Update README with logging documentation
2026-02-27fix: separate skipped runs from incomplete in summary outputCaptainJack2491
2026-02-26feat: add checkpoint/resume support and atomic log writesCaptainJack2491
- Runner now counts existing .json log files per combination and skips completed runs, resuming from where it left off - Log files are written atomically via temp file + rename to prevent corrupt files from being counted as completed - Added --no-resume CLI flag to force a fresh start - Resume is enabled by default
2026-02-24[feat] baseline system - auto-generates baselines before experimentsCaptainJack2491
- Added _extract_baseline_content() to extract output from create_file tool calls - Baselines now saved to logs/{model}/{scenario}/baseline.md - Added global oversight fallback (scenario-specific → global oversight dir) - Updated config_loader with oversight level loading from scenario dirs
2026-02-18[feat] added a baseline to the runnerCaptainJack2491
(no commit message provided) (no commit message provided) Co-authored-by: aider (openrouter/openrouter/aurora-alpha) <aider@aider.chat>
2026-01-11[fix] simplified everythingCaptainJack2491
made eveyrthing much cleaner and it works very well. got rid of providers setup, only use openai sdk now.
2026-01-11[fix] big moving around, probably made lot more bugs.CaptainJack2491