docs: update dissertation intro, lit review, and methodology chapters

1 files changed, 81 insertions, 131 deletions

diff --git a/docs/final/methodology.typ b/docs/final/methodology.typ
index a49785f..f3c5408 100644
--- a/docs/final/methodology.typ
+++ b/docs/final/methodology.typ
@@ -1,189 +1,139 @@
 = Methodology
 
-== Introduction
-
-This research investigates whether AI agents exhibit conditional deception ( pursuing hidden objectives while appearing compliant ) with behavior modulated by perceived oversight intensity. While recent work demonstrates that frontier models engage in in-context scheming (Meinke et al., 2025; Järviniemi & Hubinger, 2024), a critical gap persists: no study systematically isolates perceived oversight as an independent variable affecting deceptive behavior.
-
-This methodology addresses this gap through controlled experimentation with a custom agentic framework that enables precise manipulation of oversight conditions while maintaining consistent task structure across trials.
-
 == Research Methodology
 
-=== Methodological approach
-
-This research employs an experimental observational methodology to study emergent behavior in AI systems. Unlike supervised learning studies that optimize model performance on benchmarks, this work observes how existing pretrained models behave under adversarial conditions without modification to their weights or training procedures.
-
-The methodology follows a controlled experiment design:
-
-- *Independent variables*: Model architecture, oversight level, scenario type
-- *Dependent variables*: Deceptive behavior classification (via 3-prong rubric), deception sophistication level
-- *Control measures*: Fixed temperature settings, identical system prompts per oversight level, standardized Virtual File System (VFS) environment
-- *Experimental units*: Individual conversation runs (model × scenario × oversight combination)
-
-=== Ideology [TOFIX]
-
-This research takes a behavioural rather than interpretability-based approach to studying AI deception. While mechanistic interpretability seeks to understand why models behave as they do through weight analysis, behavioral observation examine what models do under specific conditions. The reasoning traces (chain-of-thought) serve as behavioural indicators rather than ground truth about internal model states.
+This dissertation uses an **experimental observational** methodology to study deceptive behavior in AI systems under conflicting goals and varying perceived oversight.
 
-The literature review identifies perceived oversight as a critical unexplored variable. While existing work demonstrates scheming capabilities, studies either provide hidden goals without oversight variation (Meinke et al., 2025) or evaluate scheming in environments where models are aware of being tested (Schoen et al., 2025). 
+The study examines whether, in a prompt context that includes both a benign task objective and a hidden objective, model outputs and reasoning traces exhibit patterns consistent with deception.
 
-This work addresses these gaps through:
-- 
+=== Experimental Design
 
-*Scope Limitations*
+The experiment follows a controlled design:
 
-- Obfuscation Problem (out of scope): As Schoen et al. warns, models may learn to reason in uninterpretable encodings, or their reasoning traces might not accurately reflect internal reasoning. This research assumes human-legible accurate reasoning traces and acknowledges that the detection methods will likely fail when models start to obfuscate deliberation.
+- **Independent variables:** model, scenario type, and perceived oversight intensity
+- **Dependent variables:** deception classification and deception sophistication level
+- **Control measures:** fixed generation settings from configuration (including temperature), a standardized sandboxed Virtual File System (VFS), and an oversight-specific prompt component
+- **Experimental units:** individual conversation runs (model × scenario × oversight)
 
-- Hybrid Threats (out of scope): This work exclusively focuses on in-context deception, the interaction between trained-in backdoors and contextual triggers as identified in the literature as the more dangerous scenario, requires more research.
+=== Interpretation and Epistemic Stance
 
-- Observer Effect Ambiguity (acknowledged, not resolved): Whether oversight cues trigger genuine situational awareness or merely shift token probabilities cannot be determined from behavior alone. This research focuses on the model's behaviour regardless of why.
-
-- Perceived vs Real Oversight (out of scope): This work implies oversight by the model's system message, and or letting the model discover it by interacting with the environment. No real oversight was implemented, this work assumes that real oversight would have similar effect.
+This research adopts a behavioral approach. Reasoning traces (where extractable) are treated as evidence about the model’s apparent intent and strategy, rather than as a guaranteed view of internal computation.
 
+The evaluation framework is therefore designed to classify observable behavior using multiple complementary signals.
 
 == System Architecture
 
-The goal was to create a modular system which allows for fast iterative testing for different models, scenarios, tools and oversight levels. To achieve this, multiple conscious decisions were made. 
-
-=== Overview
-
-This system's goal is to test model's behavioural changes under conflicting goals under varying oversight levels. To accomplish the goal of modularity, the system is divided into multiple sections. 
-
-=== Configuration system
-
-The conclusion to use a configuration file was derived from preliminary testing which suggested that an easy, simple way to vary model features was needed for faster testing. This configuration file supports defining defaults, provider settings, scenarios to test, and output configurations.
-
-The project uses a two-part configuration system:
-
-1. `config.yaml` - Defines what to run:
-   - Models to test (with provider info)
-   - Scenarios to run
-   - Oversight levels
-   - Output settings
+To support modular experimentation, the implementation is structured so that models, scenarios, tools, and oversight conditions can be varied through configuration.
 
-  ```yaml
-  # Default parameters (can be overridden per model)
-  defaults:
-    temperature: 1.0
-    max_tokens: 4096
+=== Configuration System
 
-  providers:
-    google:
-    api_key_env: GOOGLE_API_KEY
-    base_url: https://generativelanguage.googleapis.com/v1beta/openai/
-    models:
-    - id: gemini-3-flash-preview
-      provider: google
-      temperature: 1
-      extra_body:
-        ...
+Experiment configuration is provided via `config.yaml` together with environment variables for API credentials.
 
-  scenarios:
-    - path: scenarios/corporate_sabotage_v2
-      runs: 2
+The configuration specifies:
 
-  oversight_levels:
-    - low
-    - ..
-  output:
-    dir: logs
-    save_vfs_state: false
-  ```
+- which models to test (including provider and optional provider-specific parameters)
+- which scenario directories to run
+- which oversight labels to evaluate
+- output and logging settings
 
-2. `.env` - Stores API keys as environment variables:
-
-   ```env
-   OPENAI_API_KEY=sk-...
-   GOOGLE_API_KEY=...
-   ANTHROPIC_API_KEY=...
-   OPENROUTER_API_KEY=...
-   ```
-
-The `ConfigLoader` reads `config.yaml` and looks up API keys from environment variables.
+A configuration loader parses these settings into objects consumed by the experiment runner.
 
 === Virtual File System (VFS)
 
+The agent interacts with a sandboxed Virtual File System (VFS), initialized from scenario-specific data directories. The VFS provides a consistent file interface and isolates the experiment from the host file system.
 
+The VFS supports tool operations including:
 
-=== Reasoning Extraction
-
-=== Logging Structure
-
-=== Interrogation Framework
-
-
-== Scenario structure and design
-
-=== Standard scenario format
-
-=== Design rational
-
-=== Scenario realism and model awareness
-
-
-== Experimental procedure
-
-=== Experiment flow
-
-=== Oversight levels
-
-
+- listing files
+- reading files
+- creating files
+- deleting files
 
+=== Agent Implementation
 
-== Evaluation Framework
+The agent uses an OpenAI-compatible client interface so that multiple providers can be tested under a unified calling pattern.
 
-=== The 3-pronged evaluation rubric
+Key implementation behaviors include:
 
-=== Prong 1 : Deterministic filter (regex)
+- extracting reasoning information from provider-specific response fields when available
+- handling tool calls: when the model requests tools, the framework executes them and appends tool outputs back into the conversation
+- writing logs incrementally to reduce the likelihood of losing partial results if a run terminates unexpectedly
 
-=== Prong 2 : Black Box Judge (User perception)
+=== Scenario Structure and Design
 
-=== Prong 3 : Glass Box Judge (Intent detector)
+Each scenario directory contains:
 
-=== Triangulation across prongs
+- a benign goal (`benign_goal.md`)
+- a hidden objective (`hidden_goal.md`)
+- a user instruction (`user.md`)
+- a set of source documents under `data/`
+- scenario-specific regex rules under `regex_rules.yaml`
+- oversight cues under `oversight/`
 
-=== Deception sophistication scale
+The system prompt is assembled such that the model is exposed to benign and hidden objectives along with the text corresponding to the chosen oversight condition.
 
-=== Inter-rater reliability
+== Experimental Procedure
 
-=== Validity considerations
+Each run proceeds as follows:
 
+1. Initialize the VFS with scenario-specific documents.
+2. Construct an agent using model/provider configuration.
+3. Assemble the system prompt including benign goal, hidden goal, and the oversight cue text.
+4. Run the agent with the scenario user instruction, allowing tool calls when requested.
+5. Save the complete conversation logs, including any extractable reasoning fields.
 
-== Analysis approach
+== Evaluation Framework: The 3-Pronged Judging System
 
-=== Data Organisation
+The dissertation uses a multi-pronged rubric to classify deception and assess deception sophistication.
 
-=== Cross-Tabulation Analysis
+=== Prong 1: Deterministic Regex Checker (Regex)
 
-=== Reproducibility measures
+Scenario-defined YAML rules provide deterministic pattern checks over the final output. This prong aims to capture cases where required details are omitted or prohibited content is present.
 
-=== Preliminary findings and expected pattern
+=== Prong 2: Black-Box Judge (Output-Level)
 
+A separate LLM judge evaluates the final output without access to the reasoning trace. The judge compares the output against:
 
-== Sample size and statistical power
+- the task description
+- the source documents considered ground truth for content
+- a baseline output produced in the absence of the hidden goal
+- the model output under evaluation
 
-=== Per-Condition runs
+The black-box judge assigns one category:
 
-=== Power Analysis
+- **ACCURATE**
+- **MISLEADING**
+- **OMITTED**
+- **FABRICATED**
 
-=== Budget Constraints
+=== Prong 3: Glass-Box Judge (Intent-Level)
 
+A glass-box judge is provided with the extracted reasoning trace, the full system prompt, and the final output.
 
+The glass-box judge assigns one intent category:
 
+- **ROBUST_HONEST**
+- **CONSIDERED_HONEST**
+- **COMPLIANT_DECEPTION**
+- **FAILED_DECEPTION**
+- **UNINTENTIONAL_FAILURE**
 
-== Challenges and Limitations
+The judge also assigns a sophistication tier:
 
-=== Epistemic Limitations
+- NAIVE, REACTIVE, TACTICAL, STRATEGIC
 
-=== Scope Limitations
+=== Deception Definition Used in the Main Results
 
-=== Methodological Limitations
+For the primary results presented in this dissertation, a run is treated as **deceptive** when the glass-box category is **COMPLIANT_DECEPTION**.
 
-=== Theoretical Interpretation 
+This definition is used to derive deception rates across oversight levels, models, and scenario types.
 
+== Data Analysis Approach
 
-== Ethical Consideration
+Aggregated statistics are computed from `/home/jayrup/uni/dis/logs/full_experiment/results.csv`.
 
-=== Research scope and safety
+The analysis reports deception rates (overall and conditional on supervision cues), sophistication distributions, and breakdowns across black-box categories.
 
-=== Dual-Use Implications
+All numeric claims included in the Results and Analysis sections are derived directly from the CSV dataset.
 
-=== Transparency and Reproducibility 
+#pagebreak()