docs: dissertation draft

1 files changed, 73 insertions, 3 deletions

diff --git a/docs/final/methodology.typ b/docs/final/methodology.typ
index 0032f01..87aff01 100644
--- a/docs/final/methodology.typ
+++ b/docs/final/methodology.typ
@@ -8,6 +8,36 @@ The reasoning traces are treated as observable behavioral outputs rather than re
 
 Known limitations are declared: the obfuscation problem (methods assume human-legible reasoning), hybrid threats (trained-in backdoors are excluded), and environmental fidelity (the VFS sandbox is simplified).
 
+== Two-Phase Confirmatory Design
+
+The v2 experimental design departs from the full factorial structure employed in v1, adopting instead a two-phase confirmatory approach. This revision addresses a critical methodological limitation: the full $3 times 3 times 2$ factorial (oversight × goal type × scenario) would yield 1,620 total runs, creating substantial resource demands while introducing interpretational challenges from three-way interaction effects. The two-phase design isolates key variables into standalone, interpretable studies.
+
+=== Study 1: The Oversight Effect
+
+Study 1 examines whether increased oversight intensity reduces deceptive behavior. The design employs a $3 times 3$ structure: three models (DeepSeek V3.2, Gemini 3.1 Pro, GLM-5.1) crossed with three oversight levels (low, mid, high), with 30 replicates per cell yielding 270 total runs.
+
+The scenario selection for Study 1 is deliberate and exclusive: only the corporate_sabotage_v2 scenario is used. This isolation serves an important methodological function. By holding the scenario constant, any observed variation in deception rates can be attributed to the oversight manipulation rather than scenario-specific effects. The corporate sabotage scenario provides a clean test case because it presents explicit adversarial framing—a direct instruction to protect a specific company's reputation—which models should recognize as problematic if oversight cues are salient.
+
+Furthermore, Study 1 employs only the "bare" goal type: the hidden goal is presented without any moral or self-serving framing. This serves as a control condition, establishing baseline deceptive behavior without the complicating factor of rationalization. If models deceive when given straightforward adversarial instructions but are deterred by oversight, this provides clear evidence that oversight mechanisms can influence behavior even in the absence of narrative framing that might enable post-hoc justification.
+
+The expected contribution of Study 1 is a clean test of the oversight-deception relationship. By isolating this variable, we can establish whether the hypothesized deterrent effect exists and characterize its magnitude across different model architectures.
+
+=== Study 2: The Framing Effect
+
+Study 2 shifts focus to examine how goal framing influences deceptive behavior independent of oversight variation. The design again uses three models, but now crosses them with three goal types (bare, moral, self_serving) while holding the oversight level fixed at mid (10% random review).
+
+The mid-level oversight condition was selected for Study 2 because it represents a moderate baseline: not so lenient as to permit unfettered deception, but not so stringent as to produce ceiling effects where all models refuse. This allows us to detect variation in how different framings influence behavior under conditions approximating real-world deployment scenarios.
+
+The scenario for Study 2 is child_protection, which tests a fundamentally different deception mechanism than corporate sabotage. Where corporate sabotage tests explicit adversarial framing, child protection tests _bureaucratic rationalisation_—deception that emerges from morally justified, procedurally plausible objectives. By testing whether moral framing ("prudent resource allocation") increases compliance compared to bare instructions, Study 2 directly addresses the rationalization hypothesis: that models are more likely to deceive when they can frame their actions as serving legitimate goals.
+
+The expected contribution is a test of whether rationalization enables deception even under fixed oversight conditions. If moral framing significantly increases deception rates, this suggests that current oversight paradigms may be poorly aligned with the dominant failure modes of automated decision systems.
+
+=== Rationale for Separate Studies
+
+The decision to separate these investigations rather than combine them into a full factorial design reflects several methodological considerations. First, three-way interactions (model × oversight × framing) are notoriously difficult to interpret; main effects and two-way interactions can be obscured by complex higher-order effects. Second, each study is designed to provide a standalone, interpretable result capable of standing alone should resource constraints prevent completion of the full protocol. Third, the phased structure allows Study 2 to build upon Study 1 findings: if Study 1 establishes that oversight has limited deterrent effect, Study 2 can examine whether rationalization further erodes what little protection oversight provides.
+
+The total design yields 540 runs—substantially more efficient than the 1,620 runs required for full factorial—while maintaining statistical power for the targeted comparisons of interest.
+
 == System Architecture
 
 To evaluate agents in realistic, multi-turn loops rather than single-turn prompts, we developed a custom orchestration framework. The agent manages a continuous conversation loop (maximum 20 turns), executing tools and updating state autonomously based on the system prompt directives. The system is designed to be modular and extensible, allowing for easy integration of new models, scenarios, and oversight conditions, all of which can be configured via YAML configuration files. Every agent interaction is logged in structured JSON format for downstream evaluation.
@@ -70,6 +100,14 @@ Seven models spanning different architectures and providers were selected, all e
 
 Models were selected to represent diverse architectures and providers while maintaining access to reasoning traces required for glassbox evaluation. Temperature was fixed at 1.0, following the default settings recommended by the respective model providers. This choice reflects an ecological evaluation goal: models were tested under conditions similar to typical production deployments rather than deterministic benchmarking configurations.
 
+=== Model Selection Rationale
+
+The three models selected for v2 (DeepSeek V3.2, Gemini 3.1 Pro, GLM-5.1) were chosen based on their distinctive deception profiles observed in v1 pilot testing. DeepSeek V3.2 demonstrated near-ceiling deception rates across conditions, providing a test case for whether any oversight manipulation can deter models prone to systematic deception. Gemini 3.1 Pro showed variable behavior—deceptive under some conditions but honest under others—making it sensitive to experimental manipulations. GLM-5.1 exhibited floor effects, refusing deceptive instructions even without oversight, which helps establish whether the scenarios themselves are capable of eliciting deception when models are compliant.
+
+This diversity in baseline behavior is methodologically valuable. Ceiling and floor effects, typically viewed as limitations, become findings in this design: if oversight cannot deter a model that deceives 95% of the time, this itself constitutes important evidence about the limitations of oversight interventions. Similarly, if a model that refuses 95% of deceptive requests in v1 suddenly becomes compliant under specific framing conditions in v2, this reveals the power of rationalization.
+
+The reasoning trace requirement imposed a significant constraint on model selection. Only models exposing chain-of-thought reasoning could be considered for glassbox evaluation, excluding popular models like GPT-4o or Claude variants that do not provide reasoning traces. This trade-off was accepted because glassbox classification—determining whether models are aware they are deceiving—is central to the research questions. The selected models represent a mix of providers accessed through OpenRouter, ensuring diversity and avoiding same-family bias that might arise from testing multiple models from a single provider.
+
 == Baseline Generation
 
 For each model--scenario pair, a baseline run is generated using _only_ the benign goal (no hidden goal, no oversight). The baseline output---extracted from the last `create_file` tool call---serves as the honest reference for blackbox judging. Known limitation: if the baseline contains errors, blackbox comparisons inherit them. This is partially mitigated by including source documents in the judge's input hierarchy.
@@ -94,13 +132,45 @@ The same judge receives the full system prompt (including hidden goal), concaten
 
 All judging uses Claude Sonnet 4 (Anthropic)---a different model family from all test subjects---at temperature 0 via the Batch API (50% cost). Each run requires two judge calls, totalling 840 API requests. Known biases include narcissistic bias #cite(<panickssery2024llmevaluatorsrecognizefavor>) and verbosity bias, partially mitigated by cross-family judging.
 
+=== Judge Validation
+
+Given the centrality of judge classifications to all downstream analyses, a validation study was conducted to establish the reliability of proxy judges relative to a gold standard. The validation employed a stratified sampling approach: 20% of all runs (n=108) were selected across model, scenario, and outcome strata to ensure representative coverage.
+
+The gold standard judgments were produced by Claude Sonnet 4.6, a more capable model from the same family as the primary judge but representing a higher capability tier. This gold standard judged the stratified subset on both blackbox integrity (output classification) and glassbox dimensions (intent and sophistication).
+
+Two proxy judges were evaluated as potential substitutes: Grok-4.1-fast-reasoning (Blackbox mode, no reasoning traces provided) and GPT-4.1 (Glassbox mode, with reasoning traces). Inter-rater reliability was assessed using Cohen's Kappa ($kappa$), which measures agreement beyond chance. Results demonstrated substantial agreement between proxy judges and the gold standard:
+
+- *Blackbox classifications*: Grok-4.1-fast-reasoning achieved $kappa = 0.702$, indicating substantial agreement with the gold standard on output integrity judgments.
+- *Glassbox intent classifications*: GPT-4.1 achieved $kappa = 0.878$, indicating almost perfect agreement on deceptive intent categorization.
+- *Glassbox sophistication classifications*: GPT-4.1 achieved $kappa = 0.538$, indicating moderate agreement on sophistication tier assignment.
+
+The validation methodology employed a split configuration rationale: blackbox judgments used a different provider (xAI via OpenRouter) than glassbox judgments (OpenAI), while the gold standard remained constant (Anthropic). This design avoids same-family bias that could inflate agreement scores if proxy and gold standard judges shared training data or architectural assumptions.
+
+An acceptance threshold of $kappa > 0.6$ (substantial agreement) was established a priori based on conventions in inter-rater reliability research. Both the blackbox and glassbox intent judges exceeded this threshold, validating their use in the main evaluation pipeline. The lower sophistication agreement ($kappa = 0.538$) was deemed acceptable because sophistication classification is inherently more subjective than intent classification, involving judgment about the quality and depth of reasoning rather than categorical determination of deceptive intent.
+
 == Statistical Analysis
 
-All quantitative analyses were conducted using Python's `SciPy` and `statsmodels` libraries. Because the dependent variables are categorical (e.g., deceptive vs. honest, strategic vs. naive), non-parametric tests were employed to assess statistical significance. Replicate runs were treated as independent observations because each run was executed in a freshly initialized agent environment.
+All quantitative analyses were conducted using Python's `SciPy` and `statsmodels` libraries. The analytical strategy employs a three-layer approach designed to provide both headline results and nuanced model-specific insights while handling data characteristics that violate parametric assumptions.
+
+=== Three-Layer Analysis Strategy
+
+*Layer 1: Pooled Analysis.* The primary analysis combines all models to test the central hypotheses: whether oversight intensity reduces deception (Study 1) and whether moral framing increases deception (Study 2). This pooled approach maximizes statistical power and provides the headline result interpretable as a cross-model tendency. The pooled sample yields 90 observations per condition in Study 1 (3 models × 30 replicates) and 90 observations per goal type in Study 2.
+
+*Layer 2: Per-Model Analysis.* Secondary analyses examine each model individually to identify architecture-specific patterns. This layer is critical because models exhibited heterogeneous baseline behaviors in v1—some near ceiling, some near floor, some variable. Per-model analysis treats these heterogeneities as findings rather than nuisances. For example, if oversight significantly reduces deception in Gemini but not in DeepSeek, this suggests that oversight effectiveness depends on model-specific factors such as fine-tuning or constitutional training.
+
+*Layer 3: Sensitivity Analysis.* Tertiary analyses exclude models exhibiting ceiling or floor effects (defined as < 5% or >95% variance in the dependent variable). This sensitivity analysis tests whether the pooled findings are robust to exclusion of models at performance boundaries. If results persist after excluding ceiling/floor models, this strengthens confidence that the effects generalize to models with responsive behavior ranges.
+
+=== Justification for Non-Parametric Tests
+
+The dependent variables are categorical (deceptive vs. honest; strategic vs. naive vs. reactive vs. tactical), violating the normality assumptions required for parametric tests such as ANOVA. Furthermore, deception rates in pilot data exhibited bimodal distributions for some models (clustering near 0% and 100%), rendering mean-based comparisons misleading. Non-parametric tests were therefore employed exclusively.
+
+The Pearson Chi-Square test of independence ($chi^2$) served as the primary test for comparing deception frequencies across independent groups (e.g., between goal types in Study 2 or between oversight levels when treated as nominal categories). The Cochran-Armitage trend test evaluated ordinal relationships, specifically the hypothesis that deception decreases monotonically with oversight intensity (Low → Mid → High).
+
+=== Handling Ceiling and Floor Effects
 
-The primary test used for comparing deception frequencies across independent groups (e.g., between scenarios or between model tiers) was the Pearson Chi-Square test of independence ($chi^2$). To evaluate the impact of ordinal oversight levels (Low, Mid, High) on deception rates, we utilized the Cochran-Armitage trend test. 
+Ceiling and floor effects pose particular interpretive challenges. When a model deceives on 100% of low-oversight runs, this creates a boundary condition where oversight "improvements" have no room to manifest. Rather than excluding such models, they are analyzed explicitly: ceiling models (95-100% deception in permissive conditions) are examined for evidence of "refusal-to-deceive" transitions, while floor models (0-5% deception) are examined for "rationalization-induced compliance." The per-model analysis layer captures these dynamics, while sensitivity analyses verify that pooled conclusions do not depend on boundary models.
 
-For all tests, the threshold for statistical significance was set a priori at $alpha = 0.05$. Where statistically significant relationships were identified, we calculated Cramér's V to report the effect size (categorized as small, medium, or large) and Odds Ratios (OR) with 95% Confidence Intervals (CI) to quantify the magnitude of the vulnerability.
+For all tests, the threshold for statistical significance was set a priori at $alpha = 0.05$. Where statistically significant relationships were identified, Cramér's V reports effect size (small: 0.1, medium: 0.3, large: 0.5) and Odds Ratios (OR) with 95% Confidence Intervals (CI) quantify vulnerability magnitude. All analyses were conducted in Python using `scipy.stats` for non-parametric tests and `statsmodels` for contingency table analyses and OR calculations.
 
 == Reproducibility