Orion Investigation — Wallet Drift Root Cause¶
To: Atlas, Vesper
CC: Katja
From: Orion
Date: 2026-04-19
Priority: Production blocker — Phase 7.3 gate
DB: neo_live_stage1.db (confirmed live via S39 halt_reason match)
Script: 02 Projects/NEO Trading Engine/scripts/wallet_drift_investigation_v2.py
Executive verdict¶
The wallet drain from post-injection baseline (66.82 XRP) → real on-chain (28 XRP) is NOT a single-session engine blowout. It is the cumulative result of two separate forces, split by where the DB sees them:
- Inside captured sessions (S33–S38): cumulative inter-session drift = −3.63 XRP only. Session-bounded drift is small.
- Outside captured sessions (after S38 close at 2026-04-18T21:11Z): the remaining −45.29 XRP gap is NOT in this DB. Candidates: residual resting orders filling post-shutdown, manual/external activity, or a session whose start row never recorded. This is the dominant component of the reported drain.
S39 is a ghost. The recorded halt_reason "xrp exposure limit (115.20 > 100.0)" is stale state from the halt-reason-lifecycle bug Branch #1 was supposed to fix. Actual S39 XRP balance never exceeded 80.64 across 335 valuation snapshots; XRP balance delta was +0.0107 XRP. No phantom inventory.
The anchor-saturation hypothesis is strongly supported at aggregate level. Engine_state rollup shows 54.6% of ticks at anchor cap, 75.2% beyond the 5 bps reliability floor, median anchor error = 10.0 bps (pinned). Per-session attribution is IMPOSSIBLE with current telemetry — anchor divergence is not persisted in system_metrics or market_snapshots. This is a structural telemetry gap.
Fill-size asymmetry is real but modest and mixed-direction across sessions. S36 was net −6.15 XRP despite more buys by count (Atlas's hypothesis pattern). S37 was net +10.81 XRP. S34 was net −19.5 XRP from only 3 fills. No single session drove the drift.
INVESTIGATION RESULTS — Wallet Drift Root Cause¶
Session mapping: S36 = DB 36 | S37 = DB 37 | S39 = DB 38 (S38 labeled session never wrote a sessions row — lost to CTRL_CLOSE_EVENT)
A. Drift by session (inter-session XRP diffs)¶
Pre-FLAG-NEW-001 the sessions table has NULL ended_at and ending_xrp for sessions 1–37. Drift reconstructed as next.starting_xrp − this.starting_xrp, capturing intra-session fills PLUS any manual activity between sessions.
Post-injection cascade (S33 onward):
| session | started_at | start_xrp | start_rlusd | delta_to_next_xrp | delta_to_next_rlusd |
|---|---|---|---|---|---|
| 33 | 2026-04-18T00:48Z | 76.92 | 85.19 | 0.00 | 0.00 |
| 34 | 2026-04-18T00:51Z | 76.92 | 85.19 | −13.19 | +19.50 |
| 35 | 2026-04-18T01:05Z | 63.73 | 104.69 | +6.09 | −9.00 |
| 36 | 2026-04-18T02:18Z | 69.83 | 95.69 | −4.15 | +6.15 |
| 37 | 2026-04-18T16:35Z | 65.67 | 101.84 | +7.61 | −10.81 |
| 38 (S39) | 2026-04-18T20:40Z | 73.28 | 91.03 | — | — |
| 38 end | 2026-04-18T21:11Z | 73.29 | 91.03 | — | — |
Intra-post-injection cumulative drift: 76.92 → 73.29 XRP = −3.63 XRP (and +5.84 RLUSD, roughly consistent with selling XRP into RLUSD at ~1.45).
Reported real-wallet delta from post-injection baseline: 66.82 → 28.0 XRP = −38.82 XRP.
Gap outside DB capture: 73.29 (S39 end) − 28.0 (real wallet) = −45.29 XRP unaccounted-for after S39 shutdown. The drain happened AFTER the last session in this DB closed.
Note: the post-injection baseline Atlas cited (66.82 XRP) does not match S33 start (76.92 XRP). There is a +10 XRP gap between the reported post-injection baseline and the DB's view of the first post-injection session start. Possible explanations: (a) 66.82 was measured at a different moment than S33 start, (b) there were fills or manual moves between injection and S33 start, (c) capital_events tracking vs sessions tracking use different sources. Flagging for Vesper.
Conclusion: Sessions S33–S38 together accounted for only ~9% of the reported drain (−3.63 of −38.82 XRP). The remaining ~91% is either pre-S33 (between injection and S33 start) or post-S38-close (after 21:11Z on Apr 18).
B. Fill asymmetry (count vs notional)¶
Full post-injection table (sessions with fills):
| session | n_buys | n_sells | xrp_bought | xrp_sold | net_xrp |
|---|---|---|---|---|---|
| 34 | 1 | 2 | 10.5 | 30.0 | −19.5 |
| 35 | 1 | 1 | 19.5 | 10.5 | +9.0 |
| 36 | 11 | 10 | 103.5 | 109.65 | −6.15 |
| 37 | 20 | 11 | 111.31 | 100.5 | +10.81 |
| 38 (S39) | 4 | 4 | 30.0 | 30.0 | 0.0 |
Totals across S34–S38: 37 buys / 28 sells / net −5.84 XRP inside fills. Matches the A-block inter-session drift within rounding.
Atlas's hypothesis pattern — "more buys by count, net negative XRP": - S36 confirms: 11 buys + 10 sells by count, but avg sell size ~11.0 XRP vs avg buy size 9.4 XRP → net −6.15 XRP despite buy-count surplus. - S34 confirms more extremely: 1 buy + 2 sells only, but the sells averaged 15.0 XRP each vs the single buy at 10.5 → −19.5 XRP from 3 fills. - S37 contradicts: more buys AND net positive XRP — asymmetry works both ways. - S38 (S39) is balanced: 4/4 buys/sells, exact 30 XRP each side, net 0.
Pattern: asymmetry exists but is bidirectional and session-specific. The net-XRP-per-session has no consistent sign across the cascade — S34 (−19.5), S35 (+9), S36 (−6.15), S37 (+10.81), S38 (0). This is inconsistent with a persistent engine bias. It is consistent with anchor-direction-dependent fill skew (see Block C).
Conclusion: Fill-size asymmetry is real. S36 is a confirmed instance of Atlas's hypothesis. But it does NOT accumulate a drain in the same direction — the net-XRP signs flip session-to-session. The asymmetry is a CONDITION (produced by some upstream cause — likely anchor direction), not the root cause itself.
C. Anchor saturation — rollup only, per-session history not available¶
CRITICAL TELEMETRY GAP: The anchor divergence (last-computed-anchor-error vs CLOB mid, in bps) is NOT persisted per-tick in system_metrics or market_snapshots. Only a rolling cumulative rollup exists in engine_state.
Engine_state rollup (latest session aggregate):
| key | value |
|---|---|
| anchor.max_bps | 10.0 |
| anchor.median_bps | 10.0 |
| anchor.mean_bps | 7.63 |
| anchor.min_bps | −1.88 |
| anchor.pct_cap_applied | 54.6% |
| anchor.pct_error_above_5bps | 75.2% |
| anchor.pct_above_10bps | 0.0 |
| anchor.valid_count | 335 (= S39 tick count) |
| market.clob_mid_price | 1.431353 |
| market.clob_vs_amm_divergence_bps | 1.32 |
Reading: In the most recent session (S39, 335 ticks), more than half the time (54.6%) the anchor was pinned at the 10 bps cap — meaning CLOB and AMM were divergent enough that the anchor logic clipped the computed fair price to the cap. Three-quarters of ticks (75.2%) were beyond the 5 bps Atlas evaluation reliability floor. Median anchor error = cap value (10.0) — half the ticks were saturated.
This means S39 operated in the STRESS regime (>8 bps) more than half the time. Atlas's Apr 18 regime model: ALIGNED ≤3 / DIVERGENT 3–8 / STRESS >8 bps. S39 was NOT a controlled calibration — it was running in a regime where the fair-price anchor itself was unreliable.
Per-session correlation with drift is NOT computable from this DB. To do that we would need either:
- (a) an anchor-error column added to system_metrics or market_snapshots per tick, OR
- (b) archived log files from S33–S38 parsed for anchor values.
Branch #6 (distance-to-touch diagnostic) added distance-to-touch but NOT anchor-error-per-tick persistence. This is a gap that needs a follow-up branch (proposed name: feat/anchor-error-per-tick-telemetry) before Phase 7.3 can validate calibration choices in the anchor-regime dimension.
Conclusion: Anchor saturation is MASSIVELY present in S39 and — by strong prior — likely present in S36/S37/S38 under the same market conditions. Causal link to per-session drift cannot be proven with this DB, but the failure mode is consistent: when anchor saturates, quotes become biased relative to true mid, and fills arrive with a skew that accumulates when the market trends. This is the most plausible upstream cause of the fill-size asymmetry documented in Block B.
D. S39 phantom inventory — answered NO¶
[D.1] sessions row (after FLAG-NEW-001 fix, S39 is the first session to have ended_at populated):
| field | value |
|---|---|
| started_at | 2026-04-18T20:40:49Z |
| ended_at | 2026-04-18T21:11:01Z |
| starting_xrp | 73.2805 |
| starting_rlusd | 91.0335 |
| ending_xrp | 73.2912 |
| ending_rlusd | 91.0335 |
| halt_reason | xrp exposure limit (115.20 > 100.0) |
S39 intra-session delta: +0.0107 XRP / 0.0000 RLUSD. Session was effectively flat.
[D.2] inventory_ledger for S39 fills:
| asset | n_events | sum_change | min | max | last_balance |
|---|---|---|---|---|---|
| RLUSD | 8 | 0.0 | 80.53 | 110.53 | 91.03 |
| XRP | 8 | +0.0107 | 24.43 | 45.43 | 38.08 |
Note: inventory_ledger last_balance for XRP (38.08) differs from sessions ending_xrp (73.29) by exactly 35.21 XRP — the Apr 18 00:19 injection amount (Block E). The ledger appears to track position-from-trading without crediting the post-ledger-init deposit. The sum_change (+0.0107) matches sessions delta exactly, so the ledger's DELTA tracking is correct — only its running BASELINE lags. This is a ledger reconciliation issue to file separately; it does not indicate phantom state driving decisions.
[D.3] valuation_snapshots during S39: 335 records, xrp_balance range 59.64–80.64 XRP, total_value_rlusd 195.57–195.94. First-record xrp_balance = 73.28 (matches sessions.start); last-record = 73.29 (matches sessions.end).
[D.4] inventory_snapshots during S39: 335 records, xrp_pct range 43.53%–58.89%, xrp_balance range 59.64–80.64 XRP. Start/end match valuation_snapshots exactly.
[D.5] S39 fills (joined to orders for side):
| side | n | xrp moved | rlusd notional | avg price |
|---|---|---|---|---|
| buy | 4 | 30.0 | 42.87 | 1.4285 |
| sell | 4 | 30.0 | 42.90 | 1.4298 |
Perfectly balanced. 4 buys of 30 XRP equal 4 sells of 30 XRP, net 0 XRP, net +0.02 RLUSD (inside spread capture).
Conclusion — no phantom inventory: sessions, valuation_snapshots, inventory_snapshots, and the 4+4 fills are MUTUALLY CONSISTENT. Max observed XRP during S39 = 80.64, well below any exposure cap. The engine was NOT trading on inflated state.
But the halt_reason is fraudulent. "xrp exposure limit (115.20 > 100.0)" never happened during S39. The 115.20 figure has no source in S39 data. This is the halt-reason-lifecycle ghost — a prior session's halt reason persisted into engine_state and was written to the S39 sessions row at shutdown. Branch #1 (fix/halt-reason-lifecycle) is the fix; verify it actually cleared this path on shutdown, because the ghost still appeared.
Root cause assessment¶
Primary cause: Split between two forces. - INSIDE-SESSION component (small, ~3.63 XRP of 38.82): Fill-size asymmetry driven by anchor-regime stress. S36 is the clearest example (−6.15 XRP). Anchor saturation (54.6% at cap in S39, likely similar in S33–S37) is the most plausible upstream cause — when anchor pins at the cap, quote skew produces adverse fill selection. This is consistent with Atlas's Apr 18 regime taxonomy treating >8 bps as STRESS. - OUTSIDE-SESSION component (large, ~35 XRP of 38.82): UNACCOUNTED FOR IN THIS DB. Must have occurred either (a) between the Apr 18 00:19 injection and S33 start at 00:48Z (30-minute window — unlikely to drain 10 XRP without fills), or (b) after S39 close at 2026-04-18T21:11Z — e.g. residual resting orders filling on-ledger after engine shutdown without the engine to reflect the fills into its DB. Option (b) is the most likely source of the bulk of the drain.
Session where drift originated: No single session. The in-DB drift (−3.63 XRP) is spread across S34 (−19.5), S36 (−6.15), with partial offsets from S35 (+9) and S37 (+10.81). The bulk of the reported drain occurred outside captured sessions.
State mismatch (engine internal vs reality): NO for decision-making state. All three inventory views (sessions/valuation_snapshots/inventory_snapshots) agree during S39. The engine was NOT making decisions on phantom balances. YES for halt-reason state — S39 halt_reason is a ghost from a prior session. YES for inventory_ledger baseline tracking — the ledger never credited the Apr 18 deposit (lags by 35.21 XRP). Neither of these is phantom inventory in the decision-path sense.
Answers to Atlas's five questions¶
- What changed: The wallet moved from ~67 XRP to 28 XRP over the post-injection live period plus an unaccounted window after the last DB-captured session. The engine's internal books agree with the DB's sessions table; they do NOT agree with the on-ledger wallet from some point after S39 close.
- When it changed: Inside captured sessions: S34 (−19.5), S36 (−6.15), net of offsets = −3.63 XRP across Apr 18 00:51Z–21:11Z. Outside captured sessions: the ~35 XRP bulk drain occurred after 2026-04-18T21:11Z (S39 close) — exact timing unknown, not in this DB. The Apr 18 injection was at 00:19Z; S33 first session was at 00:48Z, so there is also a small pre-S33 window.
- How much it changed the wallet: Reported delta −38.82 XRP. DB-captured fraction: −3.63 XRP (9.4%). Out-of-capture fraction: −35.19 XRP (90.6%). The engine's in-DB view of post-injection drift is only a fraction of what actually drained.
- Whether internal state matched reality: During S39, internal state (valuation_snapshots, inventory_snapshots, sessions) matched the engine's own DB view perfectly. The engine's DB view matched reality inside captured sessions. The engine's DB view did NOT match reality after S39 close — the real wallet kept moving without the engine's DB receiving updates. Internal ≠ reality happened at the boundary of session capture, not during live trading.
- What exact protection layer would have stopped it:
- For the in-session fraction (−3.63 XRP): anchor saturation guard — halt if >X% of recent ticks at cap. This would have halted S36 before its fill-size asymmetry accumulated. Inventory corridor guard as secondary.
- For the out-of-capture fraction (−35.19 XRP): session-close order cancellation invariant — before
close_session, cancel all resting XRPL orders AND wait for confirmation that they are off-book. If residual orders fill post-shutdown, the engine's DB will not capture them. This is the dominant candidate root cause for the bulk drain and is a MISSING layer. Also: on-ledger wallet reconciliation at session start — read actual on-chain XRP/RLUSD before starting, compare against stored ending balances, halt if discrepancy exceeds threshold. This would have caught the out-of-capture drain at the next session start.
Protection layer design — three guards Atlas mandated + one more¶
1. Inventory corridor guard (session-runtime, Atlas-mandated)
- Metric: inventory_snapshot.xrp_pct every tick.
- Corridor: default [20.0, 80.0] (wider than the Branch 2 invariant band [5, 95] because this is a live guard, not a shutdown invariant). Config name: risk.inventory_corridor_min_pct / risk.inventory_corridor_max_pct.
- Action: three ticks outside corridor → halt with reason inventory_corridor_breach. Single-tick outside allows transient settling.
- Persistence: log inventory_corridor.breach_count and .last_xrp_pct to engine_state every tick.
2. Persistent directional drift guard (session-runtime, Atlas-mandated)
- Metric: rolling 5-minute window of sum(buy_xrp − sell_xrp) from fills.
- Threshold: halt if abs(net_xrp_5min) > 2 × base_size_rlusd (with current base_size=15, threshold=30 XRP of directional flow in 5 minutes). Config name: risk.directional_drift_threshold_xrp / risk.directional_drift_window_seconds.
- Action: on breach, halt with reason directional_drift_breach. Capture direction (buy-skew or sell-skew) in halt_reason.
- Rationale: would have halted S34 (−19.5 XRP in 3 fills) before completion.
3. Anchor saturation guard (session-runtime, Atlas-mandated)
- Metric: rolling 100-tick window of pct_at_cap. NOTE: requires the anchor-error-per-tick telemetry branch to be merged first, since system_metrics does not currently persist per-tick anchor.
- Threshold: halt if pct_at_cap_last_100_ticks > 40%. Config name: risk.anchor_saturation_pct_threshold / risk.anchor_saturation_window_ticks.
- Action: halt with reason anchor_saturation_breach. Log the rolling pct and the last N anchor values to engine_state.
- Prerequisite: requires a new branch feat/anchor-error-per-tick-telemetry to add the anchor column to system_metrics.
4. Session-close order cancellation invariant (NEW — not yet mandated, but primary candidate root cause)
- Action: in _shutdown, after halting tick loop and before close_session:
- call cancel_all_resting_orders() and await confirmation that XRPL reports zero open offers for the wallet.
- if any offers remain after a timeout (say 10s), write shutdown_order_cancel_invariant.status = fail:residual_orders to engine_state.
- if confirmed clean, write shutdown_order_cancel_invariant.status = ok.
- Why: without this, residual orders can fill on-ledger after the engine is dead, producing drift invisible to the engine's DB. This is the primary candidate explanation for the ~35 XRP out-of-capture drain.
- Plus: on next session start, read on-chain wallet BEFORE initializing inventory and compare against engine_state.shutdown_final_xrp / _rlusd. If delta > 2 XRP, halt with reason startup_on_chain_reconciliation_fail.
Secondary findings / anomalies¶
A1. Post-injection baseline mismatch. Atlas cited 66.82 XRP post-injection; S33 first session start = 76.92 XRP. +10 XRP gap unexplained. Either the 66.82 figure was measured at a different moment, or there is a capital event / manual move between Apr 18 00:19Z (injection) and 00:48Z (S33 start) not captured in capital_events. Suggest Vesper verify on-ledger history for this window.
A2. inventory_ledger baseline lag. The inventory_ledger running balance for XRP is 35.21 below the sessions table (= Apr 18 deposit amount). The ledger tracks deltas correctly but never credited the deposit. Not a decision-path issue (engine uses valuation_snapshots / get_snapshot, which is correct) but IS a reconciliation bug. Suggest filing as fix/inventory-ledger-credit-capital-deposits.
A3. S39 halt_reason is a ghost. Branch #1 fix/halt-reason-lifecycle was supposed to clear this; the ghost still appeared in S39. Either Branch #1 did not cover this code path, or the reset is triggered by a different event. Need to re-audit before Phase 7.3. Suggest Vesper check whether Branch #1 actually resets engine.halt_reason on engine_status = RUNNING or only at close_session.
A4. S32→S33 XRP jump of +37.19 XRP. Between S32 end (start=39.73) and S33 start (76.92) there is a +37.19 XRP jump. This corresponds to the Apr 18 00:19 injection (+35.21 XRP), consistent within a few XRP of intra-session fills in S32 (55 buys / 56 sells, net +2.9 XRP). Matches expectations.
A5. Anchor rollup is last-session-only. engine_state.anchor.* keys are overwritten each session, so we cannot attribute Block C findings to S36 or S37 specifically — only to S39. Fix is the anchor-per-tick telemetry branch proposed above.
Recommended Phase 7.3 gate sequencing¶
- Block Phase 7.3 until on-ledger reconciliation is done. The ~35 XRP out-of-capture drain has to be explained before any calibration run. Ask Katja to read current on-chain wallet now (expect: some value between 28 and 73.29 XRP depending on how much of the post-S39 drain has finalized) and diff vs S39
ending_xrp. If delta is consistent with residual-orders-filling hypothesis, that directly validates protection layer #4. - Branch
feat/anchor-error-per-tick-telemetrymust land before Phase 7.3 (or as part of Phase 7.3 setup). Without it, per-session anchor attribution is impossible and Phase 7.3 calibration results cannot be interpreted. - Branches 1+2 (distance-to-touch + inventory-invariant-at-shutdown) can still be applied — they are audit-plan deliverables that don't touch the drift mechanism. But they should not trigger an immediate S40-equivalent re-run. The re-run should come after the three protection guards land.
- Protection layers 1–3 as three separate branches, reviewed in sequence. Layer 4 (session-close order cancellation invariant + startup reconciliation) as a fourth branch.
- Phase 7.3 re-runs under the protection layers — if any guard fires during a calibration run, that run is discarded from the evaluation.
Data provenance¶
Read-only investigation against neo_live_stage1.db via SQLite URI mode. No writes. Full output at /tmp/investigation_output.txt. Script: 02 Projects/NEO Trading Engine/scripts/wallet_drift_investigation_v2.py. Schema probe + V1 script also in scripts/.
— Orion