Orion Delivery — `feat/reconciler-disappeared-order-audit-log`¶

To: Vesper From: Orion CC: Atlas, Katja (Captain) Date: 2026-04-19 Re: Pure observability for phantom-fill events — pre-FLAG-037 data gathering Branch: feat/reconciler-disappeared-order-audit-log Commits: 4 Tests: 11 new (+ 33 pre-existing reconciler-adjacent tests still green → 44 total)

What this branch does¶

Zero behavior change. Every time LedgerReconciler._handle_disappeared_active_order() takes the phantom-fill branch (order gone from ledger, no cancel_tx_hash set), we now:

Emit a structured [RECONCILER_ANOMALY] WARNING log with order + market context.
Persist one row to a new reconciler_anomaly_log table via state.record_reconciler_anomaly(action_taken="phantom_fill_applied").

Then _apply_full_fill runs exactly as before. The audit write happens BEFORE the fill apply so that if the fill apply fails and escalates to HALT via result.fill_errors, the observation is still preserved.

The dashboard's System Health strip gains an optional RECON_ANOM: n chip that only surfaces when the count is non-zero — healthy sessions show the old strip unchanged. Coloring is warn (1–2) / bad (3+).

This lets us measure shape, frequency, and market context of phantom-fill applications before changing the conservative policy in FLAG-037. Atlas's "observe → understand → then fix" sequence.

Commits¶

9365d71 test(reconciler): reconciler_anomaly_log schema + wiring + cancel-race + audit-failure (11 tests)
9c7243e feat(dashboard): reconciler anomaly count chip on System Health strip
f155973 feat(reconciler): RECONCILER_ANOMALY log + audit row on disappeared active order (pre-FLAG-037)
bf87869 feat(db): reconciler_anomaly_log table + recorder (observability for FLAG-037)

Diffstat:

 dashboard.py                         |  52 ++++-
 neo_engine/ledger_reconciler.py      | 100 ++++++++++
 neo_engine/state_manager.py          | 112 +++++++++++
 tests/test_reconciler_anomaly_log.py | 368 +++++++++++++++++++++++++++++++++++
 4 files changed, 629 insertions(+), 3 deletions(-)

Bundle path: 02 Projects/NEO Trading Engine/patches/feat-reconciler-disappeared-order-audit-log/ (4 patches).

Schema¶

Additive migration in StateManager.initialize_database():

CREATE TABLE IF NOT EXISTS reconciler_anomaly_log (
    id                INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id        INTEGER REFERENCES sessions (session_id),
    detected_at       TEXT    NOT NULL,
    order_id          TEXT    NOT NULL,
    side              TEXT    NOT NULL,
    quantity_rlusd    REAL,
    xrp_equivalent    REAL,
    age_seconds       INTEGER,
    last_status       TEXT,
    offer_sequence    INTEGER,
    cancel_tx_hash    TEXT,
    action_taken      TEXT    NOT NULL,
    notes             TEXT
);

CREATE INDEX IF NOT EXISTS idx_reconciler_anomaly_session
    ON reconciler_anomaly_log (session_id);
CREATE INDEX IF NOT EXISTS idx_reconciler_anomaly_action
    ON reconciler_anomaly_log (action_taken);

action_taken documents what the reconciler DID on the event. Today the only value is 'phantom_fill_applied'. Once FLAG-037 lands, values like 'skipped_no_tx_confirmation' or 'deferred_pending_retry' become possible and we can compare populations without a schema change.

Code shape¶

`neo_engine/state_manager.py`¶

Schema migration added between inventory_truth_snapshots and sessions blocks.
record_reconciler_anomaly(*, order_id, side, quantity_rlusd, xrp_equivalent, age_seconds, last_status, offer_sequence, cancel_tx_hash, action_taken, notes=None) — writer; auto-tags session_id from self._current_session_id.
count_reconciler_anomalies(*, session_id=None) — returns session-scoped count (used by dashboard chip loader).

`neo_engine/ledger_reconciler.py`¶

_age_seconds_since(iso) — best-effort helper; returns None on None / malformed input.
_record_disappeared_order_anomaly(order, *, state, action_taken, notes=None) — computes quantity_rlusd = order.quantity and xrp_equivalent = quantity_rlusd / mid_price (NULL when mid unavailable), computes age_seconds from order.created_at, emits the WARNING log, persists the row. Any exception during the row write is caught and logged at ERROR.
_handle_disappeared_active_order: the phantom-fill branch (lines ~675+) now calls _record_disappeared_order_anomaly before _log_fill_detected → _apply_full_fill. The cancel-race short-circuit at the top is untouched.

`dashboard.py`¶

load_reconciler_anomaly_count(conn, *, session_id, sessions_tracked) — returns int, handles legacy DBs / missing session gracefully.
_render_system_health(...) takes a new reconciler_anomaly_count=0 kwarg; appends a RECON_ANOM: n cell to the strip only when n > 0.
Main render flow threads the loader through once and passes the value into _render_system_health.

Tests (11 new, all passing)¶

test_schema_creates_reconciler_anomaly_table                                   PASSED
test_schema_has_expected_columns                                               PASSED
test_schema_has_expected_indexes                                               PASSED
test_record_reconciler_anomaly_round_trip                                      PASSED
test_count_reconciler_anomalies                                                PASSED
test_record_disappeared_order_anomaly_computes_xrp_equivalent                  PASSED
test_record_disappeared_order_anomaly_null_mid_keeps_xrp_null                  PASSED
test_age_seconds_since_handles_bad_input                                       PASSED
test_disappeared_active_order_writes_anomaly_and_still_applies_fill            PASSED
test_disappeared_active_order_cancel_race_path_writes_no_row                   PASSED
test_anomaly_write_failure_does_not_block_reconciler                           PASSED

Full run: python -m pytest tests/test_reconciler_anomaly_log.py -v → 11 passed in ~1s.

Regression check: python -m pytest tests/test_ledger_reconciler.py tests/test_flag_036_wallet_truth_reconciliation.py tests/test_reconciler_anomaly_log.py -q → 44 passed in 1.35s.

Key invariants asserted:

Schema migration creates the table, all 13 columns with correct types, both indexes.
Phantom-fill path writes exactly one row, emits WARNING with [RECONCILER_ANOMALY] prefix, AND still calls _apply_full_fill (result.full_fills == 1).
Cancel-race path (order has cancel_tx_hash) writes zero rows — that event class uses result.cancel_races and is deliberately not re-instrumented here.
Best-effort audit — a raise in record_reconciler_anomaly is caught, logged at ERROR, and the phantom-fill application still completes. The reconciler's correctness path is unchanged.
Market context — xrp_equivalent = order.quantity / mid_price when mid available; NULL when market state is unset.

Apply instructions (Windows)¶

From C:\Users\Katja\Documents\NEO GitHub\neo-2026\, assuming a clean main:

git checkout -b feat/reconciler-disappeared-order-audit-log main
git am "C:\Users\Katja\Documents\Claude Homebase Neo\02 Projects\NEO Trading Engine\patches\feat-reconciler-disappeared-order-audit-log\*.patch"
python -m pytest tests/test_reconciler_anomaly_log.py -v

Expected: 11 passed in ~1s.

Full reconciler-adjacent regression check (optional):

python -m pytest tests/test_ledger_reconciler.py tests/test_flag_036_wallet_truth_reconciliation.py tests/test_reconciler_anomaly_log.py -q

Expected: 44 passed.

Operator impact¶

Healthy sessions: dashboard strip unchanged, no new log noise.
First phantom-fill event of a session: [RECONCILER_ANOMALY] WARNING in the log; RECON_ANOM: 1 chip (warn color) appears on the dashboard strip.
≥3 phantom-fills in a session: chip switches to bad color. No behavioral escalation from the chip itself — this is observability only.
Data gathering: after a few live sessions, query reconciler_anomaly_log grouped by session_id to see per-session frequency, and join on detected_at vs. market_snapshots to understand the market conditions that produce these events.

The FLAG-037 fix (conservative vs. aggressive phantom-fill policy) becomes easier to scope once we have this population.

Status¶

Branch ready for review. No open questions this round — spec was unambiguous, ruled-out scope (cancel-race path, _apply_full_fill semantics) was explicit.

— Orion 2026-04-19

Orion Delivery — feat/reconciler-disappeared-order-audit-log¶