quant-os/runbooks/INCIDENTS.md

Incident and safe-mode runbook

This runbook describes operator actions. The current scaffold does not implement a production supervisor, automatic broker kill switch or durable reconciliation service. Do not mistake these procedures for automated guarantees.

Universal first response

For any unexplained data, order, cash or position state:

1. stop generating and submitting new opening orders;
2. do not blindly cancel or resubmit—first query the broker for current orders and fills;
3. capture local time, broker/source time, platform version, run/config/data identifiers and the last known ledger hash;
4. export a fresh read-only broker snapshot;
5. preserve logs and raw callbacks locally with credentials/account identifiers protected;
6. classify whether existing risk-reduction orders may continue; if uncertain, require human approval;
7. recover only after the exit condition in this runbook is met.

“Restart until it works” is not a recovery method. A restart can duplicate orders when submission succeeded but the acknowledgement was lost.

Severity

Severity	Examples	Posture
S0	Credential exposure, unexplained live position/cash, suspected duplicate live order, wrong account	Stop all strategy submissions; broker/operator intervention
S1	Broker disconnect during active orders, unknown order state, reconciliation difference, stale live data	Block new orders; query and reconcile
S2	Platform backtest failure, Qlib dependency/model failure, synthetic test regression	No production release; live system unaffected if isolated
S3	Documentation, optional report or non-critical metric failure	Record and repair before next release

Data stale, missing or inconsistent

Symptoms:

• latest bar/session absent;
• provider and canonical hashes unexpectedly differ;
• timestamps are in the future or timezone is unknown;
• suspended sessions were forward-filled without an explicit rule.

Actions:

1. block target publication;
2. identify the last complete immutable data version;
3. compare provider fetch time, event time and available time;
4. rerun quality checks without mutating the prior snapshot;
5. if using an older snapshot for risk reduction, mark the decision STALE_DATA_RISK_REDUCTION_ONLY.

Exit: a new immutable snapshot passes quality checks and the same input replays the same signal/target. Never patch a historical snapshot in place.

QMT/MiniQMT disconnect

Symptoms:

• native API import works but query calls fail;
• callbacks stop;
• QMT terminal logs out or session reconnects;
• order request timed out without an acknowledgement.

Actions:

1. block new submissions;
2. keep the client/terminal state and logs; do not immediately allocate a new session ID and resend;
3. restore terminal login/connectivity;
4. query assets, positions, orders and fills for the full affected window;
5. match by local idempotency key, broker order ID, symbol/side/quantity/time;
6. classify every pending local order as broker-known terminal/open or UNKNOWN;
7. escalate any UNKNOWN to a human/broker before resubmission.

Exit: no unknown order, position/cash reconcile, callback stream is fresh, and a read-only preflight passes.

Lost acknowledgement or possible duplicate order

Actions:

1. never assume a timeout means rejection;
2. query broker orders and fills first;
3. search using client strategy/order labels plus symbol, side, quantity and time;
4. if one broker order matches, bind its broker ID and continue from broker state;
5. if multiple match, stop the affected symbol/account and escalate;
6. resubmit only after proving no live/open/filled broker order represents the intent.

Exit: a one-to-one mapping exists or the operator explicitly retires the intent.

Callback duplicate, out of order or unknown state

Actions:

1. preserve the raw callback and its receive/source timestamps;
2. deduplicate by broker event identity or a documented content hash;
3. rebuild state from broker query plus the complete event stream;
4. do not force a terminal local order backwards to an earlier state;
5. map unmapped broker codes to UNKNOWN, not the nearest convenient status.

Exit: normalized event history is monotonic/idempotent and equals the broker's current query.

Reconciliation difference

Compare four sets independently:

cash
positions and sellable quantities
orders
fills and fees

Actions:

1. block new opening orders;
2. freeze the local ledger and snapshot it;
3. determine whether the difference is timing, fee, corporate action, manual trade, callback loss or wrong account;
4. append an explicit correction/reconciliation event; never edit prior ledger lines;
5. require human review for manual trades, unexplained fees or position differences.

Exit: zero unexplained difference and a signed reconciliation report. A balancing “miscellaneous P&L” line is not an explanation.

Price limit, suspension or zero liquidity

Actions:

1. treat the position as held/frozen; never mark it sold because the target is zero;
2. stop repeated cancel/resubmit loops;
3. retain exposure in risk and cash calculations;
4. schedule a new decision only when a valid quote and broker sellable quantity exist;
5. record opportunity cost separately from explicit transaction cost.

Exit: valid tradability returns or the target is deliberately withdrawn.

Insufficient cash or T+1 rejection

Actions:

1. refresh broker cash, positions, sellable quantity, open orders and fills;
2. include reserved cash/open buy orders and minimum commission;
3. do not sell today's purchase to fund another order;
4. reduce/cancel unsubmitted buys before modifying risk-reduction sells;
5. record the target-to-order shortfall reason.

Exit: pre-trade calculation based on the fresh broker snapshot passes.

Ledger integrity or disk failure

Symptoms:

• hash-chain verification fails;
• JSONL is truncated;
• disk is full;
• atomic replacement did not complete.

Actions:

1. stop submission;
2. preserve the corrupted file and filesystem diagnostics;
3. verify the last known good immutable copy;
4. query broker truth for the entire uncertain interval;
5. reconstruct a new ledger by replay, retaining a link to the incident and old head hash;
6. never delete or hand-edit the corrupt original.

Exit: reconstructed ledger verifies and reconciles to broker truth; storage monitoring and free space are restored.

Clock or timezone drift

Actions:

1. block time-sensitive target/order publication;
2. record OS, platform and broker times;
3. restore trusted time synchronization;
4. recompute as_of and executable-window eligibility;
5. discard—not rename—any decision that used future or ambiguous data, retaining it as failed evidence.

Exit: clocks are within the release tolerance and a fresh decision is computed.

Model, optimizer or Qlib failure

Actions:

1. do not silently substitute a different model or equal weight;
2. mark the decision NO_TRADE or, when separately approved, RISK_REDUCTION_ONLY;
3. preserve model/config/data versions and traceback;
4. reproduce in the isolated Python 3.12 research environment;
5. require the normal champion promotion process before restoring.

Exit: the approved model/run reproduces and its outputs pass schema/risk gates.

Credential exposure

Treat as S0. Stop order automation, rotate/revoke the credential, inspect Git history/artifacts/logs and notify the provider/broker when appropriate. Removing a file without rotation is insufficient. Follow docs/SECURITY.md.

Evidence template

Save an incident record outside credential-bearing logs:

incident_id:
severity:
opened_at / closed_at:
account_hash:
engine/mode/version:
git_commit:
config_hash / data_version / rules_version:
last_good_ledger_hash:
symptom:
orders/fills at risk:
containment:
broker queries performed:
root cause:
recovery evidence:
operator:
follow-up tests:

An incident remains open while any affected broker order or position is UNKNOWN.