210 lines
7.9 KiB
Markdown
210 lines
7.9 KiB
Markdown
# Incident and safe-mode runbook
|
|
|
|
This runbook describes operator actions. The current scaffold does not implement a production supervisor, automatic broker kill switch or durable reconciliation service. Do not mistake these procedures for automated guarantees.
|
|
|
|
## Universal first response
|
|
|
|
For any unexplained data, order, cash or position state:
|
|
|
|
1. stop generating and submitting **new opening orders**;
|
|
2. do not blindly cancel or resubmit—first query the broker for current orders and fills;
|
|
3. capture local time, broker/source time, platform version, run/config/data identifiers and the last known ledger hash;
|
|
4. export a fresh read-only broker snapshot;
|
|
5. preserve logs and raw callbacks locally with credentials/account identifiers protected;
|
|
6. classify whether existing risk-reduction orders may continue; if uncertain, require human approval;
|
|
7. recover only after the exit condition in this runbook is met.
|
|
|
|
“Restart until it works” is not a recovery method. A restart can duplicate orders when submission succeeded but the acknowledgement was lost.
|
|
|
|
## Severity
|
|
|
|
| Severity | Examples | Posture |
|
|
| --- | --- | --- |
|
|
| S0 | Credential exposure, unexplained live position/cash, suspected duplicate live order, wrong account | Stop all strategy submissions; broker/operator intervention |
|
|
| S1 | Broker disconnect during active orders, unknown order state, reconciliation difference, stale live data | Block new orders; query and reconcile |
|
|
| S2 | Platform backtest failure, Qlib dependency/model failure, synthetic test regression | No production release; live system unaffected if isolated |
|
|
| S3 | Documentation, optional report or non-critical metric failure | Record and repair before next release |
|
|
|
|
## Data stale, missing or inconsistent
|
|
|
|
Symptoms:
|
|
|
|
- latest bar/session absent;
|
|
- provider and canonical hashes unexpectedly differ;
|
|
- timestamps are in the future or timezone is unknown;
|
|
- suspended sessions were forward-filled without an explicit rule.
|
|
|
|
Actions:
|
|
|
|
1. block target publication;
|
|
2. identify the last complete immutable data version;
|
|
3. compare provider fetch time, event time and available time;
|
|
4. rerun quality checks without mutating the prior snapshot;
|
|
5. if using an older snapshot for risk reduction, mark the decision `STALE_DATA_RISK_REDUCTION_ONLY`.
|
|
|
|
Exit: a new immutable snapshot passes quality checks and the same input replays the same signal/target. Never patch a historical snapshot in place.
|
|
|
|
## QMT/MiniQMT disconnect
|
|
|
|
Symptoms:
|
|
|
|
- native API import works but query calls fail;
|
|
- callbacks stop;
|
|
- QMT terminal logs out or session reconnects;
|
|
- order request timed out without an acknowledgement.
|
|
|
|
Actions:
|
|
|
|
1. block new submissions;
|
|
2. keep the client/terminal state and logs; do not immediately allocate a new session ID and resend;
|
|
3. restore terminal login/connectivity;
|
|
4. query assets, positions, orders and fills for the full affected window;
|
|
5. match by local idempotency key, broker order ID, symbol/side/quantity/time;
|
|
6. classify every pending local order as broker-known terminal/open or `UNKNOWN`;
|
|
7. escalate any `UNKNOWN` to a human/broker before resubmission.
|
|
|
|
Exit: no unknown order, position/cash reconcile, callback stream is fresh, and a read-only preflight passes.
|
|
|
|
## Lost acknowledgement or possible duplicate order
|
|
|
|
Actions:
|
|
|
|
1. never assume a timeout means rejection;
|
|
2. query broker orders and fills first;
|
|
3. search using client strategy/order labels plus symbol, side, quantity and time;
|
|
4. if one broker order matches, bind its broker ID and continue from broker state;
|
|
5. if multiple match, stop the affected symbol/account and escalate;
|
|
6. resubmit only after proving no live/open/filled broker order represents the intent.
|
|
|
|
Exit: a one-to-one mapping exists or the operator explicitly retires the intent.
|
|
|
|
## Callback duplicate, out of order or unknown state
|
|
|
|
Actions:
|
|
|
|
1. preserve the raw callback and its receive/source timestamps;
|
|
2. deduplicate by broker event identity or a documented content hash;
|
|
3. rebuild state from broker query plus the complete event stream;
|
|
4. do not force a terminal local order backwards to an earlier state;
|
|
5. map unmapped broker codes to `UNKNOWN`, not the nearest convenient status.
|
|
|
|
Exit: normalized event history is monotonic/idempotent and equals the broker's current query.
|
|
|
|
## Reconciliation difference
|
|
|
|
Compare four sets independently:
|
|
|
|
```text
|
|
cash
|
|
positions and sellable quantities
|
|
orders
|
|
fills and fees
|
|
```
|
|
|
|
Actions:
|
|
|
|
1. block new opening orders;
|
|
2. freeze the local ledger and snapshot it;
|
|
3. determine whether the difference is timing, fee, corporate action, manual trade, callback loss or wrong account;
|
|
4. append an explicit correction/reconciliation event; never edit prior ledger lines;
|
|
5. require human review for manual trades, unexplained fees or position differences.
|
|
|
|
Exit: zero unexplained difference and a signed reconciliation report. A balancing “miscellaneous P&L” line is not an explanation.
|
|
|
|
## Price limit, suspension or zero liquidity
|
|
|
|
Actions:
|
|
|
|
1. treat the position as held/frozen; never mark it sold because the target is zero;
|
|
2. stop repeated cancel/resubmit loops;
|
|
3. retain exposure in risk and cash calculations;
|
|
4. schedule a new decision only when a valid quote and broker sellable quantity exist;
|
|
5. record opportunity cost separately from explicit transaction cost.
|
|
|
|
Exit: valid tradability returns or the target is deliberately withdrawn.
|
|
|
|
## Insufficient cash or T+1 rejection
|
|
|
|
Actions:
|
|
|
|
1. refresh broker cash, positions, sellable quantity, open orders and fills;
|
|
2. include reserved cash/open buy orders and minimum commission;
|
|
3. do not sell today's purchase to fund another order;
|
|
4. reduce/cancel unsubmitted buys before modifying risk-reduction sells;
|
|
5. record the target-to-order shortfall reason.
|
|
|
|
Exit: pre-trade calculation based on the fresh broker snapshot passes.
|
|
|
|
## Ledger integrity or disk failure
|
|
|
|
Symptoms:
|
|
|
|
- hash-chain verification fails;
|
|
- JSONL is truncated;
|
|
- disk is full;
|
|
- atomic replacement did not complete.
|
|
|
|
Actions:
|
|
|
|
1. stop submission;
|
|
2. preserve the corrupted file and filesystem diagnostics;
|
|
3. verify the last known good immutable copy;
|
|
4. query broker truth for the entire uncertain interval;
|
|
5. reconstruct a new ledger by replay, retaining a link to the incident and old head hash;
|
|
6. never delete or hand-edit the corrupt original.
|
|
|
|
Exit: reconstructed ledger verifies and reconciles to broker truth; storage monitoring and free space are restored.
|
|
|
|
## Clock or timezone drift
|
|
|
|
Actions:
|
|
|
|
1. block time-sensitive target/order publication;
|
|
2. record OS, platform and broker times;
|
|
3. restore trusted time synchronization;
|
|
4. recompute `as_of` and executable-window eligibility;
|
|
5. discard—not rename—any decision that used future or ambiguous data, retaining it as failed evidence.
|
|
|
|
Exit: clocks are within the release tolerance and a fresh decision is computed.
|
|
|
|
## Model, optimizer or Qlib failure
|
|
|
|
Actions:
|
|
|
|
1. do not silently substitute a different model or equal weight;
|
|
2. mark the decision `NO_TRADE` or, when separately approved, `RISK_REDUCTION_ONLY`;
|
|
3. preserve model/config/data versions and traceback;
|
|
4. reproduce in the isolated Python 3.12 research environment;
|
|
5. require the normal champion promotion process before restoring.
|
|
|
|
Exit: the approved model/run reproduces and its outputs pass schema/risk gates.
|
|
|
|
## Credential exposure
|
|
|
|
Treat as S0. Stop order automation, rotate/revoke the credential, inspect Git history/artifacts/logs and notify the provider/broker when appropriate. Removing a file without rotation is insufficient. Follow [`docs/SECURITY.md`](../docs/SECURITY.md).
|
|
|
|
## Evidence template
|
|
|
|
Save an incident record outside credential-bearing logs:
|
|
|
|
```text
|
|
incident_id:
|
|
severity:
|
|
opened_at / closed_at:
|
|
account_hash:
|
|
engine/mode/version:
|
|
git_commit:
|
|
config_hash / data_version / rules_version:
|
|
last_good_ledger_hash:
|
|
symptom:
|
|
orders/fills at risk:
|
|
containment:
|
|
broker queries performed:
|
|
root cause:
|
|
recovery evidence:
|
|
operator:
|
|
follow-up tests:
|
|
```
|
|
|
|
An incident remains open while any affected broker order or position is `UNKNOWN`.
|