feat: add Quant OS A-share baseline
This commit is contained in:
@@ -0,0 +1,209 @@
|
||||
# Incident and safe-mode runbook
|
||||
|
||||
This runbook describes operator actions. The current scaffold does not implement a production supervisor, automatic broker kill switch or durable reconciliation service. Do not mistake these procedures for automated guarantees.
|
||||
|
||||
## Universal first response
|
||||
|
||||
For any unexplained data, order, cash or position state:
|
||||
|
||||
1. stop generating and submitting **new opening orders**;
|
||||
2. do not blindly cancel or resubmit—first query the broker for current orders and fills;
|
||||
3. capture local time, broker/source time, platform version, run/config/data identifiers and the last known ledger hash;
|
||||
4. export a fresh read-only broker snapshot;
|
||||
5. preserve logs and raw callbacks locally with credentials/account identifiers protected;
|
||||
6. classify whether existing risk-reduction orders may continue; if uncertain, require human approval;
|
||||
7. recover only after the exit condition in this runbook is met.
|
||||
|
||||
“Restart until it works” is not a recovery method. A restart can duplicate orders when submission succeeded but the acknowledgement was lost.
|
||||
|
||||
## Severity
|
||||
|
||||
| Severity | Examples | Posture |
|
||||
| --- | --- | --- |
|
||||
| S0 | Credential exposure, unexplained live position/cash, suspected duplicate live order, wrong account | Stop all strategy submissions; broker/operator intervention |
|
||||
| S1 | Broker disconnect during active orders, unknown order state, reconciliation difference, stale live data | Block new orders; query and reconcile |
|
||||
| S2 | Platform backtest failure, Qlib dependency/model failure, synthetic test regression | No production release; live system unaffected if isolated |
|
||||
| S3 | Documentation, optional report or non-critical metric failure | Record and repair before next release |
|
||||
|
||||
## Data stale, missing or inconsistent
|
||||
|
||||
Symptoms:
|
||||
|
||||
- latest bar/session absent;
|
||||
- provider and canonical hashes unexpectedly differ;
|
||||
- timestamps are in the future or timezone is unknown;
|
||||
- suspended sessions were forward-filled without an explicit rule.
|
||||
|
||||
Actions:
|
||||
|
||||
1. block target publication;
|
||||
2. identify the last complete immutable data version;
|
||||
3. compare provider fetch time, event time and available time;
|
||||
4. rerun quality checks without mutating the prior snapshot;
|
||||
5. if using an older snapshot for risk reduction, mark the decision `STALE_DATA_RISK_REDUCTION_ONLY`.
|
||||
|
||||
Exit: a new immutable snapshot passes quality checks and the same input replays the same signal/target. Never patch a historical snapshot in place.
|
||||
|
||||
## QMT/MiniQMT disconnect
|
||||
|
||||
Symptoms:
|
||||
|
||||
- native API import works but query calls fail;
|
||||
- callbacks stop;
|
||||
- QMT terminal logs out or session reconnects;
|
||||
- order request timed out without an acknowledgement.
|
||||
|
||||
Actions:
|
||||
|
||||
1. block new submissions;
|
||||
2. keep the client/terminal state and logs; do not immediately allocate a new session ID and resend;
|
||||
3. restore terminal login/connectivity;
|
||||
4. query assets, positions, orders and fills for the full affected window;
|
||||
5. match by local idempotency key, broker order ID, symbol/side/quantity/time;
|
||||
6. classify every pending local order as broker-known terminal/open or `UNKNOWN`;
|
||||
7. escalate any `UNKNOWN` to a human/broker before resubmission.
|
||||
|
||||
Exit: no unknown order, position/cash reconcile, callback stream is fresh, and a read-only preflight passes.
|
||||
|
||||
## Lost acknowledgement or possible duplicate order
|
||||
|
||||
Actions:
|
||||
|
||||
1. never assume a timeout means rejection;
|
||||
2. query broker orders and fills first;
|
||||
3. search using client strategy/order labels plus symbol, side, quantity and time;
|
||||
4. if one broker order matches, bind its broker ID and continue from broker state;
|
||||
5. if multiple match, stop the affected symbol/account and escalate;
|
||||
6. resubmit only after proving no live/open/filled broker order represents the intent.
|
||||
|
||||
Exit: a one-to-one mapping exists or the operator explicitly retires the intent.
|
||||
|
||||
## Callback duplicate, out of order or unknown state
|
||||
|
||||
Actions:
|
||||
|
||||
1. preserve the raw callback and its receive/source timestamps;
|
||||
2. deduplicate by broker event identity or a documented content hash;
|
||||
3. rebuild state from broker query plus the complete event stream;
|
||||
4. do not force a terminal local order backwards to an earlier state;
|
||||
5. map unmapped broker codes to `UNKNOWN`, not the nearest convenient status.
|
||||
|
||||
Exit: normalized event history is monotonic/idempotent and equals the broker's current query.
|
||||
|
||||
## Reconciliation difference
|
||||
|
||||
Compare four sets independently:
|
||||
|
||||
```text
|
||||
cash
|
||||
positions and sellable quantities
|
||||
orders
|
||||
fills and fees
|
||||
```
|
||||
|
||||
Actions:
|
||||
|
||||
1. block new opening orders;
|
||||
2. freeze the local ledger and snapshot it;
|
||||
3. determine whether the difference is timing, fee, corporate action, manual trade, callback loss or wrong account;
|
||||
4. append an explicit correction/reconciliation event; never edit prior ledger lines;
|
||||
5. require human review for manual trades, unexplained fees or position differences.
|
||||
|
||||
Exit: zero unexplained difference and a signed reconciliation report. A balancing “miscellaneous P&L” line is not an explanation.
|
||||
|
||||
## Price limit, suspension or zero liquidity
|
||||
|
||||
Actions:
|
||||
|
||||
1. treat the position as held/frozen; never mark it sold because the target is zero;
|
||||
2. stop repeated cancel/resubmit loops;
|
||||
3. retain exposure in risk and cash calculations;
|
||||
4. schedule a new decision only when a valid quote and broker sellable quantity exist;
|
||||
5. record opportunity cost separately from explicit transaction cost.
|
||||
|
||||
Exit: valid tradability returns or the target is deliberately withdrawn.
|
||||
|
||||
## Insufficient cash or T+1 rejection
|
||||
|
||||
Actions:
|
||||
|
||||
1. refresh broker cash, positions, sellable quantity, open orders and fills;
|
||||
2. include reserved cash/open buy orders and minimum commission;
|
||||
3. do not sell today's purchase to fund another order;
|
||||
4. reduce/cancel unsubmitted buys before modifying risk-reduction sells;
|
||||
5. record the target-to-order shortfall reason.
|
||||
|
||||
Exit: pre-trade calculation based on the fresh broker snapshot passes.
|
||||
|
||||
## Ledger integrity or disk failure
|
||||
|
||||
Symptoms:
|
||||
|
||||
- hash-chain verification fails;
|
||||
- JSONL is truncated;
|
||||
- disk is full;
|
||||
- atomic replacement did not complete.
|
||||
|
||||
Actions:
|
||||
|
||||
1. stop submission;
|
||||
2. preserve the corrupted file and filesystem diagnostics;
|
||||
3. verify the last known good immutable copy;
|
||||
4. query broker truth for the entire uncertain interval;
|
||||
5. reconstruct a new ledger by replay, retaining a link to the incident and old head hash;
|
||||
6. never delete or hand-edit the corrupt original.
|
||||
|
||||
Exit: reconstructed ledger verifies and reconciles to broker truth; storage monitoring and free space are restored.
|
||||
|
||||
## Clock or timezone drift
|
||||
|
||||
Actions:
|
||||
|
||||
1. block time-sensitive target/order publication;
|
||||
2. record OS, platform and broker times;
|
||||
3. restore trusted time synchronization;
|
||||
4. recompute `as_of` and executable-window eligibility;
|
||||
5. discard—not rename—any decision that used future or ambiguous data, retaining it as failed evidence.
|
||||
|
||||
Exit: clocks are within the release tolerance and a fresh decision is computed.
|
||||
|
||||
## Model, optimizer or Qlib failure
|
||||
|
||||
Actions:
|
||||
|
||||
1. do not silently substitute a different model or equal weight;
|
||||
2. mark the decision `NO_TRADE` or, when separately approved, `RISK_REDUCTION_ONLY`;
|
||||
3. preserve model/config/data versions and traceback;
|
||||
4. reproduce in the isolated Python 3.12 research environment;
|
||||
5. require the normal champion promotion process before restoring.
|
||||
|
||||
Exit: the approved model/run reproduces and its outputs pass schema/risk gates.
|
||||
|
||||
## Credential exposure
|
||||
|
||||
Treat as S0. Stop order automation, rotate/revoke the credential, inspect Git history/artifacts/logs and notify the provider/broker when appropriate. Removing a file without rotation is insufficient. Follow [`docs/SECURITY.md`](../docs/SECURITY.md).
|
||||
|
||||
## Evidence template
|
||||
|
||||
Save an incident record outside credential-bearing logs:
|
||||
|
||||
```text
|
||||
incident_id:
|
||||
severity:
|
||||
opened_at / closed_at:
|
||||
account_hash:
|
||||
engine/mode/version:
|
||||
git_commit:
|
||||
config_hash / data_version / rules_version:
|
||||
last_good_ledger_hash:
|
||||
symptom:
|
||||
orders/fills at risk:
|
||||
containment:
|
||||
broker queries performed:
|
||||
root cause:
|
||||
recovery evidence:
|
||||
operator:
|
||||
follow-up tests:
|
||||
```
|
||||
|
||||
An incident remains open while any affected broker order or position is `UNKNOWN`.
|
||||
Reference in New Issue
Block a user