# Incident Response Runbook

Version 0.1, 20.10.2026. Owner: Amir Latypov, founder. Every role below is held by Amir until a second person joins.

## 1. Roles

| Role | Who | Decides | Writes |
|---|---|---|---|
| Incident lead | Amir | All decisions, including stopping services and public statements | Everything |
| First public sentence | Amir | Approves wording | Status post |
| Sign-off | Amir | Closes the incident | Post-incident note |
| Legal | Counsel, when engaged | Notification duties (GDPR 72 hours where applicable) | Notifications |

Contact points: hello@predge.io (inbound reports), X @PredgeAI (public updates), predge.io/trust (status section, not yet live).

## 2. First hour

Times are from the moment an incident is first noticed (T0).

| When | Action |
|---|---|
| T0 | Note the time, the symptom and the source in a text file kept locally. |
| T0 + 15 min | Decide severity (P1, P2 or P3, per SLA.md). |
| T0 + 30 min | Contain. For P1 signing faults: stop signing (section 3). For P1 API outage: check Railway deploy status and last change. |
| T0 + 60 min | Post the first public sentence if P1 or customer-visible. Template below. |

First public sentence template:

> We are investigating [symptom] affecting [scope] since [time UTC]. Paid calls that did not return a 2xx are not charged. Next update by [time UTC] at predge.io/trust.

Do not state a cause until it is confirmed. Do not say "compliant", "secure" or "bond-backed".

## 3. Signing key compromise

Facts from code (20.10.2026):
- If `ATTEST_SIGNING_KEY` is unset, signing routes return 503 and never sign with a fallback key.
- The key registry emits only the current signing key and the cachet-oracle key. It has no revocation or status field.
- `GET /verify` returns `valid: true` for any signature that checks against the supplied key, and only reports `key_published` separately.

Procedure:

1. Stop signing. Unset `ATTEST_SIGNING_KEY` on Railway and redeploy. Signing routes now return 503. Record the time.
2. Set the time boundary. Use the earliest time you have evidence of compromise (T_c). Anything signed by the old kid after T_c must be treated as untrusted unless it is in our database.
3. Generate a new key on a clean machine. Store a sealed offline copy. Set the new `ATTEST_SIGNING_KEY` and redeploy.
4. Publish the revocation notice on predge.io/trust and on X. It must contain: the revoked kid, T_c, the new kid, and the statement that signatures by the old kid after T_c are not endorsed.
5. List affected records. Query the verdict and signed-call tables for records under the old public key with `created_at` at or after T_c. Export the list with ids. Note that a forged record that never reached our database will not appear in this list.
6. Tell verifiers how to check existence. Records in our database can be confirmed by `GET /v1/verdicts/:id`. Forgeries will fail that check.
7. Notify affected customers by email within 4 hours of confirmation, and by the public notice.
8. Assess whether personal data was exposed. If it was, decide with counsel whether GDPR notification applies (72 hours from awareness).

Known limits, to be fixed (see TABLETOP-2026-10-20-signing-key-leak.md):
- Verifiers cannot tell from the registry that a key was revoked, so step 4 depends on people reading the notice.
- The rotation text promises that prior keys stay listed as inactive. The code does not do this yet.
- Signing time is self-asserted inside the signed payload, so it cannot bound the compromise window cryptographically.

## 4. Wrong verdict

1. Confirm the verdict is wrong by reproducing the checks on the stored evidence (hashes and status) and the job spec.
2. Stop issuing that class of verdict if the cause is systemic (route flag or feature switch).
3. Notify the affected merchant and buyer by email with the verdict id, the reason, and the corrected outcome.
4. Publish a notice on predge.io/trust that names the verdict id range.

Limit: the verdict table has no update grant for the API role, and no correction record type exists yet. Corrections are published as notices, not as signed records, until that is built (target 06.11.2026, see TRUST-PACK.md G5).

## 5. API outage

1. Check the health endpoint `GET /health` and the Railway deploy status.
2. Roll back the last deploy if it is the likely cause.
3. Post a status update within 4 hours (P1). Give next-update time.
4. After recovery, post a closing note with the duration and the cause.
5. Check the database backups are intact.

Paid calls that did not return 2xx are not charged (x402 settles on success only).

## 6. Post-incident

Within 5 working days: a short written review with timeline, cause, impact, fixes and owners, published on predge.io/trust. Use the same date-stamped format as the tabletop.

## 7. Contacts and tools

- Email: hello@predge.io
- X: @PredgeAI
- Hosting: Railway (API, database), Vercel (web)
- Keys: Railway environment (signing key), local deployer file (contracts)