Production Incident Response SOP
Technology | SOP

Use this when: Outages are handled differently each time, and customers hear about them late.

Owner: [on-call lead]

1. Declare the incident as soon as you suspect customer impact. Assign a severity.
2. Name one incident lead and one communications lead.
3. Open one channel for the incident and keep a timeline.
4. Focus on restoring service first, and find the root cause afterward.
5. Post a customer update at [interval], even if there's nothing new.
6. Before any risky fix, have a second person review it and confirm the rollback plan.
7. Confirm recovery using customer-facing checks.
8. Send a final update.
9. Hold a blameless review within [5 days], and assign each action an owner and a date.
10. If personal data may be affected, call [privacy lead] right away.

Before you use this: Privacy laws and customer contracts may set breach-notice duties and deadlines, so check them with counsel.

Fill in the items in square brackets and adapt this to your business.
Free template from SAZ: https://playbooks.saz.ca/p/technology-sop