Game Day Engineering - Exploring Incident Simulators

One of the things I've noticed over the years is that engineers being able to handle production incidents well i.e.

• communicate clearly and effectively with stakeholders
• make decisions quickly and effectively
• stay calm and composed under pressure

is often based on experience with similar situations in the past.

They've seen enough systems fail that when something starts behaving strangely, they instinctively know where to look, what questions to ask, what evidence to gather, and just as importantly, what not to do.

Such Engineers have built up an internal library of failure narratives:

• Database deadlocked.
• DNS expired.
• Certificate rotated incorrectly.
• Queue backed up.
• Cache stampede.
• Race condition.
• Feature flag left on.
• Third-party API changed.

Each incident adds another mental model to the Engineer's toolkit.

The problem is that production incidents are an incredibly expensive way to gain experience. Customers are affected, engineers are stressed, people are pulled out of bed at 3 a.m., and everyone would rather the incident had never happened.

That made me wonder a few days ago:

Can a program be designed to help an IC or an engineering team become better at handling failure scenarios? E.g. a slackbot that presents these situations once a day, perhaps based on a handbook, for handling critical scenarios?

Learning from stories

Other professions already do this.

• Pilots spend countless hours in flight simulators before flying real passengers.
• Doctors discuss clinical cases before encountering them in practice.
• Military organizations run war games.
• Security teams run red-team exercises.
• Software engineering has tabletop exercises, where teams sit together and discuss how they would respond to a hypothetical incident.

These are incredibly valuable because stories are compressed experience. They allow us to borrow lessons from situations we haven't personally lived through.

But I couldn't help wondering if we could take that idea further. Instead of discussing an incident in a tabletop exercise, what if we could simulate one?

I think the answer is yes, and that's where incident simulators come in.

What are incident simulators?

Imagine a game in the style of Dungeons and Dragons where a Game Master presents a failure scenario to the players, who then work to resolve the situation. The players must use the tools available to them to observe, investigate, act, verify, and get scored on judgment - not just on whether the incident got resolved, but on how it got resolved.

The incident simulator would need to:

• Understand the team's architecture i.e. maintain a graph of all services and their inter-dependencies.
• Know about postmortems for past failures so it can be better informed about what could go wrong.
• Know about the team's incident response plan in case of a failure so it can know what tools to give the players.
• Keep track of the player's actions and the outcome of each action so it can provide feedback and scores to the players at the end of the game.

It generates a realistic production incident and your job is to restore production.

The simulator doesn't tell you what's broken even though it knows what's broken. You have to figure it out yourself.

You investigate logs, inspect dependencies, compare deployments, check metrics, communicate with stakeholders, and decide whether to restart, roll back, scale, or simply wait for more evidence.

Every action you take advances the simulated world.

The incident evolves regardless of whether you're looking in the right place or not.

Grounded in your own architecture

One of the biggest limitations of generic incident exercises is that they're generic. They don't understand the team's architecture, their runbooks, their previous postmortems, and their incident response process.

Imagine providing it with:

• An architectural graph generated from Infrastructure as Code.
• Service dependencies.
• Runbooks.
• Historical postmortems.
• Incident response procedures.
• Team-specific operational conventions.

A simple example

☕ Engineering Scenario #142

Simulated Time: Monday, July 6, 2026 - 11:37 AM.
Severity: SEV-2 (data loss - core workflows degraded)

CPU on one prod-user-service node jumps to 100%.
5 customers have reported issues creating new users.
Latency is increasing.
Error logs are showing a lot of 500 errors on requests to the user service.

Tools:
- `logs <service-name>` to see the error logs
- `metrics <service-name>` to see the metrics
- `deployments <service-name>` to see the recent deployments
- `communicate <message>` to inform stakeholders of the issue
- `inspect <service-name>` to see the service's configuration, dependencies, dependents, etc.
- `query <database-name>` to run read-only queries
- `rollback <service-name>` to rollback the service to a specified version
- `wait <duration>` to wait for a specified duration and advance the game clock
- `document <message>` to document the issue, action taken and the outcome e.g. a postmortem.
- `resolve` to declare the incident resolved, end the game, and get feedback
- `help` to show the available commands

The players can then respond with their actions based on the tools available to them e.g.

logs prod-user-service

to which the Game Master may respond with a message like:

Error logs count (45m):
- Errors on requests to `POST /api/users`: 32

The player, noticing a spike right around the reported time, decides to check what changed recently:

deployments prod-user-service

Recent deployments for prod-user-service:
- v2.14.1 deployed 11:22 AM by @jane (feature: add email verification step)
- v2.14.0 deployed yesterday 4:03 PM by @deploy-bot (routine dependency bump)

That deployment at 11:22 AM lines up suspiciously close to when the errors started. The player digs a little deeper before jumping to conclusions:

inspect prod-user-service

prod-user-service
Dependencies: postgres-primary, email-verification-service, redis-cache
Dependents: signup-frontend, admin-console

Note: email-verification-service has no configured timeout on outbound calls from prod-user-service.

Now the picture is clearer: the new email verification step, shipped in v2.14.1, calls out to a service with no timeout, and under load that's tying up threads and spiking CPU. Rather than wait and hope, the player chooses to act:

communicate Investigating a spike in 500 errors on user creation, likely tied to the v2.14.1 deploy at 11:22 AM. Rolling back as a precaution.

Important: It shows that the player is able to communicate with stakeholders to inform them of the issue and the action being taken. It will reflect in the player's score on communication skills.

rollback prod-user-service

Rolling back prod-user-service to v2.14.0... done.

wait 5m

5 minutes pass. CPU on the affected node has dropped back to 40%.

logs prod-user-service

Error logs count (45m):
- Errors on requests to `POST /api/users`: 2 (down from 32)

Error rate on POST /api/users is back to baseline.

Satisfied that the rollback resolved the issue, the player wraps up:

communicate User creation is back to normal.

resolve

Incident resolved in 8 simulated minutes.

Feedback:
+ Correctly correlated the error spike with a recent deployment instead of guessing
+ Communicated to stakeholders before taking a mitigating action
+ Verified the fix worked before declaring resolution
- Root cause (missing timeout on email-verification-service) wasn't logged for a follow-up postmortem

Analysis: 90%
Communication: 95%
Decision Making: 85%
Documentation: 60%

Total Score: 82.5/100

This is the loop: observe, investigate, act, verify, and get scored on judgment - not just on whether the incident got resolved, but on how it got resolved.

This is a very simple and generic example, but with some real context about the team's architecture and incident response plan, the incident simulator could be made to be more specific to the team's environment.

Where could this be used?

• As a daily practice for engineers to improve their skills.
• As an onboarding tool for new engineers to learn the ropes.
• As a tool for managers to assess the readiness of their teams.
• As a tool for executives to understand the risk of their systems.
• As a tool for security teams to test the resilience of their systems.

Conclusion

The more I think about this, it's clearer that the real product is not the incident simulator itself, but quickly turning operational knowledge into experience without the pain of a production incident.

Done right, real incidents will still happen, but they will be less surprising and your Engineers will be better prepared to handle them.

I'm curious whether anyone has built something similar. Let me know in the comments below.