On-Call Practices and Runbook Design Questions

Allocate people by comparing the two incidents' business impact and required expertise, not by splitting the team evenly, and default to running them as separate incidents with separate commanders unless you find a shared root cause; escalate to a higher-level incident commander as soon as the resource conflict itself (not just the technical severity) becomes the bottleneck.

Allocating people across concurrent incidents

• Score each incident independently first: user-facing impact, revenue impact, data-integrity risk, and blast radius (one team's problem versus platform-wide). Two incidents rarely score identically, and the higher-scored one gets first claim on the strongest responders.
• Check for a shared root cause before splitting resources. If both services depend on the same failing component (a shared database, a shared auth service), that's actually one incident with two symptoms, and it should be run as a single incident with one commander, not two competing efforts pulling on the same underlying fix.
• Staff each independent incident with a minimum viable team: one incident commander, one primary responder, one communications owner. Resist over-staffing the incident that's louder or more visible if the other one is actually higher severity but quieter.
• Protect against double-booking the same expert. If one person is the only one who understands a shared piece of infrastructure, they can advise both incidents briefly but should not be the sole owner of fixing both; pull in a secondary responder even if slower.

When to escalate to a higher-level incident commander

Escalate when any of these is true, not just when severity is high:

• Resource contention itself is blocking progress: both incidents need the same scarce specialist or the same change-freeze exception, and someone above both incident commanders needs to arbitrate.
• Combined blast radius crosses an organizational boundary: the two incidents together affect enough of the business (multiple product lines, a shared customer segment) that unified external communication is needed, even if each incident alone wouldn't trigger that.
• One incident commander is starting to context-switch between both incidents. A single IC trying to run two incidents at once is a bigger risk than the incidents themselves; that's a signal to bring in a second commander or an overall coordinator, not to push through.
• Duration crosses a threshold where sustained dual-incident load starts to fatigue the responding team; a higher-level commander can pull in fresh responders or make the call to deprioritize the lower-severity incident explicitly (and communicate that decision) rather than let it silently starve.

Worked example

Two incidents fire eleven minutes apart: the checkout service is returning 500s for roughly a third of requests (revenue-impacting, high severity), and the internal analytics dashboard is showing stale data (no customer impact, low severity). The correct allocation: full incident-commander-plus-primary-plus-comms team goes to checkout immediately; analytics gets a single responder to investigate on a non-paging basis, because pulling more people onto analytics wouldn't shorten its resolution meaningfully and would strip capacity from checkout. If, twenty minutes in, the checkout investigation discovers the 500s trace back to the same message queue that feeds the analytics pipeline, the two incidents are merged under checkout's commander, because they share a root cause and running them separately would mean two people independently investigating the same queue.
Escalation in this example would trigger only if a third, unrelated incident arrived while checkout was still active and unresolved: three concurrent incidents makes single-IC-per-incident coordination itself the bottleneck, which is exactly the resource-contention trigger above.

Trade-offs and pitfalls

A common mistake is allocating headcount proportional to how loud or visible each incident is (how many people are asking about it in Slack) rather than its actual business impact; loud and low-impact will out-compete quiet and high-impact if you let it. Another is treating "escalate to a higher IC" as an admission of failure, so teams delay it past the point where a fresh coordinator would have resolved the resource conflict faster. The trade-off with merging incidents on a suspected shared root cause is real: merge too eagerly and you lose the separate investigation threads that might have found the divergence faster; the mitigation is to merge the coordination and communication, but keep separate technical workstreams until the shared cause is actually confirmed.

Direct answer

What changes between internal and external incident communication is technical depth and certainty: engineering updates include specifics, hypotheses, exact systems and logs, mitigation steps, because that audience can act on them, while external updates stick to confirmed customer-visible impact and next-update timing, because speculation shared with customers erodes trust if it turns out wrong. What stays the same is cadence discipline and honesty: both audiences get updates on a predictable schedule, and neither gets a false ETA.

Structured elaboration

Tailoring further by audience

Enterprise customers with contractual SLAs often get a more detailed, sometimes one-to-one update from their account team in addition to the public status page, while free-tier customers rely on the public page alone. The underlying facts should be identical; only the delivery channel and level of individual attention differ. Escalation also has its own trigger: open a live bridge call and involve leadership once the outage crosses a defined severity or duration threshold, such as a sustained top-severity incident running longer than the org's own SLA commitment, not just because internal chat is busy.

What never changes regardless of audience

No blame is assigned, internally or externally, while the incident is still active; naming a cause before it's confirmed just gets walked back later. No ETA gets promised that isn't actually known; "investigating" is an honest status, a fabricated timeline is not.

Worked example

Internal update at the 15-minute mark: "checkout-service 5xx rate is elevated since 14:02 UTC, correlates with a config deploy at 13:58; on-call is rolling back now, expect confirmation in a few minutes; questions in the incident channel." External status page at the 30-minute mark, built only from the confirmed facts in that internal update: "We're investigating an issue causing checkout errors for a subset of users. Our team has identified a likely cause and is applying a fix. Next update in 30 minutes." The external version omits the specific deploy detail, since it isn't customer-actionable and would need walking back if the rollback doesn't fully resolve things, but keeps the same cadence commitment as the internal update.

Trade-offs and pitfalls

• Copy-pasting the internal technical update into the external channel either leaks unnecessary detail or reads as more alarming than the plain-language version would.
• More frequent external updates build trust but risk announcing something not yet confirmed if the situation is still moving fast; anchor the cadence to confirmed milestones, not the clock alone.
• Skipping communications or product sign-off on external messages risks a technically accurate but poorly worded update going out while everyone is heads-down on the actual fix.

With sparse telemetry, reconstruct the timeline from whatever correlatable evidence exists (logs, deploy history, database audit fields, even client-side error reports), be explicit about the gaps rather than papering over them with guesses, and then rank the missing instrumentation by how much it would have shortened this specific investigation, not by a generic wishlist.

Reconstructing the timeline

• Inventory every source that exists, even partial ones: application logs, load-balancer or edge logs, deployment/CI history, database updated_at or audit-table entries, orchestrator events, and client-side error or crash reports. Sparse telemetry usually means no distributed tracing, not zero data.
• Normalize and correlate by whatever join key is available. Without trace IDs, correlate by IP plus user-agent plus a tight time window, or by user ID if logs carry it. This gives an approximate rather than exact join, so timestamp uncertainty needs to be carried forward explicitly ("between 14:02 and 14:06" rather than a false-precision single timestamp).
• Use deployment and config-change history as anchor points. Even with no application telemetry, knowing exactly when a deploy or config change happened gives a hard boundary the rest of the timeline can be built around.
• Mark every inferred gap as a gap, not a guess. If there's a 20-minute window with zero data between the last known-good log line and the first symptom report, write "unknown: 20-minute gap" in the timeline rather than smoothing over it with an assumption; the gap itself is the signal for what to instrument next.

Turning gaps into a prioritized instrumentation plan

Priority order, ranked by how much each would have cut investigation time for the incident just reconstructed, not by comprehensiveness:

1. Request-scoped correlation ID, propagated from the edge through every downstream service call and into logs. This directly closes the "approximate join" problem above and is almost always the highest-leverage single addition.
2. Structured logging on the exact boundary where the gap occurred (if the gap was inside a background job, instrument that job's start/end/error events first, not an unrelated part of the system).
3. A minimal health/heartbeat signal for anything that failed silently (a background job that stopped running with no error, a queue consumer that stalled): a periodic "I am alive and processed N items" emission would have converted a silent gap into a detectable one.
4. Everything else (full distributed tracing, high-cardinality dashboards, synthetic monitoring) is real but lower priority than closing the specific gap that made this RCA (root cause analysis) hard.

Worked example

A checkout failure is reported by users around 14:15, but the last application log line referencing checkout is at 13:52, and no logs mention it again until 14:31 when a deploy rollback log appears. The 13:52 to 14:31 window is the gap. Cross-referencing the CDN access log shows checkout page loads continuing normally through 14:20 with a rising 502 rate starting at roughly 14:08, narrowed to that window because the CDN's log granularity is per-minute. The deploy history shows a deploy landed at 13:50, two minutes before the last known-good application log. That's enough to build a bounded timeline: deploy at 13:50, unknown internal state for 18 minutes, first externally visible failure around 14:08, rollback at 14:31. The instrumentation gap this surfaces is specific: the checkout service was emitting no logs at all during a partial-failure state, meaning either logging itself failed or the failure mode (hung requests, not crashed ones) doesn't hit any log statement. The top-priority fix is a heartbeat or request-timeout log on that exact code path, not a generic "add more logging everywhere" recommendation.

Trade-offs and pitfalls

The biggest pitfall is presenting a reconstructed timeline with the same confidence as one built from complete telemetry; a bounded, gap-labeled timeline is honest and still useful, a smoothed-over one invites a wrong root cause. Instrumentation has real costs (ingest volume, storage, and potential PII exposure in high-cardinality logs), so the priority list above should stop at the point where the next item's cost clearly exceeds the diagnostic value for the failure modes actually observed, rather than instrumenting everything a textbook would recommend. A related wrong turn is chasing full distributed tracing as the first fix when the actual gap was a silently-hanging job with zero heartbeat: tracing helps correlate across services, but it doesn't help if the service in question emitted nothing at all during the failure.

Classify early based on plausible worst-case scope and reversibility, not on confirmed scope, because waiting for certainty before declaring Sev1 is exactly what turns a fast, contained incident into a slow, sprawling one; then reclassify explicitly and immediately the moment new evidence narrows or widens that scope.

Deciding severity when the picture is unclear

• Ask what the worst plausible scope is, not the confirmed scope. If a database is showing elevated errors and that database backs three services, declare based on "up to three services could be affected" until you can rule two of them out, not "one service confirmed affected so far." Early Sev1 declarations should err toward the customer-impact ceiling, not the floor.
• Weight reversibility and compounding risk, not just current visible impact. An issue that's currently small but is actively getting worse (a disk filling up, a queue backing up) deserves a higher initial severity than one that's currently larger but stable, because the stable one is easier to reason about and doesn't have a clock running against it.
• Use who needs to be in the room as a practical proxy. If resolving this plausibly requires a cross-team bridge, executive communication, or specialized on-call outside the immediate team, that need alone justifies Sev1 regardless of exact customer-count uncertainty, because the coordination cost of under-declaring and having to re-page everyone later is higher than the cost of over-declaring and standing a bridge down early.

Reclassification triggers

Reclassify down when:

• The actual blast radius is confirmed narrower than the worst-case assumption (e.g., only one of the three services was actually affected, and it has already recovered).
• The trajectory has stabilized (error rate plateaued and isn't compounding) even if full resolution is still pending.

Reclassify up when:

• A contained issue turns out to have a shared dependency that widens scope (the "isolated" service issue turns out to share a cache with a second, larger service).
• Time-in-incident crosses a threshold where customer-facing damage (SLA breach, visible outage duration) becomes material even if the technical severity hasn't changed.

Either direction, the reclassification should be a stated, timestamped decision in the incident channel, with the reasoning, not a silent drift; that record is what makes the postmortem's severity-classification section honest rather than reconstructed after the fact.

Worked example

An alert fires showing elevated 500s on the checkout API. At minute 2, scope is unknown: it could be isolated to checkout, or checkout could share a dependency (say, the pricing service) with two other revenue paths. Declare Sev1 immediately based on the plausible worst case (three revenue paths potentially affected), not Sev2 based on the one confirmed symptom, because the cost of paging two extra teams and standing them down at minute 10 is far lower than the cost of a slow escalation while pricing quietly degrades unmonitored for twenty minutes. At minute 8, the pricing-service dashboard shows it's healthy and checkout's errors trace to a checkout-only cache eviction; reclassify to Sev2 at that point, log the reclassification with the evidence ("pricing confirmed unaffected, root cause isolated to checkout cache"), and stand down the extra teams. At minute 25, if the checkout error rate is still climbing rather than plateauing, that trajectory alone (not a wider blast radius) is grounds to reclassify back up to Sev1, because a still-worsening, still-unresolved customer-facing failure at that duration crosses the SLA-relevant threshold even though the technical scope never widened.

Trade-offs and pitfalls

Erring toward the worst-case scope has a real cost: it pages more people than turn out to be needed, and doing this habitually without disciplined stand-down erodes trust in Sev1 pages the same way alert fatigue erodes trust in any alert. The mitigation is treating the down-reclassification as equally important and equally fast as the initial declaration, not an afterthought. A common wrong turn is anchoring severity to the first symptom's technical name ("it's just a cache issue") rather than its business consequence; a cache issue on checkout is a revenue incident, not a caching incident, and severity should be scored on the former.

A strong answer to this picks one real incident, states plainly what your specific role was (not "the team"), walks through how status was communicated to the people who needed it, and ends with a concrete process change that came out of the postmortem, not just a lesson learned in the abstract. The bar an interviewer is checking is whether you can drive coordination under pressure and whether the org actually got better afterward, not whether the incident itself was dramatic.

Structuring the answer

• Situation: what broke, which teams were involved, and why it needed cross-team coordination rather than one person fixing it alone.
• Task: your specific role, on-call engineer, incident commander, SME being pulled in, and what you were responsible for.
• Action: what you actually did, especially the communication piece: how often you posted updates, who you looped in and when, how you decided what to escalate.
• Result: how the incident resolved, and the one concrete thing that changed afterward as a direct result of the postmortem.

Worked example (illustrative story skeleton)

Situation: I was on-call when a payment-processing service started failing a meaningful share of transactions during a peak traffic window, and it quickly became clear the cause spanned two teams, ours and the database team, not something either of us could fix alone.

Task: As the on-call engineer, my first job was triage and coordination, not fixing it single-handedly, since the fix needed the database team's context on a recent schema change.

Action: I opened a dedicated incident channel, pulled in the database on-call and our service owner, and posted a short status update on a fixed cadence so people weren't pinging me individually for progress. We traced the failures to a missing index from a recent migration, coordinated a rollback with the database team while I kept support and the affected team's manager updated with plain-language impact summaries (not the technical detail I was sharing in the incident channel).

Result: The rollback resolved the failure rate back to its normal baseline within the same on-call shift. In the blameless postmortem, the concrete change we made was adding a required index-verification check to the deploy pipeline for any migration, so a similar schema change can't ship without it being caught automatically rather than during a live incident.

Variant: when the story is about a communication breakdown

The strongest version of this question for a communication-focused answer isn't one where everything went smoothly, it's one where something specific broke in how status was shared, and what changed as a result. For example: during a similar incident, the customer-support team found out about the outage from customers before the incident channel posted a status update, because the communications role wasn't assigned and everyone assumed someone else was posting it. The fix that came out of that postmortem wasn't "communicate better", it was specific: the incident-declaration checklist now requires naming a communications owner in the first two minutes, before technical triage even starts, so that role is never implicitly assumed.

Trade-offs and pitfalls

The most common mistake is describing only the technical fix and skipping the coordination and communication details entirely, which is exactly the part this question is testing for. A close second is ending on "we learned to communicate better" without naming the specific process change that came out of it; a blameless postmortem is only as good as the concrete, trackable action item it produces, and an answer that mirrors that (a specific fix, not a vague lesson) is what separates a senior-sounding story from a generic one.