Security Incident and Breach Response Questions
Responding to a confirmed or suspected security incident or data breach where the hard part is coordination, evidence and disclosure: deciding between rapid containment and preserving forensic artifacts (including on Kubernetes nodes and cloud accounts that are also the crime scene), chain of custody and legal holds, separating a security compromise from an operational failure, dividing ownership between operations and the security team, involving legal and compliance, deciding whether a breach is notifiable and who must be told by when (regulators, affected individuals, customers), sequencing notifications when one breach touches several regimes with different clocks, scoping exposure from a misconfiguration, an over-broad internal audience, a supply-chain dependency or a machine learning dataset, briefing executives and customers during a breach, and post-breach analysis and safeguards. The generic detect-contain-eradicate-recover lifecycle and its containment mechanics, hands-on forensic artifact analysis, privacy law as law, vulnerability triage, detection engineering and outage response are covered elsewhere.
Design the process by which your organization captures and stores incident evidence (logs, packet captures, configuration snapshots) so it can stand up in legal or compliance review. How do you control access, record custody, limit privacy exposure, and handle retention against routine deletion and legal holds?
Sample Answer
The idea in plain words
Evidence is only useful in a legal or compliance review if you can show it is genuine, complete, unaltered and handled only by people who had a reason to. So the process is not "save the logs somewhere". It is a pipeline that fixes each item at the moment of capture (hash it, time-stamp it, record who took it), keeps it in storage that cannot be quietly edited, lets only named people read it and records every read, limits how much personal data is collected and exposed, and stops routine deletion for exactly the data a lawyer says must be kept. Whether a particular item is accepted as evidence depends on the jurisdiction and the matter, which is why counsel signs off the design, but the same disciplined habits serve almost every regime. Published guidance such as ISO/IEC 27037 (guidelines for identifying, collecting, acquiring and preserving digital evidence) describes the same principles.
Terms used here:
- Chain of custody: a documented record of every hand-off of an item of evidence.
- Hash: a fixed-length fingerprint of a file, such as SHA-256. Change one bit and the hash changes, so a matching hash shows the file is unaltered.
- WORM / write-once storage: storage where objects cannot be changed or deleted until a set date (object lock is a common name).
- Legal hold: counsel's instruction to stop deleting data that may be relevant to a legal matter.
- Percentile: the 95th percentile of a set of numbers is the value that 95 percent of them are at or below. If 95 of 100 past incidents were noticed within 45 days, 45 days is the 95th percentile of detection delay. Using it sizes retention for nearly all cases instead of the average case.
- Pseudonymise: replace a person's identifiers (name, username) with a code, keeping the lookup table elsewhere under tighter access.
- Purpose limitation: a data protection principle (for example in EU GDPR Article 5) that personal data collected for one purpose is not reused for an incompatible one.
- Lifecycle rule: a storage setting that automatically deletes or moves objects once they reach an age.
- Packet capture (pcap): a recording of network traffic, which may contain the full content of communications.
The design, stage by stage
- Open a case. Declaring an incident creates a case ID. Every item, ledger entry and access request refers to it.
- Capture. Use approved tools and written procedures. Prefer the central log store over a log on a possibly compromised host, since the attacker may have edited the local copy. Record the collection method, the exact source, the start and end of the time window, and the collector. Keep clocks synchronised and express all times in UTC.
- Seal. Compute a SHA-256 hash at capture. Store the hash in the custody ledger, which is a separate system from the evidence store, so altering one without the other is detectable.
- Store. A dedicated evidence account or vault, administered by the security team and separate from production administrators. Encrypt it, with keys held by a different group than the collectors. Use write-once retention so even an administrator cannot silently delete.
- Control access.
- Access is per case and per named person, approved by the case owner (and counsel for sensitive matters).
- Originals are read-only. Analysts work on copies, and the copy's hash is checked against the original.
- Retrieving an original needs two people (one requests, one approves).
- Every read, copy and export is logged and the logs are themselves protected. Access is reviewed at case close and quarterly.
- Record custody. One ledger row per event, as below.
- Limit privacy exposure (next section).
- Retain, hold and dispose (the section after that).
Custody ledger (illustrative entries, all values are placeholders)
| Item | Event | By | To | When (UTC) | Purpose | Hash check |
|---|---|---|---|---|---|---|
| case-0412 / web01 disk image | Captured from snapshot | collector_a | evidence vault | 2026-03-02T11:40:12Z | Initial preservation | recorded at capture |
| same | Copy made for analysis | analyst_b | working share | 2026-03-02T14:05:30Z | Malware triage | copy hash matches original |
| same | Original retrieved | analyst_b, approved by lead_c | forensic firm | 2026-03-04T09:12:00Z | External examination | matches |
Hashing commands (every flag explained)
# compute the fingerprint and save it next to the file
sha256sum case-0412/web01-disk.img > case-0412/web01-disk.img.sha256
# recompute later and compare against the saved value
sha256sum -c case-0412/web01-disk.img.sha256
# current time in UTC, ISO 8601, for the ledger entry
date -u +%Y-%m-%dT%H:%M:%SZ
sha256sum prints the hash followed by the filename. -c (check) reads a file in that format, recomputes each hash and prints the filename followed by OK or FAILED. date -u asks for UTC and +%Y-%m-%dT%H:%M:%SZ is the output format string. An illustrative excerpt of the check:
case-0412/web01-disk.img: OK
OK in this check is what the "copy hash matches original" cell of the ledger records: the copy's recomputed hash equals the hash stored at capture. If one bit of the file had changed, the output would instead read case-0412/web01-disk.img: FAILED, followed by a warning that 1 computed checksum did NOT match. A FAILED result is written into the ledger as an event, the copy is not used, and the case owner and counsel are told.
Limiting privacy exposure
- Minimise at capture: a time window and a host list, not "everything". For packet captures, capture headers or use a targeted filter unless the content is needed, because full captures hold passwords, messages and personal data.
- Tier the access: most analysts see metadata and indicators; only the named examiner sees content.
- Redact or pseudonymise working copies where the analysis does not need identities.
- Purpose limitation: evidence collected for an incident is not used for unrelated employee performance review.
- Know where it is stored: location rules for personal data can restrict where copies are kept, and employee monitoring is regulated differently by country. Counsel confirms the policy.
Retention against routine deletion and legal holds
Routine deletion (a log source that keeps data for a fixed number of days) is automatic and does not know about incidents. A hold has to stop it deliberately. The mechanism is a tag: the lifecycle rule is written as "delete objects older than 30 days unless the object carries the tag legal_hold". When counsel issues hold H-17, an operator (or an automated job reading the hold register) adds legal_hold=H-17 to every object in scope, the next run of the deletion job skips those objects, and removing the tag is possible only through counsel's written release. The program below shows the effect.
- Hold register: hold ID, the systems, accounts and date range covered, who issued it (counsel), the date, and status. Deletion jobs consult the register before deleting, or the hold applies a tag that the lifecycle rule respects. The hold covers the source systems as well as the evidence vault.
- Release only by counsel, in writing. After release, dispose according to the retention schedule and record the disposal.
- Retention of evidence itself follows a schedule by class and matter type, set with counsel. The numbers are policy decisions, not engineering ones.
Worked example of the timing risk (assumptions). Assume a log source deletes entries after 30 days. The attacker's first action is on day 0 and the incident is detected on day 25. The convention used here is that an entry written on day d is deleted on day d + 30. The earliest entries therefore expire on day 30, so there are 30 minus 25 = 5 days to apply the hold. If detection happens on day 33, every entry with d + 30 <= 33 has already gone, which is days 0, 1, 2 and 3: four days of evidence. The sizing rule is:
retention≥detection delay (a high percentile)+time to place a holdIn words: keep data at least as long as it usually takes to notice a problem, plus the time it takes to freeze the data once you have noticed. Here detection delay is the number of days from the first malicious event to the day it is detected, and time to place a hold is the number of days from detection to the hold being in effect on the source. If the 95th percentile of detection delay were 45 days and a hold takes 2 days to place, retention must be at least 45 + 2 = 47 days; 90 days leaves a margin. Both figures are assumptions for illustration, and a longer window costs storage and increases privacy exposure, so the number is a trade-off decided with the privacy team.
RETENTION_DAYS = 30
def expired_days(today, first_day, last_day):
"""An entry written on day d is deleted by the job on day d + RETENTION_DAYS."""
return [d for d in range(first_day, last_day + 1) if d + RETENTION_DAYS <= today]
def cleanup(today, entries):
"""Return the entry days the deletion job removes: expired and not tagged legal_hold."""
return [day for day, tags in entries if day + RETENTION_DAYS <= today and "legal_hold" not in tags]
gone = expired_days(33, 0, 40)
print("detected on day 33, expired entry days:", gone[0], "to", gone[-1], "count", len(gone))
print("detected on day 25, days left to place a hold:", 0 + RETENTION_DAYS - 25)
no_hold = [(d, set()) for d in range(0, 36)]
with_hold = [(d, {"legal_hold"} if d <= 10 else set()) for d in range(0, 36)]
print("job on day 33, no hold tag, deletes entry days:", cleanup(33, no_hold))
print("job on day 33, days 0 to 10 tagged, deletes entry days:", cleanup(33, with_hold))
delay_p95, hold_time = 45, 2
print("minimum retention in days:", delay_p95 + hold_time)
Output:
detected on day 33, expired entry days: 0 to 3 count 4
detected on day 25, days left to place a hold: 5
job on day 33, no hold tag, deletes entry days: [0, 1, 2, 3]
job on day 33, days 0 to 10 tagged, deletes entry days: []
minimum retention in days: 47
The first two lines reproduce the day 33 and day 25 cases above, the next two show the deletion job with and without the tag, and the last line is the sizing rule.
Testing the process
Each quarter pick a past case and have someone outside the team verify a hash from the ledger, confirm every item has a complete custody trail, and confirm a test hold blocks a deletion in a non-production log store.
Judgment
I would build the ledger and the hold register first, because without them the rest cannot be proven. What would flip it: a very small company can use the same principles with a locked folder, a spreadsheet ledger and counsel on call, provided each step still has a named person and a record.
Common mistakes: holding evidence on the same host or admin group as production, hashing only at close instead of at capture, giving all of security read access to everything, and keeping logs forever "just in case" without a privacy review.
An attacker with stolen cloud credentials deleted production database snapshots. Restoration is urgent but the account is also the crime scene. How do you decide what to preserve, what logs you need, how you restore, and who you inform internally and externally?
Sample Answer
The idea in plain words
When an attacker uses stolen cloud credentials, the cloud account is both the thing you must repair and the evidence of what happened. Two moves protect both: first make sure the account's audit trail is copied somewhere the attacker cannot reach, and disable (do not delete) the stolen identity; second, restore into a clean place, not by pouring data back into the account you do not yet trust. Terms: audit log is the cloud provider's record of API calls (who, which call, from which address, when; AWS CloudTrail is one example, and every major cloud has an equivalent audit trail under its own name, so the steps below carry over); snapshot is a point-in-time copy of a database volume; blast radius is everything the stolen credential could touch; RPO (recovery point objective) is how much recent data you can afford to lose; RTO (recovery time objective) is how long you can afford to be down. IAM (identity and access management) is the cloud's system of users, roles and permissions; a principal is the identity making a call (a user, a role or a key); an access key is a long-lived credential pair for a user, while a role session is a temporary credential obtained by assuming a role, which expires on its own; a role's trust policy states who may assume it; federation settings link an outside login provider, so an attacker who adds one can sign in later without the stolen key; a user agent is the client software string recorded in the log entry; a deny-all policy is a permission rule that refuses every action and overrides any allow; write-once retention (also called a retention lock) stops stored objects being altered or deleted until a set date; log file integrity validation is a provider feature that signs log files so tampering is detectable; point-in-time recovery restores a database to any moment inside a window using its continuous change log; data-level events are log entries for reads and writes of individual objects, as opposed to management actions such as deleting a snapshot.
First question: is the urgency data loss or lost backups?
Deleted snapshots are a loss of recovery options, not necessarily of data. If the live production database is still running and healthy, nothing needs restoring yet; what is urgent is making a new protected copy before the attacker returns. If the live database was also deleted or encrypted, restoration is the emergency. The plan below handles both, in this order.
Step 1: preserve the evidence (first 30 minutes)
- Secure the audit logs. Confirm they are delivered to a separate log-archive account or storage with write-once retention, and copy the last 90 days now to a second, restricted location. If the logs live only inside the compromised account, assume the attacker can alter them and note that in the record. Check whether log file integrity validation is available and use it.
- Freeze, do not delete, the identity. Deactivate the stolen access key, attach an explicit deny-all policy to the user or role, and revoke active sessions. Deleting the identity removes details you want (creation date, attached policies, last-used data).
- Preserve configuration state. Export a listing of users, roles, access keys, trust policies, network rules, snapshot sharing settings and the list of surviving snapshots, so that later changes are comparable.
- Start the evidence register. One row per artifact: what, source, time collected, collector, SHA-256 hash, storage location. This is the chain of custody.
Step 2: decide what else the credential did (hours 1 to 3)
Query the audit log for every call by the stolen principal, ordered by time. Three illustrative entries (generic field names, invented values, not a real capture) show what each class looks like:
time=2026-03-02T10:31:05Z principal=user/ci-deployer credential=access_key:key-7f3a source_ip=198.51.100.23 user_agent=curl/8.5 action=CreateAccessKey target=user/svc-backup2 result=success
time=2026-03-02T10:36:50Z principal=user/ci-deployer credential=access_key:key-7f3a source_ip=198.51.100.23 user_agent=curl/8.5 action=ShareSnapshot resource=snap-0051 shared_with=account-777 result=success
time=2026-03-02T10:40:12Z principal=user/ci-deployer credential=access_key:key-7f3a source_ip=198.51.100.23 user_agent=curl/8.5 action=DeleteSnapshot resource=snap-0042 result=success
The first is persistence (a new key on another user, so revoking the stolen key alone would not remove the attacker), the second is exfiltration (a snapshot shared with an account that is not ours), and the third is the deletion that was reported. The same source address and user agent across all three ties them to one actor, and the earliest entry from that address marks the first use. Persistence means any way for the attacker to keep access after you revoke the stolen credential. Look for: the first use and the source address and user agent; reconnaissance calls (list and describe); the deletions; persistence (new users, access keys, roles, changed trust policies, new federation settings); exfiltration (snapshots shared to or copied into another account, database exports, object storage reads, new public access); and cost abuse (large compute launches). The questions the log answers decide the response: if there are only deletions, this is mainly an availability and integrity incident; if there are copies or reads of personal data, it is also a confidentiality breach with notification duties. If the log cannot tell you whether data was read (some data-level events are off by default), say that plainly to counsel: "no evidence of access" is different from "evidence of no access".
Step 3: restore without trusting the crime scene
- Find what survives. Candidates: automated backups and point-in-time recovery logs, cross-region or cross-account snapshot copies, logical dumps in object storage, replicas. Check each for attacker changes, since the same credential may have touched them.
- If the live database is intact: take a fresh snapshot now and copy it to a separate, hardened account whose permissions are separate from the compromised one. Add retention locks and a deletion delay so one credential cannot delete everything again.
- If the database is gone: restore from the newest surviving copy into a new environment (new account or a rebuilt, re-keyed network) rather than into the account with unknown persistence. Replay transaction logs if they survive to reduce data loss.
- Do not reuse the old credentials or keys in the restored environment, and rotate secrets the database held (application passwords, connection strings).
Worked example (assumptions: daily cross-account copy at 02:00, deletion at 10:40, no transaction logs): data loss = 10:40 - 02:00 = 8 h 40 min = 520 minutes of writes. If the business RPO is 60 minutes, this recovery path misses it by 520 - 60 = 460 minutes, which is why replaying transaction logs, if any exist, or rebuilding the missing rows from application event streams, belongs in the plan. RTO is the other half. Assume the business RTO is 4 hours (240 minutes), a clean environment takes 90 minutes to build and the restore takes 120 minutes: 90 + 120 = 210 minutes, which fits within 240 with 30 minutes to spare, so restoring into a clean environment does not break the RTO, while the RPO of 60 minutes is the objective that fails and needs the extra replay work.
Step 4: who is told, by whom, when
- Immediately (internal): incident commander, security lead, the owner of the account, and legal. Executives get a short brief once restoration options are known.
- Legal and privacy (first hour): they decide notification duties. If personal data was read or copied, the GDPR Article 33 clock (72 hours from awareness to the supervisory authority, where feasible, unless the breach is unlikely to result in a risk to people) and other regimes' clocks may apply; deletion of backups on its own may not trigger them if the live data is intact, and counsel makes that call.
- Outside, as legal approves: the cloud provider's security or abuse team (they can help with logs and hold actions), cyber insurance (policies often require prompt notice), law enforcement if counsel decides, affected customers and regulators if personal data or customer-contract duties apply.
- Customers: state what was affected, what is not known yet, and when the next update comes.
Judgment
I would preserve first for 30 minutes, because the logs are the only way to tell "backups deleted" from "data stolen", and the answer to that changes who must be told. I would then restore into a clean environment, even if it costs time, because rebuilding inside an account with unmapped persistence invites a second deletion. What would flip it: if the business cannot operate and the attacker's persistence has been fully enumerated and removed, restoring in place is acceptable.
Common mistakes
- Deleting the compromised user or key before recording its history.
- Restoring into the same account before listing every persistence mechanism.
- Treating deleted backups as "just an availability problem" without checking for data reads.
- Relying on logs stored in the compromised account.
- Forgetting to rotate the secrets inside the restored database.
During an incident that may involve personal data exposure, when do you bring in legal and compliance, and why does it matter whether counsel directs the investigation? What do you preserve, and how do you handle the pull between fast containment and keeping evidence intact for regulators and possible litigation?
Sample Answer
The idea in plain words
Bring legal and compliance in at the first credible sign that personal data may have been accessed, not after the technical team is sure. A breach is notifiable when a law or contract requires telling a regulator, the affected people or a customer about it; whether a given breach is notifiable is a legal judgement, not a technical one. Three things depend on that early call: legal deadlines can start running from the moment the organisation is aware, so someone with legal authority has to judge when that is (for example, under the EU GDPR, Article 33, a regulator must be told within 72 hours of becoming aware of a personal data breach, where feasible, unless it is unlikely to result in a risk to people; US state laws, sector rules and contracts add their own clocks, and counsel confirms which apply); the organisation may want its investigation to be privileged (legal communications and work prepared for legal advice that a court may allow it to withhold from an opponent or regulator); and a legal hold (an instruction to stop deleting anything that may be relevant to an investigation or lawsuit) has to be placed before the evidence expires. Fast containment and intact evidence are reconciled by sequencing: copy first, contain second, document both.
When to involve counsel
The trigger is "reasonable suspicion that personal data was accessed, disclosed, lost or altered", not "confirmed breach". Examples that meet it: an alert that a database holding customer records was queried from an unknown address; a laptop with unencrypted personal data reported stolen; a misconfigured storage bucket found to be publicly readable. At that point the incident commander tells the privacy or general counsel and the data protection officer (DPO, the person responsible for privacy compliance in organisations that have one). Waiting for certainty is the common error: by the time everyone is sure, a notification deadline may already be partly spent, and the written record may already contain speculation that counsel would have advised against.
Why it matters whether counsel directs the investigation
- Privilege. In some legal systems, communications between a lawyer and client made to obtain legal advice, and materials prepared in anticipation of litigation, are protected from disclosure. If outside counsel hires the forensic firm under a counsel engagement (the law firm signs the contract with the forensic firm, directs its scope, and receives its report), and the firm reports to counsel, so that the work is carried out to help counsel advise on legal exposure, there is a stronger basis to claim protection for the analysis and the draft conclusions. Whether and how far this works depends on the jurisdiction and on the facts: a report used for ordinary business purposes, or shared widely, may lose protection, and the underlying facts (what logs show, which files were accessed) are never privileged. A concrete contrast: in one structure, the outside law firm hires the forensic firm, the forensic report is addressed to the law firm, and the law firm tells the board and company what it needs to give legal advice. In the other, the IT department hires the forensic firm directly, the report goes to IT and is forwarded to the board, the insurer and the customers; the first has a stronger basis for a privilege claim, the second a weak one. So counsel decides the engagement structure; engineers should not assume privilege exists because a document is labelled "privileged".
- Legal judgement is needed anyway. Whether an event is a notifiable breach, when the organisation became aware, what the regulator and contracts require, and what to say to affected people are legal decisions.
- Control of the narrative record. Counsel can shape who receives the sensitive early drafts and can stop speculative conclusions from circulating in writing.
How to word emails and tickets before counsel has assessed
Write observations, not conclusions. Do not delete or avoid writing; the aim is an accurate, restrained record.
| Avoid | Write instead |
|---|---|
| "We got hacked and the customer data is stolen" | "At 02:10 UTC the audit log shows 1,200 queries to the customers table from account svc-reports at an address not seen before. Investigating." |
| "This happened because Priya forgot to patch" | "Host web-14 was running version 2.3; the fix is in 2.4. Cause under investigation." |
| "No data left the building" | "No evidence yet of data leaving. Logs for the egress path are being reviewed." |
| Forwarding the thread to a wide list | Post in the incident channel with a limited membership and tell counsel where it is |
Keep the working record of the investigation in one place, and mark communications that seek legal advice, for example "Prepared at the request of counsel". Understand that the label alone does not create privilege.
What to preserve
- Logs: authentication, VPN, cloud control-plane audit logs, database audit logs, network flow logs (records of which addresses talked to which, when and how much, without the content), web and application logs, endpoint security tool data, for a window covering well before the first suspected activity.
- System state: disk images or snapshots of affected systems, memory captures and the list of open network connections where the host is still running, configuration and deployment history.
- Business and communication records of the incident: tickets, chat and email of the response, call notes, decision log, status-page history.
- Third-party material: vendor logs, backups and old snapshots that may hold the older copies of data.
- The chain of custody record: for every item, who collected it, when, how, a hash (a fingerprint that changes if the file changes), where it is stored, and each hand-off.
The effect of a legal hold issued mid-incident
A hold suspends any automatic deletion of in-scope material. In practice that means three decisions made at once:
- Log retention. Rolling deletion has to stop for in-scope sources. Arithmetic example with assumed inputs: logs are kept for 30 days, and the first suspected attacker activity was 41 days ago. Logs from the first 41 - 30 = 11 days of the intrusion window have already been deleted. If the hold is issued three days later, a further 3 days are lost, 14 in total, because every day one more day of the window ages out. So the hold goes out the same day, and someone checks cold storage (cheap long-term storage that holds older logs, often beyond the normal retention period), the security information and event management (SIEM) archive (the central log-collection and alerting system, whose older data is often archived separately), and backups for older copies.
- Rebuilds and wipes. Systems in scope are imaged first, and wiped or rebuilt only with a written release. Routine rebuilds on the affected hosts pause; rebuilds on clean replacements proceed.
- Scope and cost. Counsel sets the scope, since over-broad holds are costly and under-broad holds are risky. Retained personal data stays under access control and has a defined review and release date.
Containment versus intact evidence
The pull is real but narrow. Most containment steps (blocking an address, disabling an account, moving a host into a network segment that blocks its traffic, which cuts the egress path, the route by which data leaves the organisation) do not destroy evidence if a snapshot and a record come first. Steps that do destroy evidence (reimaging, terminating instances, wiping devices, restarting services that hold in-memory state) wait for the copy. If a step cannot wait because data is leaving now, take it and document why, what was lost, and who approved. A regulator or a court sees a documented, reasoned trade-off far better than silent loss.
What I would do in order
- Tell counsel and the DPO as soon as personal data is plausibly involved.
- Issue the legal hold with counsel the same day.
- Image and snapshot before changing anything that destroys state; start the chain-of-custody register.
- Contain with the least destructive step that stops the harm.
- Keep written communications factual and limited; route cause and impact questions through counsel.
- Keep a decision log so the choice to contain before the full image can be defended.
Common mistakes
- Waiting for confirmation before calling legal.
- Assuming a "privileged" label protects a document, or sharing a privileged report widely.
- Letting log rotation continue because the hold "has not been formalised".
- Writing a root-cause opinion in a ticket on day one.
- Rebuilding first and documenting later.
During a high-severity incident, legal and compliance ask for logs, system artifacts and data extracts for a regulator inquiry. How do you coordinate evidence collection, keep chain of custody on each artifact, protect personal data while preserving investigatory value, and communicate findings to legal and compliance without speculating?
Sample Answer
The idea in plain words
When legal asks for evidence during a live incident, you are changing from "responder" to "witness". Everything you hand over may later be examined by a regulator, so each item needs three properties: it is complete enough to answer the request, provably unchanged since collection, and limited to the personal data the request actually needs. The way to get there is one intake channel, one evidence register, and a strict separation between what was observed, what is inferred, and what is unknown. Terms: chain of custody is the written record of who held each piece of evidence, when, and what they did with it; a legal hold is a counsel instruction to stop deleting relevant material; privilege is legal protection that can keep communications with counsel from being disclosed, and its scope depends on the jurisdiction, so counsel decides how to protect it; a hash (here SHA-256) is a fixed-length fingerprint of a file that changes if any byte changes.
Step 1: one written request, one owner
- Take the request in writing from legal: what is asked, which regulator or inquiry, the date range and systems in scope, the deadline, and who may receive the results. Verbal asks become a ticket.
- Name one evidence coordinator (typically the forensic examiner) who is the only person who collects and logs. Responders keep fixing the incident; they do not hand over files ad hoc.
- Ask legal to issue the legal hold at once: suspend log rotation (the routine that deletes or overwrites old log files to save space), automatic deletion and device re-imaging for the in-scope systems. Logs that roll off after 30 days under normal policy are the commonest loss. Retention jobs, the scheduled tasks that enforce those deletion rules, must be paused or exempted too, and a cloud service delivered over the internet (SaaS, software as a service) usually has its own retention setting that only its admin console can change.
Step 2: collect with custody on every artifact
For each artifact (log export, disk or memory image, configuration, database extract):
- Collect a copy and leave the original in place. Use read-only access or an export from the source system, and note the exact query or command used.
- Hash it immediately and store the digest separately from the file. Example (
sha256sumprints the SHA-256 digest then the file name; the sample file is created first so the digest can be reproduced):
$ printf 'user_id,event_time,source_ip\n1042,2026-03-09T14:02:11Z,203.0.113.7\n1042,2026-03-09T14:02:19Z,203.0.113.7\n' > auth_events.csv
$ sha256sum auth_events.csv
5961661a8c6abf41021a5fbf31fe4e2d9b233e0c11f5be88a7785b35d7e5b98b auth_events.csv
- Write the register row. Columns: artifact ID, description, source system, time collected (UTC), collected by, method, SHA-256, storage location, who has accessed it. Here the identity provider is the service that signs users in, so an identity provider export is its sign-in log downloaded as a file, and write-once storage accepts new files but refuses edits and deletes. Example row:
A-003 | authentication events, 2026-03-09 | identity provider export | 2026-03-09 16:40 UTC | J. Rivera | console export, date filter 14:00-15:00 | 5961661a...b98b | restricted evidence store, write-once | J. Rivera, M. Chen. - Move it only with a hand-off entry: from, to, time, reason, hash re-verified on receipt. If the hash does not match, stop and record it.
- Work on a copy. Analysts use a working copy; the master stays locked and untouched.
- Record time sources: note which clock produced each timestamp and any offset; mismatched clocks are a common cause of wrong timelines.
Step 3: protect personal data and keep the value
Because the evidence is for a regulator inquiry, collecting more personal data than needed creates its own exposure. Apply data minimization (collect only what the request needs):
- Scope by field and time. Example (assumed): the database extract the request names has 2,400 rows and 9 columns, of which only 4 are needed to show the event sequence (event time, source address, action, record ID). Those four are the ones that answer the regulator's question of what happened and in what order; the other five (name, email, phone, address, account notes) identify the person but do not change the sequence. Extract those 4 columns for all 2,400 rows: 4 of 9 columns is 44 percent of the fields (4 / 9 = 0.444), so the copy that analysts and legal circulate carries about 2,400 x 4 = 9,600 values instead of 2,400 x 9 = 21,600, and the master still holds the full set if the regulator asks for it.
- Pseudonymize in the analyst copy. To pseudonymize is to replace names and emails with a stable token (for example
user_7f3a), so events from one person still link together without showing who the person is. Keep the lookup key with legal or the privacy officer, not with analysts. - Keep the unmodified master under tight access, because a regulator may want the originals, and a redacted copy is a derived artifact with its own register row stating exactly what was removed and by whom.
- Check the transfer: if the request comes from another country's authority, or the evidence leaves the country where it is stored, legal decides whether it is allowed and by what route.
- Encrypt in transit and at rest, and limit who can open the evidence store to the named people.
Step 4: communicate findings without speculating
Findings go to legal, not around legal. If counsel wants the analysis kept privileged, the analysis is requested by counsel and shared in the channel counsel specifies; marking a document "privileged" does not make it so.
Write every status in three labelled parts:
- Observed: facts with artifact IDs. "Artifact A-003 shows 2 authentication events from 203.0.113.7 between 14:02:11 and 14:02:19 UTC on 2026-03-09." (The address is from a documentation-reserved range.)
- Inferred: what the facts suggest and how strong the support is. "This may indicate a single session; it does not show whether the user or an attacker was behind it."
- Unknown, and when we expect to know: "Whether other accounts were used from this address; we will check by 18:00 UTC."
Words to avoid until the evidence supports them: "attack", "stolen", "no data was accessed", the name of an actor, the number of affected people. Correct earlier statements in writing when new facts arrive, with the time and reason, instead of quietly replacing them.
Common mistakes
- Handing over a live system's files that were opened and re-saved (changing timestamps).
- Sending the full database extract because it was easier than choosing the columns.
- Mixed time zones in the timeline.
- Informal chat replies to legal that state guesses as facts.
- Letting retention jobs run while the hold is still being drafted.
After an incident involving suspected data theft, how does the post-incident review and evidence handling differ from a normal operational postmortem? Cover who is involved, what is preserved and for how long, what can be written down and shared, and how recovery proceeds without destroying evidence.
Sample Answer
The idea in plain words
A normal operational postmortem asks "what broke and how do we stop it happening again", and it is written openly, blamelessly (focused on systems, not on punishing people) and shared widely. When data theft is suspected, the same event may also become a legal matter: regulators, customer contracts, lawsuits, insurers and possibly police. So the review changes shape. Evidence is preserved before anyone tidies up, counsel (the company's lawyers) steers the investigation, and what is written and who reads it are controlled. The insight: there are two outputs, a privileged investigation and a shareable lessons-learned document, and the order is evidence first, recovery in parallel without touching the evidence, lessons last.
Terms used here:
- Legal hold: a written instruction, normally issued by counsel, to stop deleting any record that may be relevant to a legal matter. It overrides routine deletion until counsel releases it.
- Chain of custody: a documented record of every hand-off of a piece of evidence (who held it, when, why), so nobody can claim it was altered.
- Forensic image: a bit-for-bit copy of a disk or snapshot of a volume, with a cryptographic hash (a fingerprint such as SHA-256) recorded so any later change is detectable.
- Write-once location: storage that accepts new files but does not allow existing ones to be changed or deleted for a set period, so saved evidence cannot be altered.
- Disk image and memory image: a disk image is a complete copy of a drive's contents. A memory image is a copy of what was in the machine's RAM at one moment, including running programs, encryption keys and open network connections, all of which vanish when the machine powers off.
- Bastion host: a hardened server that administrators log in through to reach internal systems, so its logs show who entered.
- Autoscaling teardown: automation that shuts down and deletes servers when load drops, which would erase an affected server along with its evidence.
- Credential metadata: records about a credential (when it was created, last used, from where, what it can reach), not the secret itself.
- Discoverable document: a written record that a court or opposing party can require you to hand over in a legal dispute.
- Privilege: protection that can keep confidential communications made to obtain legal advice out of a court or regulator's reach. Whether a given document is protected depends on the country and the facts, so counsel decides how the work is structured.
How the two reviews differ
| Dimension | Normal operational postmortem | Suspected data theft review |
|---|---|---|
| Who leads | The service owner or an SRE facilitator | Counsel directs; an incident response (IR) lead and a forensic examiner (internal, or an outside firm engaged through counsel) run the work |
| Who is involved | Responders, service owners, managers | The same technical people for system knowledge, plus counsel, security, privacy or compliance, communications, an executive sponsor, and HR only if an insider is suspected. Access is need-to-know |
| What is preserved | Timeline, graphs, tickets, chat export | Disk and memory images with hashes, all logs covering the attack window and the weeks before it, cloud audit trails, the responders' own chat and tickets, access records, and the custody ledger |
| For how long | Under the normal document retention schedule | Under legal hold until counsel releases it in writing, which happens when the regulatory, litigation and insurance matters are closed. The length is a counsel and retention-schedule decision, not an engineering one |
| What is written | Open narrative, contributing factors, opinions, "we should have..." | Facts with their source and time. No speculation, no legal conclusions ("we were negligent", "this breach violated..."), no guesses about what data was taken until the evidence supports it |
| Who can read it | The whole engineering organisation | Investigation report held by counsel; the broad lessons document only after counsel reviews it |
| How recovery proceeds | Fix and redeploy quickly | Image first, rebuild beside the affected system, keep the original isolated and intact |
Method, in the order you would do it
- Bring counsel in at the first suspicion of theft, before the timeline is written. Work done at counsel's direction for the purpose of legal advice has a better chance of being privileged than the same work done on engineering's own initiative.
- Issue the legal hold within the first hours. Name the systems, accounts, mailboxes and chat channels. Suspend automated deletion for exactly those sources: log expiry rules, ticket archiving, autoscaling teardown, disk recycling.
- Preserve before touching. Take snapshots or images, hash them, store them in an access-controlled, write-once location, and open a custody ledger entry for each.
- Recover in parallel, not in place. Build a clean replacement from known-good code in a new account or network segment. Leave the affected system isolated (no network) but powered and intact until the forensic examiner releases it. Rotate or disable stolen credentials right after the current credential metadata and logs have been exported (a matter of minutes), because rotation changes what the logs show afterwards, and not later than that, because the credential is the attacker's access.
- Investigate under counsel. Findings are recorded as facts with sources. Opinions and hypotheses go to counsel, not into open chat.
- Produce two documents. The privileged investigation report stays with counsel. The lessons-learned document describes control changes (for example "alert on bulk reads by service accounts") derived from established facts, and goes to the wider organisation after counsel review.
- Close the hold only on counsel's release, then delete or archive according to the retention schedule.
Worked scenario (all inputs are assumptions)
Assume a database service account read the customer table from an unfamiliar address over about three hours. Timeline of one possible response:
| Time | Action | Why |
|---|---|---|
| Day 0, 14:00 | Alert fires; on-call confirms the reads are not a known job | Detection |
| 14:30 | IR lead declares a suspected data theft incident and calls counsel | Counsel from the start |
| 15:00 | Legal hold issued for the database, the service account's logs, the bastion host, the responders' chat channel; log expiry suspended for those sources | Stops routine deletion |
| 15:15 | The service account's credential metadata and access logs are exported (minutes of work), then the credential is disabled, ending the unfamiliar reads | Containment as soon as the records it would change are captured |
| 15:30 | Database volume snapshot taken and hashed; custody entry 1 written | Evidence fixed before change |
| 15:45 | Snapshot copied to the evidence store and the hash compared; custody entry 2 written | Working copy kept apart from the original |
| 16:00 | SRE starts a clean replacement stack in a new account; the old one is cut from the network but left intact | Recovery without overwriting |
| Day 1, 09:00 | Forensic examiner given read access to the copy; custody entry 3 written | Analysis under counsel's direction |
| Day 1, 12:00 | Counsel assesses, from the facts so far, whether any notification duty has been triggered and notes the moment of awareness (for example, under EU GDPR Article 33 the supervisory authority is told within 72 hours of the controller becoming aware, where feasible, and information may follow in phases; other regimes and contracts set their own clocks) | Legal clocks run from awareness, not from the end of the investigation |
| Day 10 | Forensic examiner finishes the factual timeline; counsel updates any notices with the confirmed facts | Facts first for the report, not for the clock |
| Day 14 | Lessons-learned review with owners and dates for control changes; counsel clears it for wide sharing | Learning without exposure |
The days in the last three rows are assumptions for illustration. The point is the ordering: hold, then export the records the credential change would alter, then disable the credential, then image and rebuild beside, then lessons. The export takes minutes, so it delays containment only briefly; containment never waits for the image, the rebuild or the investigation, because the theft may still be running.
Illustrative custody ledger rows for the 15:30 snapshot (hash shortened for display):
| Entry | Item | Hash (SHA-256) | From | To | When (UTC) | Purpose |
|---|---|---|---|---|---|---|
| 1 | Snapshot of the database volume | 9f2c...e41a | SRE engineer A (created it) | IR lead | Day 0, 15:30 | Preserve state before any change |
| 2 | Copy of the snapshot in the evidence store | 9f2c...e41a | IR lead | Evidence store (write-once) | Day 0, 15:45 | Working copy kept apart from the original; same hash confirms it is identical |
| 3 | Read access to the copy | 9f2c...e41a | Evidence store | Forensic examiner | Day 1, 09:00 | Analysis under counsel's direction |
The same finding written two ways
| Open postmortem style | Counsel-controlled style |
|---|---|
| "The attacker got in because Priya never rotated the key, and they probably took every customer's record." | "At 14:02 UTC the service account read the customer table from an address not previously seen (source: database audit log). The key's last rotation date is recorded in the key metadata (source: key management record). The volume of data read has not yet been determined." |
The second version states what was observed, where the proof lives, and what is still unknown, and it names no person and draws no conclusion about fault or about what was taken.
How long evidence is kept
There is no single number. The hold lasts until the last open matter closes, and regulatory enquiries and lawsuits can run for years, so in practice a hold is measured in months to years, not weeks. Routine retention, such as 30-day log expiry, would have deleted the evidence long before then, which is why the hold suspends it. Counsel sets the end date and releases the hold in writing.
Judgment and pitfalls
I would commit to counsel-led evidence handling from the first hour, accept that the engineering write-up goes out later and in a narrower form, and never trade evidence for speed of cleanup. What would flip this: if the facts show a pure outage with no data access, the review reverts to the normal blameless format, and the hold is released on counsel's say-so.
Common mistakes:
- Speculating about "what they took" or "who is at fault" in a public chat channel, which then becomes a discoverable document, as defined above.
- Terminating the compromised instance to stop the cloud bill, destroying the memory and disk state.
- Sharing the first draft with the whole company, then being unable to correct it.
- Treating "blameless" as a reason to hide facts. Blamelessness governs how people are treated, while the hold governs what must be kept. If an insider is suspected, HR and counsel handle individual conduct separately, and the technical review stays about systems.
- Deleting or rewriting messages "for tidiness" after the hold, which can itself create legal exposure.
Unlock Full Question Bank
Get access to all 7 Security Incident and Breach Response interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.