Data Reliability and Fault Tolerance Questions

Design and operate data pipelines and stream processing systems to guarantee correctness, durability, and predictable recovery under partial failures, network partitions, and node crashes. Topics include delivery semantics such as at most once, at least once, and exactly once and the trade offs among latency, throughput, and complexity. Candidates should understand idempotent processing, deduplication techniques using unique identifiers or sequence numbers, transactional and atomic write strategies, and coordinator based or two phase commit approaches when appropriate. State management topics include checkpointing, snapshotting, write ahead logs, consistent snapshots for aggregations and joins, recovery of operator state, and handling out of order events. Operational practices include safe retries, retry and circuit breaker patterns for downstream dependencies, dead letter queues and reconciliation processes, strategies for replay and backfill, runbooks and automation for incident response, and failure mode testing and chaos experiments. Data correctness topics include validation and data quality checks, schema evolution and compatibility strategies, lineage and provenance, and approaches to detect and remediate data corruption and schema drift. Observability topics cover metrics, logs, tracing, alerting for pipeline health and state integrity, and designing alerts and dashboards to detect and diagnose processing errors. The topic also includes reasoning about when exactly once semantics are achievable versus when at least once with compensating actions or idempotent sinks is preferable given operational and performance trade offs.