Debugging and Testing ML Systems Questions

Direct answer

A chaos-engineering plan for an ML system deliberately injects a specific, named failure, feature corruption, missing upstream data, a sudden spike in one label's frequency, or delayed inputs, into a small, capped slice of real or shadow traffic, then measures how the system actually degrades against a hypothesis stated before the experiment runs. The two disciplines that separate this from just breaking things in production are a hard blast-radius cap with an automatic abort if the observed error rate crosses a threshold, and a mandatory follow-up step: every experiment that reveals a real weakness produces a tracked fix, not just a finding written down and forgotten.

Structured elaboration

The four failure injections and what each one tests.

• Feature corruption: replace a feature's value with an out-of-distribution or garbage value (large-magnitude noise, a wrong-typed value) for a fraction of requests, simulating an upstream bug that silently mangles a field rather than failing loudly. This tests whether the model (or the pipeline wrapping it) has any input-sanity check at all, or whether it will confidently score garbage.
• Missing upstream data: force a feature to fall back to its default (null, zero, or a mean-imputation value), simulating an upstream service outage or a schema field that stopped being populated. This specifically tests the fallback path, which in a healthy system runs rarely and is therefore the least-exercised code path in the whole pipeline.
• A spike in one label's frequency: flood the system with requests that skew heavily toward one predicted class or one input segment, simulating a real-world event that shifts the traffic mix suddenly (a fraud ring probing one product category, a viral post driving one query type). This tests capacity and downstream-system assumptions that were tuned for an average traffic mix, a fraud-review queue sized for a steady 2% flag rate will behave very differently if flags briefly spike to 40%.
• Delayed inputs: hold back a feature's arrival past its normal latency budget, simulating a slow upstream dependency. This tests the timeout and fallback logic specifically, does the system wait indefinitely (a latency failure), serve a stale cached value (a correctness failure), or fail the request outright (an availability failure), and which of those three is actually the intended, chosen behavior versus an accident of how the code happens to be written.

Capturing and interpreting the system's response. Before running anything, write down the falsifiable hypothesis the experiment tests ("if feature X is corrupted for 10% of traffic, end-to-end accuracy on that slice stays within Y of baseline, because the input-validation layer should catch and reject it"), then measure the actual metric on the actual injected slice against a same-window, non-injected control group, not against a historical average that could be confounded by unrelated factors. Interpretation is comparing the observed result to the hypothesis, not just reporting a number: a result that matches the hypothesis exactly is informative in a different way than one that badly misses it, both are useful, but they call for different follow-ups.

Safety controls so the experiment cannot cause real user harm. Three layers, from least to most severe:

1. A hard blast-radius cap: the experiment is only allowed to touch a small, pre-approved percentage of traffic (single digits for a first run of any new experiment type), enforced in code before the experiment starts, not as a policy someone is trusted to follow.
2. An automatic abort trigger: monitor the injected slice's error rate (or a business metric like conversion) in real time during the experiment, and kill it automatically the moment it crosses a pre-set threshold, rather than waiting for a human to notice.
3. A kill switch and rollback path tested in advance: the mechanism that stops the experiment must itself be tested before the experiment runs, an abort trigger that has never been exercised is not a safety control, it is an unverified assumption.

After a test reveals a real weakness. Treat the finding the same way a production incident would be treated: file it as a tracked issue with an owner and a severity, not just a note in a doc; prioritize the fix by the same criteria as any other production risk (how likely is this failure to occur outside of an experiment, and how bad is it if it does); and, once fixed, re-run the exact same experiment to confirm the fix actually closes the gap, rather than assuming the fix worked because it looks correct in code review.

Worked example

Training a small classifier and measuring the actual impact of the first two injection types on a genuinely held-out test set. The setup is shipped in full rather than assumed, because the accuracy drops below only mean something if you can see how strongly the label depends on the feature being corrupted (here feature 0 carries most of the signal, which is why injecting into it moves the numbers at all):

import torch
import torch.nn as nn

def make_dataset(n, seed):
    gen = torch.Generator().manual_seed(seed)
    X = torch.randn(n, 4, generator=gen)
    logits = 2.5 * X[:, 0] + 0.6 * X[:, 1] - 0.4 * X[:, 2]
    y = (logits > 0).long()
    return X, y

def accuracy(model, X, y):
    with torch.no_grad():
        return (model(X).argmax(dim=1) == y).float().mean().item()

torch.manual_seed(0)
X_train, y_train = make_dataset(2000, seed=1)
model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 2))
opt = torch.optim.Adam(model.parameters(), lr=0.02)
for _ in range(400):
    opt.zero_grad()
    nn.functional.cross_entropy(model(X_train), y_train).backward()
    opt.step()
model.eval()

g = torch.Generator().manual_seed(3)   # separate seeded generator for injection sampling

X_test, y_test = make_dataset(300, seed=2)
baseline_acc = accuracy(model, X_test, y_test)
print(f"baseline accuracy (no injection): {baseline_acc:.3f}")

def corrupt(frac):                      # Experiment A: garbage value
    X = X_test.clone()
    idx = torch.randperm(len(X_test), generator=g)[:int(frac * len(X_test))]
    X[idx, 0] = torch.randn(len(idx), generator=g) * 10
    return X

def drop_to_default(frac):              # Experiment B: fallback to the default 0.0
    X = X_test.clone()
    idx = torch.randperm(len(X_test), generator=g)[:int(frac * len(X_test))]
    X[idx, 0] = 0.0
    return X

for title, inject in [("Experiment A: feature corruption (feature 0 replaced with noise)", corrupt),
                      ("Experiment B: missing upstream data (feature 0 falls back to 0.0)", drop_to_default)]:
    print()
    print(title)
    for frac in [0.1, 0.5, 1.0]:
        acc = accuracy(model, inject(frac), y_test)
        print(f"  injected fraction={frac:.0%}: accuracy={acc:.3f} (drop={baseline_acc - acc:.3f})")

Actual output:

baseline accuracy (no injection): 0.993

Experiment A: feature corruption (feature 0 replaced with noise)
  injected fraction=10%: accuracy=0.943 (drop=0.050)
  injected fraction=50%: accuracy=0.753 (drop=0.240)
  injected fraction=100%: accuracy=0.547 (drop=0.447)

Experiment B: missing upstream data (feature 0 falls back to 0.0)
  injected fraction=10%: accuracy=0.953 (drop=0.040)

At every injected fraction, garbage-value corruption hurts accuracy more than a sane zero-fallback (0.050 against 0.040 at 10%, 0.240 against 0.207 at 50%, and 0.447 against 0.423 at 100%), a concrete, falsifiable version of the intuition that a defined default is safer than an undefined one, and exactly the kind of result a chaos experiment is supposed to produce: a number to argue from instead of an assumption. Note how small the gap is at 100% injection, 0.447 against 0.423. Both injections destroy the feature's signal; the defined default is better, but only slightly, which is a more useful finding than the intuition alone, since it says the real mitigation is not a better default value but detecting that the feature is broken at all.

The blast-radius safety control, exercised on all three of its outcomes: a request that violates the cap, one that respects it and completes, and one that respects the cap but trips the abort threshold mid-experiment. The third case matters most and is the one usually left untested, because an abort trigger that has never fired is an assumption rather than a control:

def run_capped_experiment(inject_fn, frac, max_blast_radius=0.05, abort_error_rate=0.5):
    if frac > max_blast_radius:
        return {"status": "REJECTED_BEFORE_START",
                "reason": f"requested {frac:.0%} exceeds blast-radius cap {max_blast_radius:.0%}"}
    error_rate = 1 - accuracy(model, inject_fn(frac), y_test)
    if error_rate > abort_error_rate:
        return {"status": "ABORTED_MID_EXPERIMENT", "error_rate": round(error_rate, 3)}
    return {"status": "COMPLETED", "error_rate": round(error_rate, 3)}

print("requesting a 50% blast radius (should be rejected before it ever runs):")
print(run_capped_experiment(corrupt, 0.50))
print("requesting a 3% blast radius (within cap, should run):")
print(run_capped_experiment(corrupt, 0.03))
print("a 4% experiment whose injected slice errors past the abort threshold:")
print(run_capped_experiment(corrupt, 0.04, abort_error_rate=0.01))

Output:

  injected fraction=50%: accuracy=0.787 (drop=0.207)
  injected fraction=100%: accuracy=0.570 (drop=0.423)
requesting a 50% blast radius (should be rejected before it ever runs):
{'status': 'REJECTED_BEFORE_START', 'reason': 'requested 50% exceeds blast-radius cap 5%'}
requesting a 3% blast radius (within cap, should run):
{'status': 'COMPLETED', 'error_rate': 0.023}
a 4% experiment whose injected slice errors past the abort threshold:
{'status': 'ABORTED_MID_EXPERIMENT', 'error_rate': 0.02}

Trade-offs and pitfalls

• A capped blast radius trades statistical power for safety, and that is the correct trade. A 3% slice gives a noisier estimate of the true impact than 50% would, but the entire point of the cap is that you are not yet confident enough in the outcome to accept the larger slice's risk; run small first, and only widen the blast radius once the small run's result matches the hypothesis.
• Injecting on shadow or synthetic traffic first, and only later on a capped slice of real traffic, catches a different class of bug than either alone. Shadow traffic is safe but might not reproduce real timing and concurrency conditions; a real capped slice is realistic but risks a real (small) user impact, so a mature program runs both, in that order, rather than picking one.
• The 'spike in one label' injection is the easiest of the four to get wrong by testing the model instead of the system. The model itself usually handles a skewed prediction mix without complaint; the actual fragile point is almost always a downstream system (a review queue, a rate limiter, a notification budget) that was capacity-planned around the normal mix, so point this experiment at the whole pipeline, not just at the model's own accuracy.
• A finding that reveals no weakness is not a wasted experiment. It is tempting to only report and act on experiments that find a problem; a clean result that matches the stated hypothesis is evidence the system's fallback behavior works as designed, which is exactly what justifies the next experiment's larger blast radius.

Direct answer

The single most useful move is to compare metrics at matched label maturity, never a fully-resolved older cohort against a still-maturing recent one, because a cohort that has not yet had time for all its true positives to be labeled will structurally look worse on precision and recall even if the model's real behavior has not changed at all. Alongside that, check the labeling pipeline's own operational health directly (label volume, lag, recent changes) as an independent signal, and watch whether the apparent drop is confined to the newest, least-mature cohort or also shows up in cohorts old enough that their labels should already be complete.

Structured elaboration

Distinguishing genuine degradation from a label-delay artifact. The core diagnostic is a maturity-matched comparison: instead of comparing today's partially-labeled recent cohort against an old cohort's final, fully-resolved metric, compute what that same old cohort's metric would have looked like back when it was the same number of days old as the recent cohort is now. If the retroactively-truncated old cohort shows the same apparent drop the recent cohort shows today, the drop is a structural artifact of label maturity, not a real change, since the same computation applied to a cohort now known to be healthy reproduces the same symptom. If the old cohort's truncated-maturity metric looks fine while the recent cohort's does not, something genuinely changed. A second, independent check is the labeling pipeline's own health: label ingestion volume, lag between the predicted event and its label arriving, and any recent deploys or schema changes to the labeling job itself, correlated against exactly when the apparent drop started. If the drop's onset lines up precisely with how far back the label-delay window reaches from today, that timing itself is strong evidence of an artifact rather than a coincidence.

Monitoring adjustments. Report the primary accuracy metric only on cohorts that have reached full label maturity, with the dashboard's most recent visible point lagged by the label delay rather than showing an always-current but structurally biased number. Alongside that mature metric, show a separately labeled, explicitly provisional in-progress metric for cohorts still maturing, so a viewer is never left assuming a low number is final when it is not. Track the labeling pipeline's own volume and lag as a first-class monitored signal in its own right, not merely as a debugging tool reached for after a metric alarm fires, since a labeling-pipeline problem is itself an operational incident worth catching directly.

Proxy metrics usable before true labels arrive. Two families, both label-free: input and score stability, and short-horizon leading behavior. For input and score stability, compare the model's predicted-positive rate and score distribution for the recent cohort against the same statistics from historically healthy cohorts; this needs no labels at all, since it only depends on what the model produced, and it directly catches genuine model or feature drift, which a maturity-matching check on the label side cannot see. For short-horizon leading behavior, identify an earlier, faster-arriving signal correlated with the eventual true label (for a churn model, a near-term drop in product usage ahead of an eventual cancellation), but validate any such proxy by backtesting it against true historical labels first, confirming it actually predicts the outcome before relying on it operationally, rather than assuming a plausible-sounding proxy is automatically a good one.

Concrete diagnostic plan. In order: (1) confirm the labeling pipeline's own health, volume and lag, against its own history; (2) run the maturity-matched comparison, an old cohort truncated to the recent cohort's current age, against the recent cohort itself; (3) if step 2 shows the old cohort's truncated metric reproduces the same apparent drop, conclude artifact, and switch the dashboard to a maturity-lagged view while relying on the label-free proxies for a faster read; (4) if step 2 shows the old cohort's truncated metric looks normal while the recent cohort does not, treat this as a genuine signal and begin the same root-cause investigation any other unexplained metric drop deserves, checking upstream data and feature pipelines, not the labeling process.

Worked example

Simulate 20,000 customers, a 10% true churn rate with each churner's churn day spread evenly across a 30-day window, and a model with a stable, known-real precision of 70% and recall of 65% on fully-resolved data (every churn event, wherever in the 30-day window it happened, has had time to be labeled). The artifact only becomes legible in COUNTS, so start there rather than with the rates:

• 2,000 eventual churners (10% of 20,000).
• 1,857 predicted positives, the customers the model flags as likely churners. This count comes from the model's own output and never changes as labels arrive.
• 1,300 of those predictions are correct once every label has landed, giving precision 1300/1857 = 0.700 and recall 1300/2000 = 0.650.

Now compute the identical metrics for that same cohort observed for only 5 of its 30 days, treating any churner who has not churned yet (because their churn day falls after day 5) as if they were a labeled negative, exactly what a naive dashboard does by default. At day 5, 309 of the 2,000 churners have actually churned (15.45% of them, in line with the 5/30 = 16.7% expected), and only 201 of the model's 1,300 true positives are among that 309. The seed is pinned, so every count below is reproducible rather than a one-off draw:

import numpy as np

N_CUSTOMERS, CHURN_RATE, WINDOW_DAYS, OBSERVED_DAYS = 20_000, 0.10, 30, 5
PRECISION, RECALL = 0.70, 0.65

n_churners = int(N_CUSTOMERS * CHURN_RATE)          # 2000 eventual churners
n_true_pos = int(round(n_churners * RECALL))        # 1300 correct predictions
n_predicted_pos = int(round(n_true_pos / PRECISION))  # 1857 predicted positives

rng = np.random.default_rng(2564)                   # pinned, so these counts reproduce
churn_day = rng.integers(1, WINDOW_DAYS + 1, n_churners)   # spread evenly over 30 days
true_pos = rng.choice(n_churners, n_true_pos, replace=False)

matured_churners = int((churn_day <= OBSERVED_DAYS).sum())
matured_true_pos = int((churn_day[true_pos] <= OBSERVED_DAYS).sum())

print(f"eventual churners:        {n_churners}")
print(f"predicted positives:      {n_predicted_pos}")
print(f"true positives (resolved): {n_true_pos}"
      f"  -> precision {n_true_pos/n_predicted_pos:.3f}, recall {n_true_pos/n_churners:.3f}")
print(f"\nat day {OBSERVED_DAYS} of {WINDOW_DAYS}:")
print(f"  churners labeled so far: {matured_churners}"
      f" ({matured_churners/n_churners:.2%}, expected {OBSERVED_DAYS/WINDOW_DAYS:.1%})")
print(f"  matured true positives:  {matured_true_pos}")
p_now = matured_true_pos / n_predicted_pos
r_sofar = matured_true_pos / matured_churners
r_eventual = matured_true_pos / n_churners
print(f"  precision                        = {matured_true_pos}/{n_predicted_pos} = {p_now:.3f}"
      f"  (down {100*(1 - p_now/(n_true_pos/n_predicted_pos)):.1f}%)")
print(f"  recall vs labels that exist      = {matured_true_pos}/{matured_churners} = {r_sofar:.4f}"
      f"  (was {n_true_pos/n_churners:.4f})")
print(f"  recall vs eventual churners      = {matured_true_pos}/{n_churners} = {r_eventual:.4f}"
      f"  (down {100*(1 - r_eventual/(n_true_pos/n_churners)):.1f}%)")

Output:

eventual churners:        2000
predicted positives:      1857
true positives (resolved): 1300  -> precision 0.700, recall 0.650

at day 5 of 30:
  churners labeled so far: 309 (15.45%, expected 16.7%)
  matured true positives:  201
  precision                        = 201/1857 = 0.108  (down 84.5%)
  recall vs labels that exist      = 201/309 = 0.6505  (was 0.6500)
  recall vs eventual churners      = 201/2000 = 0.1005  (down 84.5%)

Nothing about the model changed between the resolved numbers and any row of that table; the same 1,857 predictions and the same 1,300 correct ones are being counted, just earlier. Which denominator the dashboard happens to use for recall is what decides whether recall appears to collapse or appears fine, which is why a metric whose definition does not name its denominator is not really defined at all, just a number with a familiar label on it.

The invariant that makes this artifact recognizable in the wild. Only the NUMERATOR waits for labels. Write the matured fraction as $f$ (here 201/1300 = 0.155): it multiplies the true-positive count and nothing else.

• Precision's denominator, predicted positives, does not wait for labels, so precision falls by exactly $f$: 0.700 becomes 0.108, an 84.5% relative drop.
• Recall against the labels that exist so far carries $f$ in the numerator and in the denominator, so it cancels and the metric barely moves: 0.650 stays 0.650.
• Recall against a fixed count of eventual churners keeps its denominator, so it falls by exactly the same $f$ as precision: 0.650 becomes 0.1005, also an 84.5% relative drop.

So a pure label-maturity artifact produces one of exactly two signatures, depending on the denominator convention: precision collapsing with recall flat, or precision and recall collapsing by the SAME relative amount. Two different, unequal relative drops are not what maturity alone does, and seeing them is itself evidence that something besides maturity is also going on and the investigation should not stop at "it is just the label lag."

This is precisely the artifact the maturity-matched comparison from the structured elaboration exists to catch, and the second block above IS that comparison run on a cohort already known to be healthy: it is the fully-resolved cohort's own metrics recomputed from only its first 5 days, and it reproduces the collapse, which is what confirms the drop is structural rather than a real change in the model.

Trade-offs and pitfalls

• Waiting for full label maturity is honest but slow. If the label delay is 30 days, a maturity-matched, artifact-free signal is also up to 30 days behind a genuine regression; the label-free proxies exist specifically to shrink that detection lag, at the cost of needing their own validation before they can be trusted.
• The maturity-matched comparison only works if enough history is stored to reconstruct it. It requires knowing, for a historical cohort, exactly when each label arrived, not just the final aggregate metric; a system that only ever stores the final rolled-up accuracy number has thrown away the information needed to do this diagnostic after the fact, so the label-arrival timestamp itself needs to be a first-class stored field, not an afterthought.
• Treating every metric drop on a delayed-label model as a labeling artifact by default is its own bias. A team that gets used to "it's probably just the label lag" can miss a genuine regression that happens to coincide with a period of normal label immaturity; the labeling-pipeline health check and the maturity-matched comparison both have to actually be run each time, not assumed.
• A proxy metric that looks plausible is not automatically valid. A leading indicator has to be backtested against true historical labels and shown to actually correlate with the eventual outcome before it is trusted as an early-warning signal; an unvalidated proxy can itself drift independently of the true label relationship and quietly stop meaning what it once meant.

Direct answer. An intermittent multi-GPU failure (one node's CUDA OOM, or results diverging across otherwise-identical nodes) needs evidence gathered from every rank, not just the one that visibly failed, because the actual fault often originates on a DIFFERENT rank than the one that shows the symptom.

Debugging plan.

1. Collect NCCL, CUDA, and system logs from every rank, not just the failing one, and turn them on BEFORE the run, since none of this can be recovered retroactively. A rank that silently produces a NaN, for instance, often only shows up as an OOM or a hang on a DIFFERENT rank once the collective operation (all-reduce) tries to synchronize with it. The three layers each need their own switch.
   • NCCL / collectives: NCCL_DEBUG=INFO (plus NCCL_DEBUG_SUBSYS=ALL when you need the ring topology) shows which ranks joined which communicator and where a collective stalled. Pair it with TORCH_NCCL_ASYNC_ERROR_HANDLING=1 so a rank that dies tears the job down with an error instead of leaving every other rank blocked forever in the all-reduce, and TORCH_DISTRIBUTED_DEBUG=DETAIL to get shape and dtype mismatches across ranks reported as errors rather than as silent corruption.
   • CUDA: on a REPRO run (not the production one, because it serializes kernel launches and changes timing), set CUDA_LAUNCH_BLOCKING=1 so the traceback points at the kernel that actually failed rather than at whatever later call happened to synchronize. For the OOM specifically, wrap the step in a handler that dumps torch.cuda.memory_summary() at the moment of failure, and for a repeat offender turn on the allocator trace with torch.cuda.memory._record_memory_history() and dump it with torch.cuda.memory._dump_snapshot(), which gives you the call sites holding every live block instead of a single total.
   • System: check the kernel log on the suspect node (dmesg -T) for Xid errors and for the host OOM killer, and nvidia-smi -q -d ECC,TEMPERATURE,POWER for uncorrectable ECC counts, retired pages and thermal or power throttling. This layer is the one people skip and it is the one that answers the question actually being asked, which is why THIS node. A repeating Xid or a rising uncorrectable-ECC count on exactly one node turns a week of software debugging into a node replacement, and a host-side OOM kill explains a node dying with no GPU memory problem at all.
2. Separate the three memory failure shapes: a genuine leak, fragmentation, and a legitimate peak-allocation spike. They look identical at the moment of the OOM and completely different across many steps, so log TWO numbers per step, not one: memory ALLOCATED (the bytes currently held by live tensors) and memory RESERVED (the bytes the caching allocator is holding from the driver, including free-but-cached blocks). In PyTorch those are torch.cuda.memory_allocated() and torch.cuda.memory_reserved(). On a 16 GiB device the three shapes read like this:

step              1      50     100    150    200
LEAK        alloc 8.1    8.4    8.7    9.0    9.3   (GiB, monotonic climb)
            resv  8.6    8.9    9.2    9.5    9.8
SPIKE       alloc 6.2    9.8    6.2    9.8    6.2   (sawtooth, returns to baseline)
            resv  9.9    9.9    9.9    9.9    9.9
FRAGMENT    alloc 6.0    6.0    6.1    6.0    6.0   (flat, far below capacity)
            resv  9.5   10.2   10.9   11.4   11.8   (climbs away from alloc)

• Leak: allocated climbs monotonically across steps and never returns to baseline. Something is retaining references (a loss tensor accumulated into a Python list without .detach() or .item(), a growing cache, a hook holding activations). Fix the retention. Raising the memory ceiling only buys steps.
• Spike: allocated is a sawtooth that returns to the same baseline every step, with the peak driven by a particularly large batch or an activation-checkpointing boundary. Reserved sits flat at the high-water mark. This is normal behavior against an unlucky ceiling, and reducing batch size or enabling activation checkpointing genuinely fixes it.
• Fragmentation: allocated stays flat and well below device capacity while reserved climbs away from it, and the OOM message itself gives it away, reading something like "tried to allocate 2.00 GiB, 3.50 GiB free": there IS enough free memory in total, just not in one contiguous block. The other tell is that failure depends on allocation ORDER rather than step count, so it can fire at step 12 on one run and step 400 on the next, and it is strongly associated with varying tensor shapes (variable sequence lengths, ragged batches) that make each allocation a slightly different size. The fixes are different in kind from the other two: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (or tune max_split_size_mb) so the allocator stops carving unusable slivers, call torch.cuda.empty_cache() at a safe step boundary to hand cached blocks back to the driver, bucket or pad variable-length inputs so allocation sizes repeat instead of drifting, and pre-allocate the large buffers once at start-up. Note what is NOT on that list: reducing batch size, which is the standard reflex and does not reliably help here, because the problem was never total capacity.

Confusing any two of the three leads to the wrong fix, and the fragmentation case is the one where the wrong fix is most tempting, since the allocator is reporting plenty of free memory while the allocation still fails.
3. Validate that batch size and sharding are actually consistent across ranks. A configuration bug (one node launched with a stale config, or an uneven data-sharding split) can silently give one rank a larger effective batch than the others; log the actual batch size and shard boundaries each rank believes it has, and diff them across ranks rather than assuming the launch config was applied uniformly everywhere.
4. Isolate WHICH rank produces an anomaly (for example a NaN) in a multi-GPU job: add a lightweight per-rank check right after the forward pass (before the collective all-reduce) that logs a boolean "this rank saw a NaN" flag, tagged with the rank ID, and aggregate these flags centrally. This turns "the job produced a NaN somewhere" into "rank 3 produced the NaN, ranks 0/1/2 were clean," which narrows the investigation from the whole cluster to one node's data shard, environment, or hardware.

Short-term mitigation to keep the job running while investigating. Reduce the per-rank batch size (or enable gradient accumulation to compensate) to lower peak memory pressure and buy headroom while you investigate the OOM's root cause, and enable periodic checkpointing (if not already in place) so an eventual failure doesn't lose the whole run's progress. This is explicitly a stopgap, not a fix: if the true cause is a genuine leak or a sharding bug, reducing batch size only delays the eventual failure, and if the cause is fragmentation it may not delay it at all, since a smaller batch changes allocation sizes without making the free memory any more contiguous. In that case the cheapest stopgap is instead the allocator setting plus a periodic empty_cache() at a step boundary.

Direct answer

A test oracle (the mechanism a test uses to decide pass or fail) for a probabilistic, non-deterministic pipeline cannot compare a single output to a single expected value, because there is no single correct output to compare against, run the same input twice and you legitimately get two different scores. The oracle instead has to compare a distribution of candidate outputs against a distribution of expected outputs, using a statistical test built for exactly that comparison, and the oracle itself needs to be validated against a synthetic dataset with known ground truth before it is trusted, the same way any measurement instrument gets calibrated before it is used to measure something real.

Structured elaboration

What "acceptance criteria" means with no single correct answer. For a deterministic function, acceptance criteria is "does the output equal the expected value." For a probabilistic ranking or scoring pipeline, acceptance criteria has to be reframed as a statement about the output's distribution: does the distribution of scores this candidate produces look statistically consistent with what a non-regressed system should produce, given the same inputs. This is a genuinely weaker, probabilistic claim than exact-match equality, and being explicit about that weakness (a passing oracle says "consistent with no regression," not "definitely correct") is part of specifying the acceptance criteria honestly.

Statistical tests for comparing a candidate distribution against an expected one. The two-sample Kolmogorov-Smirnov (KS) test compares the empirical cumulative distribution functions (CDFs, functions giving the probability a random draw is less than or equal to a value) of two samples and reports the largest gap between them:

$D = \sup_x \left| F_n(x) - G_m(x) \right|$

where $F_n$ and $G_m$ are the empirical CDFs of the expected and candidate samples. It is a good default choice here because it is distribution-free, it makes no assumption the scores are normally distributed or any other specific shape, and it is sensitive to any difference in distribution shape, not only a shift in the mean. The test returns a p-value; below a chosen significance threshold ($\alpha$, conventionally 0.05), declare a regression.

Building a synthetic dataset with known ground truth to validate the oracle itself. An oracle is only trustworthy once you know its false-positive rate (how often it cries regression when there truly is none) and its statistical power (how often it correctly catches a regression that is really there), and you can only measure both of those with a dataset where you, the test author, control the ground truth. Concretely: generate two samples from the exact same known distribution (there is truly no regression, by construction) and run the oracle many times; a well-built oracle at significance level $\alpha$ should flag a "regression" close to $\alpha$ of the time, not much more (a broken, overly-sensitive oracle) and not much less (a broken, underpowered oracle that has no real ability to detect anything). Then generate a second pair from two deliberately different known distributions (there truly is a regression, by construction) and confirm the oracle's detection rate is high. Only once both of these checks pass do you have evidence the oracle itself works, as opposed to evidence about whatever real pipeline you eventually point it at.

Detecting a regression despite run-to-run variance. A single trial's p-value can land in the roughly-5%-of-the-time false-alarm band purely by chance, even with no real regression, or, more rarely, miss a real one. Require the same candidate build to fail the oracle on a majority of several independent re-samples (for example, 3 of 5) before declaring a confirmed regression, rather than acting on a single p-value below threshold. This trades a small amount of detection latency (you need several re-samples, not just one) for a large reduction in both false alarms and missed detections.

Worked example

The oracle, and validating it on a known-null synthetic dataset (both samples drawn from the identical distribution, so there is truly no regression by construction):

from scipy import stats
import numpy as np

ALPHA = 0.05

def ks_oracle(expected_scores, candidate_scores, alpha=ALPHA):
    stat, p_value = stats.ks_2samp(expected_scores, candidate_scores)
    # bool(...) matters: p_value < alpha on a numpy scalar returns np.bool_, which
    # reprs as np.True_ and quietly makes the oracle's output non-JSON-serializable.
    return {"ks_statistic": float(stat), "p_value": float(p_value),
            "flag_regression": bool(p_value < alpha)}

rng = np.random.default_rng(seed=123)
n_trials, n_samples = 500, 300
false_positives = 0
for _ in range(n_trials):
    expected = rng.beta(2, 5, size=n_samples)   # models a ranking model's relevance scores
    candidate = rng.beta(2, 5, size=n_samples)  # SAME distribution, independent draw: no real regression
    if ks_oracle(expected, candidate)["flag_regression"]:
        false_positives += 1

fp_rate = false_positives / n_trials
print(f"trials: {n_trials}, samples/trial: {n_samples}")
print(f"empirical false-positive rate: {fp_rate:.3f}  (target: ~{ALPHA:.3f})")
print(f"within [0.02, 0.08] (a reasonable sanity band around alpha={ALPHA}): {0.02 <= fp_rate <= 0.08}")

Actual output:

trials: 500, samples/trial: 300
empirical false-positive rate: 0.040  (target: ~0.050)
within [0.02, 0.08] (a reasonable sanity band around alpha=0.05): True

The oracle's empirical false-positive rate (4.0%) lands close to the target 5% significance level, which is exactly what a correctly-built oracle should do on a known-null dataset, neither near 0% (which would mean the test has essentially no power) nor far above 5% (which would mean it is over-flagging).

Now the same oracle on a known-regressed synthetic dataset (candidate distribution deliberately shifted from Beta(2,5) to Beta(4,4), moving probability mass toward higher scores):

detected, example = 0, None
for _ in range(50):
    expected = rng.beta(2, 5, size=n_samples)
    candidate = rng.beta(4, 4, size=n_samples)  # genuinely different distribution
    result = ks_oracle(expected, candidate)
    if example is None:
        example = result
    if result["flag_regression"]:
        detected += 1

print(f"one example trial: ks_statistic={example['ks_statistic']:.4f}, "
      f"p_value={example['p_value']:.2e}, flag_regression={example['flag_regression']}")
print(f"detection rate (statistical power) over 50 trials: {detected / 50:.3f}")

Actual output:

one example trial: ks_statistic=0.4667, p_value=7.06e-30, flag_regression=True
detection rate (statistical power) over 50 trials: 1.000

With a genuine distribution shift this large, the oracle detects it on every single trial, confirming real statistical power, not just a low false-positive rate on the null case.

Finally, the repeated-trial rule for run-to-run variance, requiring 3 of 5 independent re-samples to agree before confirming a regression:

def repeated_trial_oracle(expected_scores, sample_candidate_fn, n_repeats=5, majority=3):
    flags = [ks_oracle(expected_scores, sample_candidate_fn())["flag_regression"] for _ in range(n_repeats)]
    return {"flags": flags, "regression_confirmed": sum(flags) >= majority}

baseline = rng.beta(2, 5, size=n_samples)
null_case = repeated_trial_oracle(baseline, lambda: rng.beta(2, 5, size=n_samples))
regressed = repeated_trial_oracle(baseline, lambda: rng.beta(4, 4, size=n_samples))
print(f"no real regression, 5 resamples: flags={null_case['flags']}, confirmed={null_case['regression_confirmed']}")
print(f"real regression,    5 resamples: flags={regressed['flags']}, confirmed={regressed['regression_confirmed']}")

Output:

no real regression, 5 resamples: flags=[False, False, False, False, False], confirmed=False
real regression,    5 resamples: flags=[True, True, True, True, True], confirmed=True

Trade-offs and pitfalls

• The KS test is sensitive to shape differences generically, which is a strength and a limitation at once. It will flag a shift in variance or skew just as readily as a shift in mean, which is usually what you want for a regression detector, but it also means the p-value alone does not tell you which kind of change occurred; pair it with a specific summary statistic (mean, a chosen quantile) when you need to communicate what actually changed, not just that something did.
• A larger sample size makes the KS test more powerful, which can become a problem, not just a benefit. With enough samples, the test will eventually flag arbitrarily small, practically meaningless distribution differences as statistically significant. Pair the p-value threshold with a minimum effect-size threshold (a floor on the KS statistic itself, not just the p-value) so trivial differences do not trigger the majority-vote rule as readily as meaningful ones.
• Validating the oracle on synthetic data proves the oracle's statistical machinery is correct, not that the real pipeline's assumptions hold. The synthetic validation above assumes independent, identically distributed samples; a real production pipeline can violate that (autocorrelated traffic, a slowly drifting input distribution during the sampling window itself), which can inflate the real-world false-positive rate above what the synthetic validation measured. Re-validate periodically against a realistic traffic sample, not only once at build time.
• The majority-vote rule adds latency to detecting a real regression, and that cost is a genuine trade-off, not a free win. Five independent re-samples take five times as long to collect as one; for a regression severe enough to matter immediately (a fully broken pipeline), a single very-low p-value trial is often already conclusive enough to act on without waiting for the full majority vote, so a practical oracle typically short-circuits to "confirmed" immediately on an extreme statistic rather than always waiting for all five.

Direct answer

Structure the test strategy as a funnel of increasingly expensive, increasingly realistic checks: cheap, fast, deterministic tests run on every pull request and block the merge; slower, statistical tests run on a schedule against a larger held-out sample; and full production validation (shadow traffic, then a small live canary) runs only on release candidates, since that stage is the only one that measures real user-facing impact but also the only one with real cost and real risk. The cadence of both testing and retraining should be set from the same question: how fast does the world the model describes actually change, not from a fixed calendar habit like "we retrain every Monday."

Structured elaboration

The speed/quality/cost funnel, stage by stage.

Why the probabilistic output specifically changes the fast-tier design. For a model producing probabilities rather than hard labels, "does the output look reasonable" cannot be a single fixed expected value the way it can for a deterministic function; the fast tier needs a property check (is this a valid probability distribution, does the highest-probability class match a small set of known-unambiguous golden examples) rather than an exact-match assertion, since two legitimate model versions can differ slightly in their exact probability values while both being correct.

Cost trade-offs of retraining and test-execution cadence. These are two different cadences with two different cost drivers, and conflating them is a common mistake:

• Test-execution cadence trades CI (continuous integration, the automated pipeline that runs checks on every proposed change) wall-clock time and compute cost against how quickly a regression is caught. Running the full statistical-oracle suite on every commit instead of nightly catches a regression a day earlier, at the cost of running an expensive statistical comparison dozens of times a day instead of once; most teams land on "fast tier on every commit, expensive tier nightly or per-release-candidate" because the fast tier alone already catches the majority of defects (data leakage, a broken shape, a NaN, short for not-a-number, an invalid floating-point value) cheaply, and the expensive tier's marginal catch rate does not justify its cost at a higher frequency.
• Retraining cadence trades the cost of training compute against how stale the model's view of the world is allowed to get, worked through with real numbers below.

Where to put the retraining decision in the funnel. Retraining should never happen automatically off the calendar alone; a new candidate model, however it was triggered (schedule, drift alert, or a manual retrain), still has to pass through the exact same funnel above, pre-merge checks, scheduled statistical checks, shadow, then canary, before it replaces the production model. Treating "retrained on schedule" as equivalent to "validated and safe to serve" is exactly the assumption a rigorous test strategy exists to remove.

Worked example

A concrete illustration of the retraining-cadence trade-off's shape (hypothetical numbers, chosen to be easy to verify by hand, not a measured result from any real system): suppose a single retraining run costs 10 GPU-hours.

$\text{monthly cost}_{\text{weekly}} = 10 \times 4 = 40 \text{ GPU-hours}$
$\text{monthly cost}_{\text{daily}} = 10 \times 30 = 300 \text{ GPU-hours}$
$\text{ratio} = \frac{300}{40} = 7.5$

(Those two lines round the month differently, 4 weeks on one and 30 days on the other. On a consistent 30-day month the weekly cadence is 30/7 = 4.3 retrains, or 42.9 GPU-hours, and the ratio is 7.0 rather than 7.5. Nothing below turns on which rounding you pick, but it is worth being able to say which one you used.)

The benefit side needs its own arithmetic rather than an eyeballed number, because it is the half of this comparison that is easiest to overstate. A prediction request arrives at a uniformly random moment between two retrains, so the model's average staleness is half the retraining interval:

$\text{average staleness} = \frac{\text{interval}}{2} \;\Rightarrow\; \text{weekly} = \frac{7}{2} = 3.5 \text{ days}, \quad \text{daily} = \frac{1}{2} = 0.5 \text{ days}$
$\text{average freshness gain} = 3.5 - 0.5 = 3 \text{ days}$

Six days is a real number about this system, but it is a different one: it is the worst-case gap, comparing the moment just before each retrain, when the weekly model is 7 days stale and the daily model is 1. Quoting that as the average silently doubles the benefit side of the trade, which is the kind of error that flips a cost decision without anyone noticing. Dividing the two sides gives the unit the decision actually needs:

$\frac{300 - 40 \text{ GPU-hours}}{3 \text{ days}} \approx 87 \text{ GPU-hours per day of average freshness gained}$

So the move from weekly to daily buys three days of average freshness at roughly 87 GPU-hours each. Whether that 7.5x is worth paying depends entirely on how fast the underlying data distribution actually drifts: a fraud model facing an adaptive adversary might need daily retraining to stay ahead of attackers actively probing for the current decision boundary, while a model predicting a slow-moving demographic trend would see no measurable quality gain from anything faster than monthly, making the extra compute pure waste for that specific model. The right cadence is the one where the marginal cost of one more retrain equals the marginal value of the freshness it buys, not a cadence chosen because it is a round number on a calendar.

Trade-offs and pitfalls

• A fast tier that is too permissive defeats the purpose of having a funnel at all. If the pre-merge checks let obviously broken candidates through because they only check for a valid probability distribution and nothing else, every regression gets pushed to the expensive, slow, less frequent tier, which is the opposite of the intended cost structure; the fast tier needs at least a few golden-input exact-behavior checks alongside the property checks, not property checks alone.
• Shadow testing catches a different class of bug than a canary does, and skipping straight to canary to save time is a common, costly shortcut. Shadow traffic is safe (nothing is served to real users) and lets a comparison run at full traffic volume, which surfaces the rare-input bugs a limited canary slice might miss for weeks; a canary is the only stage that measures real business-metric impact, which shadow cannot. Neither stage substitutes for the other.
• Retraining cadence set purely by calendar habit, rather than by measured drift, wastes compute in the common case and under-reacts in the rare case that actually matters. A drift-triggered retraining policy (retrain when a monitored input or prediction distribution crosses a threshold, not on a fixed schedule) generally beats a fixed calendar cadence on both cost and responsiveness, at the cost of needing a reliable drift-detection system in the first place, which is itself something that has to be tested and monitored.
• The cost side of this trade-off is not only compute. A daily retraining cadence also means daily exposure to the full validation funnel's risk surface (a bad candidate reaching canary more often), and daily operational load on whoever is on call for release approvals; a slower, better-justified cadence can be the right call even when compute cost alone would favor going faster.