Network Monitoring and Performance Questions

Direct answer

A threshold alert looks at one minute at a time, so it cannot see a trend. Instead, boil each week of one-minute data down to one robust "busy period" number, fit a trend line that outliers cannot bend, and turn the line into a date: "this link reaches its planning threshold in N days". Warn when N is smaller than the time it takes to buy and install more capacity. The output is a ticket with a date on it, not a page.

Why the alert never fires

Utilization here means bits per second on the link divided by link speed, per direction (a full-duplex link has its own capacity each way, so keep in and out as separate series). A cycle of day and night, plus weekends at roughly half the weekday load, moves the signal by tens of percentage points every day. A slow rise of 0.08 percentage points per day moves it by about 29 points in a year, yet no single minute need cross a 90 percent alert line. The example below reproduces this: the busiest minute of the whole year is 68.6 percent.

Steps

1. Build clean utilization. Interface octet counters only ever increase, so take the difference between consecutive polls divided by the interval, using the 64-bit counters (32-bit counters wrap in about 34 seconds at 1 Gbit/s: $2^{32} \times 8 / 10^9 \approx 34.4$ s). Drop the sample after a counter reset or device reboot. Leave missing minutes as gaps, never as zero, or the trend sinks.
2. Remove the seasonality by comparing like with like. Take one number per week from the weekday data only. Use the 95th percentile (p95: 95 percent of minutes are at or below it) rather than the mean, which hides the busy hours that matter, or the maximum, which one backup job can set.
3. Fit a robust trend. Use the Theil-Sen slope (the median of the slopes between every pair of weekly points). Ordinary least squares chases outliers, Theil-Sen does not. Confirm the trend is real with the Mann-Kendall test, which counts how many pairs of points rise versus fall (a large positive z means a monotonic upward trend, not noise). On four weekly values 10, 12, 11, 15 there are six pairs with slopes 2, 0.5, 1.67, -1, 1.5 and 4; the median is (1.5 + 1.67) / 2 = 1.58 points per week (the Theil-Sen slope), and five pairs rise and one falls, so the Mann-Kendall statistic S is 5 - 1 = 4.
4. Convert to a date and an uncertainty. Days to threshold = (threshold minus latest weekly p95) divided by slope. Use the median slope for the planning date and a higher-quartile slope (the slope that 75 percent of the pairwise slopes fall below) for the early warning, since a steeper slope gives an earlier date; the example computes both.
5. Choose the threshold and lead time from the business. The planning threshold (here 70 percent of busy-period p95, leaving room for bursts and for failover load) and the lead time (here 90 days to procure and install, plus a 30 day margin, so warn at under 120 days) are inputs, not constants.
6. Handle step changes. A customer migrated onto the link produces a jump, not a slope. Annotate known change events, and when the slope is dominated by one jump forecast from the level after it, not the slope across it. When a link is upgraded or members are added to a bundle, restart its history.
7. Back-test the rule. Replay it week by week on history (as below) and count how often it would have fired and how early. Across a fleet of links also require a minimum slope, since with enough links a few will pass the significance test by chance.

Worked example

Synthetic year for a 1 Gbit/s link: base load grows 0.08 points per day, a day/night cycle, weekends at 0.55 of weekday load, Gaussian noise. The script is complete and uses only the standard library.

import math, random, statistics
random.seed(57)
CAP = 1000.0  # Mbit/s link
DAYS = 365
GROWTH = 0.08  # percentage points of capacity per day added to the busy-hour level
samples = []   # (day, minute_of_day, util_pct)
for d in range(DAYS):
    weekday = d % 7
    wk = 0.55 if weekday in (5, 6) else 1.0
    base = 35 + GROWTH * d
    for m in range(1440):
        diurnal = 0.5 + 0.5 * math.sin((m - 600) / 1440 * 2 * math.pi)
        u = (base * (0.4 + 0.6 * diurnal) * wk) + random.gauss(0, 1.5)
        samples.append((d, m, max(0.0, min(100.0, u))))
print("points:", len(samples))

def p95(v):
    v = sorted(v); return v[int(math.ceil(0.95 * len(v))) - 1]

# 1. one busy-hour-style number per week: weekday p95 of the 1-min data
weekly = []
for w in range(DAYS // 7):
    vals = [u for (d, m, u) in samples if d // 7 == w and d % 7 < 5]
    weekly.append(p95(vals))
print("weeks:", len(weekly), "first/last weekly p95: %.1f %.1f" % (weekly[0], weekly[-1]))

# 2. Theil-Sen slope (median of pairwise slopes), robust to outliers
slopes = [(weekly[j] - weekly[i]) / (j - i) for i in range(len(weekly)) for j in range(i + 1, len(weekly))]
ts = statistics.median(slopes)
# 3. Mann-Kendall S statistic and z
S = sum((weekly[j] > weekly[i]) - (weekly[j] < weekly[i]) for i in range(len(weekly)) for j in range(i + 1, len(weekly)))
n = len(weekly); var = n * (n - 1) * (2 * n + 5) / 18
z = (S - 1) / math.sqrt(var) if S > 0 else 0
print("Theil-Sen slope: %.3f pct points/week, Mann-Kendall z: %.1f" % (ts, z))

# 4. forecast: weeks until weekly p95 reaches 70 and 90
last = weekly[-1]
for thr in (70, 90):
    print("weeks to %d%%: %.1f" % (thr, (thr - last) / ts))
# 4b. early warning: the upper-quartile (75th percentile) pairwise slope, a steeper but still plausible trend
fast = statistics.quantiles(slopes, n=4)[2]
print("upper-quartile slope: %.3f pct points/week -> days to 70%%: %.0f" % (fast, (70 - last) / fast * 7))
# 5. why a threshold alert never fired: max of any single minute vs a 90% alert
print("max 1-min util in year: %.1f" % max(u for _, _, u in samples))
# alert rule: fire when forecast days-to-70% < 120 (procurement lead time 90 days + 30 margin)
days_to_70 = (70 - last) / ts * 7
print("days to 70%%: %.0f -> warning=%s" % (days_to_70, days_to_70 < 120))

# 6. replay the rule week by week: slope from the trailing 12 weeks only
def ts_slope(v):
    return statistics.median([(v[j]-v[i])/(j-i) for i in range(len(v)) for j in range(i+1,len(v))])
for w in range(12, len(weekly) + 1):
    win = weekly[w - 12:w]
    sl = ts_slope(win)
    eta = (70 - win[-1]) / sl * 7 if sl > 0 else float("inf")
    if eta < 120:
        print("first warning after week %d: weekly p95 %.1f, slope %.2f/wk, ETA %.0f days" % (w, win[-1], sl, eta))
        break

Output:

points: 525600
weeks: 52 first/last weekly p95: 35.6 63.8
Theil-Sen slope: 0.552 pct points/week, Mann-Kendall z: 10.5
weeks to 70%: 11.3
weeks to 90%: 47.5
upper-quartile slope: 0.555 pct points/week -> days to 70%: 78
max 1-min util in year: 68.6
days to 70%: 79 -> warning=True
first warning after week 47: weekly p95 61.1, slope 0.55/wk, ETA 114 days

Reading the script block by block: the first loop builds a year of synthetic one-minute samples (525,600 of them); weekly keeps one weekday p95 per week; slopes holds every pairwise slope and its median is the Theil-Sen slope; S and z are the Mann-Kendall test; the for thr loop converts the slope into weeks to each level; the last loop replays the rule on a trailing 12 week window as if run each week.

Reading the output: 525,600 points collapse to 52 weekly values. The Theil-Sen slope is 0.55 points per week (the underlying growth is 0.08 x 7 = 0.56), z is 10.5 so the trend is not noise, and the forecast says the 70 percent line is 79 days away. The upper-quartile slope is 0.555, almost the same as the median, so the early-warning date is 78 days: this synthetic link grows steadily, and on a real link with noisier weeks the two slopes would spread further apart. Replaying the rule on a trailing 12 week window, the first warning appears after week 47 with 114 days of lead time, while the busiest single minute of the year (68.6 percent) never came near a 90 percent alert.

Trade-offs and pitfalls

• One-minute averages hide microbursts (very short spikes that overflow queues). Pair the capacity trend with output-drop counters, since a link can drop packets at 40 percent average utilization.
• A linear forecast fits steady growth. Seasonal businesses (a retail peak in December) need a year-over-year comparison, which is exactly why a year of data is valuable: compare this November's p95 with last November's.
• Report the forecast with its range, not as a single date, and say which link, current level, slope and evidence chart in the ticket so the owner can act without re-deriving it.
• Do not page on this. A page means act now, and a 79 day warning is planning work.

Direct answer

eBPF (extended Berkeley Packet Filter) is a Linux feature that loads small programs into the kernel; the kernel's verifier checks each one for safety (it must finish, and may not read outside its memory) before it runs when an event happens, here a packet passing a network hook. The programs keep results in maps, kernel tables that ordinary user-space programs can also read. The design therefore has two halves: tiny eBPF programs in the kernel do the per-packet work (count, timestamp, pick a histogram bucket), and one normal user-space program reads and resets the maps every 10 seconds and exports the numbers.

Attach tc (Linux traffic control) eBPF programs on the clsact hook, a tc attachment point with both an ingress side (packets arriving at the NIC, the network interface card) and an egress side (packets leaving it), so one program set sees data segments going out and acknowledgements (ACKs, the receiver's confirmations) coming back. Keep flow counters in a per-CPU LRU hash map (a table keyed by flow that discards the least recently used entry when full, with a separate copy per CPU core so cores never fight over one entry) and latency in a per-CPU histogram array (a fixed list of counters, one per latency range, each counting how many samples landed in that range; sub-bucketed log scale, 250 buckets reaching 10 seconds, worst midpoint error 6.25 percent). Under load, drop whole flows by hashing the 5-tuple (source address, destination address, source port, destination port, protocol), never histogram updates, and export counts of everything dropped. XDP (eXpress Data Path, the earliest ingress hook in the driver) is a fallback for ingress counting only.

Where to attach

XDP runs earlier (before the sk_buff is allocated, in native driver mode, meaning inside the NIC driver itself rather than after the kernel has wrapped the packet) but sees only ingress, so it cannot timestamp egress segments. Start with tc and move counting to XDP only if measurement says so.

What to aggregate in the kernel

• Flow table: BPF_MAP_TYPE_LRU_PERCPU_HASH keyed by 5-tuple, value packets, bytes, first time and last time (8 bytes each) and the OR of TCP flags (1 byte), 33 bytes of fields padded to 40. Per-CPU slots avoid cross-CPU contention. The LRU type evicts the least recently used entry at capacity, so inserts do not fail. Cost: memory is multiplied by CPU count. With 262,144 entries, a 16 B key (the 13 byte 5-tuple padded), 40 B values and 64 CPUs, that is 262,144 x (16 + 40 x 64) = 675,282,944 bytes (0.68 GB), and one million entries would be 2.58 GB, so size the table deliberately. Real maps add per-entry overhead on top, so treat this as a lower bound.
• Latency: at tc egress, store a timestamp keyed by flow and expected acknowledgement number (sequence number plus payload length). At ingress, when an ACK covers it, the difference is one sample. A sequence number is TCP's running count of bytes sent on a connection, so a segment that starts at sequence 1,000 and carries 500 bytes of payload is acknowledged by an ACK number of 1,500 (illustrative numbers). The egress program sees that segment leave at time 10,000,000 ns and stores 10,000,000 under the key (flow, 1500). The ingress program later sees an ACK for that flow with ACK number 1,500 at 10,250,000 ns, looks up the key, and subtracts: 250,000 ns, a sample of 250 microseconds, which goes into the histogram (the trace below follows exactly this value). Skip retransmitted segments (the ACK is ambiguous) and expire entries by age. Delayed acknowledgements inflate samples, so label the metric "ACK delay as seen at the NIC".
• Histogram: a BPF_MAP_TYPE_PERCPU_ARRAY (preallocated, one slot per CPU). User space sums the per-CPU values. Keyed by traffic class, not per flow, so memory stays bounded.
• Ring buffer (a one-way queue from the kernel to user space) only for rare events such as flow end. When it is full, reservation fails without blocking, so it cannot stall the datapath.

Accurate tails

Histograms with only powers of two are up to 50 percent off at midpoint. The version below splits each power of two into 8 linear sub-buckets. Each value picks its bucket in two steps: the position of its highest set bit says which power-of-two range (octave) it is in, and the next 3 bits below that bit pick one of 8 equal slices of the range. Trace 250,000 ns: in binary it has 18 bits (bit_length() is 18), so the highest set bit is bit 17 and octave is 17, the range 131,072 to 262,143. Shifting right by octave - 3 = 14 bits keeps the top 4 bits, 250,000 >> 14 = 15 (binary 1111); & 7 keeps the last 3 of them, giving sub = 7. The index is (17 - 2) x 8 + 7 = 127, matching the printed output below. Going back, bucket 127 covers (8 + 7) << 14 = 245,760 up to (9 + 7) << 14 = 262,144 ns, whose midpoint 253,952 is 1.6 percent from 250,000. Values below 8 ns just use themselves as the index. This script runs unchanged:

# Latency histogram: 8 linear sub-buckets inside each power-of-two range (values in nanoseconds)
def bucket_index(ns):
    if ns < 8:
        return ns
    octave = ns.bit_length() - 1          # position of the highest set bit
    sub = (ns >> (octave - 3)) & 7        # next 3 bits pick the sub-bucket
    return (octave - 2) * 8 + sub

def bucket_bounds(i):
    if i < 8:
        return i, i + 1
    octave = i // 8 + 2
    sub = i % 8
    return (8 + sub) << (octave - 3), (9 + sub) << (octave - 3)

worst = 0.0
for ns in list(range(8, 200000)) + [10**9, 3 * 10**9]:
    lo, hi = bucket_bounds(bucket_index(ns))
    assert lo <= ns < hi
    worst = max(worst, abs((lo + hi) / 2 - ns) / ns)
print("worst midpoint error for values from 8 ns up: %.2f%%" % (worst * 100))
print("bucket index of 250 us, 20 ms, 10 s: %d %d %d" % (bucket_index(250_000), bucket_index(20_000_000), bucket_index(10**10)))
print("buckets needed up to 10 s: %d" % (bucket_index(10**10) + 1))
print("per-CPU histogram bytes at 64 CPUs: %d" % ((bucket_index(10**10) + 1) * 8 * 64))
cpus, entries, value = 64, 262_144, 40
print("per-CPU flow table bytes: %d" % (entries * (16 + value * cpus)))

Output:

worst midpoint error for values from 8 ns up: 6.25%
bucket index of 250 us, 20 ms, 10 s: 127 177 249
buckets needed up to 10 s: 250
per-CPU histogram bytes at 64 CPUs: 128000
per-CPU flow table bytes: 675282944

Merge by adding counts, never by averaging percentiles. The per-CPU histogram is 250 x 8 x 64 = 128,000 bytes.

What to drop under load

1. New-flow detail: track only flows where hash(5-tuple) mod N is 0, so a tracked flow is complete. Keep global packet and byte counters for all traffic.
2. Pending-ACK entries older than the expiry.
3. Events.
   Never silently. Export the sampling fraction, map-insert failures and ring-buffer reservation failures with every export, as sFlow carries its sampling rate in each flow sample, so downstream can scale and see loss.

Budget and validation

At 3 Mpps over 8 receive queues each core sees 375,000 packets per second, so per-packet work is a few map operations and no loops over payload. Validate by replaying a capture whose latencies are known (compare against offline analysis with the same matching rule), then measure CPU with and without the programs at target rates, and test a link with a known added delay to confirm the histogram shifts by exactly that amount.

Trade-offs

Per-CPU maps trade memory for no contention. Per-flow sampling trades small-flow visibility for stability. Latency from ACKs measures the path plus the peer's ACK behaviour, not application latency.

Direct answer

A fixed threshold fires when a metric crosses a number you chose in advance (for example link utilization above 85 percent). Anomaly detection fires when a metric deviates from what is normal for that metric at that time (for example Tuesday 03:00 traffic at four times its usual level). Fixed thresholds are simple, predictable and right for limits that are physical or contractual, but blind to anything that is abnormal yet below the line. Anomaly detection catches those, but needs history, is harder to explain, and can quietly learn a problem as the new normal. For network metrics I would use both: fixed thresholds for hard limits and error counters, anomaly detection for volume and latency shapes with no natural limit.

Where each works and where each fails

Executable comparison

This script builds hourly utilization percent with a daily cycle (12 percent overnight, up to 40 percent in the early afternoon) plus Gaussian noise (standard deviation 1.5, seed 7). The anomaly detector compares each hour with the median of the same hour over the previous 14 days and alerts when the gap exceeds 4 times a robust spread, with a floor of 1.5 points. A robust spread is a measure of how much the values normally vary that, unlike the standard deviation, is not thrown off by a few extreme values. Here it is the median absolute deviation (the median of each value's distance from the median) multiplied by 1.4826, the factor that makes it match the standard deviation when the data is bell-shaped, so "4 times" reads like four standard deviations. The floor stops a very quiet hour from making the rule over-sensitive. The fixed rule alerts above 85 percent. The scenarios mirror the table: each one injects a fault into otherwise clean data and asks which detector fires.

import random, math, statistics as st

def series(days, ramp_per_day=0.0, seed=7):
    r = random.Random(seed)
    out = []
    for d in range(days):
        for h in range(24):
            daily = 12 + 28 * max(0.0, math.sin(math.pi * (h - 6) / 14)) if 6 <= h <= 20 else 12
            out.append(daily + ramp_per_day * d + r.gauss(0, 1.5))
    return out

def baseline(x, i, window_days=14):
    same = [x[i - 24 * k] for k in range(1, window_days + 1)]
    med = st.median(same)
    mad = st.median(abs(v - med) for v in same) * 1.4826
    return med, max(mad, 1.5)

def evaluate(x, start_day, fixed=85.0, k=4.0):
    fixed_hits, anom_hits = [], []
    for i in range(start_day * 24, len(x)):
        med, scale = baseline(x, i)
        if x[i] > fixed: fixed_hits.append(i)
        if abs(x[i] - med) / scale > k: anom_hits.append(i)
    return fixed_hits, anom_hits

def fmt(i): return f"day{i // 24:02d} {i % 24:02d}:00"

x = series(44)
f, a = evaluate(x, 14)
print("clean 30 days:", "fixed alerts", len(f), "| anomaly alerts", len(a))

x = series(44); i = 40 * 24 + 3; x[i] = 45.0
f, a = evaluate(x, 14)
print("03:00 job at 45%:", "fixed", [fmt(j) for j in f if j == i], "| anomaly", [fmt(j) for j in a if j == i])

x = series(44); i = 40 * 24 + 14; x[i] = 2.0
f, a = evaluate(x, 14)
print("14:00 collapse to 2%:", "fixed", [fmt(j) for j in f if j == i], "| anomaly", [fmt(j) for j in a if j == i])

x = series(150, ramp_per_day=0.5)
f, a = evaluate(x, 14)
print("ramp 0.5pt/day, 150 days: first fixed alert", fmt(f[0]) if f else None, "| anomaly alerts total", len(a))
print("util at first fixed alert: %.1f" % x[f[0]])

Output:

clean 30 days: fixed alerts 0 | anomaly alerts 0
03:00 job at 45%: fixed [] | anomaly ['day40 03:00']
14:00 collapse to 2%: fixed [] | anomaly ['day40 14:00']
ramp 0.5pt/day, 150 days: first fixed alert day85 13:00 | anomaly alerts total 22
util at first fixed alert: 86.8

Reading it. On clean data neither detector alarms, so the later results are signal. The 45 percent overnight job and the 2 percent collapse are invisible to the fixed rule (45 and 2 are both below 85) and obvious to the anomaly rule, which knows the 03:00 and 14:00 norms. The slow ramp reverses it: utilization rises half a point per day, the fixed rule fires cleanly on day 85 at 86.8 percent, a clear "this link is about to saturate" signal, while the anomaly detector produced 22 isolated alerts across the 136 evaluated days (the longest run was two consecutive hours) and never a sustained one. Its 14-day median baseline lags the ramp by about 3.75 points (the window's median is about 7.5 days old, and the ramp adds 0.5 points per day), under the threshold of at least 6 points, so the trend is absorbed as normal and only noise occasionally tips an hour over. The demo uses a daily cycle only. Real traffic also has a weekly cycle, so a production baseline must compare the same hour of the same weekday.

Recommendation and pitfalls

• Use fixed thresholds for anything with a physical or contractual ceiling and for counters that should be zero (discards, cyclic redundancy check or CRC errors). Require the condition to persist, for example 10 minutes, to avoid paging on one noisy sample.
• Use anomaly detection for volume and latency shapes, set a minimum absolute change as well as a statistical one so a quiet link going from 0.1 to 0.3 percent does not page, and send those alerts to a ticket or a review queue first until false positives (alerts when nothing is actually wrong) are measured.
• Cover the anomaly blind spot (slow drift) with a forecast or a fixed capacity line, as the ramp case shows.
• What would flip the choice: with no history (new site), start with fixed; with metrics that have no safe absolute number, lean on anomaly detection.

Direct answer

Collect five things: interface counters (traffic, errors, discards), device health (CPU, memory, temperature, power, fans), reachability, active probe results between sites (latency, jitter, loss), and syslog. Poll counters every 60 seconds, run probes every 10 seconds, and start with duration-based thresholds: warn at 70% link utilisation and alert at 85%, both sustained for 10 minutes; flag any rising error counter; flag discards above 0.1% of packets. These are starting values to tune against 30 days of your own baseline, not standards.

What to collect, by failure type

Why each choice:

• Utilisation must be read per direction. A full-duplex 1 Gbit/s link can carry 1 Gbit/s in each direction, so an input average of 40% and an output average of 90% means the output is the problem.
• Discards and errors differ. RFC 2863 defines errors as packets that contained errors preventing delivery, and discards as packets dropped even though no error was detected (typically a full queue). Errors suggest a physical fault; discards suggest congestion or policy.
• Interface counters cannot see loss inside a provider network. Only probes between your endpoints see it.
• Jitter needs a baseline, not a universal number. A path that is naturally variable (for example a satellite or a congested internet VPN) would alert constantly on a fixed value, so compare to its own history.
• A duration requirement stops flapping. One 60 s burst to 95% is normal; ten minutes is a trend.

Worked example

A 1 Gbit/s uplink shows these deltas over one 60 s poll: 6,000,000,000 output octets and 6,000,000 output packets, of which 9,000 were discarded; on the input side 4,000,000 packets and 600 errors.

• Output utilisation: 6,000,000,000 x 8 / 60 = 800,000,000 bit/s = 80%. Above the 70% warning level, but the rule needs 10 consecutive minutes, so one poll does not alert.
• Output discards: 9,000 / 6,000,000 = 0.15%, above the 0.1% threshold. If it persists for 10 minutes it alerts, and it is the stronger signal that the queue is overflowing.
• Input errors: 600 / 4,000,000 = 0.015%, above the 0.01% critical level, so this points to a physical fault on the receive side: the cable, the optic (the pluggable laser module in the port), or the transmitter of the device at the other end of the cable.

Trade-offs and pitfalls

• Alert on rates and ratios computed from two successive counter readings, never on the raw counter. A 32-bit octet counter wraps in about 34 seconds at 1 Gbit/s; use the 64-bit ifHC objects.
• Probe loss and interface errors do not always agree: run both and look at where they diverge.
• After a month, replace fixed numbers with per-link baselines (for example hour-of-week percentiles: for each link, keep the readings taken in the same hour of the same weekday over the past several weeks, such as every Tuesday 09:00 to 10:00, and alert when the current value is above, say, the 95th percentile of those readings, meaning higher than 95 out of every 100 of them), so a branch that always runs hot does not alert and a quiet link that suddenly doubles does.
• Do not alert on every port. Server-facing ports get status history; infrastructure links and uplinks get thresholds.

Direct answer

Design the alerting around dependencies, not devices. Give every alert a label that says what it depends on, make the failure of the upstream device suppress the "unreachable" alerts of everything behind it, group what remains, and page only on the cause. When the core switch fails, the network operations centre (NOC) should see one clear alert (core switch down) at the top, with the downstream unreachable devices still recorded but not notified. This is dependency-based suppression, called inhibition in Prometheus Alertmanager (the open-source alert router I use for the example).

The design, step by step

1. Topology as data. Pull parent relationships from the source of truth (the inventory or CMDB, the system that records which device connects to which). Each device's reachability alert gets a blocked_by label naming the upstream device whose failure would hide it. The core switch's own down alert carries its own name in the same label.
2. Separate cause from effect. Two alert types: DeviceDown for a device that fails its own health check from an independent vantage point (a place the probe runs from, other than behind that device), and DeviceUnreachable for devices whose probes fail. Only the cause class is allowed to inhibit, and the cause alert also carries a role label (here role="core") so that only devices of a role you trust as an upstream can suppress others.
3. Inhibition rule. A source alert (the cause) suppresses target alerts (the effects) when the labels listed in equal match, per the Alertmanager configuration reference.
4. Group and wait. Group by site and blocked_by, meaning alerts that share the same value of both labels are bundled into one notification. group_wait (default 30s) is how long to wait before the first notification for a new group, giving the cause alert time to arrive before the group is sent. group_interval (default 5m) spaces later notifications and repeat_interval (default 4h) controls reminders.
5. Route by severity. The cause alert pages; downstream effects appear on the dashboard and in the ticket with the incident, not in the pager.
6. Redundant paths. A device dual-homed (connected to two upstream devices) to two cores is unreachable only if both are down. Emit its blocked_by as the pair (for example core-pair-a) and have a rule that raises a pair-level cause alert only when both members are down. A single-valued label would otherwise suppress a real, separate failure.

Configuration and executed proof

route:
  receiver: noc
  group_by: [site, blocked_by]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
receivers:
  - name: noc
inhibit_rules:
  - source_matchers:
      - alertname="DeviceDown"
      - role="core"
    target_matchers:
      - alertname="DeviceUnreachable"
    equal: [blocked_by]

Reading the rule line by line. route sends everything to the noc receiver and bundles alerts by site and blocked_by (group_by), waiting 30 s before the first notification of a new group (group_wait), 5 minutes between later notifications for it (group_interval) and 4 hours between reminders (repeat_interval). inhibit_rules holds the suppression rule. source_matchers selects the alerts that are allowed to suppress: a DeviceDown alert whose role label is core. target_matchers selects the alerts that can be suppressed: any DeviceUnreachable. equal: [blocked_by] adds the condition that a suppression happens only when the source and the target carry the same value for blocked_by. So a DeviceDown for core-sw1 suppresses every DeviceUnreachable labelled blocked_by=core-sw1, but not acc-900, which is labelled blocked_by=core-sw2 and so is still reported.

I ran this on Alertmanager 0.34.1 in a container: amtool check-config reported SUCCESS. I then posted 300 alerts to /api/v2/alerts: one DeviceDown for core-sw1 (blocked_by=core-sw1), 298 DeviceUnreachable access switches with blocked_by=core-sw1, and one DeviceUnreachable (acc-900) behind a different core, core-sw2, that is still up. Result: 2 alerts not inhibited (core-sw1 and acc-900) and 298 inhibited; querying with inhibited alerts included still returned all 300, and grouping produced two groups (blocked_by=core-sw1 with 299 members, blocked_by=core-sw2 with 1). So the NOC gets 300 alerts reduced to 2 notifications, and the unrelated failure behind the healthy core was not hidden.

I then resolved the core-sw1 alert while leaving the other 298 firing: the number of non-inhibited alerts went from 2 to 299. That is the right behaviour (when the cause clears, anything still broken re-surfaces) and it means the NOC should expect a second round of alerts if devices stay down after the core returns.

Trade-offs and pitfalls

• Suppression hides symptoms by design. Keep inhibited alerts visible in the UI and the API (they were in the test) and review the dependency labels after every topology change, because a stale blocked_by either suppresses a real fault or lets the storm through.
• The cause must be detected independently of the path it sits on. If the only monitor sits behind the core, its loss looks like unreachability for everything, so monitor the core from at least two vantage points.
• Polling gaps create ordering races: children can fail a probe before the core is flagged, which is what group_wait covers. Do not shorten it to speed the first page.
• Keep a count of suppressed alerts on the incident so a post-incident review can see the blast radius (how many devices and users one failure touched; here 298 devices).