Network Troubleshooting and Diagnostics Questions
Systematically diagnosing network problems: a layered troubleshooting methodology, diagnostic tools and commands (ping, traceroute, tcpdump, packet capture), root-cause analysis, and connectivity and user-access issues. Covers isolating faults across the stack, reading packet-level evidence, and driving from symptom to root cause. The diagnostic discipline that spans all networking layers.
A financial firm's multicast market feed stopped reaching subscribers after a topology change. Describe how you would validate IGMP/MLD membership, PIM-SM or PIM-DM state, RPF checks, Rendezvous Point configuration, and how to inspect multicast routing tables ('show ip mroute') and IGMP snooping. Include steps to capture and analyze multicast traffic for subscribers.
Sample Answer
Direct answer
A multicast market-data feed failing after a topology change is diagnosed by walking the multicast control plane in order: confirm IGMP/MLD group membership is still correctly registered from subscribers, confirm the PIM (Protocol Independent Multicast) state that builds the actual distribution tree, and confirm the RPF (Reverse Path Forwarding) check, the mechanism that prevents loops, still passes given the NEW topology, since a topology change specifically can break RPF even when everything else about multicast configuration is untouched.
Structured elaboration
- Confirm IGMP/MLD membership is still current from subscribers:
show ip igmp groups(or the MLD/IPv6 equivalent) on the router closest to subscribers confirms they're still correctly signaling interest in the multicast group; if a topology change moved subscribers to a DIFFERENT router or interface, and IGMP snooping/membership wasn't correctly re-established on the new path, subscribers may appear to have "unsubscribed" even though their own IGMP behavior didn't actually change. - Confirm PIM state reflects the NEW topology correctly:
show ip pim neighborandshow ip mroute(multicast routing table) confirm whether PIM has correctly rebuilt its adjacencies and distribution tree given the topology change; PIM-SM (Sparse Mode) versus PIM-DM (Dense Mode) behave differently here, PIM-SM depends on correctly locating and maintaining state through a Rendezvous Point (RP), so also confirm the RP's own reachability and configuration are still correct post-change. - Confirm the Rendezvous Point configuration specifically, for PIM-SM: if the topology change altered the path to the RP (or, in a worse case, the RP itself is now unreachable due to the topology change), the entire PIM-SM tree-building process for NEW or re-established memberships can fail, even if existing state briefly persisted before finally timing out.
- Check the RPF check specifically, since topology changes are its classic failure trigger: multicast forwarding validates that traffic for a given source arrives via the INTERFACE that unicast routing would use to reach that source (the RPF check), specifically to prevent loops; a topology change that altered the UNICAST routing path to the multicast source, without a corresponding update propagating correctly through the multicast routing state, can cause the RPF check to fail, silently dropping otherwise-valid multicast traffic even though IGMP and basic connectivity look fine.
- Inspect multicast routing tables directly (
show ip mroute) for the specific group, checking the incoming interface list against what unicast routing NOW says the correct path to the source should be, which directly reveals an RPF mismatch if one exists. - Capture and analyze multicast traffic directly at points along the expected (and actual) distribution path to confirm definitively whether traffic is being correctly forwarded, dropped due to RPF failure, or simply never reaching a specific segment due to an IGMP/PIM state issue further upstream.
Worked example
show ip igmp groups confirms subscribers are still correctly signaling membership for the affected multicast group; that part of the chain is healthy. show ip mroute for the specific group shows the multicast routing entry's incoming interface no longer matches what show ip route (unicast routing, for the source's address) says the correct path should be, following the topology change; this mismatch is a direct RPF failure, meaning traffic arriving via the actual (now-correct, per updated unicast routing) path is being REJECTED by multicast forwarding because it doesn't match the STALE incoming-interface expectation still held in the multicast routing table. Clearing and allowing the mroute entry to correctly rebuild against the current, accurate unicast routing state resolves the RPF mismatch and restores the feed.
Trade-offs & pitfalls
It's easy to focus purely on IGMP (since that's the layer closest to and most visible from the subscriber's perspective) and miss that the topology change's actual impact is on RPF, a mechanism most engineers who don't work with multicast daily aren't as familiar with reasoning about; always check the multicast routing table's incoming-interface expectation against CURRENT unicast routing specifically after any topology change, since RPF mismatches are exactly the kind of failure a topology change is prone to introduce, even when every other piece of multicast configuration remains completely untouched.
Users report packet loss but interface counters on involved devices show no errors or drops. Describe advanced areas to investigate: per-queue egress drops/tail drops, microbursts leading to transient drops, QoS shaping/policing, bufferbloat and large buffers increasing latency, hardware offload masking counters, and how to gather high-resolution telemetry (ASIC counters, per-queue stats) to find the root cause.
Sample Answer
Direct answer
When packet loss is reported but interface counters on the involved devices show no errors, look above and below where standard counters measure: transient microbursts and per-queue tail drops that come and go faster than a counter's polling interval can capture, QoS shaping or policing discarding traffic by policy rather than by fault, and hardware-level buffering behavior (bufferbloat, offload features) that hides the real picture from a simple errors/drops counter.
Structured elaboration
- Understand what standard interface counters actually measure, and their blind spot: most polled counters (SNMP or show interface) sample at intervals of seconds; a microburst that fills a queue and causes a tail drop for a few milliseconds, then clears, can produce zero visible increment in a counter polled every 30 or 60 seconds, even though real packets were genuinely dropped.
- Look for per-queue, high-resolution telemetry instead: many modern switch ASICs expose per-queue drop counters (as opposed to aggregate interface-level counters) at a finer resolution; if available, these can reveal drops on a specific priority queue that never surface in the aggregate interface statistics.
- Check whether QoS shaping or policing is discarding traffic by design, not by fault: a policer enforces a committed rate and deliberately discards or remarks traffic above that rate at ingress, usually incrementing a policy-specific counter (a conform/exceed/violate counter on the policy-map) rather than the generic interface error/drop counter an engineer checks first; a shaper, by contrast, delays and queues excess traffic rather than dropping it outright, so it manifests as added latency and jitter rather than loss, unless its own buffer also overflows. A recent QoS policy change (a lowered committed rate, or a class reclassified into a stricter policer) is a common, entirely policy-driven cause of loss that will never show up as an interface error.
- Consider bufferbloat as a related but distinct pattern: an oversized buffer does not drop packets outright, but holds them long enough to inflate latency dramatically under load; this can look like loss to an application with a tight timeout (the packet was never actually dropped, but arrived too late to be useful), so distinguish true loss from excessive queuing delay using timestamps, not just counters.
- Check for hardware offload masking the real picture: some NICs and switch ASICs handle certain processing (checksums, some queueing decisions) in hardware in ways that are not reflected in the counters the OS or standard management interface exposes; a discrepancy between what the application experiences and what standard counters report can be a sign that the relevant activity is happening below where those counters look.
- Correlate timing precisely: gather the highest-resolution telemetry available (ASIC-level counters, per-queue stats, or policy-map conform/exceed/violate counters if accessible) and correlate the exact timestamps of reported application-level loss against any spike in queue depth, utilization, or policing activity at that same moment, even a spike too brief for a standard 30-second poll to register.
Worked example
An application reports occasional lost requests. Standard show interface counters on every device in the path show zero errors or drops over the reporting period. The policy-map attached to that egress interface, however, shows a nonzero and growing exceed counter under a QoS policer applied to this traffic class; a recent change lowered the committed rate for that class as part of a broader capacity reallocation. Enabling per-queue statistics on the relevant egress interface (polled every 1 second instead of every 60) corroborates this, showing brief spikes where the policed class's queue hits its maximum and experiences tail drops lasting under two seconds, precisely correlated with the timestamps of the application's reported failures; neither the standard 60-second interface counters nor a naive check of the interface's own drop counter would have surfaced this, since the drop is a deliberate policy action recorded in a QoS-specific counter.
Trade-offs & pitfalls
'The counters are clean' is often treated as proof there is no network-side loss, but standard interface counters have a real, specific blind spot for both short-duration events and policy-driven drops recorded elsewhere (in QoS policy-map counters, not the interface's own error/drop counters); before concluding the network is innocent, confirm you have looked at the highest time-resolution telemetry actually available on that hardware, and at any QoS policy applied to the affected traffic class, not just the generic interface counters. Distinguishing true drops (tail drop, policing) from bufferbloat-induced delay matters because the fixes are different: one calls for capacity, queue-management, or policy-rate changes, the other for buffer-sizing and queue-discipline tuning.
A production OSPF network shows repeated SPF recalculations causing transient reachability issues in area 1. Outline a troubleshooting approach with specific commands/telemetry to collect, likely root causes (LSA flaps, flapping links, unstable adjacencies), and mitigation steps (timers, interface dampening, BFD). Include how you would validate your fix without risking broader outages.
Sample Answer
Direct answer
Repeated SPF (Shortest Path First) recalculations causing transient reachability issues in one OSPF area point at something in that area generating frequent LSAs (Link-State Advertisements), each one forcing every router in the area to rerun SPF. Find the specific source of instability generating those LSAs using named commands and telemetry, rather than treating the symptom (frequent SPF runs) as itself the thing to fix.
Structured elaboration
- Confirm SPF recalculation frequency and correlate with LSA activity:
show ip ospf statisticsshows the SPF run history and the trigger reason for each run;show ip ospf databaseshows the current LSAs and their sequence numbers/age, and running it repeatedly (or diffing successive outputs) reveals which specific LSA is being regenerated frequently. - Check for flapping links specifically:
show interfaces(checking the "last input/output," "carrier transitions," and error counters) andshow logging | include %LINK|%LINEPROTOon the suspect interface reveal repeated up/down transitions directly. - Check for unstable adjacencies short of full flapping:
show ip ospf neighborrepeated over time, orshow logging | include OSPF-5-ADJCHG, reveals an adjacency repeatedly transitioning through intermediate states without ever fully going down. - Tune SPF-throttling timers as a mitigation, understanding its trade-off:
timers throttle spf <start-ms> <hold-ms> <max-ms>(for exampletimers throttle spf 200 1000 5000) delays how quickly SPF reruns after an LSA and backs off further if LSAs keep arriving, reducing the immediate CPU/instability impact; this is appropriate as a stopgap while investigating, not as a substitute for fixing the actual source. - Apply interface dampening to the specific flapping interface:
dampeningunder interface configuration (or the platform-equivalent, e.g.interface GigabitEthernet0/1thendampening) suppresses further LSA generation for a penalty period after repeated flaps on that one link, scoped to the identified source rather than the whole area. - Consider BFD (Bidirectional Forwarding Detection) as a complementary tool:
bfd interval 300 min_rx 300 multiplier 3under the interface, combined withip ospf bfdto bind OSPF to that BFD session, provides fast, deterministic failure detection so a genuinely failing link fails over cleanly once to a stable backup path rather than repeatedly flapping on a marginal primary one. - Validate any fix without risking a broader outage: apply dampening or timer changes to the specific identified flapping link/interface first, in a controlled way, and confirm via
show ip ospf statisticsthat SPF recalculation frequency actually drops as a direct result, rather than applying an area-wide timer change that could mask a real, future instability elsewhere.
Worked example
show ip ospf database run at short intervals shows one specific link's LSA (between two routers on the edge of area 1) incrementing its sequence number roughly every 90 seconds, correlated precisely with the reported SPF recalculation frequency shown in show ip ospf statistics. show interfaces on that link shows a nonzero, climbing "carrier transitions" counter, consistent with a marginal physical connection. Applying dampening to that specific interface immediately reduces SPF recalculation frequency across the whole area (confirmed again via show ip ospf statistics), while the physical link itself is scheduled for replacement as the durable fix; bfd interval 300 min_rx 300 multiplier 3 is also configured on the replacement link so any future failure is detected and failed over quickly rather than flapping.
Trade-offs & pitfalls
Tuning SPF timers or dampening broadly, across the whole area, without first identifying the specific flapping source via show ip ospf database and interface counters, treats the symptom everywhere rather than the cause at its actual location, and risks making the area's response to a genuinely new, real future instability slower than it should be. Identify the specific source first with the commands above, apply a targeted fix there, and reserve area-wide timer changes for cases where the instability is genuinely diffuse rather than traceable to one link.
A host can reach other systems but cannot reach its default gateway. Describe how to use ARP and interface commands (for example 'ip link', 'ip addr', 'ip neigh' or 'arp -n') to determine whether the issue is a Layer 2 problem (missing ARP entry, bad link) or a Layer 3 routing problem. Explain the outputs you expect in each case.
Sample Answer
Direct answer
When a host can reach other systems but not its default gateway, the fault is almost always local: either the host never learned the gateway's MAC address (a Layer 2 / ARP problem) or the routing table itself is wrong (a Layer 3 problem). The fastest way to tell these apart is to check the ARP cache before touching routing at all.
Structured elaboration
- Check the ARP/neighbor table first:
ip neigh show <gateway-ip>(orarp -non older systems). An entry in stateREACHABLEorSTALEwith a real MAC address means Layer 2 is fine and the problem is elsewhere (routing, or the gateway itself is down). An entry in stateFAILEDorINCOMPLETE, or no entry at all, means the host sent an ARP request and got no reply: a Layer 2 problem (bad cable/port, VLAN mismatch, the gateway's interface down, or a switch not forwarding broadcast/ARP traffic between the host and gateway). - Check the interface and link state:
ip link showshould showstate UPand a carrier. A down link explains missing ARP replies trivially, and rules out anything more subtle. - Check the routing table:
ip route showand confirm the host's subnet mask and default route point at the correct gateway IP, on the correct interface. A wrong subnet mask is a classic cause: the host computes the gateway as "on-link" when it is not (or vice versa), so it never even sends the ARP request that would establish reachability. - Force a fresh ARP resolution:
ip neigh flush <gateway-ip>followed by a ping will re-trigger ARP and let you watch the outcome cleanly, useful when a stale, wrong MAC is cached (for example after the gateway's NIC was replaced).
Worked example
Say ip route show reports default via 10.0.0.1 dev eth0 and ip addr show eth0 reports 10.0.0.55/25. The /25 mask covers 10.0.0.0-10.0.0.127, so .1 is correctly on-link and the host should ARP for it directly. Now ip neigh show 10.0.0.1 returns nothing, or FAILED. Because the route calculation is fine, this narrows the problem to Layer 2: check ip link show eth0 for carrier, and if that's up, suspect the switch port (wrong VLAN, port down, or a security policy dropping ARP) rather than anything on the host's IP configuration.
Trade-offs & pitfalls
The order matters: many engineers immediately start a packet capture, which is unnecessary overhead for what a single ip neigh command already answers. The most common mistake is assuming "can't reach the gateway" is a routing problem and jumping straight to route tables, missing that an incorrect subnet mask is itself a routing misconfiguration that manifests as an ARP failure. On a switch, also confirm the gateway's own interface and VLAN membership; a host-side fix cannot repair a gateway that has silently gone down or been moved to the wrong VLAN.
A set of firewall or security-group rules is intermittently blocking legitimate traffic, and you don't yet know which rule or which layer (stateful firewall, ACL, NAT) is responsible. Describe a systematic, minimally invasive way to narrow this down using captures and logs, and how your approach differs between an on-prem stateful firewall and a cloud model where security groups and network ACLs behave differently.
Sample Answer
Direct answer
When you suspect a firewall or security-group rule set is intermittently blocking legitimate traffic but do not know which specific rule, a controlled bisection, splitting the rule set in half and testing each half's effect, finds the offending rule in a bounded number of steps without needing to understand every rule up front. The mechanics of that bisection differ meaningfully between an on-prem stateful firewall and a cloud model split across security groups and network ACLs.
Structured elaboration
- Reproduce the failure reliably first: confirm you can trigger the block on demand (a specific request, from a specific source, at a specific time) before starting bisection; bisecting against a failure you cannot reliably reproduce will produce misleading results.
- Split the rule set and test each half: apply the first half of the current rule set in a test/staging path (or, if testing in production is unavoidable, during a low-traffic window with a fast rollback plan) and check whether the failure still occurs; whichever half reproduces the failure contains the offending rule, and you discard the other half from consideration.
- Repeat on the implicated half: continue splitting the remaining candidate rules in half each time; this converges to a single rule in roughly log2(N) test cycles rather than testing N rules one at a time.
- Use packet captures alongside bisection, not instead of it: a capture at each side of the firewall during a test cycle can confirm which packets are actually being dropped at that stage, corroborating (or correcting) what the bisection result implies.
- Adapt the mechanics to the platform: on an on-prem stateful firewall, the rule set is usually one ordered list evaluated top to bottom with implicit session/connection tracking, so bisecting means temporarily disabling or reordering blocks of rules on the same device, and a rollback is a single configuration revert on that device; because the firewall is stateful, return traffic is handled automatically once the forward direction is allowed. In a cloud model, you are usually bisecting across at least two independently evaluated layers, security groups (stateful, return traffic implicit) and network ACLs (stateless, every direction must be explicitly allowed), often owned by different teams; bisection needs to identify not just which rule but which layer first, since a security group change and a NACL change can look identical from the traffic's perspective but require editing entirely different resources. Cloud rule changes are frequently safer to test than on-prem ones because you can often clone a security group or duplicate a NACL and attach it to a single test instance or subnet without touching the shared, in-place resource that live traffic depends on.
- Roll back safely: keep every intermediate rule-set version so you can restore the last known-good state immediately if a test step causes unexpected impact, and always have an explicit, tested rollback path, not just a plan to revert if needed.
Worked example
A rule set of 32 rules is suspected on an on-prem firewall. Applying rules 1 through 16 only: failure still occurs. Applying rules 1 through 8: failure occurs. Applying rules 1 through 4: failure does not occur. Applying rules 5 through 8: failure occurs. Applying rule 5 alone: failure occurs. Five test cycles isolated the single offending rule out of 32, versus up to 32 cycles testing them one at a time, and because it is one on-prem device, each test cycle is a single config push and revert. Contrast this with the equivalent cloud case: the same intermittent block could be caused by either the instance's security group or the subnet's NACL. Here the first bisection step is not 'which rule' but 'which layer': temporarily attach a maximally permissive NACL to a cloned test subnet while leaving the security group untouched; if the failure disappears, the NACL (a stateless, easy-to-misconfigure return-path rule) is implicated, and only then do you bisect within that specific NACL's rule list, without ever needing to touch the security group or risk the shared production subnet.
Trade-offs & pitfalls
Bisection assumes the failure is caused by exactly one rule (or a small, testable subset); if two rules interact (each individually harmless, together causing the block), naive bisection can miss it, since removing either half individually may appear to fix the symptom while the real interaction goes undetected. This risk is higher in the cloud case specifically, because a security group rule and a NACL rule can interact (one allows the connection out, the other silently drops the stateless return), so a bisection that only ever tests one layer at a time can converge on a wrong or incomplete answer; test across both layers, not just within one. State this assumption explicitly, and if bisection does not converge cleanly, consider that you may be looking at an interaction rather than a single rule. Testing rule-set changes in production carries real risk on-prem, where there is often only one shared device; the cloud model's ability to clone a security group or NACL onto a single test resource lowers that risk considerably and should be preferred over testing directly against the shared production resource whenever it is available.
Unlock Full Question Bank
Get access to all Network Troubleshooting and Diagnostics interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.