feat: implement Pillars 2-4 and fix a production false-positive in Pillar 1

Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as
beaconing: on short captures, grid periods longer than the observed
span never complete a cycle, so significance saturates to 1 and the
score sticks at exactly 0.5. v1.2 caps the grid at a third of the
observed span, requires a minimum number of completed cycles, and
excludes protocols that are periodic by nature (NTP, DNS, mDNS,
ICMP). Validated against 300 seeds per scenario: false positives on
bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300,
with every beacon-detection scenario scoring unchanged.

Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented
TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling
via Shannon-entropy information volume per subdomain per hour; the
Jensen-Shannon divergence was measured and does not separate tunnels
from benign CDN subdomains, so it stays in the evidence at zero
weight instead of in the score), and Pillar 4 (deterministic
noisy-OR fusion across the three pillars).

The fusion formula was validated against 8 labeled real-traffic
vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known
limitation is kept, not hidden: a benign Philips HUE bulb's strict
periodic polling and a 21.9h session score it above three real C2
hosts. No weight or threshold was tuned to mask this; it is
documented in the architecture docs instead.

Also fixes real bugs found along the way: ingest_dir's partition
count summed file names instead of distinct dates, DuckDB errors
were counted as bad rows, /api/alerts always returned an empty list
regardless of the last analysis, and the dashboard still claimed
every pillar was unimplemented.

Restores four design corrections that were missing from
docs/architecture.md (KL to Jensen-Shannon, the persistent-session
detector, the JA4 licensing boundary, and deterministic reduction
order), and catalogs the public datasets used for validation
(IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256
checksums under research/validation/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD
This commit is contained in:
Lucas RayanandClaude Sonnet 5 committed 2026-11-01 00:28:54 -03:00
1 parent 8a80504c45
commit c64bbbf775
41 files changed
+4034 -348

No files matched your search

+13
View File
@@ -0,0 +1,13 @@
Synthetic test fixture for Zeek idle TCP split verification.
Generated with:
1. python3 research/validation/mkpcap_idle.py /tmp/zeek_split_gen/pcap/idle.pcap
2. docker run --rm --user "$(id -u):$(id -g)" \
-v /tmp/zeek_split_gen/pcap:/pcap:ro -v /tmp/zeek_split_gen/out:/out -w /out \
zeek/zeek:9.0.1 zeek -C -r /pcap/idle.pcap
Contains one TCP session (10.0.0.5:40000 -> 203.0.113.9:443) split by Zeek 9.0.1 across a 600s idle gap into two conn.log lines:
- First line: ts=1793400000.000000, duration=1741.050000, history=ShADd
- Second line: ts=1793402401.000000, duration=1800.050000, history=DdFfA (no 'S' in history, gap = 659.95s <= 3600s)
Expected chained session: L = 4201.05s, parts = 2.
+11
View File
@@ -0,0 +1,11 @@
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2026-11-01-00-28-56
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents ip_proto
#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] count
1793400000.000000 Cr5nhu4rvbFFhLnlC7 10.0.0.5 40000 203.0.113.9 443 tcp - 1741.050000 600 1200 S1 T T 0 ShADd 32 1880 31 2440 - 6
1793402401.000000 Cn4avD2d83MCmnH4xl 10.0.0.5 40000 203.0.113.9 443 tcp - 1800.050000 600 1200 SF T T 0 DdFfA 32 1880 31 2440 - 6
#close 2026-11-01-00-28-56