feat: implement Pillars 2-4 and fix a production false-positive in Pillar 1

Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as
beaconing: on short captures, grid periods longer than the observed
span never complete a cycle, so significance saturates to 1 and the
score sticks at exactly 0.5. v1.2 caps the grid at a third of the
observed span, requires a minimum number of completed cycles, and
excludes protocols that are periodic by nature (NTP, DNS, mDNS,
ICMP). Validated against 300 seeds per scenario: false positives on
bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300,
with every beacon-detection scenario scoring unchanged.

Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented
TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling
via Shannon-entropy information volume per subdomain per hour; the
Jensen-Shannon divergence was measured and does not separate tunnels
from benign CDN subdomains, so it stays in the evidence at zero
weight instead of in the score), and Pillar 4 (deterministic
noisy-OR fusion across the three pillars).

The fusion formula was validated against 8 labeled real-traffic
vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known
limitation is kept, not hidden: a benign Philips HUE bulb's strict
periodic polling and a 21.9h session score it above three real C2
hosts. No weight or threshold was tuned to mask this; it is
documented in the architecture docs instead.

Also fixes real bugs found along the way: ingest_dir's partition
count summed file names instead of distinct dates, DuckDB errors
were counted as bad rows, /api/alerts always returned an empty list
regardless of the last analysis, and the dashboard still claimed
every pillar was unimplemented.

Restores four design corrections that were missing from
docs/architecture.md (KL to Jensen-Shannon, the persistent-session
detector, the JA4 licensing boundary, and deterministic reduction
order), and catalogs the public datasets used for validation
(IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256
checksums under research/validation/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD
This commit is contained in:
Lucas RayanandClaude Sonnet 5 committed 2026-11-01 00:28:54 -03:00
1 parent 8a80504c45
commit c64bbbf775
41 files changed
+4034 -348

No files matched your search

+10
View File
@@ -0,0 +1,10 @@
dataset url license file sha256
IoT-23 34-1 (Mirai) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-34-1/ CC BY 2.0 2018-12-21-15-50-14-192.168.1.195.pcap 92ec7e2f6658ee4b007d0b816986c46cc0338bc5e2bec6ceaaca566c695e4699
IoT-23 20-1 (Torii) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-20-1/ CC BY 2.0 2018-10-02-13-12-30-192.168.100.103.pcap 4d0a00c33a4bf11228158baddd47c38f702731ea8d75ef1201061fc8bbd878b9
IoT-23 8-1 (Hakai) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-8-1/ CC BY 2.0 2018-07-31-15-15-09-192.168.100.113.pcap 80dcc2602519479ddcde889fa902fee19a76696630811452f8df38888af894f2
IoT-23 42-1 (Trojan) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-42-1/ CC BY 2.0 2019-01-10-14-34-38-192.168.1.197.pcap 7573797f17aae96e3804a3cfb22d531b74c17bbae740de1e14048f89130fba37
IoT-23 4-1 (Philips HUE) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-Honeypot-Capture-4-1/ CC BY 2.0 2018-10-25-14-06-32-192.168.1.132.pcap.xz bc5a57306d9a5f614b09d9c0976f007667fee5b6de3427fbab094fa3084b9b64
CTU-Normal-20 https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-20/ unspecified 2017-04-30_win-normal.pcap c78a00728f115ab8fc495fd395894c43fd90cb61054eb9ef4f2d024658d5ffc5
CTU-Normal-18 https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-18/ unspecified 2017-09-05_capture-win4.pcap e09df1ac62d9753ed2360373915427432292192dc06e20572e45716a2932c8ce
CTU-Normal-26 https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-26/ unspecified 2017-04-28_normal.pcap b66103fd89c8f0fc53b76e718f53320f3af27edd302c58695fc1870ede3663c5
Mendeley dtqbc-m-test https://data.mendeley.com/public-api/datasets/mzn9hvdcxg CC BY 4.0 dtqbc-m-test.csv 48f109f501e01696e78d22623e4f7938784071219acbf8924ea2211f3473a4f8
1 dataset url license file sha256
2 IoT-23 34-1 (Mirai) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-34-1/ CC BY 2.0 2018-12-21-15-50-14-192.168.1.195.pcap 92ec7e2f6658ee4b007d0b816986c46cc0338bc5e2bec6ceaaca566c695e4699
3 IoT-23 20-1 (Torii) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-20-1/ CC BY 2.0 2018-10-02-13-12-30-192.168.100.103.pcap 4d0a00c33a4bf11228158baddd47c38f702731ea8d75ef1201061fc8bbd878b9
4 IoT-23 8-1 (Hakai) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-8-1/ CC BY 2.0 2018-07-31-15-15-09-192.168.100.113.pcap 80dcc2602519479ddcde889fa902fee19a76696630811452f8df38888af894f2
5 IoT-23 42-1 (Trojan) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-42-1/ CC BY 2.0 2019-01-10-14-34-38-192.168.1.197.pcap 7573797f17aae96e3804a3cfb22d531b74c17bbae740de1e14048f89130fba37
6 IoT-23 4-1 (Philips HUE) https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-Honeypot-Capture-4-1/ CC BY 2.0 2018-10-25-14-06-32-192.168.1.132.pcap.xz bc5a57306d9a5f614b09d9c0976f007667fee5b6de3427fbab094fa3084b9b64
7 CTU-Normal-20 https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-20/ unspecified 2017-04-30_win-normal.pcap c78a00728f115ab8fc495fd395894c43fd90cb61054eb9ef4f2d024658d5ffc5
8 CTU-Normal-18 https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-18/ unspecified 2017-09-05_capture-win4.pcap e09df1ac62d9753ed2360373915427432292192dc06e20572e45716a2932c8ce
9 CTU-Normal-26 https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-26/ unspecified 2017-04-28_normal.pcap b66103fd89c8f0fc53b76e718f53320f3af27edd302c58695fc1870ede3663c5
10 Mendeley dtqbc-m-test https://data.mendeley.com/public-api/datasets/mzn9hvdcxg CC BY 4.0 dtqbc-m-test.csv 48f109f501e01696e78d22623e4f7938784071219acbf8924ea2211f3473a4f8