feat: implement Pillars 2-4 and fix a production false-positive in Pillar 1

Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as
beaconing: on short captures, grid periods longer than the observed
span never complete a cycle, so significance saturates to 1 and the
score sticks at exactly 0.5. v1.2 caps the grid at a third of the
observed span, requires a minimum number of completed cycles, and
excludes protocols that are periodic by nature (NTP, DNS, mDNS,
ICMP). Validated against 300 seeds per scenario: false positives on
bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300,
with every beacon-detection scenario scoring unchanged.

Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented
TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling
via Shannon-entropy information volume per subdomain per hour; the
Jensen-Shannon divergence was measured and does not separate tunnels
from benign CDN subdomains, so it stays in the evidence at zero
weight instead of in the score), and Pillar 4 (deterministic
noisy-OR fusion across the three pillars).

The fusion formula was validated against 8 labeled real-traffic
vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known
limitation is kept, not hidden: a benign Philips HUE bulb's strict
periodic polling and a 21.9h session score it above three real C2
hosts. No weight or threshold was tuned to mask this; it is
documented in the architecture docs instead.

Also fixes real bugs found along the way: ingest_dir's partition
count summed file names instead of distinct dates, DuckDB errors
were counted as bad rows, /api/alerts always returned an empty list
regardless of the last analysis, and the dashboard still claimed
every pillar was unimplemented.

Restores four design corrections that were missing from
docs/architecture.md (KL to Jensen-Shannon, the persistent-session
detector, the JA4 licensing boundary, and deterministic reduction
order), and catalogs the public datasets used for validation
(IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256
checksums under research/validation/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD

research/validation/p3real.py

@@ -0,0 +1,18 @@
+import sys, os, importlib.util
+here = os.path.dirname(os.path.abspath(__file__))
+sp = importlib.util.spec_from_file_location('p3lib', os.path.join(here, 'p3lib.py')); L = importlib.util.module_from_spec(sp); sp.loader.exec_module(L)
+def read(path):
+    fields = None
+    for line in open(path, errors='replace'):
+        line = line.rstrip('\n')
+        if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        yield dict(zip(fields, line.split('\t')))
+for path in sys.argv[1:]:
+    pairs = [(r['id.orig_h'], r['query']) for r in read(path) if r.get('query') not in (None, '-', '')]
+    g = L.groups_from(pairs); f = L.features(g)
+    top = sorted(f.items(), key=lambda kv: -kv[1]['U'])[:6]
+    print(f"== {path}: queries={len(pairs)} groups={len(f)}")
+    for k, v in top: print(f"  {k} U={v['U']} H={v['H']:.2f} JSD={v['JSD']:.3f} meanlen={v['L']:.1f} maxlabel={v['maxlab']}")
+    hj = sorted(f.items(), key=lambda kv: -kv[1]['JSD'])[:3]
+    for k, v in hj: print(f"  topJSD {k} U={v['U']} H={v['H']:.2f} JSD={v['JSD']:.3f} meanlen={v['L']:.1f}")