feat: implement Pillars 2-4 and fix a production false-positive in Pillar 1

Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as
beaconing: on short captures, grid periods longer than the observed
span never complete a cycle, so significance saturates to 1 and the
score sticks at exactly 0.5. v1.2 caps the grid at a third of the
observed span, requires a minimum number of completed cycles, and
excludes protocols that are periodic by nature (NTP, DNS, mDNS,
ICMP). Validated against 300 seeds per scenario: false positives on
bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300,
with every beacon-detection scenario scoring unchanged.

Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented
TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling
via Shannon-entropy information volume per subdomain per hour; the
Jensen-Shannon divergence was measured and does not separate tunnels
from benign CDN subdomains, so it stays in the evidence at zero
weight instead of in the score), and Pillar 4 (deterministic
noisy-OR fusion across the three pillars).

The fusion formula was validated against 8 labeled real-traffic
vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known
limitation is kept, not hidden: a benign Philips HUE bulb's strict
periodic polling and a 21.9h session score it above three real C2
hosts. No weight or threshold was tuned to mask this; it is
documented in the architecture docs instead.

Also fixes real bugs found along the way: ingest_dir's partition
count summed file names instead of distinct dates, DuckDB errors
were counted as bad rows, /api/alerts always returned an empty list
regardless of the last analysis, and the dashboard still claimed
every pillar was unimplemented.

Restores four design corrections that were missing from
docs/architecture.md (KL to Jensen-Shannon, the persistent-session
detector, the JA4 licensing boundary, and deterministic reduction
order), and catalogs the public datasets used for validation
(IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256
checksums under research/validation/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD

docs/architecture.md

@@ -80,24 +80,106 @@ It is bounded in `[-1.0, 1.0]` by definition and resistant to extreme outliers.
 When `Q3 == Q1` (concentrated data), the implementation assigns `0.0` with a
 sentinel flag.
 
-### 4. Isolate the Python research workspace
+### 4. Jensen-Shannon divergence instead of Kullback-Leibler
+
+Kullback-Leibler divergence diverges to infinity when `Q(x) = 0` on non-empty `P(x)`.
+Acauã adopts the Jensen-Shannon Divergence (JSD): bounded in `[0, 1]`, symmetric,
+and strictly finite even on disjoint supports. Measured in real DNS tunneling and
+CDN traffic, JSD alone does not separate benign CDNs from tunnels; hence in v1 it
+is recorded in alert evidence with weight 0.0 rather than driving the score.
+
+### 5. Dedicated persistent session detector (Pillar 2)
+
+A long-lived interactive session or steady C2 channel does not generate frequent
+reconnections (`Δt`), making it invisible to point-process periodicity analysis.
+Acauã implements a dedicated persistent session detector with TCP session chaining
+that re-aggregates sessions split by Zeek's 5-minute inactivity timeouts.
+
+### 6. JA4 licensing boundary
+
+Only the original TLS client fingerprint JA4 is BSD-3 licensed. Other modules
+(JA4S, JA4H, JA4X, JA4SSH, JA4L) use the proprietary FoxIO License 1.1 (non-OSI).
+Acauã does not bundle or depend on FoxIO-licensed modules by default in an
+open-source architecture.
+
+### 7. Determinism and fixed reduction order
+
+Parallel reductions (e.g. via `rayon`) reorder floating-point additions and
+multiplications across threads, breaking bit-level reproducibility. All reductions,
+periodogram summations, and host evidence combinations are computed in a fixed,
+deterministic sequential order, producing byte-for-byte identical JSON evidence.
+
+### 8. Isolate the Python research workspace
 
 `research/` has its own virtual environment and dependencies. It is excluded
 from the Cargo workspace, ignored by Docker builds and CI releases, and tested
 only when research validation scripts are invoked directly.
 
-### 5. Documented degraded mode for NetFlow/IPFIX
+### 9. Documented degraded mode for NetFlow/IPFIX
 
 NetFlow v9 and IPFIX do not carry DNS query names, TLS SNI, certificate chains
 or HTTP headers. Any analysis that depends on protocol features is unavailable.
 Acauã documents this as a degraded mode and never claims parity between flow
 records and full Zeek logs.
 
-### 6. Single HTTP and RPC boundary
+### 10. Single HTTP and RPC boundary
 
 The daemon exposes an Axum API on loopback by default. The embedded web UI
 communicates with the daemon exclusively via standard HTTP calls.
 
+## Detection (v1)
+
+### Pillar 1: Low-and-slow C2 Beaconing (v1.2)
+Evaluates session starts per `(orig_h, resp_h, resp_p, proto)` using a Rayleigh/Schuster
+periodogram over an effective grid bounded by `span / 3` (requiring at least 3 full cycles
+observed). Significance is gated by dispersion regularity (`gate = min(1, regularity / 0.25)`),
+preventing false positives on bursty benign browsing. Alert threshold: `score >= 0.40`.
+Non-TCP/UDP protocols, multicast/broadcast addresses, and infrastructure service ports
+(e.g., NTP 123, DNS 53, mDNS 5353, NetBIOS, DHCP) are audited and excluded.
+
+### Pillar 2: Persistent Sessions (v1.0)
+Chains TCP connection segments across Zeek inactivity timeouts (gap <= 3600s with no `S`
+SYN flag in Zeek history). Persistence score is logarithmic:
+`persistence = clamp(ln(L / 3600) / ln(24), 0, 1)`. Sessions with `L >= 7200s`
+(`score >= 0.2181`) trigger an alert. Producer-Consumer Ratio (PCR), byte transfer rates,
+and failed connection attempts are captured in evidence.
+
+### Pillar 3: DNS Subdomain Information Capacity (v1.0)
+Measures the information payload capacity pushed across DNS queries by aggregating unique subdomains
+per `(orig_h, base_domain, utc_hour)`. Normalizes queries to lowercase, strips trailing dots, and excludes
+reverse lookups (`.in-addr.arpa`, `.ip6.arpa`), `.local`, and discovery ports (5353, 5355, 137):
+- Information capacity: $I = H \times C$ bits, where $H = -\sum p_c \log_2 p_c$ (Shannon entropy per char)
+  and $C$ is the sum of characters across unique subdomains.
+- Evaluated at peak hour: $I_{\max} = \max_h I$. Requires $U_{\text{peak}} \ge 10$ unique subdomains;
+  otherwise status is `insufficient_data` with score 0.0.
+- Scoring: $\text{score} = \text{clamp}(\log_2(I_{\max} / 8192) / \log_2(32), 0, 1)$ ($8\text{ kbit} \to 0.0$,
+  $32\text{ kbit} \to 0.40$, $256\text{ kbit} \to 1.0$).
+- Alert threshold: $\text{score} \ge 0.40$ ($\iff I_{\max} \ge 32768$ bits/h).
+- Jensen-Shannon Divergence (JSD) against the mixture of other active domains in the window is stored in
+  evidence with weight 0.0 (tunnels measured 0.157–0.193 vs. benign CDNs 0.226–0.227, inverting rank).
+
+### Pillar 4: Per-Host Evidence Fusion (v1.0)
+Combines alerts per `orig_h` using a weighted noisy-OR model across pillars 1, 2, and 3:
+- For each host, $m_i$ is the maximum alert score from pillar $i$ ($0.0$ if no alert exists).
+- Fixed weights $w_1 = w_2 = w_3 = 0.75$.
+- Score formula: $S = 1 - \prod_{i=1}^3 (1 - w_i m_i)$ evaluated in fixed sequential order $[1, 2, 3]$.
+- Properties: $S \in [0, 1]$, monotonic in every $m_i$, single pillar capped at $0.75$, two pillars capped at
+  $0.9375$, three pillars capped at $0.984375$. Invariant under alert input permutations.
+- Fusion ranks hosts in `AnalysisReport.hosts` by $(S \text{ desc}, orig\_h \text{ asc})$; it provides investigation
+  prioritization without fabricating synthetic alert classes.
+
+### Known Limitations
+1. In live captures, open sessions remain unobserved until Zeek flushes them on closure or timeout.
+2. Benign cloud polling and persistent services (e.g., smart home devices like Philips HUE, websockets, push notifications) exhibit genuine periodicity and persistence, ranking high in fused scores without network-wide context.
+3. Pillar 3 tunnel detection sensitivity is approximately 100 unique queries per hour.
+4. Base domain extraction uses an effective heuristic without a full embedded Public Suffix List (PSL).
+
+### Validation Datasets
+Public datasets used for empirical verification (detailed in `research/validation/datasets.tsv`):
+- **IoT-23** (captures 34-1, 20-1, 8-1, 42-1, Honeypot 4-1): CC BY 2.0 ground truth for C2 beaconing, persistence, and benign IoT controls.
+- **CTU-Normal** (captures 20, 18, 26): public captures representing benign browsing and Facebook traffic.
+- **Mendeley dtqbc-m-test** (Y. Bubnov): CC BY 4.0 dataset with real DNS tunneling payloads (dns2tcp, dnscapy, tuns).
+
 ## API
 
 The HTTP API is served by `acaua serve`: