feat: implement Pillars 2-4 and fix a production false-positive in Pillar 1

Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as
beaconing: on short captures, grid periods longer than the observed
span never complete a cycle, so significance saturates to 1 and the
score sticks at exactly 0.5. v1.2 caps the grid at a third of the
observed span, requires a minimum number of completed cycles, and
excludes protocols that are periodic by nature (NTP, DNS, mDNS,
ICMP). Validated against 300 seeds per scenario: false positives on
bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300,
with every beacon-detection scenario scoring unchanged.

Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented
TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling
via Shannon-entropy information volume per subdomain per hour; the
Jensen-Shannon divergence was measured and does not separate tunnels
from benign CDN subdomains, so it stays in the evidence at zero
weight instead of in the score), and Pillar 4 (deterministic
noisy-OR fusion across the three pillars).

The fusion formula was validated against 8 labeled real-traffic
vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known
limitation is kept, not hidden: a benign Philips HUE bulb's strict
periodic polling and a 21.9h session score it above three real C2
hosts. No weight or threshold was tuned to mask this; it is
documented in the architecture docs instead.

Also fixes real bugs found along the way: ingest_dir's partition
count summed file names instead of distinct dates, DuckDB errors
were counted as bad rows, /api/alerts always returned an empty list
regardless of the last analysis, and the dashboard still claimed
every pillar was unimplemented.

Restores four design corrections that were missing from
docs/architecture.md (KL to Jensen-Shannon, the persistent-session
detector, the JA4 licensing boundary, and deterministic reduction
order), and catalogs the public datasets used for validation
(IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256
checksums under research/validation/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD

CHANGELOG.md

@@ -6,13 +6,27 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versio
 
 ## [Unreleased]
 
+### Added
+- **Pillar 1 (v1.2):** Rayleigh/Schuster periodogram with grid bounded by the observed span ($T \le \text{span}/3$), regularity gating, and infrastructure exclusions (NTP, mDNS/LLMNR/NBNS, SSDP, non-TCP/UDP).
+- **Pillar 2 (v1.0):** Persistent TCP session detector with inactivity chaining (gaps $\le 3600$s without SYN), logarithmic duration scoring, and PCR / failed attempt metrics in evidence.
+- **Pillar 3 (v1.0):** DNS subdomain information capacity detector ($I = H \times C$ bits/h) to identify high-entropy tunnels, with Jensen-Shannon Divergence (JSD) recorded in evidence.
+- **Pillar 4 (v1.0):** Per-host weighted noisy-OR fusion ($w_1=w_2=w_3=0.75$) with deterministic evidence linkage, ranking hosts in `acaua analyze` human output via the `hosts:` section.
+- **Zeek `dns.log` ingestion:** Daily Hive Parquet partitioning for DNS queries with deterministic column mapping by TSV header.
+- **Parquet storage schema (v1.2):** 15-column schema for `conn.log` supporting `conn_state`, `history`, `orig_pkts`, `resp_pkts`, `orig_ip_bytes`, `resp_ip_bytes`.
+- **In-memory alert reporting:** Web daemon `/api/alerts` route now serves alerts from the most recent analysis run.
+
 ### Changed
+- **Breaking (0.x):** Pillar 1 formula v1.2 modifies scores and changes `Alert.id` generation to reflect the corrected bounded grid and session-based aggregation.
 - Distribution pivoted to 100% Docker and docker-compose (server daemon and web dashboard); native binary tarballs and desktop installers are discontinued.
 - DuckDB is now a mandatory runtime dependency, dynamically linked against the official prebuilt `libduckdb.so` (v1.5.6) with fixed SHA-256 via `scripts/fetch-libduckdb.sh`.
 - Upgraded workspace MSRV to Rust 1.88 (required by `duckdb ~1.10506.1`).
 - Upgraded container runtime and base image to Debian 13 trixie.
 
 ### Fixed
+- Fixed critical false positives in short captures where period candidates exceeded the observed span.
+- Corrected partition counting in `ingest_dir` to take the union of partition dates instead of summing file counts.
+- DuckDB execution errors are now tracked in `files_failed` instead of miscounted as bad rows.
+- Updated `doctor` check status from "skeleton" to "ok" / "degraded".
 - Graceful shutdown on `SIGTERM` when running as PID 1 in container environments.
 
 ### Removed

CHANGELOG.pt-BR.md

@@ -6,13 +6,27 @@ Formato baseado no [Keep a Changelog](https://keepachangelog.com/pt-BR/1.1.0/);
 
 ## [Unreleased]
 
+### Adicionado
+- **Pilar 1 (v1.2):** Periodograma de Rayleigh/Schuster com grade limitada ao intervalo observado ($T \le \text{span}/3$), portão de significância por regularidade e exclusões de infraestrutura (NTP, mDNS/LLMNR/NBNS, SSDP, não-TCP/UDP).
+- **Pilar 2 (v1.0):** Detector de sessões TCP persistentes com encadeamento de timeouts de inatividade (intervalos $\le 3600$s sem novo SYN), score logarítmico de persistência e métricas de PCR e tentativas falhas na evidência.
+- **Pilar 3 (v1.0):** Detector de capacidade de informação em subdomínios DNS ($I = H \times C$ bits/h) para identificar túneis de alta entropia, com Divergência Jensen-Shannon (JSD) registrada na evidência.
+- **Pilar 4 (v1.0):** Fusão noisy-OR ponderada por host ($w_1=w_2=w_3=0.75$) com vínculo determinístico de evidências, ranqueando hosts na saída humana do `acaua analyze` via seção `hosts:`.
+- **Ingestão de `dns.log` do Zeek:** Particionamento diário Hive Parquet para consultas DNS com mapeamento determinístico de colunas pelo cabeçalho TSV.
+- **Esquema de armazenamento Parquet (v1.2):** Esquema de 15 colunas para `conn.log` suportando `conn_state`, `history`, `orig_pkts`, `resp_pkts`, `orig_ip_bytes`, `resp_ip_bytes`.
+- **Relato de alertas em memória:** Rota `/api/alerts` do daemon web agora serve os alertas da análise mais recente.
+
 ### Modificado
+- **Breaking (0.x):** A fórmula v1.2 do Pilar 1 altera pontuações e a geração de `Alert.id` para refletir a grade limitada e a agregação por inícios de sessão.
 - Distribuição migrada para 100% Docker e docker-compose (daemon servidor e dashboard web); tarballs binários nativos e instaladores desktop descontinuados.
 - DuckDB agora é dependência obrigatória em tempo de execução, linkada dinamicamente contra a `libduckdb.so` oficial pré-compilada (v1.5.6) com SHA-256 fixado via `scripts/fetch-libduckdb.sh`.
 - MSRV do workspace atualizado para Rust 1.88 (exigido pelo `duckdb ~1.10506.1`).
 - Runtime do container e imagem base atualizados para Debian 13 trixie.
 
 ### Corrigido
+- Correção de falsos positivos críticos em capturas curtas onde candidatos a período excediam a duração observada.
+- Correção na contagem de partições em `ingest_dir` para utilizar a união dos conjuntos de datas em vez de somar contagens de arquivos.
+- Erros de execução do DuckDB agora são contabilizados em `files_failed` em vez de linhas ruins.
+- Atualização do status da checagem do `doctor` de "skeleton" para "ok" / "degraded".
 - Encerramento limpo em `SIGTERM` ao rodar como PID 1 em ambientes de container.
 
 ### Removido

README.md

@@ -8,11 +8,11 @@
 
 ---
 
-**Status: pre-alpha skeleton.** The project structure, documentation, CI and
+**Status: pre-release (initial version).** The project structure, documentation, CI,
-compiling stubs exist. The four mathematical detection pillars, the real Zeek
+Zeek ingestion for `conn.log` and `dns.log`, DuckDB/Parquet analytical storage, and
-parser, DuckDB storage and the alert engine are **under development** and are
+initial versions of detection pillars 1 to 4 are implemented and validated against
-not shipped yet. Nothing on this page should be read as a claim that detection
+public traffic. Automatic continuous ingestion and cross-restart alert persistence are
-already works.
+planned for upcoming releases.
 
 Acauã is a network threat hunting engine built around deterministic evidence,
 not an opaque model. The goal is to answer not only "this looks suspicious" but
@@ -36,30 +36,26 @@ not an opaque model. The goal is to answer not only "this looks suspicious" but
 
 | Area | State |
 |---|---|
-| Rust workspace and CLI stubs | ✅ compiles |
+| Rust workspace and CLI (`acaua`) | ✅ functional |
-| Axum API with health, status, alerts and analyze routes | ✅ stub |
+| Axum API with health, status, alerts and analyze routes | ✅ functional |
-| Embedded web dashboard | ✅ UI shell, served by daemon |
+| Embedded web dashboard | ✅ UI served by daemon |
 | Documentation and CI/release workflows | ✅ |
 | Docker multi-arch image and compose deployment | ✅ |
 | DuckDB analytical store | ✅ official prebuilt library linked |
-| Four detection pillars | 🚧 under development |
+| Four detection pillars (P1–P4) | ✅ initial version |
-| Zeek log parser | 🚧 stub |
+| Zeek log parser (`conn.log`, `dns.log`) | ✅ functional |
-| Alert evidence and scoring | 🚧 stub |
+| Alert evidence and scoring | ✅ deterministic JSON + noisy-OR fusion |
 | Published releases | ❌ none yet |
 
 ## The four pillars
 
-All four are **under development**. The roadmap keeps them as future work, not
+All four pillars are implemented in their **initial version (v1)** and validated against synthetic fixtures and public captures:
-as implemented features.
 
-1. **Beaconing:** periodicity on event timestamps with a Rayleigh/Schuster
+1. **Beaconing (Pilar 1 v1.2):** periodicity on event timestamps with a Rayleigh/Schuster
-   periodogram, and generalized Lomb–Scargle only where a real `y` exists.
+   periodogram bounded by the observed span, regularity significance gating, and infrastructure exclusions.
-2. **Persistent sessions:** long connections that never produce a short `Δt`
+2. **Persistent sessions (Pilar 2 v1.0):** long TCP connections chained across Zeek inactivity timeouts (up to 3600s gap without new SYN), log-scale persistence score, and PCR/failed attempt tracking.
-   and would otherwise stay invisible to a beaconing detector.
+3. **Protocol anomalies (Pilar 3 v1.0):** deterministic information capacity (Shannon entropy $\times$ unique subdomain characters per UTC hour) in DNS queries to detect tunneling.
-3. **Protocol anomalies:** deterministic features over Zeek protocol logs,
+4. **Evidence fusion (Pilar 4 v1.0):** per-host weighted noisy-OR combination ($w_1=w_2=w_3=0.75$) with deterministic JSON evidence and ranking.
-   including identity and lateral-movement signals.
-4. **Evidence fusion:** bounded, symmetric combination of pillar scores with a
-   reproducible evidence record per alert.
 
 The design decisions behind these pillars, including the six corrections to the
 original blueprint, are in **[docs/architecture.md](docs/architecture.md)**.

README.pt-BR.md

@@ -8,11 +8,11 @@
 
 ---
 
-**Estado: esqueleto pré-alfa.** A estrutura do projeto, a documentação, o CI e
+**Estado: pré-lançamento (versão inicial).** A estrutura do projeto, a documentação,
-os stubs que compilam existem. Os quatro pilares matemáticos de detecção, o
+o CI, a ingestão de `conn.log` e `dns.log` do Zeek, o armazenamento analítico DuckDB/Parquet
-parser real de logs Zeek, o armazenamento DuckDB e o motor de alertas estão
+e as versões iniciais dos pilares 1 a 4 estão implementados e validados contra
-**em desenvolvimento** e ainda não são distribuídos. Nada nesta página deve ser
+tráfego público. Ingestão contínua automática e persistência de alertas entre reinícios
-lido como promessa de que a detecção já funciona.
+estão planejadas para os próximos lançamentos.
 
 O Acauã é um motor de caça a ameaças em rede construído em torno de evidência
 determinística, não de um modelo opaco. O objetivo é responder não só "isto
@@ -37,30 +37,26 @@ produziram esta pontuação".
 
 | Área | Estado |
 |---|---|
-| Workspace Rust e stubs da CLI | ✅ compila |
+| Workspace Rust e CLI (`acaua`) | ✅ funcional |
-| API Axum com rotas de health, status, alertas e análise | ✅ stub |
+| API Axum com rotas de health, status, alertas e análise | ✅ funcional |
-| Dashboard web embutido | ✅ shell da UI, servido pelo daemon |
+| Dashboard web embutido | ✅ UI servida pelo daemon |
 | Documentação e workflows de CI/release | ✅ |
 | Imagem Docker multi-arch e deployment com Compose | ✅ |
 | Armazenamento analítico DuckDB | ✅ biblioteca oficial pré-compilada linkada |
-| Quatro pilares de detecção | 🚧 em desenvolvimento |
+| Quatro pilares de detecção (P1–P4) | ✅ versão inicial |
-| Parser de logs Zeek | 🚧 stub |
+| Parser de logs Zeek (`conn.log`, `dns.log`) | ✅ funcional |
-| Evidência e pontuação de alertas | 🚧 stub |
+| Evidência e pontuação de alertas | ✅ JSON determinístico + fusão noisy-OR |
 | Releases publicadas | ❌ nenhuma ainda |
 
 ## Os quatro pilares
 
-Todos os quatro estão **em desenvolvimento**. O roadmap os mantém como trabalho
+Todos os quatro pilares estão implementados em sua **versão inicial (v1)** e validados contra fixtures sintéticas e capturas públicas:
-futuro, não como recurso implementado.
 
-1. **Beaconing:** periodicidade sobre timestamps de eventos com periodograma de
+1. **Beaconing (Pilar 1 v1.2):** periodicidade sobre timestamps de eventos com periodograma de
-   Rayleigh/Schuster, e Lomb–Scargle generalizado só onde existe um `y` real.
+   Rayleigh/Schuster limitado ao intervalo observado, portão de significância por regularidade e exclusões de infraestrutura.
-2. **Sessões persistentes:** conexões longas que nunca produzem um `Δt` curto e
+2. **Sessões persistentes (Pilar 2 v1.0):** conexões TCP longas encadeadas através de timeouts de inatividade do Zeek (gap de até 3600s sem novo SYN), score logarítmico de persistência e rastreamento de PCR e tentativas falhas.
-   ficariam invisíveis para um detector de beaconing.
+3. **Anomalias de protocolo (Pilar 3 v1.0):** capacidade determinística de informação (entropia de Shannon $\times$ caracteres de subdomínios únicos por hora UTC) em consultas DNS para detectar tunelamento.
-3. **Anomalias de protocolo:** features determinísticas sobre logs de protocolo
+4. **Fusão de evidências (Pilar 4 v1.0):** combinação noisy-OR ponderada por host ($w_1=w_2=w_3=0.75$) com evidência determinística em JSON e ordenação de prioridade.
-   do Zeek, incluindo sinais de identidade e movimento lateral.
-4. **Fusão de evidências:** combinação limitada e simétrica das pontuações dos
-   pilares, com um registro de evidência reproduzível por alerta.
 
 As decisões de projeto por trás desses pilares, incluindo as seis correções ao
 blueprint original, estão em **[docs/pt-BR/arquitetura.md](docs/pt-BR/arquitetura.md)**.

ROADMAP.md

@@ -2,40 +2,37 @@
 
 **English** · [Português](ROADMAP.pt-BR.md)
 
-This roadmap separates what exists from what is planned. The four pillars are
+This roadmap separates what exists from what is planned. The four detection pillars
-**under development**; they represent the core analytical work.
+are implemented in their **initial version (v1)** and validated against public traffic.
 
 ## Done
 
 - Rust workspace with `acaua-core` and `acaua-cli`.
-- CLI commands, Axum API and embedded web dashboard.
+- CLI commands (`analyze`, `serve`, `doctor`, `update`), Axum API and embedded web dashboard.
 - 100% Docker containerization, compose deployment, non-root hardened profile.
-- Prebuilt DuckDB linking and Parquet storage foundations.
+- Prebuilt DuckDB linking and daily Hive-partitioned Parquet storage with sandbox isolation.
-- Stubs for the four pillars, evidence fusion, and Zeek ingestion.
+- Real Zeek log parser for `conn.log` and `dns.log` mapping columns by name.
+- **Pillar 1 (v1.2):** Rayleigh/Schuster periodogram with bounded grid ($T \le \text{span}/3$), regularity gating, and infrastructure exclusions.
+- **Pillar 2 (v1.0):** Persistent TCP sessions with inactivity timeout chaining (gaps $\le 3600$s without SYN), log persistence score, and PCR/failed attempts in evidence.
+- **Pillar 3 (v1.0):** DNS information capacity (Shannon entropy $\times$ unique subdomain chars per UTC hour) to detect tunneling, with JSD in evidence.
+- **Pillar 4 (v1.0):** Per-host weighted noisy-OR fusion ($w_1=w_2=w_3=0.75$) with deterministic JSON evidence and priority ranking (`hosts:` output).
+- In-memory alert reporting in `/api/alerts` and updated web interface.
 - English/Portuguese documentation, CI/release workflows and `scripts/check.sh`.
-- Deterministic smoke tests and bilingual UI coverage checks.
+- Deterministic test suite and validation scripts against IoT-23 and CTU-Normal captures.
 
 ## Next
 
-1. **Pillar 1 — beaconing.** Implement the Rayleigh/Schuster periodogram on
+1. **Continuous ingestion:** automated log rotation and directory watching in daemon mode.
-   event timestamps, add harmonic resistance, and verify with deterministic fixtures.
+2. **Persistent alerts:** SQLite/DuckDB persistence of alerts across daemon restarts.
-2. **Pillar 2 — persistent sessions.** Detect long connections that never
+3. **Live long connection tracking:** sensor-side long connection logging before TCP teardown.
-   produce a short interval, with explicit thresholds and no overlap with
+4. **Public Suffix List (PSL):** replace base domain heuristic with official PSL.
-   pillar 1.
+5. **Allowlisting & baselining:** reduce benign cloud IoT polling (e.g. Philips HUE) false positives.
-3. **Real Zeek parser.** Read `conn.log`, `dns.log`, `ssl.log` and protocol logs
-   with bounded memory, path confinement and deterministic ordering.
-4. **Evidence record.** Emit a JSON record per alert with inputs, intermediate
-   values, code version and dataset hash.
 
 ## Later
 
-- Pillar 3 — deterministic protocol anomalies, including identity and
+- UDP sessions and additional protocol parsers (`ssl.log`, `smb_mapping.log`, `dce_rpc.log`, `rdp.log`, `ssh.log`, `kerberos.log`, `ntlm.log`).
-  lateral-movement features.
+- Identity tracking, lateral movement detection, and pivot graphs.
-- Pillar 4 — bounded, symmetric fusion with reproducible weights.
+- NetFlow/IPFIX as a documented degraded mode and Suricata EVE JSON as an optional secondary input.
-- Analytical store optimizations, daily Parquet partitioning and TTL pruning.
-- Scheduled ingestion and automated log rotation handling in daemon mode.
-- NetFlow/IPFIX as a documented degraded mode and Suricata EVE JSON as an
-  optional secondary input.
 - Signed release metadata, SBOM and reproducible builds.
 
 ## Non-goals

ROADMAP.pt-BR.md

@@ -2,38 +2,37 @@
 
 [English](ROADMAP.md) · **Português**
 
-Este roadmap separa o que existe do que está planejado. Os quatro pilares estão
+Este roadmap separa o que existe do que está planejado. Os quatro pilares de detecção
-**em desenvolvimento**; eles representam o trabalho analítico principal.
+estão implementados em sua **versão inicial (v1)** e validados contra tráfego público.
 
 ## Feito
 
 - Workspace Rust com `acaua-core` e `acaua-cli`.
-- Comandos CLI, API Axum e dashboard web embutido.
+- Comandos da CLI (`analyze`, `serve`, `doctor`, `update`), API Axum e dashboard web embutido.
 - Containerização 100% Docker, implantação Compose, perfil endurecido não-root.
-- Linkagem da DuckDB pré-compilada e fundações de armazenamento Parquet.
+- Linkagem da DuckDB pré-compilada e armazenamento Parquet particionado diariamente por Hive com isolamento sandbox.
-- Stubs dos quatro pilares, fusão de evidências e ingestão Zeek.
+- Parser real de logs Zeek para `conn.log` e `dns.log` mapeando colunas por nome.
+- **Pilar 1 (v1.2):** Periodograma de Rayleigh/Schuster com grade limitada ($T \le \text{span}/3$), portão de significância por regularidade e exclusões de infraestrutura.
+- **Pilar 2 (v1.0):** Sessões TCP persistentes com encadeamento de timeouts de inatividade (intervalos $\le 3600$s sem novo SYN), score logarítmico de persistência e métricas de PCR e tentativas falhas na evidência.
+- **Pilar 3 (v1.0):** Capacidade de informação DNS (entropia de Shannon $\times$ caracteres de subdomínio únicos por hora UTC) para detecção de túneis, com JSD na evidência.
+- **Pilar 4 (v1.0):** Fusão noisy-OR ponderada por host ($w_1=w_2=w_3=0.75$) com evidência determinística em JSON e ordenação de prioridade (saída `hosts:`).
+- Relato de alertas em memória em `/api/alerts` e interface web atualizada.
 - Documentação em inglês/português, workflows de CI/release e `scripts/check.sh`.
-- Testes determinísticos de fumaça e checagem de tradução bilíngue da UI.
+- Suíte determinística de testes e scripts de validação contra capturas do IoT-23 e CTU-Normal.
 
 ## Próximo
 
-1. **Pilar 1 — beaconing.** Implementar o periodograma de Rayleigh/Schuster sobre
+1. **Ingestão contínua:** rotação automática de logs e monitoramento de diretório em modo daemon.
-   timestamps de eventos, resistência a harmônicos e validação com fixtures determinísticas.
+2. **Alertas persistentes:** persistência de alertas em SQLite/DuckDB entre reinícios do daemon.
-2. **Pilar 2 — sessões persistentes.** Detectar conexões longas que nunca
+3. **Rastreamento de conexões longas ao vivo:** registro de conexões longas no sensor antes do encerramento TCP.
-   produzem intervalos curtos, com limiares explícitos sem sobreposição ao pilar 1.
+4. **Public Suffix List (PSL):** substituição da heurística de domínio-base pela PSL oficial.
-3. **Parser Zeek real.** Ler `conn.log`, `dns.log`, `ssl.log` e logs de protocolo
+5. **Allowlist e baselining:** redução de falsos positivos em polling de dispositivos IoT benignos em nuvem (ex.: Philips HUE).
-   com memória delimitada, confinamento de caminhos e ordem determinística.
-4. **Registro de evidência.** Emitir registro JSON por alerta contendo entradas,
-   valores intermediários, versão do código e hash do dataset.
 
 ## Depois
 
-- Pilar 3 — anomalias determinísticas de protocolo, incluindo identidade e
+- Sessões UDP e parsers adicionais de protocolo (`ssl.log`, `smb_mapping.log`, `dce_rpc.log`, `rdp.log`, `ssh.log`, `kerberos.log`, `ntlm.log`).
-  movimentação lateral.
+- Rastreamento de identidade, detecção de movimentação lateral e grafos de pivô.
-- Pilar 4 — fusão limitada e simétrica com pesos reproduzíveis.
+- NetFlow/IPFIX como modo degradado documentado e Suricata EVE JSON como entrada secundária opcional.
-- Otimizações no store analítico, particionamento diário Parquet e limpeza por TTL.
-- Ingestão agendada e tratamento automático de rotação de logs em modo daemon.
-- NetFlow/IPFIX como modo degradado documentado e Suricata EVE JSON como entrada secundária.
 - Metadados assinados de release, SBOM e builds reproduzíveis.
 
 ## Fora de plano

apps/gui/ui/i18n-en.js

@@ -11,28 +11,28 @@ export const EN = {
   'Configurações': 'Settings',
   'idioma e tema': 'language and theme',
   'Conectando…': 'Connecting…',
-  'esqueleto': 'skeleton',
+  'pré-lançamento': 'pre-release',
   'checar atualização': 'check for updates',
   'Caça a ameaças em rede': 'Network threat hunting',
   'Pesquisa determinística, sem IA opaca': 'Deterministic research, no opaque AI',
-  'Aviso de escopo.': 'Scope notice.',
+  'Estado atual.': 'Current state.',
-  'Este é um esqueleto pré-implementação. Os quatro pilares matemáticos, o parser Zeek e o armazenamento DuckDB ainda não estão implementados.': 'This is a pre-implementation skeleton. The four mathematical pillars, the Zeek parser and the DuckDB storage are not implemented yet.',
+  'Ingestão de conn.log e dns.log do Zeek, armazenamento Parquet/DuckDB e os pilares 1 a 4 em versão inicial, validados com tráfego público real. Ainda não: ingestão contínua automática, alertas persistidos entre reinícios, movimentação lateral e sessões UDP.': 'Zeek conn.log and dns.log ingestion, Parquet/DuckDB storage and pillars 1 to 4 in an initial version, validated against real public traffic. Not yet: automatic continuous ingestion, alerts persisted across restarts, lateral movement and UDP sessions.',
   'Uso autorizado.': 'Authorized use.',
   'Use somente em redes próprias ou com autorização explícita. Nenhum pacote é enviado para fora.': 'Use only on your own networks or with explicit authorization. No packet leaves the machine.',
   'Pilares de detecção': 'Detection pillars',
   'Pilar {n}': 'Pillar {n}',
   'Pilar 1': 'Pillar 1',
-  'Periodograma de Rayleigh/Schuster. Em desenvolvimento.': 'Rayleigh/Schuster periodogram. Under development.',
+  'Periodograma de Rayleigh/Schuster com grade limitada ao intervalo observado.': 'Rayleigh/Schuster periodogram with a grid bounded by the observed span.',
-  'em desenvolvimento': 'under development',
+  'versão inicial': 'initial version',
   'Pilar 2': 'Pillar 2',
   'Sessões persistentes': 'Persistent sessions',
-  'Detector de conexões longas. Em desenvolvimento.': 'Long-connection detector. Under development.',
+  'Sessões TCP longas, inclusive as que o Zeek divide por inatividade.': 'Long TCP sessions, including those Zeek splits on inactivity.',
   'Pilar 3': 'Pillar 3',
   'Anomalias de protocolo': 'Protocol anomalies',
-  'Features determinísticas por protocolo. Em desenvolvimento.': 'Deterministic per-protocol features. Under development.',
+  'Volume de informação (entropia de Shannon) em subdomínios DNS.': 'Information volume (Shannon entropy) in DNS subdomains.',
   'Pilar 4': 'Pillar 4',
   'Fusão de evidências': 'Evidence fusion',
-  'Jensen-Shannon e prova por alerta. Em desenvolvimento.': 'Jensen-Shannon and per-alert proof. Under development.',
+  'Prioridade por host com evidência por alerta.': 'Per-host priority with per-alert evidence.',
   'Analisar dataset': 'Analyze dataset',
   'Aponte para um diretório de logs Zeek ou para um dataset local.': 'Point to a Zeek log directory or a local dataset.',
   'Diretório de logs': 'Log directory',
@@ -40,9 +40,9 @@ export const EN = {
   'Atualizar status': 'Refresh status',
   'Nenhuma análise executada.': 'No analysis has run yet.',
   'Alertas recentes': 'Recent alerts',
-  'Cada alerta futuro carregará a evidência determinística que o sustenta.': 'Every future alert will carry the deterministic evidence behind it.',
+  'Cada alerta carrega a evidência determinística que o sustenta.': 'Each alert carries the deterministic evidence behind it.',
-  'Nenhum alerta ainda. O motor de detecção será ligado nas próximas specs.': 'No alerts yet. The detection engine will be enabled in upcoming specs.',
+  'Nenhum alerta ainda. Analise um diretório de logs do Zeek.': 'No alerts yet. Analyze a Zeek log directory.',
-  'Acauã · esqueleto pré-implementação': 'Acauã · pre-implementation skeleton',
+  'Acauã · pré-lançamento': 'Acauã · pre-release',
   'Fechar configurações': 'Close settings',
   'Fechar': 'Close',
   'Idioma': 'Language',

apps/gui/ui/index.html

@@ -41,7 +41,7 @@
     <div class="side-foot">
       <div class="status-row">
         <span id="connection" class="pill"><span class="dot"></span><span id="connectionText">Conectando…</span></span>
-        <span class="pill muted">esqueleto</span>
+        <span class="pill muted">pré-lançamento</span>
       </div>
       <div class="status-row">
         <span id="version" class="mono" data-i18n-skip>—</span>
@@ -60,9 +60,9 @@
       <button id="settingsBtn" type="button" class="icon-btn" aria-label="Configurações" title="Configurações">⚙</button>
     </header>
 
-    <section class="notice notice-warn">
+    <section class="notice">
-      <strong>Aviso de escopo.</strong>
+      <strong>Estado atual.</strong>
-      <span>Este é um esqueleto pré-implementação. Os quatro pilares matemáticos, o parser Zeek e o armazenamento DuckDB ainda não estão implementados.</span>
+      <span>Ingestão de conn.log e dns.log do Zeek, armazenamento Parquet/DuckDB e os pilares 1 a 4 em versão inicial, validados com tráfego público real. Ainda não: ingestão contínua automática, alertas persistidos entre reinícios, movimentação lateral e sessões UDP.</span>
     </section>
 
     <section class="notice">
@@ -74,26 +74,26 @@
       <article class="card">
         <span class="card-kicker">Pilar 1</span>
         <h2>Beaconing</h2>
-        <p>Periodograma de Rayleigh/Schuster. Em desenvolvimento.</p>
+        <p>Periodograma de Rayleigh/Schuster com grade limitada ao intervalo observado.</p>
-        <span class="tag">em desenvolvimento</span>
+        <span class="tag">versão inicial</span>
       </article>
       <article class="card">
         <span class="card-kicker">Pilar 2</span>
         <h2>Sessões persistentes</h2>
-        <p>Detector de conexões longas. Em desenvolvimento.</p>
+        <p>Sessões TCP longas, inclusive as que o Zeek divide por inatividade.</p>
-        <span class="tag">em desenvolvimento</span>
+        <span class="tag">versão inicial</span>
       </article>
       <article class="card">
         <span class="card-kicker">Pilar 3</span>
         <h2>Anomalias de protocolo</h2>
-        <p>Features determinísticas por protocolo. Em desenvolvimento.</p>
+        <p>Volume de informação (entropia de Shannon) em subdomínios DNS.</p>
-        <span class="tag">em desenvolvimento</span>
+        <span class="tag">versão inicial</span>
       </article>
       <article class="card">
         <span class="card-kicker">Pilar 4</span>
         <h2>Fusão de evidências</h2>
-        <p>Jensen-Shannon e prova por alerta. Em desenvolvimento.</p>
+        <p>Prioridade por host com evidência por alerta.</p>
-        <span class="tag">em desenvolvimento</span>
+        <span class="tag">versão inicial</span>
       </article>
     </section>
 
@@ -119,16 +119,16 @@
       <div class="panel-head">
         <div>
           <h2>Alertas recentes</h2>
-          <p class="hint">Cada alerta futuro carregará a evidência determinística que o sustenta.</p>
+          <p class="hint">Cada alerta carrega a evidência determinística que o sustenta.</p>
         </div>
       </div>
       <ul id="alertList" class="alert-list">
-        <li class="empty">Nenhum alerta ainda. O motor de detecção será ligado nas próximas specs.</li>
+        <li class="empty">Nenhum alerta ainda. Analise um diretório de logs do Zeek.</li>
       </ul>
     </section>
 
     <footer class="foot">
-      <span>Acauã · esqueleto pré-implementação</span>
+      <span>Acauã · pré-lançamento</span>
       <span class="mono">MIT OR Apache-2.0</span>
     </footer>
   </main>

crates/acaua-cli/src/main.rs

@@ -65,6 +65,33 @@ async fn run(cli: Cli) -> acaua_core::Result<()> {
                 for alert in &report.alerts {
                     println!("  [{:.3}] {}", alert.score, alert.summary);
                 }
+                if !report.hosts.is_empty() {
+                    println!("hosts:");
+                    for host in report.hosts.iter().take(10) {
+                        let m1 = host
+                            .terms
+                            .iter()
+                            .find(|t| t.pillar == 1)
+                            .map(|t| t.m)
+                            .unwrap_or(0.0);
+                        let m2 = host
+                            .terms
+                            .iter()
+                            .find(|t| t.pillar == 2)
+                            .map(|t| t.m)
+                            .unwrap_or(0.0);
+                        let m3 = host
+                            .terms
+                            .iter()
+                            .find(|t| t.pillar == 3)
+                            .map(|t| t.m)
+                            .unwrap_or(0.0);
+                        println!(
+                            "  [{:.3}] {:<18} {:.3}/{:.3}/{:.3}",
+                            host.score, host.orig_h, m1, m2, m3
+                        );
+                    }
+                }
                 println!("estado: {}", report.status);
             }
             Ok(())

crates/acaua-cli/src/web.rs

@@ -27,6 +27,7 @@ struct Assets;
 #[derive(Clone)]
 struct AppState {
     storage: Arc<Storage>,
+    last_report: Arc<std::sync::Mutex<Option<AnalysisReport>>>,
 }
 
 #[derive(Serialize)]
@@ -70,8 +71,13 @@ async fn status(State(state): State<AppState>) -> Json<Status> {
     })
 }
 
-async fn alerts() -> Json<Vec<Alert>> {
+async fn alerts(State(state): State<AppState>) -> Json<Vec<Alert>> {
-    Json(Vec::new())
+    let report_lock = state.last_report.lock().unwrap();
+    let list = report_lock
+        .as_ref()
+        .map(|r| r.alerts.clone())
+        .unwrap_or_default();
+    Json(list)
 }
 
 async fn analyze(
@@ -79,7 +85,12 @@ async fn analyze(
     Json(request): Json<AnalyzeRequest>,
 ) -> std::result::Result<Json<AnalysisReport>, (StatusCode, Json<ErrorBody>)> {
     zeek::analyze_dir(&request.path, state.storage.root())
-        .map(Json)
+        .map(|report| {
+            if let Ok(mut lock) = state.last_report.lock() {
+                *lock = Some(report.clone());
+            }
+            Json(report)
+        })
         .map_err(|error| {
             (
                 StatusCode::BAD_REQUEST,
@@ -148,6 +159,7 @@ pub async fn serve(bind: &str, data_dir: PathBuf) -> Result<()> {
     std::fs::create_dir_all(&data_dir)?;
     let state = AppState {
         storage: Arc::new(Storage::open(&data_dir)?),
+        last_report: Arc::new(std::sync::Mutex::new(None)),
     };
     let app = Router::new()
         .route("/api/health", get(health))

crates/acaua-core/src/dns_store.rs

@@ -0,0 +1,454 @@
+//! `dns.log` ingestion for Pilar 3.
+//!
+//! The generic Zeek TSV header (`#separator`, `#unset_field`, `#empty_field`,
+//! `#fields`) is honored and columns are mapped by name, never by position.
+//! `query` is stored exactly as Zeek wrote it: `\xNN` escapes are **not**
+//! decoded, because the entropy is computed over that deterministic
+//! representation. `-` and `(empty)` become SQL NULL.
+
+use crate::error::{AcauaError, Result};
+use crate::pillar3::DnsQuery;
+use crate::storage::Storage;
+use sha2::{Digest, Sha256};
+use std::collections::HashSet;
+use std::fs;
+use std::path::Path;
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct DnsRecord {
+    pub ts: f64,
+    pub uid: String,
+    pub orig_h: String,
+    pub orig_p: u16,
+    pub resp_h: String,
+    pub resp_p: u16,
+    pub proto: String,
+    pub query: Option<String>,
+    pub qtype_name: Option<String>,
+    pub rcode_name: Option<String>,
+}
+
+#[derive(Debug, Clone, Default, serde::Serialize)]
+pub struct DnsIngestionStats {
+    pub files_seen: usize,
+    pub records_ingested: u64,
+    pub lines_skipped: u64,
+    pub partitions_affected: usize,
+}
+
+fn has_parquet(conn: &duckdb::Connection, glob: &str) -> bool {
+    let query = format!("SELECT count(*) FROM read_parquet('{glob}', hive_partitioning = true);");
+    conn.query_row(&query, [], |row| row.get::<_, i64>(0))
+        .map(|count| count > 0)
+        .unwrap_or(false)
+}
+
+impl Storage {
+    fn dns_glob(&self) -> String {
+        self.root()
+            .join("raw")
+            .join("dns")
+            .join("*")
+            .join("*.parquet")
+            .to_string_lossy()
+            .replace('\'', "''")
+    }
+
+    fn conn_glob_string(&self) -> String {
+        self.root()
+            .join("raw")
+            .join("conn")
+            .join("*")
+            .join("*.parquet")
+            .to_string_lossy()
+            .replace('\'', "''")
+    }
+
+    /// Parses a Zeek `dns.log`. Returns the records, the number of skipped
+    /// lines and the set of UTC day partitions touched.
+    pub fn parse_dns_lines(content: &[u8]) -> (Vec<DnsRecord>, u64, HashSet<String>) {
+        let mut records = Vec::new();
+        let mut skipped = 0u64;
+        let mut dates = HashSet::new();
+
+        let mut separator = '\t';
+        let mut unset_field = String::from("-");
+        let mut empty_field = String::from("(empty)");
+        let mut fields: Vec<String> = Vec::new();
+        let mut idx: std::collections::HashMap<&'static str, usize> =
+            std::collections::HashMap::new();
+
+        let text = String::from_utf8_lossy(content);
+
+        for line in text.lines() {
+            let line = line.trim_end_matches(['\r', '\n']);
+            if line.is_empty() {
+                continue;
+            }
+
+            if line.starts_with('#') {
+                if let Some(rest) = line.strip_prefix("#separator ") {
+                    let value = rest.trim();
+                    if let Some(hex) = value.strip_prefix("\\x") {
+                        if let Ok(byte) = u8::from_str_radix(hex, 16) {
+                            separator = byte as char;
+                        }
+                    } else if let Some(c) = value.chars().next() {
+                        separator = c;
+                    }
+                } else if let Some(rest) = line.strip_prefix("#unset_field") {
+                    unset_field = rest.trim().to_string();
+                } else if let Some(rest) = line.strip_prefix("#empty_field") {
+                    empty_field = rest.trim().to_string();
+                } else if let Some(rest) = line.strip_prefix("#fields") {
+                    let rest = rest.trim();
+                    fields = rest
+                        .split(separator)
+                        .map(|s| s.trim().to_string())
+                        .filter(|s| !s.is_empty())
+                        .collect();
+                    idx.clear();
+                    for (i, name) in fields.iter().enumerate() {
+                        match name.as_str() {
+                            "ts" => idx.insert("ts", i),
+                            "uid" => idx.insert("uid", i),
+                            "id.orig_h" => idx.insert("orig_h", i),
+                            "id.orig_p" => idx.insert("orig_p", i),
+                            "id.resp_h" => idx.insert("resp_h", i),
+                            "id.resp_p" => idx.insert("resp_p", i),
+                            "proto" => idx.insert("proto", i),
+                            "query" => idx.insert("query", i),
+                            "qtype_name" => idx.insert("qtype_name", i),
+                            "rcode_name" => idx.insert("rcode_name", i),
+                            _ => None,
+                        };
+                    }
+                }
+                continue;
+            }
+
+            let required = ["ts", "orig_h", "orig_p", "resp_h", "resp_p", "query"];
+            if required.iter().any(|name| !idx.contains_key(name)) {
+                skipped += 1;
+                continue;
+            }
+
+            let parts: Vec<&str> = line.split(separator).collect();
+            if parts.len() < fields.len() {
+                skipped += 1;
+                continue;
+            }
+
+            let value = |name: &str| idx.get(name).and_then(|&i| parts.get(i).copied());
+            let text_value = |name: &str| -> Option<String> {
+                let raw = value(name)?;
+                if raw == unset_field || raw == "-" || raw == empty_field {
+                    None
+                } else {
+                    Some(raw.to_string())
+                }
+            };
+
+            let Some(ts) = value("ts").and_then(|v| v.parse::<f64>().ok()) else {
+                skipped += 1;
+                continue;
+            };
+            let Some(orig_p) = value("orig_p").and_then(|v| v.parse::<u16>().ok()) else {
+                skipped += 1;
+                continue;
+            };
+            let Some(resp_p) = value("resp_p").and_then(|v| v.parse::<u16>().ok()) else {
+                skipped += 1;
+                continue;
+            };
+
+            if let Some(day) = chrono::DateTime::from_timestamp(ts.floor() as i64, 0) {
+                dates.insert(day.format("%Y-%m-%d").to_string());
+            }
+
+            records.push(DnsRecord {
+                ts,
+                uid: text_value("uid").unwrap_or_default(),
+                orig_h: text_value("orig_h").unwrap_or_default(),
+                orig_p,
+                resp_h: text_value("resp_h").unwrap_or_default(),
+                resp_p,
+                proto: text_value("proto").unwrap_or_default(),
+                query: text_value("query"),
+                qtype_name: text_value("qtype_name"),
+                rcode_name: text_value("rcode_name"),
+            });
+        }
+
+        (records, skipped, dates)
+    }
+
+    /// Ingests one `dns.log` into `raw/dns` (daily partitions, idempotent by
+    /// content hash).
+    pub fn ingest_dns_log(&self, path: impl AsRef<Path>) -> Result<(u64, u64, HashSet<String>)> {
+        let path = path.as_ref();
+        let content = fs::read(path)?;
+        let mut hasher = Sha256::new();
+        hasher.update(&content);
+        let hash = hasher.finalize();
+        let mut sha16 = String::with_capacity(16);
+        for b in &hash[..8] {
+            use std::fmt::Write;
+            let _ = write!(sha16, "{b:02x}");
+        }
+
+        let (records, skipped, dates) = Self::parse_dns_lines(&content);
+        if records.is_empty() {
+            return Ok((0, skipped, HashSet::new()));
+        }
+
+        let dns_dir = self.root().join("raw").join("dns");
+        fs::create_dir_all(&dns_dir)?;
+
+        let conn = self.open_connection()?;
+        conn.execute_batch(
+            "CREATE TEMP TABLE dns_rows (
+                ts DOUBLE,
+                uid VARCHAR,
+                orig_h VARCHAR,
+                orig_p USMALLINT,
+                resp_h VARCHAR,
+                resp_p USMALLINT,
+                proto VARCHAR,
+                query VARCHAR,
+                qtype_name VARCHAR,
+                rcode_name VARCHAR
+            );",
+        )?;
+
+        {
+            let mut appender = conn.appender("dns_rows")?;
+            for r in &records {
+                appender.append_row(duckdb::params![
+                    r.ts,
+                    r.uid,
+                    r.orig_h,
+                    r.orig_p,
+                    r.resp_h,
+                    r.resp_p,
+                    r.proto,
+                    r.query,
+                    r.qtype_name,
+                    r.rcode_name
+                ])?;
+            }
+        }
+
+        let dns_dir_sql = dns_dir.to_string_lossy().replace('\'', "''");
+        let copy_sql = format!(
+            "COPY (
+                SELECT *, CAST(to_timestamp(ts) AS DATE) AS date FROM dns_rows
+            ) TO '{dns_dir_sql}' (
+                FORMAT parquet,
+                PARTITION_BY (date),
+                FILENAME_PATTERN 'src_{sha16}_{{i}}',
+                OVERWRITE_OR_IGNORE
+            );"
+        );
+        conn.execute_batch(&copy_sql)?;
+        conn.execute_batch("DROP TABLE dns_rows;")?;
+
+        Ok((records.len() as u64, skipped, dates))
+    }
+
+    /// Ingests every `dns.log` / `dns.*.log` directly inside `directory`.
+    pub fn ingest_dns_directory(&self, directory: impl AsRef<Path>) -> Result<DnsIngestionStats> {
+        let directory = directory.as_ref();
+        if !directory.is_dir() {
+            return Err(AcauaError::InvalidInput(format!(
+                "{} is not a directory",
+                directory.display()
+            )));
+        }
+        let canonical_dir = fs::canonicalize(directory)?;
+        let mut stats = DnsIngestionStats::default();
+        let mut dates = HashSet::new();
+
+        for entry in fs::read_dir(&canonical_dir)?.flatten() {
+            let path = entry.path();
+            if let Ok(canon) = fs::canonicalize(&path) {
+                if !canon.starts_with(&canonical_dir) {
+                    continue;
+                }
+            } else {
+                continue;
+            }
+            let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
+                continue;
+            };
+            if name.ends_with(".gz") {
+                continue;
+            }
+            let is_dns = name == "dns.log" || (name.starts_with("dns.") && name.ends_with(".log"));
+            if !is_dns || !path.is_file() {
+                continue;
+            }
+
+            stats.files_seen += 1;
+            match self.ingest_dns_log(&path) {
+                Ok((ingested, skipped, file_dates)) => {
+                    stats.records_ingested += ingested;
+                    stats.lines_skipped += skipped;
+                    dates.extend(file_dates);
+                }
+                Err(_) => {
+                    stats.lines_skipped += 1;
+                }
+            }
+        }
+
+        stats.partitions_affected = dates.len();
+        Ok(stats)
+    }
+
+    /// Reads the DNS rows inside the same 7-day window used by the other
+    /// pillars. The window is relative to the most recent day present in
+    /// `conn` OR `dns`.
+    pub fn query_window_dns(&self) -> Result<Vec<DnsQuery>> {
+        let conn = self.open_connection()?;
+        let dns_glob = self.dns_glob();
+        if !has_parquet(&conn, &dns_glob) {
+            return Ok(Vec::new());
+        }
+
+        let mut max_date: Option<String> = conn
+            .query_row(
+                &format!(
+                    "SELECT max(date)::VARCHAR FROM read_parquet('{dns_glob}', hive_partitioning = true);"
+                ),
+                [],
+                |row| row.get(0),
+            )
+            .unwrap_or(None);
+
+        let conn_glob = self.conn_glob_string();
+        if has_parquet(&conn, &conn_glob) {
+            let conn_max: Option<String> = conn
+                .query_row(
+                    &format!(
+                        "SELECT max(date)::VARCHAR FROM read_parquet('{conn_glob}', hive_partitioning = true);"
+                    ),
+                    [],
+                    |row| row.get(0),
+                )
+                .unwrap_or(None);
+            max_date = match (max_date, conn_max) {
+                (Some(a), Some(b)) => Some(a.max(b)),
+                (a, b) => a.or(b),
+            };
+        }
+
+        let Some(max_date) = max_date.filter(|d| !d.is_empty()) else {
+            return Ok(Vec::new());
+        };
+
+        let query = format!(
+            "SELECT ts, orig_h, resp_p, query, qtype_name, rcode_name
+             FROM read_parquet('{dns_glob}', hive_partitioning = true, union_by_name = true)
+             WHERE query IS NOT NULL
+               AND query <> ''
+               AND date > CAST('{max_date}' AS DATE) - INTERVAL 7 DAY
+             ORDER BY orig_h, ts, query;"
+        );
+
+        let mut stmt = conn.prepare(&query)?;
+        let rows = stmt.query_map([], |row| {
+            Ok(DnsQuery {
+                ts: row.get(0)?,
+                orig_h: row.get(1)?,
+                resp_p: row.get(2)?,
+                query: row.get(3)?,
+                qtype_name: row.get(4)?,
+                rcode_name: row.get(5)?,
+            })
+        })?;
+
+        let mut out = Vec::new();
+        for row in rows {
+            out.push(row?);
+        }
+        Ok(out)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::fs;
+
+    // Header copied from a real Zeek 9.0.1 dns.log.
+    const DNS_LOG: &str = "#separator \\x09\n\
+#set_separator\t,\n\
+#empty_field\t(empty)\n\
+#unset_field\t-\n\
+#path\tdns\n\
+#open\t2026-10-31-12-00-00\n\
+#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\ttrans_id\trtt\tquery\tqclass\tqclass_name\tqtype\tqtype_name\trcode\trcode_name\tAA\tTC\tRD\tRA\tZ\tanswers\tTTLs\trejected\topcode\topcode_name\n\
+#types\ttime\tstring\taddr\tport\taddr\tport\tenum\tcount\tinterval\tstring\tcount\tstring\tcount\tstring\tcount\tstring\tbool\tbool\tbool\tbool\tcount\tvector[string]\tvector[interval]\tbool\tcount\tstring\n\
+1800000000.000000\tC1\t10.0.2.15\t51000\t10.0.2.3\t53\tudp\t1\t0.001\taaa.bbb.example.com\t1\tC_INET\t1\tA\t0\tNOERROR\tF\tF\tT\tT\t0\t93.184.216.34\t60.000000\tF\t0\tQUERY\n\
+1800000030.000000\tC2\t10.0.2.15\t51001\t10.0.2.3\t53\tudp\t2\t0.001\t\\x01tunnel.example.com\t1\tC_INET\t16\tTXT\t0\tNOERROR\tF\tF\tT\tT\t0\t-\t-\tF\t0\tQUERY\n\
+1800000060.000000\tC3\t10.0.2.15\t51002\t10.0.2.3\t53\tudp\t3\t-\t-\t1\tC_INET\t1\tA\t3\tNXDOMAIN\tF\tF\tT\tT\t0\t-\t-\tF\t0\tQUERY\n\
+1800000090.000000\tC4\t10.0.2.15\t51003\t10.0.2.3\t53\tudp\t4\t0.001\t(empty)\t1\tC_INET\t1\tA\t3\tNXDOMAIN\tF\tF\tT\tT\t0\t-\t-\tF\t0\tQUERY\n";
+
+    #[test]
+    fn dns_parser_maps_by_name_and_preserves_zeek_escapes() {
+        let (records, skipped, dates) = Storage::parse_dns_lines(DNS_LOG.as_bytes());
+        assert_eq!(skipped, 0);
+        assert_eq!(records.len(), 4);
+        assert_eq!(dates.len(), 1);
+
+        assert_eq!(records[0].query.as_deref(), Some("aaa.bbb.example.com"));
+        assert_eq!(records[0].qtype_name.as_deref(), Some("A"));
+        assert_eq!(records[0].rcode_name.as_deref(), Some("NOERROR"));
+
+        // `\x01` is kept literally, not decoded.
+        assert_eq!(records[1].query.as_deref(), Some("\\x01tunnel.example.com"));
+
+        // `-` (unset) and `(empty)` become NULL.
+        assert_eq!(records[2].query, None);
+        assert_eq!(records[3].query, None);
+    }
+
+    #[test]
+    fn dns_roundtrip_is_idempotent() {
+        let temp = tempfile::tempdir().unwrap();
+        let store = Storage::open(temp.path()).unwrap();
+        let log = temp.path().join("dns.log");
+        fs::write(&log, DNS_LOG).unwrap();
+
+        let (first, skipped, partitions) = store.ingest_dns_log(&log).unwrap();
+        assert_eq!(first, 4);
+        assert_eq!(skipped, 0);
+        assert_eq!(partitions.len(), 1);
+
+        let queries = store.query_window_dns().unwrap();
+        assert_eq!(queries.len(), 2);
+        assert_eq!(queries[0].orig_h, "10.0.2.15");
+
+        let (second, _, _) = store.ingest_dns_log(&log).unwrap();
+        assert_eq!(second, 4);
+        let queries = store.query_window_dns().unwrap();
+        assert_eq!(queries.len(), 2, "re-ingest must not duplicate rows");
+    }
+
+    #[test]
+    fn dns_directory_ingestion_counts_partitions() {
+        let temp = tempfile::tempdir().unwrap();
+        let store = Storage::open(temp.path()).unwrap();
+        let logs = temp.path().join("logs");
+        fs::create_dir_all(&logs).unwrap();
+        fs::write(logs.join("dns.log"), DNS_LOG).unwrap();
+        fs::write(logs.join("dns.1.log"), DNS_LOG).unwrap();
+        fs::write(logs.join("conn.log"), "not dns").unwrap();
+
+        let stats = store.ingest_dns_directory(&logs).unwrap();
+        assert_eq!(stats.files_seen, 2);
+        assert_eq!(stats.records_ingested, 8);
+        assert_eq!(stats.partitions_affected, 1);
+    }
+}

crates/acaua-core/src/doctor.rs

@@ -93,8 +93,9 @@ pub fn run() -> DoctorReport {
             detail: "research workspace is isolated from the product".to_string(),
         },
     ];
+    let all_ok = checks.iter().all(|c| c.ok);
     DoctorReport {
         checks,
-        status: "skeleton",
+        status: if all_ok { "ok" } else { "degraded" },
     }
 }

crates/acaua-core/src/fusion.rs

@@ -1,7 +1,9 @@
 use crate::error::Result;
+use crate::pillar4::{self, FusionInput, HostScore};
 use serde::{Deserialize, Serialize};
+use std::collections::BTreeSet;
 
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
 pub struct Alert {
     pub id: String,
     pub pillar: u8,
@@ -11,15 +13,199 @@ pub struct Alert {
     pub created_at: String,
 }
 
-#[derive(Debug, Clone, Serialize)]
+pub fn alert_orig_h(alert: &Alert) -> String {
-pub struct FusionReport {
+    if let Some(h) = alert.evidence.get("orig_h").and_then(|v| v.as_str()) {
-    pub alerts: Vec<Alert>,
+        if !h.is_empty() {
-    pub status: &'static str,
+            return h.to_string();
+        }
+    }
+    if let Some(rest) = alert.summary.strip_prefix("Beaconing detected: ") {
+        if let Some(h) = rest.split(':').next() {
+            return h.to_string();
+        }
+    }
+    if let Some(rest) = alert.summary.strip_prefix("Persistent session: ") {
+        if let Some(h) = rest.split(" -> ").next() {
+            return h.to_string();
+        }
+    }
+    if let Some(rest) = alert
+        .summary
+        .strip_prefix("High-information DNS subdomains: ")
+    {
+        if let Some(h) = rest.split(" -> ").next() {
+            return h.to_string();
+        }
+    }
+    String::new()
 }
 
-pub fn fuse(_scores: &[f64]) -> Result<FusionReport> {
+pub fn fuse(alerts: &[Alert]) -> Result<Vec<HostScore>> {
-    Ok(FusionReport {
+    let mut hosts_set = BTreeSet::new();
-        alerts: Vec::new(),
+    for alert in alerts {
-        status: "skeleton",
+        let h = alert_orig_h(alert);
-    })
+        if !h.is_empty() {
+            hosts_set.insert(h);
+        }
+    }
+
+    let mut host_scores = Vec::new();
+
+    for orig_h in hosts_set {
+        let mut m1 = 0.0_f64;
+        let mut alert_id_p1 = None;
+        let mut m2 = 0.0_f64;
+        let mut alert_id_p2 = None;
+        let mut m3 = 0.0_f64;
+        let mut alert_id_p3 = None;
+
+        for alert in alerts {
+            if alert_orig_h(alert) == orig_h {
+                match alert.pillar {
+                    1 => {
+                        if alert.score > m1 || (alert.score == m1 && alert_id_p1.is_none()) {
+                            m1 = alert.score;
+                            alert_id_p1 = Some(alert.id.clone());
+                        } else if alert.score == m1 {
+                            if let Some(prev) = &alert_id_p1 {
+                                if alert.id < *prev {
+                                    alert_id_p1 = Some(alert.id.clone());
+                                }
+                            }
+                        }
+                    }
+                    2 => {
+                        if alert.score > m2 || (alert.score == m2 && alert_id_p2.is_none()) {
+                            m2 = alert.score;
+                            alert_id_p2 = Some(alert.id.clone());
+                        } else if alert.score == m2 {
+                            if let Some(prev) = &alert_id_p2 {
+                                if alert.id < *prev {
+                                    alert_id_p2 = Some(alert.id.clone());
+                                }
+                            }
+                        }
+                    }
+                    3 => {
+                        if alert.score > m3 || (alert.score == m3 && alert_id_p3.is_none()) {
+                            m3 = alert.score;
+                            alert_id_p3 = Some(alert.id.clone());
+                        } else if alert.score == m3 {
+                            if let Some(prev) = &alert_id_p3 {
+                                if alert.id < *prev {
+                                    alert_id_p3 = Some(alert.id.clone());
+                                }
+                            }
+                        }
+                    }
+                    _ => {}
+                }
+            }
+        }
+
+        let input = FusionInput {
+            m1,
+            m2,
+            m3,
+            alert_id_p1,
+            alert_id_p2,
+            alert_id_p3,
+        };
+
+        let hs = pillar4::score(&orig_h, &input)?;
+        host_scores.push(hs);
+    }
+
+    host_scores.sort_by(|a, b| {
+        b.score
+            .total_cmp(&a.score)
+            .then_with(|| a.orig_h.cmp(&b.orig_h))
+    });
+
+    Ok(host_scores)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    struct SplitMix64 {
+        state: u64,
+    }
+
+    impl SplitMix64 {
+        fn new(seed: u64) -> Self {
+            Self { state: seed }
+        }
+        fn next_u64(&mut self) -> u64 {
+            self.state = self.state.wrapping_add(0x9e3779b97f4a7c15);
+            let mut z = self.state;
+            z = (z ^ (z >> 30)).wrapping_mul(0xbf58476d1ce4e5b9);
+            z = (z ^ (z >> 27)).wrapping_mul(0x94d049bb133111eb);
+            z ^ (z >> 31)
+        }
+    }
+
+    #[test]
+    fn test_permutation_invariance_and_evidence() {
+        let alerts = vec![
+            Alert {
+                id: "alert_p1_a".to_string(),
+                pillar: 1,
+                score: 0.85,
+                summary: "Beaconing detected: 10.0.0.1:any -> 1.1.1.1:443 (period ~14400s, score 0.850)".to_string(),
+                evidence: serde_json::json!({ "orig_h": "10.0.0.1" }),
+                created_at: "2026-10-31T12:00:00Z".to_string(),
+            },
+            Alert {
+                id: "alert_p2_a".to_string(),
+                pillar: 2,
+                score: 0.60,
+                summary: "Persistent session: 10.0.0.1 -> 1.1.1.1:443 (6.0h, score 0.600)".to_string(),
+                evidence: serde_json::json!({ "orig_h": "10.0.0.1" }),
+                created_at: "2026-10-31T12:00:00Z".to_string(),
+            },
+            Alert {
+                id: "alert_p3_a".to_string(),
+                pillar: 3,
+                score: 0.50,
+                summary: "High-information DNS subdomains: 10.0.0.1 -> *.example.com (50000 bits/h, 100 unique, score 0.500)".to_string(),
+                evidence: serde_json::json!({ "orig_h": "10.0.0.1" }),
+                created_at: "2026-10-31T12:00:00Z".to_string(),
+            },
+            Alert {
+                id: "alert_p1_b".to_string(),
+                pillar: 1,
+                score: 0.45,
+                summary: "Beaconing detected: 10.0.0.2:any -> 2.2.2.2:80 (period ~900s, score 0.450)".to_string(),
+                evidence: serde_json::json!({ "orig_h": "10.0.0.2" }),
+                created_at: "2026-10-31T12:00:00Z".to_string(),
+            },
+        ];
+
+        let reference = fuse(&alerts).unwrap();
+        let ref_json = serde_json::to_vec(&reference).unwrap();
+
+        // Check that each term.alert_id exists in original alerts
+        for hs in &reference {
+            for term in &hs.terms {
+                if let Some(id) = &term.alert_id {
+                    assert!(alerts.iter().any(|a| &a.id == id));
+                }
+            }
+        }
+
+        // Test 10 random permutations
+        let mut rng = SplitMix64::new(42);
+        for _ in 0..10 {
+            let mut permuted = alerts.clone();
+            for i in (1..permuted.len()).rev() {
+                let j = (rng.next_u64() as usize) % (i + 1);
+                permuted.swap(i, j);
+            }
+            let fused = fuse(&permuted).unwrap();
+            let fused_json = serde_json::to_vec(&fused).unwrap();
+            assert_eq!(ref_json, fused_json);
+        }
+    }
 }

crates/acaua-core/src/lib.rs

@@ -1,3 +1,4 @@
+pub mod dns_store;
 pub mod doctor;
 pub mod error;
 pub mod fusion;

crates/acaua-core/src/pillar1.rs

@@ -4,6 +4,8 @@ use serde::{Deserialize, Serialize};
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
 pub struct BeaconingInput {
     pub timestamps: Vec<f64>,
+    #[serde(default)]
+    pub proto: String,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
@@ -34,6 +36,13 @@ pub struct BeaconingScore {
     pub weights: WeightsInfo,
     pub formula_version: &'static str,
     pub status: &'static str,
+    pub span_seconds: f64,
+    pub min_cycles: usize,
+    pub significance: f64,
+    pub regularity: f64,
+    pub significance_gate: f64,
+    pub gate_regularity: f64,
+    pub proto: String,
 }
 
 fn quantile_cont(sorted: &[f64], q: f64) -> f64 {
@@ -55,18 +64,24 @@ fn quantile_cont(sorted: &[f64], q: f64) -> f64 {
 }
 
 pub fn score(input: &BeaconingInput) -> Result<BeaconingScore> {
-    let grid_info = GridInfo {
+    let n = input.timestamps.len();
-        min_period_s: 600.0,
+    let min_cycles = 3;
-        max_period_s: 43200.0,
+    let gate_regularity = 0.25;
-        points: 200,
+
-    };
     let weights_info = WeightsInfo {
         significance: 0.5,
         regularity: 0.5,
     };
 
-    let n = input.timestamps.len();
     if n < 20 {
+        let span = if n >= 2 {
+            let mut sorted = input.timestamps.clone();
+            sorted.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
+            sorted[n - 1] - sorted[0]
+        } else {
+            0.0
+        };
+
         return Ok(BeaconingScore {
             score: 0.0,
             observations: n,
@@ -77,16 +92,29 @@ pub fn score(input: &BeaconingInput) -> Result<BeaconingScore> {
             skew_bowley: 0.0,
             degenerate_quartiles: false,
             method: "rayleigh-schuster",
-            grid: grid_info,
+            grid: GridInfo {
+                min_period_s: 600.0,
+                max_period_s: 43200.0,
+                points: 0,
+            },
             weights: weights_info,
-            formula_version: "1.1",
+            formula_version: "1.2",
             status: "insufficient_data",
+            span_seconds: span,
+            min_cycles,
+            significance: 0.0,
+            regularity: 0.0,
+            significance_gate: 0.0,
+            gate_regularity,
+            proto: input.proto.clone(),
         });
     }
 
     let mut ts = input.timestamps.clone();
     ts.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
 
+    let span = ts[n - 1] - ts[0];
+
     let mut deltas = Vec::with_capacity(n - 1);
     for i in 1..n {
         deltas.push(ts[i] - ts[i - 1]);
@@ -110,21 +138,85 @@ pub fn score(input: &BeaconingInput) -> Result<BeaconingScore> {
     abs_devs.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
     let madm = quantile_cont(&abs_devs, 0.50);
 
-    let (madm_normalized, degenerate_deltas) = if q2 <= EPSILON {
+    // Step 2: degenerate deltas check
-        (0.0, true)
+    if q2 <= EPSILON {
-    } else {
+        return Ok(BeaconingScore {
-        (madm / q2, false)
+            score: 0.0,
-    };
+            observations: n,
+            period_detected_seconds: 0.0,
+            p_value: 1.0,
+            peak_power: 0.0,
+            madm_normalized: 0.0,
+            skew_bowley,
+            degenerate_quartiles,
+            method: "rayleigh-schuster",
+            grid: GridInfo {
+                min_period_s: 600.0,
+                max_period_s: 0.0,
+                points: 0,
+            },
+            weights: weights_info,
+            formula_version: "1.2",
+            status: "degenerate_deltas",
+            span_seconds: span,
+            min_cycles,
+            significance: 0.0,
+            regularity: 0.0,
+            significance_gate: 0.0,
+            gate_regularity,
+            proto: input.proto.clone(),
+        });
+    }
 
-    // Rayleigh/Schuster periodogram with M = 200 log-spaced periods from 600s to 43200s
+    let madm_normalized = madm / q2;
-    let t0 = ts[0];
-    let num_periods = 200;
-    let mut periods = Vec::with_capacity(num_periods);
-    let mut powers = Vec::with_capacity(num_periods);
 
-    for j in 0..num_periods {
+    // Step 3: Effective grid (periods <= span / min_cycles)
+    let max_allowed_period = span / (min_cycles as f64);
+    let num_grid_points = 200;
+    let mut periods = Vec::new();
+    for j in 0..num_grid_points {
         let period = 600.0 * 72.0_f64.powf(j as f64 / 199.0);
-        periods.push(period);
+        if period <= max_allowed_period {
+            periods.push(period);
+        }
+    }
+
+    let m_eff = periods.len();
+    if m_eff == 0 {
+        let regularity = (1.0 - 2.0 * madm_normalized).clamp(0.0, 1.0);
+        return Ok(BeaconingScore {
+            score: 0.0,
+            observations: n,
+            period_detected_seconds: 0.0,
+            p_value: 1.0,
+            peak_power: 0.0,
+            madm_normalized,
+            skew_bowley,
+            degenerate_quartiles,
+            method: "rayleigh-schuster",
+            grid: GridInfo {
+                min_period_s: 600.0,
+                max_period_s: 0.0,
+                points: 0,
+            },
+            weights: weights_info,
+            formula_version: "1.2",
+            status: "insufficient_span",
+            span_seconds: span,
+            min_cycles,
+            significance: 0.0,
+            regularity,
+            significance_gate: 0.0,
+            gate_regularity,
+            proto: input.proto.clone(),
+        });
+    }
+
+    // Step 4: Rayleigh/Schuster periodogram on effective grid
+    let t0 = ts[0];
+    let mut powers = Vec::with_capacity(m_eff);
+
+    for &period in &periods {
         let f = 1.0 / period;
         let mut sum_cos = 0.0;
         let mut sum_sin = 0.0;
@@ -144,35 +236,37 @@ pub fn score(input: &BeaconingInput) -> Result<BeaconingScore> {
         }
     }
 
-    // Harmonic resistance: largest period in the grid with P >= 0.5 * P_max
+    // Step 9: Harmonic resistance: largest period in effective grid with P >= 0.5 * P_max
     let threshold_power = 0.5 * p_max;
     let mut detected_period = periods[0];
-    for j in (0..num_periods).rev() {
+    for j in (0..m_eff).rev() {
         if powers[j] >= threshold_power {
             detected_period = periods[j];
             break;
         }
     }
 
-    let m_points = num_periods as f64;
+    let m_eff_f64 = m_eff as f64;
-    let p_value = (m_points * (-p_max).exp()).min(1.0);
+    let p_value = (m_eff_f64 * (-p_max).exp()).min(1.0);
 
-    // Formula v1.1
+    // Step 5: Significance
-    let ln_m = m_points.ln();
+    let ln_m = m_eff_f64.ln();
     let six_ln_10 = 6.0 * 10.0_f64.ln();
     let sig = ((p_max - ln_m) / six_ln_10).clamp(0.0, 1.0);
+
+    // Step 6: Regularity
     let regularity = (1.0 - 2.0 * madm_normalized).clamp(0.0, 1.0);
 
-    let status = if degenerate_deltas {
+    // Step 7: Gate
-        "degenerate_deltas"
+    let gate = (regularity / gate_regularity).min(1.0);
-    } else {
-        "ok"
-    };
 
-    let score = if degenerate_deltas {
+    // Step 8: Score
-        0.5 * sig
+    let score = 0.5 * sig * gate + 0.5 * regularity;
-    } else {
+
-        0.5 * sig + 0.5 * regularity
+    let grid_info = GridInfo {
+        min_period_s: periods[0],
+        max_period_s: periods[m_eff - 1],
+        points: m_eff,
     };
 
     Ok(BeaconingScore {
@@ -187,8 +281,15 @@ pub fn score(input: &BeaconingInput) -> Result<BeaconingScore> {
         method: "rayleigh-schuster",
         grid: grid_info,
         weights: weights_info,
-        formula_version: "1.1",
+        formula_version: "1.2",
-        status,
+        status: "ok",
+        span_seconds: span,
+        min_cycles,
+        significance: sig,
+        regularity,
+        significance_gate: gate,
+        gate_regularity,
+        proto: input.proto.clone(),
     })
 }
 
@@ -233,7 +334,11 @@ mod tests {
         for k in 0..42 {
             timestamps.push(k as f64 * 14400.0);
         }
-        let res = score(&BeaconingInput { timestamps }).unwrap();
+        let res = score(&BeaconingInput {
+            timestamps,
+            proto: "tcp".to_string(),
+        })
+        .unwrap();
         assert!(res.score >= 0.99, "score: {}", res.score);
         let rel_err = (res.period_detected_seconds - 14400.0).abs() / 14400.0;
         assert!(
@@ -243,6 +348,7 @@ mod tests {
             rel_err
         );
         assert_eq!(res.status, "ok");
+        assert_eq!(res.formula_version, "1.2");
     }
 
     #[test]
@@ -257,7 +363,11 @@ mod tests {
                 let jitter = rng.uniform(-0.18, 0.18) * 14400.0;
                 timestamps.push(t0 + k as f64 * 14400.0 + jitter);
             }
-            let res = score(&BeaconingInput { timestamps }).unwrap();
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
             assert!(res.score >= 0.80, "seed {seed} score {} < 0.80", res.score);
             if res.score < min_score {
                 min_score = res.score;
@@ -284,7 +394,11 @@ mod tests {
                 curr += 14400.0 * (1.0 + rng.uniform(-0.18, 0.18));
                 timestamps.push(curr);
             }
-            let res = score(&BeaconingInput { timestamps }).unwrap();
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
             assert!(res.score >= 0.40, "seed {seed} score {} < 0.40", res.score);
             if res.score < min_score {
                 min_score = res.score;
@@ -320,7 +434,11 @@ mod tests {
                 curr += rng.exponential(mean);
                 timestamps.push(curr);
             }
-            let res = score(&BeaconingInput { timestamps }).unwrap();
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
             assert!(res.score < 0.30, "seed {seed} score {} >= 0.30", res.score);
             if res.score < min_score {
                 min_score = res.score;
@@ -338,7 +456,11 @@ mod tests {
         for k in 0..25 {
             timestamps.push(k as f64 * 1000.0);
         }
-        let res = score(&BeaconingInput { timestamps }).unwrap();
+        let res = score(&BeaconingInput {
+            timestamps,
+            proto: "tcp".to_string(),
+        })
+        .unwrap();
         assert_eq!(res.skew_bowley, 0.0);
         assert!(res.degenerate_quartiles);
         assert!(!res.score.is_nan() && !res.score.is_infinite());
@@ -347,15 +469,24 @@ mod tests {
     #[test]
     fn test_6_median_delta_zero() {
         let timestamps = vec![1000.0; 25];
-        let res = score(&BeaconingInput { timestamps }).unwrap();
+        let res = score(&BeaconingInput {
+            timestamps,
+            proto: "tcp".to_string(),
+        })
+        .unwrap();
         assert_eq!(res.status, "degenerate_deltas");
+        assert_eq!(res.score, 0.0);
         assert!(!res.score.is_nan() && !res.score.is_infinite());
     }
 
     #[test]
     fn test_7_insufficient_data() {
         let timestamps: Vec<f64> = (0..19).map(|k| k as f64 * 1000.0).collect();
-        let res = score(&BeaconingInput { timestamps }).unwrap();
+        let res = score(&BeaconingInput {
+            timestamps,
+            proto: "tcp".to_string(),
+        })
+        .unwrap();
         assert_eq!(res.status, "insufficient_data");
         assert_eq!(res.score, 0.0);
     }
@@ -365,11 +496,172 @@ mod tests {
         let timestamps: Vec<f64> = (0..42).map(|k| k as f64 * 14400.0).collect();
         let s1 = score(&BeaconingInput {
             timestamps: timestamps.clone(),
+            proto: "tcp".to_string(),
+        })
+        .unwrap();
+        let s2 = score(&BeaconingInput {
+            timestamps,
+            proto: "tcp".to_string(),
         })
         .unwrap();
-        let s2 = score(&BeaconingInput { timestamps }).unwrap();
         let j1 = serde_json::to_vec(&s1).unwrap();
         let j2 = serde_json::to_vec(&s2).unwrap();
         assert_eq!(j1, j2);
     }
+
+    fn generate_bursty(rng: &mut SplitMix64, span: f64) -> Vec<f64> {
+        let mut out = Vec::new();
+        let mut t = 1_790_000_000.0;
+        let end = t + span;
+        loop {
+            t += rng.exponential(900.0);
+            if t > end {
+                break;
+            }
+            let mut k = 1;
+            while rng.next_f64() < 0.8 {
+                k += 1;
+            }
+            let mut u = t;
+            for _ in 0..k {
+                out.push(u);
+                u += rng.exponential(2.0);
+            }
+        }
+        out
+    }
+
+    #[test]
+    fn test_9_bursty_3h() {
+        let mut max_score = f64::MIN;
+        for seed in 1..=50 {
+            let mut rng = SplitMix64::new(seed);
+            let timestamps = generate_bursty(&mut rng, 3.0 * 3600.0);
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
+            assert!(
+                res.score < 0.05,
+                "seed {seed} bursty3h score {} >= 0.05 (status: {})",
+                res.score,
+                res.status
+            );
+            if res.score > max_score {
+                max_score = res.score;
+            }
+        }
+        println!("Test 9 (bursty 3h) 50 seeds: max={max_score:.4}");
+    }
+
+    #[test]
+    fn test_10_bursty_24h() {
+        let mut max_score = f64::MIN;
+        for seed in 1..=50 {
+            let mut rng = SplitMix64::new(seed);
+            let timestamps = generate_bursty(&mut rng, 86400.0);
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
+            assert!(
+                res.score < 0.05,
+                "seed {seed} bursty24h score {} >= 0.05 (status: {})",
+                res.score,
+                res.status
+            );
+            if res.score > max_score {
+                max_score = res.score;
+            }
+        }
+        println!("Test 10 (bursty 24h) 50 seeds: max={max_score:.4}");
+    }
+
+    #[test]
+    fn test_11_fast_short() {
+        let mut min_score = f64::MAX;
+        let mut max_score = f64::MIN;
+        let mut period_within_10pct = 0;
+
+        for seed in 1..=50 {
+            let mut rng = SplitMix64::new(seed);
+            let mut timestamps = Vec::new();
+            let mut t = 1_790_000_000.0;
+            let end = t + 86400.0;
+            while t < end {
+                timestamps.push(t);
+                t += 900.0 * (1.0 + rng.uniform(-0.1, 0.1));
+            }
+
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
+            assert!(
+                res.score >= 0.40,
+                "seed {seed} fast_short score {} < 0.40",
+                res.score
+            );
+            if res.score < min_score {
+                min_score = res.score;
+            }
+            if res.score > max_score {
+                max_score = res.score;
+            }
+            let rel_err = (res.period_detected_seconds - 900.0).abs() / 900.0;
+            if rel_err < 0.10 {
+                period_within_10pct += 1;
+            }
+        }
+        println!(
+            "Test 11 (fast short) 50 seeds: min={min_score:.4}, max={max_score:.4}, period_hits={period_within_10pct}/50"
+        );
+        assert!(
+            period_within_10pct >= 45,
+            "period hits: {period_within_10pct}/50"
+        );
+    }
+
+    #[test]
+    fn test_12_dropout_30pct() {
+        let mut hits = 0;
+        for seed in 1..=50 {
+            let mut rng = SplitMix64::new(seed);
+            let mut timestamps = Vec::new();
+            let mut t = 1_790_000_000.0;
+            for _ in 0..60 {
+                if rng.next_f64() > 0.3 {
+                    timestamps.push(t);
+                }
+                t += 14400.0 * (1.0 + rng.uniform(-0.1, 0.1));
+            }
+
+            let res = score(&BeaconingInput {
+                timestamps,
+                proto: "tcp".to_string(),
+            })
+            .unwrap();
+            if res.score >= 0.40 {
+                hits += 1;
+            }
+        }
+        println!("Test 12 (dropout 30%) 50 seeds: hits={hits}/50");
+        assert!(hits >= 48, "dropout hits {hits} < 48");
+    }
+
+    #[test]
+    fn test_13_insufficient_span() {
+        // 30 events in 25 min (1500s < 1800s)
+        let timestamps: Vec<f64> = (0..30).map(|k| 1000.0 + k as f64 * 50.0).collect();
+        let res = score(&BeaconingInput {
+            timestamps,
+            proto: "tcp".to_string(),
+        })
+        .unwrap();
+        assert_eq!(res.status, "insufficient_span");
+        assert_eq!(res.score, 0.0);
+    }
 }

crates/acaua-core/src/pillar2.rs

@@ -1,33 +1,309 @@
 use crate::error::Result;
 use serde::{Deserialize, Serialize};
 
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
-pub struct SessionInput {
+pub struct SessionCandidate {
-    pub durations_secs: Vec<f64>,
+    pub start_ts: f64,
-    pub bytes: Vec<u64>,
+    pub end_ts: f64,
+    pub duration: f64,
+    pub parts: usize,
+    pub orig_bytes: u64,
+    pub resp_bytes: u64,
 }
 
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct SessionInput {
+    pub sessions: Vec<SessionCandidate>,
+    pub failed_attempts: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct ChainRuleInfo {
+    pub max_gap_s: f64,
+    pub continuation: &'static str,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct SessionThresholds {
+    pub zero_at_s: f64,
+    pub saturate_at_s: f64,
+    pub alert_at_s: f64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
 pub struct SessionScore {
     pub score: f64,
-    pub observations: usize,
-    pub long_connections: usize,
-    pub bowley_sentinel: Option<f64>,
-    pub method: &'static str,
     pub status: &'static str,
+    pub longest_session_seconds: f64,
+    pub session_start_ts: f64,
+    pub session_end_ts: f64,
+    pub parts: usize,
+    pub orig_bytes: u64,
+    pub resp_bytes: u64,
+    pub pcr: f64,
+    pub bytes_per_hour: f64,
+    pub failed_attempts: usize,
+    pub sessions_over_1h: usize,
+    pub method: &'static str,
+    pub chain_rule: ChainRuleInfo,
+    pub thresholds: SessionThresholds,
+    pub formula_version: &'static str,
 }
 
 pub fn score(input: &SessionInput) -> Result<SessionScore> {
+    let chain_rule = ChainRuleInfo {
+        max_gap_s: 3600.0,
+        continuation: "history without 'S'",
+    };
+    let thresholds = SessionThresholds {
+        zero_at_s: 3600.0,
+        saturate_at_s: 86400.0,
+        alert_at_s: 7200.0,
+    };
+
+    let valid_sessions: Vec<&SessionCandidate> = input
+        .sessions
+        .iter()
+        .filter(|s| s.duration > 0.0 && !s.duration.is_nan() && !s.duration.is_infinite())
+        .collect();
+
+    if valid_sessions.is_empty() {
+        return Ok(SessionScore {
+            score: 0.0,
+            status: "insufficient_data",
+            longest_session_seconds: 0.0,
+            session_start_ts: 0.0,
+            session_end_ts: 0.0,
+            parts: 0,
+            orig_bytes: 0,
+            resp_bytes: 0,
+            pcr: 0.0,
+            bytes_per_hour: 0.0,
+            failed_attempts: input.failed_attempts,
+            sessions_over_1h: 0,
+            method: "chained-tcp-session-duration",
+            chain_rule,
+            thresholds,
+            formula_version: "1.0",
+        });
+    }
+
+    let longest = valid_sessions
+        .iter()
+        .max_by(|a, b| a.duration.partial_cmp(&b.duration).unwrap())
+        .unwrap();
+
+    let l = longest.duration;
+    let persistence = if l > 3600.0 {
+        ((l / 3600.0).ln() / 24.0_f64.ln()).clamp(0.0, 1.0)
+    } else {
+        0.0
+    };
+
+    let tot_bytes = longest.orig_bytes + longest.resp_bytes;
+    let pcr = if tot_bytes > 0 {
+        (longest.orig_bytes as f64 - longest.resp_bytes as f64) / (tot_bytes as f64)
+    } else {
+        0.0
+    };
+
+    let bytes_per_hour = if l > 0.0 {
+        (tot_bytes as f64) / (l / 3600.0)
+    } else {
+        0.0
+    };
+
+    let sessions_over_1h = input
+        .sessions
+        .iter()
+        .filter(|s| s.duration >= 3600.0)
+        .count();
+
     Ok(SessionScore {
-        score: 0.0,
+        score: persistence,
-        observations: input.durations_secs.len(),
+        status: "ok",
-        long_connections: input
+        longest_session_seconds: l,
-            .durations_secs
+        session_start_ts: longest.start_ts,
-            .iter()
+        session_end_ts: longest.end_ts,
-            .filter(|duration| **duration >= 300.0)
+        parts: longest.parts,
-            .count(),
+        orig_bytes: longest.orig_bytes,
-        bowley_sentinel: None,
+        resp_bytes: longest.resp_bytes,
-        method: "long-connection-stub",
+        pcr,
-        status: "skeleton",
+        bytes_per_hour,
+        failed_attempts: input.failed_attempts,
+        sessions_over_1h,
+        method: "chained-tcp-session-duration",
+        chain_rule,
+        thresholds,
+        formula_version: "1.0",
     })
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_1_formula_values() {
+        let calc = |l: f64| -> f64 {
+            let input = SessionInput {
+                sessions: vec![SessionCandidate {
+                    start_ts: 1000.0,
+                    end_ts: 1000.0 + l,
+                    duration: l,
+                    parts: 1,
+                    orig_bytes: 100,
+                    resp_bytes: 100,
+                }],
+                failed_attempts: 0,
+            };
+            score(&input).unwrap().score
+        };
+
+        assert_eq!(calc(3600.0), 0.0);
+        let s7200 = calc(7200.0);
+        assert!((s7200 - 0.218147).abs() < 1e-4, "s7200: {s7200}");
+        assert_eq!(calc(86400.0), 1.0);
+        assert_eq!(calc(200000.0), 1.0);
+        assert_eq!(calc(0.0), 0.0);
+        assert!(!calc(0.0).is_nan() && !calc(0.0).is_infinite());
+    }
+
+    #[test]
+    fn test_2_fixture_zeek_split_values() {
+        // L = 4201.05s (between 1h and 2h)
+        let input = SessionInput {
+            sessions: vec![SessionCandidate {
+                start_ts: 1793400000.0,
+                end_ts: 1793402401.0 + 1800.05,
+                duration: 4201.05,
+                parts: 2,
+                orig_bytes: 1200,
+                resp_bytes: 2400,
+            }],
+            failed_attempts: 0,
+        };
+        let res = score(&input).unwrap();
+        assert!((res.score - 0.0486).abs() < 1e-4, "score: {}", res.score);
+        assert!(res.score < 0.2181, "should not trigger alert (< 2h)");
+        assert_eq!(res.parts, 2);
+    }
+
+    #[test]
+    fn test_3_c2_persistent_synthetic() {
+        // 12.3h session in 9 parts + 13 failed attempts
+        let duration = 12.3 * 3600.0;
+        let input = SessionInput {
+            sessions: vec![SessionCandidate {
+                start_ts: 1000.0,
+                end_ts: 1000.0 + duration,
+                duration,
+                parts: 9,
+                orig_bytes: 19431,
+                resp_bytes: 19431,
+            }],
+            failed_attempts: 13,
+        };
+        let res = score(&input).unwrap();
+        assert!(res.score >= 0.78, "score: {}", res.score);
+        assert_eq!(res.failed_attempts, 13);
+        assert_eq!(res.parts, 9);
+        assert_eq!(res.status, "ok");
+    }
+
+    #[test]
+    fn test_4_reverse_shell_synthetic() {
+        // 2.5h session with PCR < 0 (more received than sent from victim perspective)
+        let duration = 2.5 * 3600.0;
+        let input = SessionInput {
+            sessions: vec![SessionCandidate {
+                start_ts: 1000.0,
+                end_ts: 1000.0 + duration,
+                duration,
+                parts: 1,
+                orig_bytes: 5000,
+                resp_bytes: 50000,
+            }],
+            failed_attempts: 0,
+        };
+        let res = score(&input).unwrap();
+        assert!(
+            res.score >= 0.2181,
+            "score should alert for 2.5h session: {}",
+            res.score
+        );
+        assert!(res.pcr < 0.0, "PCR should be negative: {}", res.pcr);
+    }
+
+    #[test]
+    fn test_5_web_browsing_synthetic() {
+        // 200 HTTPS connections of 5s to 20 min in 3h
+        let mut sessions = Vec::new();
+        for i in 0..200 {
+            let dur = 5.0 + (i as f64 % 1200.0);
+            sessions.push(SessionCandidate {
+                start_ts: 1000.0 + (i as f64 * 50.0),
+                end_ts: 1000.0 + (i as f64 * 50.0) + dur,
+                duration: dur,
+                parts: 1,
+                orig_bytes: 1000,
+                resp_bytes: 10000,
+            });
+        }
+        let input = SessionInput {
+            sessions,
+            failed_attempts: 0,
+        };
+        let res = score(&input).unwrap();
+        assert_eq!(res.score, 0.0);
+        assert!(res.score < 0.2181);
+    }
+
+    #[test]
+    fn test_6_ephemeral_port_reuse() {
+        // Two distinct sessions with SYN on same 4-tuple, separated by 30 min
+        let s1 = SessionCandidate {
+            start_ts: 1000.0,
+            end_ts: 2000.0,
+            duration: 1000.0,
+            parts: 1,
+            orig_bytes: 100,
+            resp_bytes: 100,
+        };
+        let s2 = SessionCandidate {
+            start_ts: 3800.0,
+            end_ts: 4800.0,
+            duration: 1000.0,
+            parts: 1,
+            orig_bytes: 100,
+            resp_bytes: 100,
+        };
+        let input = SessionInput {
+            sessions: vec![s1, s2],
+            failed_attempts: 0,
+        };
+        let res = score(&input).unwrap();
+        assert_eq!(res.longest_session_seconds, 1000.0);
+        assert_eq!(res.score, 0.0);
+    }
+
+    #[test]
+    fn test_7_determinism() {
+        let input = SessionInput {
+            sessions: vec![SessionCandidate {
+                start_ts: 1793400000.123456,
+                end_ts: 1793444000.654321,
+                duration: 44000.530865,
+                parts: 4,
+                orig_bytes: 12345,
+                resp_bytes: 67890,
+            }],
+            failed_attempts: 3,
+        };
+        let s1 = score(&input).unwrap();
+        let s2 = score(&input).unwrap();
+        let j1 = serde_json::to_vec(&s1).unwrap();
+        let j2 = serde_json::to_vec(&s2).unwrap();
+        assert_eq!(j1, j2);
+    }
+}

crates/acaua-core/src/pillar3.rs

@@ -1,30 +1,671 @@
 use crate::error::Result;
+use crate::fusion::Alert;
 use serde::{Deserialize, Serialize};
+use sha2::{Digest, Sha256};
+use std::collections::{BTreeMap, BTreeSet};
+use time::OffsetDateTime;
+
+/// A single DNS query as read from the Zeek `dns.log`.
+///
+/// `query` is kept exactly as Zeek wrote it (including `\xNN` escapes); DNS
+/// names are not unescaped before the entropy calculation.
+#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)]
+pub struct DnsQuery {
+    pub ts: f64,
+    pub orig_h: String,
+    pub resp_p: u16,
+    pub query: String,
+    pub qtype_name: Option<String>,
+    pub rcode_name: Option<String>,
+}
 
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
 pub struct ProtocolInput {
-    pub events: usize,
+    pub queries: Vec<DnsQuery>,
-    pub protocols: Vec<String>,
 }
 
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct Thresholds {
+    pub zero_bits: f64,
+    pub alert_bits: f64,
+    pub saturate_bits: f64,
+    pub min_unique: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct Weights {
+    pub information: f64,
+    pub jsd: f64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
 pub struct ProtocolScore {
     pub score: f64,
-    pub observations: usize,
-    pub distinct_protocols: usize,
-    pub method: &'static str,
     pub status: &'static str,
+    pub orig_h: String,
+    pub base_domain: String,
+    pub peak_hour_start_ts: f64,
+    pub unique_subdomains: usize,
+    pub subdomain_chars: usize,
+    pub entropy_bits_per_char: f64,
+    pub information_bits: f64,
+    pub mean_subdomain_len: f64,
+    pub max_label_len: usize,
+    /// Sorted by key; never a `HashMap`, so the evidence is deterministic.
+    pub qtype_counts: BTreeMap<String, u64>,
+    pub nxdomain_ratio: f64,
+    pub jsd: f64,
+    pub jsd_reference_groups: usize,
+    pub method: &'static str,
+    pub thresholds: Thresholds,
+    pub weights: Weights,
+    pub formula_version: &'static str,
 }
 
-pub fn score(input: &ProtocolInput) -> Result<ProtocolScore> {
+const ZERO_BITS: f64 = 8192.0;
-    let mut protocols = input.protocols.clone();
+const ALERT_BITS: f64 = 32768.0;
-    protocols.sort();
+const SATURATE_BITS: f64 = 262144.0;
-    protocols.dedup();
+const MIN_UNIQUE: usize = 10;
-    Ok(ProtocolScore {
+const ALERT_SCORE: f64 = 0.40;
-        score: 0.0,
+
-        observations: input.events,
+const SLD2: [&str; 7] = ["co", "com", "net", "org", "gov", "edu", "ac"];
-        distinct_protocols: protocols.len(),
+
-        method: "jensen-shannon-stub",
+fn thresholds() -> Thresholds {
-        status: "skeleton",
+    Thresholds {
+        zero_bits: ZERO_BITS,
+        alert_bits: ALERT_BITS,
+        saturate_bits: SATURATE_BITS,
+        min_unique: MIN_UNIQUE,
+    }
+}
+
+fn weights() -> Weights {
+    Weights {
+        information: 1.0,
+        jsd: 0.0,
+    }
+}
+
+/// Normalizes an ASCII DNS name (lowercase, trailing dot removed) and splits
+/// it into `(base_domain, subdomain)` with the spec heuristic (no Public
+/// Suffix List in v1). Returns `None` when there is no subdomain left.
+pub fn split_base_domain(query: &str) -> Option<(String, String)> {
+    let normalized = query.trim_end_matches('.').to_ascii_lowercase();
+    if normalized.is_empty() {
+        return None;
+    }
+    let labels: Vec<&str> = normalized.split('.').collect();
+    if labels.iter().any(|label| label.is_empty()) {
+        return None;
+    }
+    let k = if labels.len() >= 3
+        && labels[labels.len() - 1].len() == 2
+        && SLD2.contains(&labels[labels.len() - 2])
+    {
+        3
+    } else {
+        2
+    };
+    if labels.len() <= k {
+        return None;
+    }
+    let base = labels[labels.len() - k..].join(".");
+    let sub = labels[..labels.len() - k].join(".");
+    if sub.is_empty() {
+        None
+    } else {
+        Some((base, sub))
+    }
+}
+
+/// Full pre-filter and normalization for one `dns.log` row.
+///
+/// Excluded, with the reason documented in the spec:
+/// - `resp_p` 5353/5355/137: mDNS, LLMNR and NBNS are local discovery traffic.
+/// - bases ending in `in-addr.arpa`, `ip6.arpa` or `.local`: reverse and local
+///   names are structured and long by nature.
+pub fn classify_query(query: &str, resp_p: u16) -> Option<(String, String)> {
+    if matches!(resp_p, 5353 | 5355 | 137) {
+        return None;
+    }
+    let (base, sub) = split_base_domain(query)?;
+    if base.ends_with("in-addr.arpa") || base.ends_with("ip6.arpa") || base.ends_with(".local") {
+        return None;
+    }
+    Some((base, sub))
+}
+
+/// Shannon entropy in bits per character over an aggregated character count.
+pub fn entropy_bits_per_char(counts: &BTreeMap<char, u64>) -> f64 {
+    let total: u64 = counts.values().sum();
+    if total == 0 {
+        return 0.0;
+    }
+    let n = total as f64;
+    let mut h = 0.0;
+    for &count in counts.values() {
+        let p = count as f64 / n;
+        if p > 0.0 {
+            h -= p * p.log2();
+        }
+    }
+    h
+}
+
+/// Jensen-Shannon divergence (base 2) between two normalized distributions.
+/// Keys are visited in sorted order and `0 * log2(0)` terms are skipped, so
+/// the result is finite and symmetric bit-for-bit.
+pub fn jsd(p: &BTreeMap<char, f64>, q: &BTreeMap<char, f64>) -> f64 {
+    let keys: BTreeSet<char> = p.keys().chain(q.keys()).copied().collect();
+    let mut total = 0.0;
+    for key in keys {
+        let a = p.get(&key).copied().unwrap_or(0.0);
+        let b = q.get(&key).copied().unwrap_or(0.0);
+        let m = (a + b) / 2.0;
+        let term_a = if a > 0.0 && m > 0.0 {
+            0.5 * a * (a / m).log2()
+        } else {
+            0.0
+        };
+        let term_b = if b > 0.0 && m > 0.0 {
+            0.5 * b * (b / m).log2()
+        } else {
+            0.0
+        };
+        // Fixed summation order independent of the argument order: required
+        // for JSD(P, Q) == JSD(Q, P) bit for bit.
+        let (low, high) = if term_a <= term_b {
+            (term_a, term_b)
+        } else {
+            (term_b, term_a)
+        };
+        total += low;
+        total += high;
+    }
+    total
+}
+
+fn normalized_counts(counts: &BTreeMap<char, u64>) -> BTreeMap<char, f64> {
+    let total: u64 = counts.values().sum();
+    if total == 0 {
+        return BTreeMap::new();
+    }
+    let n = total as f64;
+    counts.iter().map(|(&c, &v)| (c, v as f64 / n)).collect()
+}
+
+/// Information volume in bits for one unique-subdomain set:
+/// `C` = characters without the dots, `H` = Shannon entropy over the
+/// aggregated character counts, `I = H * C`.
+fn information(subdomains: &BTreeSet<String>) -> (usize, usize, f64, f64) {
+    let mut counts: BTreeMap<char, u64> = BTreeMap::new();
+    let mut chars = 0usize;
+    for sub in subdomains {
+        for c in sub.chars().filter(|c| *c != '.') {
+            *counts.entry(c).or_insert(0) += 1;
+            chars += 1;
+        }
+    }
+    let h = entropy_bits_per_char(&counts);
+    (subdomains.len(), chars, h, h * chars as f64)
+}
+
+fn score_from_bits(information_bits: f64) -> f64 {
+    if information_bits <= 0.0 {
+        return 0.0;
+    }
+    let raw = (information_bits / ZERO_BITS).log2() / (SATURATE_BITS / ZERO_BITS).log2();
+    raw.clamp(0.0, 1.0)
+}
+
+pub fn score(input: &ProtocolInput) -> Result<Vec<ProtocolScore>> {
+    // (orig_h, base) -> hour -> unique subdomains
+    let mut hourly: BTreeMap<(String, String), BTreeMap<i64, BTreeSet<String>>> = BTreeMap::new();
+    // (orig_h, base) -> unique subdomains over the whole window (used by JSD)
+    let mut window: BTreeMap<(String, String), BTreeSet<String>> = BTreeMap::new();
+    // (orig_h, base) -> qtype counts and NXDOMAIN counters
+    let mut qtypes: BTreeMap<(String, String), BTreeMap<String, u64>> = BTreeMap::new();
+    let mut rcode_total: BTreeMap<(String, String), u64> = BTreeMap::new();
+    let mut rcode_nx: BTreeMap<(String, String), u64> = BTreeMap::new();
+
+    for query in &input.queries {
+        if query.query == "-" || query.query == "(empty)" || query.query.is_empty() {
+            continue;
+        }
+        let Some((base, sub)) = classify_query(&query.query, query.resp_p) else {
+            continue;
+        };
+        let key = (query.orig_h.clone(), base.clone());
+        let hour = (query.ts / 3600.0).floor() as i64;
+        hourly
+            .entry(key.clone())
+            .or_default()
+            .entry(hour)
+            .or_default()
+            .insert(sub.clone());
+        window.entry(key.clone()).or_default().insert(sub);
+
+        let qtype = query.qtype_name.clone().unwrap_or_else(|| "-".to_string());
+        *qtypes
+            .entry(key.clone())
+            .or_default()
+            .entry(qtype)
+            .or_insert(0) += 1;
+        *rcode_total.entry(key.clone()).or_insert(0) += 1;
+        if query.rcode_name.as_deref() == Some("NXDOMAIN") {
+            *rcode_nx.entry(key).or_insert(0) += 1;
+        }
+    }
+
+    // JSD reference: equal-weight mixture of every OTHER (orig_h, base) group
+    // in the window. It is evidence only (weight 0.0) and never enters score.
+    let window_counts: BTreeMap<(String, String), BTreeMap<char, u64>> = window
+        .iter()
+        .map(|(key, subs)| {
+            let mut counts: BTreeMap<char, u64> = BTreeMap::new();
+            for sub in subs {
+                for c in sub.chars().filter(|c| *c != '.') {
+                    *counts.entry(c).or_insert(0) += 1;
+                }
+            }
+            (key.clone(), counts)
+        })
+        .collect();
+    let distributions: BTreeMap<(String, String), BTreeMap<char, f64>> = window_counts
+        .iter()
+        .map(|(key, counts)| (key.clone(), normalized_counts(counts)))
+        .collect();
+    let group_count = distributions.len();
+
+    let mut results = Vec::with_capacity(hourly.len());
+    for ((orig_h, base_domain), hours) in &hourly {
+        let mut best_bits = -1.0f64;
+        let mut best_hour = 0i64;
+        let mut best_unique = 0usize;
+        let mut best_chars = 0usize;
+        let mut best_entropy = 0.0f64;
+        for (&hour, subs) in hours {
+            let (unique, chars, entropy, bits) = information(subs);
+            if bits > best_bits {
+                best_bits = bits;
+                best_hour = hour;
+                best_unique = unique;
+                best_chars = chars;
+                best_entropy = entropy;
+            }
+        }
+        let information_bits = best_bits.max(0.0);
+        let key = (orig_h.clone(), base_domain.clone());
+        let window_subs = &window[&key];
+
+        let mean_subdomain_len = if window_subs.is_empty() {
+            0.0
+        } else {
+            let total: usize = window_subs.iter().map(|s| s.len()).sum();
+            total as f64 / window_subs.len() as f64
+        };
+        let max_label_len = window_subs
+            .iter()
+            .flat_map(|s| s.split('.'))
+            .map(|label| label.len())
+            .max()
+            .unwrap_or(0);
+
+        let (jsd_value, jsd_reference_groups) = if group_count > 1 {
+            let own = &distributions[&key];
+            let divisor = (group_count - 1) as f64;
+            let mut reference: BTreeMap<char, f64> = BTreeMap::new();
+            for (other_key, dist) in &distributions {
+                if other_key == &key {
+                    continue;
+                }
+                for (&c, &v) in dist {
+                    *reference.entry(c).or_insert(0.0) += v / divisor;
+                }
+            }
+            reference.retain(|_, v| *v > 1e-15);
+            (jsd(own, &reference), group_count - 1)
+        } else {
+            (0.0, 0)
+        };
+
+        let total_rcode = *rcode_total.get(&key).unwrap_or(&0);
+        let nxdomain_ratio = if total_rcode == 0 {
+            0.0
+        } else {
+            *rcode_nx.get(&key).unwrap_or(&0) as f64 / total_rcode as f64
+        };
+
+        let status = if best_unique < MIN_UNIQUE {
+            "insufficient_data"
+        } else {
+            "ok"
+        };
+        let score = if best_unique < MIN_UNIQUE {
+            0.0
+        } else {
+            score_from_bits(information_bits)
+        };
+
+        results.push(ProtocolScore {
+            score,
+            status,
+            orig_h: orig_h.clone(),
+            base_domain: base_domain.clone(),
+            peak_hour_start_ts: best_hour as f64 * 3600.0,
+            unique_subdomains: best_unique,
+            subdomain_chars: best_chars,
+            entropy_bits_per_char: best_entropy,
+            information_bits,
+            mean_subdomain_len,
+            max_label_len,
+            qtype_counts: qtypes.get(&key).cloned().unwrap_or_default(),
+            nxdomain_ratio,
+            jsd: jsd_value,
+            jsd_reference_groups,
+            method: "dns-subdomain-information",
+            thresholds: thresholds(),
+            weights: weights(),
+            formula_version: "1.0",
+        });
+    }
+
+    Ok(results)
+}
+
+/// Stable `Alert.id`: SHA-256 of `3|orig_h|base_domain|peak_hour_start_ts`,
+/// truncated to 16 hex characters.
+pub fn alert_id(score: &ProtocolScore) -> String {
+    let mut hasher = Sha256::new();
+    hasher.update(
+        format!(
+            "3|{}|{}|{:.6}",
+            score.orig_h, score.base_domain, score.peak_hour_start_ts
+        )
+        .as_bytes(),
+    );
+    let hash = hasher.finalize();
+    let mut id = String::with_capacity(16);
+    for b in &hash[..8] {
+        use std::fmt::Write;
+        let _ = write!(id, "{b:02x}");
+    }
+    id
+}
+
+/// Builds the alert for a `ProtocolScore` that reached the 0.40 threshold.
+/// JSD travels in the evidence with weight 0.0; it never changes the score.
+pub fn alert_for(score: &ProtocolScore) -> Result<Alert> {
+    let summary = format!(
+        "High-information DNS subdomains: {} -> *.{} ({} bits/h, {} unique, score {:.3})",
+        score.orig_h,
+        score.base_domain,
+        score.information_bits.round() as u64,
+        score.unique_subdomains,
+        score.score
+    );
+    Ok(Alert {
+        id: alert_id(score),
+        pillar: 3,
+        score: score.score,
+        summary,
+        evidence: serde_json::to_value(score)?,
+        created_at: OffsetDateTime::now_utc().to_string(),
     })
 }
+
+pub fn alerts(scores: &[ProtocolScore]) -> Result<Vec<Alert>> {
+    let mut out = Vec::new();
+    for score in scores {
+        if score.status == "ok" && score.score >= ALERT_SCORE {
+            out.push(alert_for(score)?);
+        }
+    }
+    Ok(out)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    struct SplitMix64 {
+        state: u64,
+    }
+
+    impl SplitMix64 {
+        fn new(seed: u64) -> Self {
+            Self { state: seed }
+        }
+        fn next_u64(&mut self) -> u64 {
+            self.state = self.state.wrapping_add(0x9E37_79B9_7F4A_7C15);
+            let mut z = self.state;
+            z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+            z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+            z ^ (z >> 31)
+        }
+        fn below(&mut self, n: u64) -> u64 {
+            self.next_u64() % n
+        }
+    }
+
+    fn base32_char(rng: &mut SplitMix64) -> char {
+        const ALPHABET: &[u8] = b"abcdefghijklmnopqrstuvwxyz234567";
+        ALPHABET[rng.below(ALPHABET.len() as u64) as usize] as char
+    }
+
+    fn hex_char(rng: &mut SplitMix64) -> char {
+        const ALPHABET: &[u8] = b"0123456789abcdef";
+        ALPHABET[rng.below(ALPHABET.len() as u64) as usize] as char
+    }
+
+    fn base32_label(rng: &mut SplitMix64) -> String {
+        let len = 50 + rng.below(14) as usize;
+        (0..len).map(|_| base32_char(rng)).collect()
+    }
+
+    #[test]
+    fn entropy_of_a_repeated_character_is_zero() {
+        let mut counts = BTreeMap::new();
+        counts.insert('a', 4);
+        assert_eq!(entropy_bits_per_char(&counts), 0.0);
+    }
+
+    #[test]
+    fn entropy_of_uniform_32_symbols_is_five_bits() {
+        let mut counts = BTreeMap::new();
+        for i in 0..32u8 {
+            counts.insert((b'a' + i) as char, 1000);
+        }
+        assert!((entropy_bits_per_char(&counts) - 5.0).abs() < 1e-9);
+    }
+
+    #[test]
+    fn jsd_equal_distributions_is_zero_and_disjoint_is_one() {
+        let mut p = BTreeMap::new();
+        p.insert('a', 0.5);
+        p.insert('b', 0.5);
+        let same = p.clone();
+        assert!(jsd(&p, &same).abs() < 1e-12);
+
+        let mut q = BTreeMap::new();
+        q.insert('c', 1.0);
+        assert!((jsd(&p, &q) - 1.0).abs() < 1e-12);
+    }
+
+    #[test]
+    fn jsd_is_symmetric_and_finite_with_zero_support() {
+        let mut p = BTreeMap::new();
+        p.insert('a', 0.75);
+        p.insert('b', 0.25);
+        let mut q = BTreeMap::new();
+        q.insert('b', 1.0);
+        let forward = jsd(&p, &q);
+        let backward = jsd(&q, &p);
+        assert_eq!(forward.to_bits(), backward.to_bits());
+        assert!(forward.is_finite());
+    }
+
+    #[test]
+    fn base_domain_heuristic_matches_the_spec() {
+        assert_eq!(
+            split_base_domain("a.b.example.co.uk"),
+            Some(("example.co.uk".to_string(), "a.b".to_string()))
+        );
+        assert_eq!(
+            split_base_domain("x.example.com"),
+            Some(("example.com".to_string(), "x".to_string()))
+        );
+        assert_eq!(split_base_domain("example.com"), None);
+        assert_eq!(classify_query("foo.local", 53), None);
+        assert_eq!(classify_query("1.0.0.10.in-addr.arpa", 53), None);
+        assert_eq!(classify_query("x.example.com", 5353), None);
+    }
+
+    #[test]
+    fn synthetic_base32_tunnel_is_detected_on_50_seeds() {
+        for seed in 1..=50u64 {
+            let mut rng = SplitMix64::new(seed * 1000 + 100);
+            let mut queries = Vec::new();
+            for i in 0..100 {
+                let labels = 2 + (i % 2);
+                let mut sub = Vec::new();
+                for _ in 0..labels {
+                    sub.push(base32_label(&mut rng));
+                }
+                queries.push(DnsQuery {
+                    ts: 1_800_000_000.0 + (i as f64) * 30.0,
+                    orig_h: "10.66.66.66".to_string(),
+                    resp_p: 53,
+                    query: format!("{}.t.example-tunnel.net", sub.join(".")),
+                    qtype_name: Some("TXT".to_string()),
+                    rcode_name: Some("NOERROR".to_string()),
+                });
+            }
+            let scores = score(&ProtocolInput { queries }).unwrap();
+            let best = scores
+                .iter()
+                .max_by(|a, b| a.score.total_cmp(&b.score))
+                .unwrap();
+            assert!(best.score >= 0.40, "seed {seed}: score {}", best.score);
+        }
+    }
+
+    #[test]
+    fn synthetic_hex_tunnel_is_detected_on_50_seeds() {
+        for seed in 1..=50u64 {
+            let mut rng = SplitMix64::new(seed * 1000 + 101);
+            let mut queries = Vec::new();
+            for i in 0..100 {
+                let len = 60 + rng.below(4) as usize;
+                let label: String = (0..len).map(|_| hex_char(&mut rng)).collect();
+                let len2 = 60 + rng.below(4) as usize;
+                let label2: String = (0..len2).map(|_| hex_char(&mut rng)).collect();
+                queries.push(DnsQuery {
+                    ts: 1_800_000_000.0 + (i as f64) * 30.0,
+                    orig_h: "10.66.66.67".to_string(),
+                    resp_p: 53,
+                    query: format!("{label}.{label2}.example.com"),
+                    qtype_name: Some("TXT".to_string()),
+                    rcode_name: Some("NOERROR".to_string()),
+                });
+            }
+            let scores = score(&ProtocolInput { queries }).unwrap();
+            let best = scores
+                .iter()
+                .max_by(|a, b| a.score.total_cmp(&b.score))
+                .unwrap();
+            assert!(best.score >= 0.40, "seed {seed}: score {}", best.score);
+        }
+    }
+
+    #[test]
+    fn synthetic_cdn_cloudfront_is_zero_and_amazonaws_stays_below_alert() {
+        let mut rng = SplitMix64::new(42);
+        let mut cloudfront = Vec::new();
+        for i in 0..45 {
+            let mut name = String::from("d");
+            for _ in 0..13 {
+                let alphabet = b"abcdefghijklmnopqrstuvwxyz0123456789";
+                name.push(alphabet[rng.below(alphabet.len() as u64) as usize] as char);
+            }
+            cloudfront.push(DnsQuery {
+                ts: 1_800_000_000.0 + (i as f64) * 60.0,
+                orig_h: "10.0.2.15".to_string(),
+                resp_p: 53,
+                query: format!("{name}.cloudfront.net"),
+                qtype_name: Some("A".to_string()),
+                rcode_name: Some("NOERROR".to_string()),
+            });
+        }
+        let scores = score(&ProtocolInput {
+            queries: cloudfront,
+        })
+        .unwrap();
+        let best = scores
+            .iter()
+            .max_by(|a, b| a.score.total_cmp(&b.score))
+            .unwrap();
+        assert_eq!(best.score, 0.0);
+        assert!(best.information_bits < ZERO_BITS);
+
+        let mut amazon = Vec::new();
+        for i in 0..70 {
+            let a = rng.below(256);
+            let b = rng.below(256);
+            let c = rng.below(256);
+            let d = rng.below(256);
+            amazon.push(DnsQuery {
+                ts: 1_800_000_000.0 + (i as f64) * 40.0,
+                orig_h: "10.0.2.15".to_string(),
+                resp_p: 53,
+                query: format!("ec2-{a}-{b}-{c}-{d}.compute-1.amazonaws.com"),
+                qtype_name: Some("A".to_string()),
+                rcode_name: Some("NOERROR".to_string()),
+            });
+        }
+        let scores = score(&ProtocolInput { queries: amazon }).unwrap();
+        let best = scores
+            .iter()
+            .max_by(|a, b| a.score.total_cmp(&b.score))
+            .unwrap();
+        assert!(
+            best.score < 0.40,
+            "amazonaws score {} ({} bits)",
+            best.score,
+            best.information_bits
+        );
+        assert!(best.status == "ok");
+    }
+
+    #[test]
+    fn evidence_is_byte_for_byte_deterministic() {
+        let mut rng = SplitMix64::new(7);
+        let mut queries = Vec::new();
+        for i in 0..200 {
+            let name = base32_label(&mut rng);
+            let host = if i % 3 == 0 { "10.0.0.1" } else { "10.0.0.2" };
+            let base = if i % 5 == 0 {
+                "example.com"
+            } else {
+                "example.net"
+            };
+            queries.push(DnsQuery {
+                ts: 1_800_000_000.0 + (i as f64) * 17.0,
+                orig_h: host.to_string(),
+                resp_p: 53,
+                query: format!("{name}.{base}"),
+                qtype_name: Some("A".to_string()),
+                rcode_name: Some(if i % 11 == 0 { "NXDOMAIN" } else { "NOERROR" }.to_string()),
+            });
+        }
+        let first = serde_json::to_vec(
+            &score(&ProtocolInput {
+                queries: queries.clone(),
+            })
+            .unwrap(),
+        )
+        .unwrap();
+        let second = serde_json::to_vec(&score(&ProtocolInput { queries }).unwrap()).unwrap();
+        assert_eq!(first, second);
+    }
+}

crates/acaua-core/src/pillar4.rs

@@ -1,26 +1,275 @@
 use crate::error::Result;
 use serde::{Deserialize, Serialize};
 
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub const WEIGHT_P1: f64 = 0.75;
+pub const WEIGHT_P2: f64 = 0.75;
+pub const WEIGHT_P3: f64 = 0.75;
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct FusionWeights {
+    pub p1: f64,
+    pub p2: f64,
+    pub p3: f64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct FusionTerm {
+    pub pillar: u8,
+    pub alert_id: Option<String>,
+    pub m: f64,
+    pub w: f64,
+    pub weighted: f64,
+    pub complement: f64,
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)]
 pub struct FusionInput {
-    pub beaconing: f64,
+    pub m1: f64,
-    pub sessions: f64,
+    pub m2: f64,
-    pub protocol: f64,
+    pub m3: f64,
+    #[serde(default)]
+    pub alert_id_p1: Option<String>,
+    #[serde(default)]
+    pub alert_id_p2: Option<String>,
+    #[serde(default)]
+    pub alert_id_p3: Option<String>,
 }
 
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
-pub struct FusionScore {
+pub struct HostScore {
+    pub orig_h: String,
     pub score: f64,
-    pub inputs: usize,
+    pub terms: Vec<FusionTerm>,
+    pub product: f64,
     pub method: &'static str,
-    pub status: &'static str,
+    pub weights: FusionWeights,
+    pub order: [u8; 3],
+    pub formula_version: &'static str,
 }
 
-pub fn score(input: &FusionInput) -> Result<FusionScore> {
+pub fn score(orig_h: &str, input: &FusionInput) -> Result<HostScore> {
-    Ok(FusionScore {
+    let t1_weighted = WEIGHT_P1 * input.m1;
-        score: 0.0,
+    let t1_comp = 1.0 - t1_weighted;
-        inputs: [input.beaconing, input.sessions, input.protocol].len(),
+    let term1 = FusionTerm {
-        method: "deterministic-fusion-stub",
+        pillar: 1,
-        status: "skeleton",
+        alert_id: input.alert_id_p1.clone(),
+        m: input.m1,
+        w: WEIGHT_P1,
+        weighted: t1_weighted,
+        complement: t1_comp,
+    };
+
+    let t2_weighted = WEIGHT_P2 * input.m2;
+    let t2_comp = 1.0 - t2_weighted;
+    let term2 = FusionTerm {
+        pillar: 2,
+        alert_id: input.alert_id_p2.clone(),
+        m: input.m2,
+        w: WEIGHT_P2,
+        weighted: t2_weighted,
+        complement: t2_comp,
+    };
+
+    let t3_weighted = WEIGHT_P3 * input.m3;
+    let t3_comp = 1.0 - t3_weighted;
+    let term3 = FusionTerm {
+        pillar: 3,
+        alert_id: input.alert_id_p3.clone(),
+        m: input.m3,
+        w: WEIGHT_P3,
+        weighted: t3_weighted,
+        complement: t3_comp,
+    };
+
+    // Product computed in fixed order 1, 2, 3
+    let product = t1_comp * t2_comp * t3_comp;
+    let s = (1.0 - product).clamp(0.0, 1.0);
+
+    Ok(HostScore {
+        orig_h: orig_h.to_string(),
+        score: s,
+        terms: vec![term1, term2, term3],
+        product,
+        method: "weighted-noisy-or",
+        weights: FusionWeights {
+            p1: WEIGHT_P1,
+            p2: WEIGHT_P2,
+            p3: WEIGHT_P3,
+        },
+        order: [1, 2, 3],
+        formula_version: "1.0",
     })
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_1_noisy_or_properties() {
+        // Zero input -> score 0
+        let zero = score(
+            "10.0.0.1",
+            &FusionInput {
+                m1: 0.0,
+                m2: 0.0,
+                m3: 0.0,
+                ..Default::default()
+            },
+        )
+        .unwrap();
+        assert_eq!(zero.score, 0.0);
+
+        // Exact ceilings
+        let one_pillar = score(
+            "10.0.0.1",
+            &FusionInput {
+                m1: 1.0,
+                m2: 0.0,
+                m3: 0.0,
+                ..Default::default()
+            },
+        )
+        .unwrap();
+        assert_eq!(one_pillar.score, 0.75);
+
+        let two_pillars = score(
+            "10.0.0.1",
+            &FusionInput {
+                m1: 1.0,
+                m2: 1.0,
+                m3: 0.0,
+                ..Default::default()
+            },
+        )
+        .unwrap();
+        assert_eq!(two_pillars.score, 0.9375);
+
+        let three_pillars = score(
+            "10.0.0.1",
+            &FusionInput {
+                m1: 1.0,
+                m2: 1.0,
+                m3: 1.0,
+                ..Default::default()
+            },
+        )
+        .unwrap();
+        assert_eq!(three_pillars.score, 0.984375);
+
+        // Monotonicity over 11x11x11 grid
+        for i in 0..=10 {
+            let m1 = i as f64 / 10.0;
+            for j in 0..=10 {
+                let m2 = j as f64 / 10.0;
+                for k in 0..=10 {
+                    let m3 = k as f64 / 10.0;
+                    let base = score(
+                        "10.0.0.1",
+                        &FusionInput {
+                            m1,
+                            m2,
+                            m3,
+                            ..Default::default()
+                        },
+                    )
+                    .unwrap()
+                    .score;
+
+                    if i < 10 {
+                        let step = score(
+                            "10.0.0.1",
+                            &FusionInput {
+                                m1: m1 + 0.1,
+                                m2,
+                                m3,
+                                ..Default::default()
+                            },
+                        )
+                        .unwrap()
+                        .score;
+                        assert!(step >= base);
+                    }
+                    if j < 10 {
+                        let step = score(
+                            "10.0.0.1",
+                            &FusionInput {
+                                m1,
+                                m2: m2 + 0.1,
+                                m3,
+                                ..Default::default()
+                            },
+                        )
+                        .unwrap()
+                        .score;
+                        assert!(step >= base);
+                    }
+                    if k < 10 {
+                        let step = score(
+                            "10.0.0.1",
+                            &FusionInput {
+                                m1,
+                                m2,
+                                m3: m3 + 0.1,
+                                ..Default::default()
+                            },
+                        )
+                        .unwrap()
+                        .score;
+                        assert!(step >= base);
+                    }
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn test_2_validation_table_vectors() {
+        let cases = [
+            ("192.168.1.132", 1.000, 0.971, 0.0, 0.932), // Philips HUE benign
+            ("192.168.100.113", 0.829, 0.0, 0.0, 0.622), // Hakai C&C
+            ("192.168.1.195", 0.0, 0.821, 0.0, 0.616),   // Mirai C&C
+            ("192.168.100.103", 0.0, 0.790, 0.0, 0.593), // Torii C&C
+            ("10.0.2.15", 0.0, 0.582, 0.0, 0.436),       // CTU-Normal-26
+            ("192.168.1.114", 0.471, 0.0, 0.0, 0.353),   // CTU-Normal-18
+            ("192.168.1.197", 0.0, 0.0, 0.0, 0.0),       // 42-1 Trojan
+            ("10.0.2.15-20", 0.0, 0.0, 0.0, 0.0),        // CTU-Normal-20
+        ];
+
+        for (host, m1, m2, m3, expected_s) in cases {
+            let res = score(
+                host,
+                &FusionInput {
+                    m1,
+                    m2,
+                    m3,
+                    ..Default::default()
+                },
+            )
+            .unwrap();
+            assert!(
+                (res.score - expected_s).abs() < 1e-3,
+                "host {host} score {} != expected {}",
+                res.score,
+                expected_s
+            );
+        }
+    }
+
+    #[test]
+    fn test_3_determinism() {
+        let input = FusionInput {
+            m1: 0.829,
+            m2: 0.790,
+            m3: 0.450,
+            alert_id_p1: Some("p1_id".to_string()),
+            alert_id_p2: Some("p2_id".to_string()),
+            alert_id_p3: Some("p3_id".to_string()),
+        };
+        let s1 = score("10.0.0.1", &input).unwrap();
+        let s2 = score("10.0.0.1", &input).unwrap();
+        let j1 = serde_json::to_vec(&s1).unwrap();
+        let j2 = serde_json::to_vec(&s2).unwrap();
+        assert_eq!(j1, j2);
+    }
+}

crates/acaua-core/src/storage.rs

@@ -1,8 +1,8 @@
 use crate::error::{AcauaError, Result};
 use chrono::NaiveDate;
-use serde::Serialize;
+use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
 use std::fs;
 use std::path::{Path, PathBuf};
 
@@ -27,12 +27,14 @@ impl DatasetSummary {
     }
 }
 
-#[derive(Debug, Clone, Default)]
+#[derive(Debug, Clone, Default, Serialize)]
 pub struct IngestionStats {
     pub files_seen: usize,
     pub records_ingested: u64,
     pub lines_skipped: u64,
     pub partitions_affected: usize,
+    pub files_failed: usize,
+    pub failed_errors: Vec<String>,
 }
 
 #[derive(Debug, Clone)]
@@ -40,9 +42,35 @@ pub struct TupleTimestamps {
     pub orig_h: String,
     pub resp_h: String,
     pub resp_p: u16,
+    pub proto: String,
     pub timestamps: Vec<f64>,
 }
 
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ChainedSession {
+    pub orig_h: String,
+    pub orig_p: u16,
+    pub resp_h: String,
+    pub resp_p: u16,
+    pub proto: String,
+    pub start_ts: f64,
+    pub end_ts: f64,
+    pub duration: f64,
+    pub parts: usize,
+    pub orig_bytes: u64,
+    pub resp_bytes: u64,
+}
+
+#[derive(Debug, Clone)]
+pub struct TupleSessions {
+    pub orig_h: String,
+    pub resp_h: String,
+    pub resp_p: u16,
+    pub proto: String,
+    pub sessions: Vec<ChainedSession>,
+    pub failed_attempts: usize,
+}
+
 #[derive(Debug, Clone)]
 pub struct Storage {
     root: PathBuf,
@@ -59,6 +87,12 @@ pub struct ConnRecord {
     pub proto: String,
     pub service: Option<String>,
     pub duration: Option<f64>,
+    pub orig_bytes: Option<u64>,
+    pub resp_bytes: Option<u64>,
+    pub conn_state: Option<String>,
+    pub history: Option<String>,
+    pub orig_pkts: Option<u64>,
+    pub resp_pkts: Option<u64>,
 }
 
 impl Storage {
@@ -133,7 +167,7 @@ impl Storage {
 
         let query = format!(
             "SELECT count(*)::BIGINT, count(DISTINCT date)::BIGINT
-             FROM read_parquet('{parquet_glob}', hive_partitioning = true);"
+             FROM read_parquet('{parquet_glob}', hive_partitioning = true, union_by_name = true);"
         );
 
         let row = conn.query_row(&query, [], |r| {
@@ -193,6 +227,26 @@ impl Storage {
             }
         }
 
+        // Also prune raw/dns if exists
+        let dns_dir = self.root.join("raw").join("dns");
+        if let Ok(dns_entries) = fs::read_dir(&dns_dir) {
+            for entry in dns_entries.flatten() {
+                let path = entry.path();
+                if !path.is_dir() {
+                    continue;
+                }
+                if let Some(name) = path.file_name().and_then(|n| n.to_str()) {
+                    if let Some(date_str) = name.strip_prefix("date=") {
+                        if let Ok(d) = NaiveDate::parse_from_str(date_str, "%Y-%m-%d") {
+                            if d <= cutoff && fs::remove_dir_all(&path).is_ok() {
+                                pruned += 1;
+                            }
+                        }
+                    }
+                }
+            }
+        }
+
         Ok(pruned)
     }
 
@@ -203,6 +257,7 @@ impl Storage {
 
         let mut separator = '\t';
         let mut unset_field = String::from("-");
+        let mut empty_field = String::from("(empty)");
         let mut fields: Vec<String> = Vec::new();
 
         let mut idx_ts = None;
@@ -214,6 +269,12 @@ impl Storage {
         let mut idx_proto = None;
         let mut idx_service = None;
         let mut idx_duration = None;
+        let mut idx_orig_bytes = None;
+        let mut idx_resp_bytes = None;
+        let mut idx_conn_state = None;
+        let mut idx_history = None;
+        let mut idx_orig_pkts = None;
+        let mut idx_resp_pkts = None;
 
         let text = String::from_utf8_lossy(content);
 
@@ -235,6 +296,8 @@ impl Storage {
                     }
                 } else if let Some(rest) = line.strip_prefix("#unset_field") {
                     unset_field = rest.trim().to_string();
+                } else if let Some(rest) = line.strip_prefix("#empty_field") {
+                    empty_field = rest.trim().to_string();
                 } else if let Some(rest) = line.strip_prefix("#fields") {
                     let rest = rest.trim();
                     fields = rest
@@ -251,11 +314,16 @@ impl Storage {
                     idx_proto = fields.iter().position(|s| s == "proto");
                     idx_service = fields.iter().position(|s| s == "service");
                     idx_duration = fields.iter().position(|s| s == "duration");
+                    idx_orig_bytes = fields.iter().position(|s| s == "orig_bytes");
+                    idx_resp_bytes = fields.iter().position(|s| s == "resp_bytes");
+                    idx_conn_state = fields.iter().position(|s| s == "conn_state");
+                    idx_history = fields.iter().position(|s| s == "history");
+                    idx_orig_pkts = fields.iter().position(|s| s == "orig_pkts");
+                    idx_resp_pkts = fields.iter().position(|s| s == "resp_pkts");
                 }
                 continue;
             }
 
-            // Must have fields mapped
             let (i_ts, i_uid, i_oh, i_op, i_rh, i_rp, i_pr) = match (
                 idx_ts, idx_uid, idx_orig_h, idx_orig_p, idx_resp_h, idx_resp_p, idx_proto,
             ) {
@@ -298,25 +366,29 @@ impl Storage {
                 }
             };
 
-            let service = idx_service.and_then(|idx| {
+            let get_val = |idx: Option<usize>| -> Option<&str> {
-                let val = parts.get(idx)?;
+                let i = idx?;
-                if *val == unset_field.as_str() || *val == "-" {
+                let val = parts.get(i)?;
+                if *val == unset_field.as_str()
+                    || *val == "-"
+                    || *val == empty_field.as_str()
+                    || *val == "(empty)"
+                {
                     None
                 } else {
-                    Some(val.to_string())
+                    Some(*val)
                 }
-            });
+            };
 
-            let duration = idx_duration.and_then(|idx| {
+            let service = get_val(idx_service).map(|s| s.to_string());
-                let val = parts.get(idx)?;
+            let duration = get_val(idx_duration).and_then(|s| s.parse::<f64>().ok());
-                if *val == unset_field.as_str() || *val == "-" {
+            let orig_bytes = get_val(idx_orig_bytes).and_then(|s| s.parse::<u64>().ok());
-                    None
+            let resp_bytes = get_val(idx_resp_bytes).and_then(|s| s.parse::<u64>().ok());
-                } else {
+            let conn_state = get_val(idx_conn_state).map(|s| s.to_string());
-                    val.parse::<f64>().ok()
+            let history = get_val(idx_history).map(|s| s.to_string());
-                }
+            let orig_pkts = get_val(idx_orig_pkts).and_then(|s| s.parse::<u64>().ok());
-            });
+            let resp_pkts = get_val(idx_resp_pkts).and_then(|s| s.parse::<u64>().ok());
 
-            // Track day partition (UTC)
             let date_ts = chrono::DateTime::from_timestamp(ts.floor() as i64, 0);
             if let Some(dt) = date_ts {
                 dates.insert(dt.format("%Y-%m-%d").to_string());
@@ -332,13 +404,19 @@ impl Storage {
                 proto: parts[i_pr].to_string(),
                 service,
                 duration,
+                orig_bytes,
+                resp_bytes,
+                conn_state,
+                history,
+                orig_pkts,
+                resp_pkts,
             });
         }
 
         (records, skipped, dates)
     }
 
-    pub fn ingest_file(&self, path: impl AsRef<Path>) -> Result<(u64, u64, usize)> {
+    pub fn ingest_file(&self, path: impl AsRef<Path>) -> Result<(u64, u64, HashSet<String>)> {
         let path = path.as_ref();
         let content = fs::read(path)?;
         let mut hasher = Sha256::new();
@@ -352,7 +430,7 @@ impl Storage {
 
         let (records, skipped, dates) = Self::parse_conn_log(&content);
         if records.is_empty() {
-            return Ok((0, skipped, 0));
+            return Ok((0, skipped, dates));
         }
 
         let conn = self.open_connection()?;
@@ -366,7 +444,13 @@ impl Storage {
                 resp_p USMALLINT,
                 proto VARCHAR,
                 service VARCHAR,
-                duration DOUBLE
+                duration DOUBLE,
+                orig_bytes UBIGINT,
+                resp_bytes UBIGINT,
+                conn_state VARCHAR,
+                history VARCHAR,
+                orig_pkts UBIGINT,
+                resp_pkts UBIGINT
             );",
         )?;
 
@@ -374,8 +458,21 @@ impl Storage {
             let mut appender = conn.appender("t")?;
             for r in &records {
                 appender.append_row(duckdb::params![
-                    r.ts, r.uid, r.orig_h, r.orig_p, r.resp_h, r.resp_p, r.proto, r.service,
+                    r.ts,
-                    r.duration
+                    r.uid,
+                    r.orig_h,
+                    r.orig_p,
+                    r.resp_h,
+                    r.resp_p,
+                    r.proto,
+                    r.service,
+                    r.duration,
+                    r.orig_bytes,
+                    r.resp_bytes,
+                    r.conn_state,
+                    r.history,
+                    r.orig_pkts,
+                    r.resp_pkts
                 ])?;
             }
         }
@@ -401,7 +498,7 @@ impl Storage {
         conn.execute_batch(&copy_sql)?;
         conn.execute_batch("DROP TABLE t;")?;
 
-        Ok((records.len() as u64, skipped, dates.len()))
+        Ok((records.len() as u64, skipped, dates))
     }
 
     pub fn ingest_dir(&self, directory: impl AsRef<Path>) -> Result<IngestionStats> {
@@ -421,7 +518,6 @@ impl Storage {
         let entries = fs::read_dir(&canonical_dir)?;
         for entry in entries.flatten() {
             let path = entry.path();
-            // Path confinement: do not follow symlink to outside directory
             if let Ok(canon) = fs::canonicalize(&path) {
                 if !canon.starts_with(&canonical_dir) {
                     continue;
@@ -435,27 +531,26 @@ impl Storage {
                 None => continue,
             };
 
-            // Skip .gz archives with note
             if file_name.ends_with(".gz") {
                 continue;
             }
 
-            // Only conn.log and conn.*.log directly in directory
             let is_conn = file_name == "conn.log"
                 || (file_name.starts_with("conn.") && file_name.ends_with(".log"));
 
             if is_conn && path.is_file() {
                 stats.files_seen += 1;
                 match self.ingest_file(&path) {
-                    Ok((recs, sk, affected)) => {
+                    Ok((recs, sk, dates)) => {
                         stats.records_ingested += recs;
                         stats.lines_skipped += sk;
-                        for _ in 0..affected {
+                        all_dates.extend(dates);
-                            all_dates.insert(file_name.to_string());
-                        }
                     }
-                    Err(_) => {
+                    Err(e) => {
-                        stats.lines_skipped += 1;
+                        stats.files_failed += 1;
+                        if stats.failed_errors.len() < 10 {
+                            stats.failed_errors.push(e.to_string());
+                        }
                     }
                 }
             }
@@ -465,7 +560,7 @@ impl Storage {
         Ok(stats)
     }
 
-    pub fn query_7day_tuples(&self) -> Result<Vec<TupleTimestamps>> {
+    pub fn query_7day_sessions(&self) -> Result<Vec<TupleSessions>> {
         if !self.has_parquet_files() {
             return Ok(Vec::new());
         }
@@ -483,7 +578,7 @@ impl Storage {
         // Find max date present in data
         let max_date_query = format!(
             "SELECT max(date)::VARCHAR
-             FROM read_parquet('{parquet_glob}', hive_partitioning = true);"
+             FROM read_parquet('{parquet_glob}', hive_partitioning = true, union_by_name = true);"
         );
 
         let max_date: Option<String> = conn
@@ -495,47 +590,193 @@ impl Storage {
             _ => return Ok(Vec::new()),
         };
 
+        // Query all conn rows in 7-day window ordered by orig_h, orig_p, resp_h, resp_p, ts
         let query = format!(
-            "WITH daily AS (
+            "SELECT ts, orig_h, orig_p, resp_h, resp_p, proto, duration, orig_bytes, resp_bytes, conn_state, history
-               SELECT date, orig_h, resp_h, resp_p, list(ts ORDER BY ts) AS tss
+             FROM read_parquet('{parquet_glob}', hive_partitioning = true, union_by_name = true)
-               FROM read_parquet('{parquet_glob}', hive_partitioning = true)
+             WHERE date > CAST('{max_date_str}' AS DATE) - INTERVAL 7 DAY
-               WHERE date > CAST('{max_date_str}' AS DATE) - INTERVAL 7 DAY
+             ORDER BY orig_h, orig_p, resp_h, resp_p, ts;"
-               GROUP BY ALL
-             )
-             SELECT orig_h, resp_h, resp_p, list_sort(flatten(list(tss))) AS timestamps
-             FROM daily
-             GROUP BY ALL
-             ORDER BY orig_h, resp_h, resp_p;"
         );
 
+        struct RawRow {
+            ts: f64,
+            orig_h: String,
+            orig_p: u16,
+            resp_h: String,
+            resp_p: u16,
+            proto: String,
+            duration: Option<f64>,
+            orig_bytes: Option<u64>,
+            resp_bytes: Option<u64>,
+            conn_state: Option<String>,
+            history: Option<String>,
+        }
+
         let mut stmt = conn.prepare(&query)?;
         let rows = stmt.query_map([], |row| {
-            let orig_h: String = row.get(0)?;
+            Ok(RawRow {
-            let resp_h: String = row.get(1)?;
+                ts: row.get(0)?,
-            let resp_p: u16 = row.get(2)?;
+                orig_h: row.get(1)?,
-            let timestamps_val: duckdb::types::Value = row.get(3)?;
+                orig_p: row.get(2)?,
-            let mut timestamps = Vec::new();
+                resp_h: row.get(3)?,
-            if let duckdb::types::Value::List(items) = timestamps_val {
+                resp_p: row.get(4)?,
-                for item in items {
+                proto: row.get(5)?,
-                    if let duckdb::types::Value::Double(t) = item {
+                duration: row.get(6)?,
-                        timestamps.push(t);
+                orig_bytes: row.get(7)?,
-                    }
+                resp_bytes: row.get(8)?,
-                }
+                conn_state: row.get(9)?,
-            }
+                history: row.get(10)?,
-            Ok(TupleTimestamps {
-                orig_h,
-                resp_h,
-                resp_p,
-                timestamps,
             })
         })?;
 
-        let mut result = Vec::new();
+        let mut raw_rows = Vec::new();
+        let mut failed_map: HashMap<(String, String, u16), usize> = HashMap::new();
+
         for r in rows {
-            result.push(r?);
+            let row = r?;
+            if row.proto == "tcp" {
+                if let Some(cs) = &row.conn_state {
+                    if cs == "S0" || cs == "REJ" || cs == "RSTOS0" || cs == "SH" {
+                        *failed_map
+                            .entry((row.orig_h.clone(), row.resp_h.clone(), row.resp_p))
+                            .or_insert(0) += 1;
+                    }
+                }
+            }
+            raw_rows.push(row);
         }
+
+        // Group rows by 4-tuple (orig_h, orig_p, resp_h, resp_p, proto)
+        // Since rows are already ordered by (orig_h, orig_p, resp_h, resp_p, ts),
+        // we can chain sequentially.
+        let mut all_sessions: Vec<ChainedSession> = Vec::new();
+        let mut current_chain: Option<ChainedSession> = None;
+
+        for r in raw_rows {
+            let row_end = r.ts + r.duration.unwrap_or(0.0);
+            if r.proto == "tcp" {
+                let matches_current_4tuple = match &current_chain {
+                    Some(curr) => {
+                        curr.orig_h == r.orig_h
+                            && curr.orig_p == r.orig_p
+                            && curr.resp_h == r.resp_h
+                            && curr.resp_p == r.resp_p
+                            && curr.proto == r.proto
+                    }
+                    None => false,
+                };
+
+                let is_continuation = if matches_current_4tuple {
+                    let curr = current_chain.as_ref().unwrap();
+                    match &r.history {
+                        Some(h) => !h.contains('S') && (r.ts - curr.end_ts <= 3600.0),
+                        None => false,
+                    }
+                } else {
+                    false
+                };
+
+                if is_continuation {
+                    let curr = current_chain.as_mut().unwrap();
+                    if row_end > curr.end_ts {
+                        curr.end_ts = row_end;
+                    }
+                    curr.duration = curr.end_ts - curr.start_ts;
+                    curr.parts += 1;
+                    curr.orig_bytes += r.orig_bytes.unwrap_or(0);
+                    curr.resp_bytes += r.resp_bytes.unwrap_or(0);
+                } else {
+                    if let Some(finished) = current_chain.take() {
+                        all_sessions.push(finished);
+                    }
+                    current_chain = Some(ChainedSession {
+                        orig_h: r.orig_h,
+                        orig_p: r.orig_p,
+                        resp_h: r.resp_h,
+                        resp_p: r.resp_p,
+                        proto: r.proto,
+                        start_ts: r.ts,
+                        end_ts: row_end,
+                        duration: row_end - r.ts,
+                        parts: 1,
+                        orig_bytes: r.orig_bytes.unwrap_or(0),
+                        resp_bytes: r.resp_bytes.unwrap_or(0),
+                    });
+                }
+            } else {
+                if let Some(finished) = current_chain.take() {
+                    all_sessions.push(finished);
+                }
+                all_sessions.push(ChainedSession {
+                    orig_h: r.orig_h,
+                    orig_p: r.orig_p,
+                    resp_h: r.resp_h,
+                    resp_p: r.resp_p,
+                    proto: r.proto,
+                    start_ts: r.ts,
+                    end_ts: row_end,
+                    duration: row_end - r.ts,
+                    parts: 1,
+                    orig_bytes: r.orig_bytes.unwrap_or(0),
+                    resp_bytes: r.resp_bytes.unwrap_or(0),
+                });
+            }
+        }
+        if let Some(finished) = current_chain {
+            all_sessions.push(finished);
+        }
+
+        // Group sessions by (orig_h, resp_h, resp_p, proto)
+        let mut grouped: HashMap<(String, String, u16, String), Vec<ChainedSession>> =
+            HashMap::new();
+        for session in all_sessions {
+            let key = (
+                session.orig_h.clone(),
+                session.resp_h.clone(),
+                session.resp_p,
+                session.proto.clone(),
+            );
+            grouped.entry(key).or_default().push(session);
+        }
+
+        let mut result = Vec::new();
+        for ((orig_h, resp_h, resp_p, proto), mut sessions) in grouped {
+            sessions.sort_by(|a, b| a.start_ts.partial_cmp(&b.start_ts).unwrap());
+            let failed_attempts = failed_map
+                .get(&(orig_h.clone(), resp_h.clone(), resp_p))
+                .copied()
+                .unwrap_or(0);
+            result.push(TupleSessions {
+                orig_h,
+                resp_h,
+                resp_p,
+                proto,
+                sessions,
+                failed_attempts,
+            });
+        }
+
+        result.sort_by(|a, b| {
+            (&a.orig_h, &a.resp_h, a.resp_p, &a.proto)
+                .cmp(&(&b.orig_h, &b.resp_h, b.resp_p, &b.proto))
+        });
+
         Ok(result)
     }
+
+    pub fn query_7day_tuples(&self) -> Result<Vec<TupleTimestamps>> {
+        let tuple_sessions = self.query_7day_sessions()?;
+        Ok(tuple_sessions
+            .into_iter()
+            .map(|ts| TupleTimestamps {
+                orig_h: ts.orig_h,
+                resp_h: ts.resp_h,
+                resp_p: ts.resp_p,
+                proto: ts.proto,
+                timestamps: ts.sessions.into_iter().map(|s| s.start_ts).collect(),
+            })
+            .collect())
+    }
 }
 
 #[cfg(test)]
@@ -548,15 +789,15 @@ mod tests {
 #unset_field\t-\n\
 #path\tconn\n\
 #open\t2026-10-31-12-00-00\n\
-#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\tservice\tduration\tip_proto\n\
+#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\tservice\tduration\torig_bytes\tresp_bytes\tconn_state\thistory\torig_pkts\tresp_pkts\n\
-#types\ttime\tstring\taddr\tport\taddr\tport\tenum\tstring\tinterval\tcount\n\
+#types\ttime\tstring\taddr\tport\taddr\tport\tenum\tstring\tinterval\tcount\tcount\tstring\tstring\tcount\tcount\n\
-1793461200.000000\tCuid1\t192.168.1.10\t45000\t10.0.0.1\t443\ttcp\tssl\t1.234\t6\n\
+1793461200.000000\tCuid1\t192.168.1.10\t45000\t10.0.0.1\t443\ttcp\tssl\t1.234\t100\t200\tSF\tShADadFf\t10\t12\n\
-1793461205.000000\tCuid2\t192.168.1.10\t45001\t10.0.0.1\t443\ttcp\t-\t-\t6\n\
+1793461205.000000\tCuid2\t192.168.1.10\t45001\t10.0.0.1\t443\ttcp\t-\t-\t-\t-\tS0\tS\t1\t0\n\
 #close\t2026-10-31-12-01-00\n";
 
     const CONN_LOG_PERMUTED_COLS: &str = "#separator \\x09\n\
-#fields\tid.resp_p\tid.orig_h\tservice\tuid\tduration\tproto\tts\tid.resp_h\tid.orig_p\n\
+#fields\tid.resp_p\tid.orig_h\tservice\tuid\tduration\tproto\tts\tid.resp_h\tid.orig_p\tconn_state\thistory\n\
-443\t192.168.1.10\thttp\tCuid3\t0.5\ttcp\t1793461210.000000\t10.0.0.1\t45002\n";
+443\t192.168.1.10\thttp\tCuid3\t0.5\ttcp\t1793461210.000000\t10.0.0.1\t45002\tSF\tShAD\n";
 
     #[test]
     fn test_parser_extracts_correct_fields_and_unset_as_none() {
@@ -573,10 +814,19 @@ mod tests {
         assert_eq!(records[0].proto, "tcp");
         assert_eq!(records[0].service.as_deref(), Some("ssl"));
         assert_eq!(records[0].duration, Some(1.234));
+        assert_eq!(records[0].orig_bytes, Some(100));
+        assert_eq!(records[0].resp_bytes, Some(200));
+        assert_eq!(records[0].conn_state.as_deref(), Some("SF"));
+        assert_eq!(records[0].history.as_deref(), Some("ShADadFf"));
+        assert_eq!(records[0].orig_pkts, Some(10));
+        assert_eq!(records[0].resp_pkts, Some(12));
 
         assert_eq!(records[1].uid, "Cuid2");
         assert_eq!(records[1].service, None);
         assert_eq!(records[1].duration, None);
+        assert_eq!(records[1].orig_bytes, None);
+        assert_eq!(records[1].conn_state.as_deref(), Some("S0"));
+        assert_eq!(records[1].history.as_deref(), Some("S"));
     }
 
     #[test]
@@ -611,7 +861,6 @@ mod tests {
         assert_eq!(summary1.records, 2);
         assert_eq!(summary1.partitions, 1);
 
-        // Idempotence: re-ingesting the exact same file does not duplicate records
         let (ingested2, skipped2, _) = store.ingest_file(&log_file).unwrap();
         assert_eq!(ingested2, 2);
         assert_eq!(skipped2, 0);
@@ -619,7 +868,6 @@ mod tests {
         let summary2 = store.summary().unwrap();
         assert_eq!(summary2.records, 2);
 
-        // Ingesting a different file for the same tuple/day adds to it
         let log_file_2 = temp_dir.path().join("conn.extra.log");
         fs::write(&log_file_2, CONN_LOG_PERMUTED_COLS).unwrap();
         store.ingest_file(&log_file_2).unwrap();
@@ -632,6 +880,7 @@ mod tests {
         assert_eq!(tuples[0].orig_h, "192.168.1.10");
         assert_eq!(tuples[0].resp_h, "10.0.0.1");
         assert_eq!(tuples[0].resp_p, 443);
+        assert_eq!(tuples[0].proto, "tcp");
         assert_eq!(tuples[0].timestamps.len(), 3);
         assert_eq!(tuples[0].timestamps[0], 1793461200.0);
         assert_eq!(tuples[0].timestamps[1], 1793461205.0);
@@ -643,7 +892,6 @@ mod tests {
         let temp_dir = tempfile::tempdir().unwrap();
         let store = Storage::open(temp_dir.path()).unwrap();
 
-        // 2026-10-31 02:00:00 UTC = 1793412000
         let log = "#separator \\x09\n#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\n1793412000.000000\tu1\t10.0.0.1\t1000\t10.0.0.2\t80\ttcp\n";
         let log_file = temp_dir.path().join("conn.log");
         fs::write(&log_file, log).unwrap();
@@ -676,7 +924,6 @@ mod tests {
         let store = Storage::open(temp_dir.path()).unwrap();
 
         let raw_conn = temp_dir.path().join("raw").join("conn");
-        // Day 1 (10 days ago) and Day 10 (recent)
         let old_part = raw_conn.join("date=2026-10-01");
         let new_part = raw_conn.join("date=2026-10-11");
         fs::create_dir_all(&old_part).unwrap();
@@ -687,4 +934,154 @@ mod tests {
         assert!(!old_part.exists());
         assert!(new_part.exists());
     }
+
+    #[test]
+    fn test_union_by_name_v11_and_v12_compatibility() {
+        let temp_dir = tempfile::tempdir().unwrap();
+        let store = Storage::open(temp_dir.path()).unwrap();
+        let conn = store.open_connection().unwrap();
+
+        let part_dir = store
+            .root()
+            .join("raw")
+            .join("conn")
+            .join("date=2026-10-31");
+        fs::create_dir_all(&part_dir).unwrap();
+        let part_str = part_dir.to_string_lossy().replace('\'', "''");
+
+        // Write v1.1 format (no history, no orig_bytes etc.)
+        conn.execute_batch(&format!(
+            "CREATE TEMP TABLE v11 (ts DOUBLE, uid VARCHAR, orig_h VARCHAR, orig_p USMALLINT, resp_h VARCHAR, resp_p USMALLINT, proto VARCHAR, date DATE);
+             INSERT INTO v11 VALUES (1793461200.0, 'u1', '10.0.0.1', 1000, '10.0.0.2', 80, 'tcp', '2026-10-31');
+             COPY v11 TO '{part_str}/v11.parquet' (FORMAT parquet);
+             DROP TABLE v11;"
+        )).unwrap();
+
+        // Write v1.2 format (with history, orig_bytes)
+        conn.execute_batch(&format!(
+            "CREATE TEMP TABLE v12 (ts DOUBLE, uid VARCHAR, orig_h VARCHAR, orig_p USMALLINT, resp_h VARCHAR, resp_p USMALLINT, proto VARCHAR, history VARCHAR, orig_bytes UBIGINT, date DATE);
+             INSERT INTO v12 VALUES (1793461210.0, 'u2', '10.0.0.1', 1001, '10.0.0.2', 80, 'tcp', 'ShAD', 500, '2026-10-31');
+             COPY v12 TO '{part_str}/v12.parquet' (FORMAT parquet);
+             DROP TABLE v12;"
+        )).unwrap();
+
+        let parquet_glob = store
+            .root()
+            .join("raw")
+            .join("conn")
+            .join("*")
+            .join("*.parquet")
+            .to_string_lossy()
+            .replace('\'', "''");
+
+        let read_sql = format!(
+            "SELECT count(*)::BIGINT, count(history)::BIGINT
+             FROM read_parquet('{parquet_glob}', hive_partitioning = true, union_by_name = true);"
+        );
+        let (total, with_history): (i64, i64) = conn
+            .query_row(&read_sql, [], |r| Ok((r.get(0)?, r.get(1)?)))
+            .unwrap();
+        assert_eq!(total, 2);
+        assert_eq!(with_history, 1);
+    }
+
+    #[test]
+    fn test_zeek_split_real_fixture_chaining() {
+        let temp_dir = tempfile::tempdir().unwrap();
+        let store = Storage::open(temp_dir.path()).unwrap();
+
+        let fixture_path = Path::new(env!("CARGO_MANIFEST_DIR"))
+            .join("tests")
+            .join("fixtures")
+            .join("zeek_split")
+            .join("conn.log");
+
+        store.ingest_file(&fixture_path).unwrap();
+
+        let sessions = store.query_7day_sessions().unwrap();
+        assert_eq!(sessions.len(), 1);
+        assert_eq!(sessions[0].orig_h, "10.0.0.5");
+        assert_eq!(sessions[0].resp_h, "203.0.113.9");
+        assert_eq!(sessions[0].resp_p, 443);
+        assert_eq!(sessions[0].proto, "tcp");
+        assert_eq!(sessions[0].sessions.len(), 1);
+
+        let s = &sessions[0].sessions[0];
+        assert_eq!(s.parts, 2);
+        assert!(
+            (s.duration - 4201.05).abs() < 0.01,
+            "duration: expected 4201.05, got {}",
+            s.duration
+        );
+        assert_eq!(s.orig_bytes, 1200);
+        assert_eq!(s.resp_bytes, 2400);
+    }
+
+    #[test]
+    fn test_chaining_negative_cases() {
+        let temp_dir = tempfile::tempdir().unwrap();
+        let store = Storage::open(temp_dir.path()).unwrap();
+
+        // 1. Second line has 'S' in history -> 2 sessions
+        let log_syn = "#separator \\x09\n#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\tduration\thistory\n\
+1793400000.0\tu1\t10.0.0.1\t40000\t10.0.0.2\t80\ttcp\t100.0\tShAD\n\
+1793400200.0\tu2\t10.0.0.1\t40000\t10.0.0.2\t80\ttcp\t100.0\tShAD\n";
+        let f_syn = temp_dir.path().join("conn.syn.log");
+        fs::write(&f_syn, log_syn).unwrap();
+        store.ingest_file(&f_syn).unwrap();
+
+        let sess_syn = store.query_7day_sessions().unwrap();
+        assert_eq!(
+            sess_syn[0].sessions.len(),
+            2,
+            "history with 'S' must not chain"
+        );
+
+        // 2. Gap of 3601s -> 2 sessions
+        let temp_dir2 = tempfile::tempdir().unwrap();
+        let store2 = Storage::open(temp_dir2.path()).unwrap();
+        let log_gap = "#separator \\x09\n#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\tduration\thistory\n\
+1793400000.0\tu1\t10.0.0.1\t40000\t10.0.0.2\t80\ttcp\t100.0\tShAD\n\
+1793403702.0\tu2\t10.0.0.1\t40000\t10.0.0.2\t80\ttcp\t100.0\tAD\n";
+        let f_gap = temp_dir2.path().join("conn.gap.log");
+        fs::write(&f_gap, log_gap).unwrap();
+        store2.ingest_file(&f_gap).unwrap();
+
+        let sess_gap = store2.query_7day_sessions().unwrap();
+        assert_eq!(sess_gap[0].sessions.len(), 2, "gap > 3600s must not chain");
+
+        // 3. history is NULL -> does not chain
+        let temp_dir3 = tempfile::tempdir().unwrap();
+        let store3 = Storage::open(temp_dir3.path()).unwrap();
+        let log_null = "#separator \\x09\n#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\tduration\thistory\n\
+1793400000.0\tu1\t10.0.0.1\t40000\t10.0.0.2\t80\ttcp\t100.0\tShAD\n\
+1793400200.0\tu2\t10.0.0.1\t40000\t10.0.0.2\t80\ttcp\t100.0\t-\n";
+        let f_null = temp_dir3.path().join("conn.null.log");
+        fs::write(&f_null, log_null).unwrap();
+        store3.ingest_file(&f_null).unwrap();
+
+        let sess_null = store3.query_7day_sessions().unwrap();
+        assert_eq!(
+            sess_null[0].sessions.len(),
+            2,
+            "history NULL must not chain"
+        );
+    }
+
+    #[test]
+    fn test_partitions_affected_and_files_failed() {
+        let temp_dir = tempfile::tempdir().unwrap();
+        let store = Storage::open(temp_dir.path()).unwrap();
+
+        // 2 lines across 2 days (2026-10-30 and 2026-10-31)
+        let log = "#separator \\x09\n#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\n\
+1793325600.0\tu1\t10.0.0.1\t1000\t10.0.0.2\t80\ttcp\n\
+1793412000.0\tu2\t10.0.0.1\t1000\t10.0.0.2\t80\ttcp\n";
+        let f = temp_dir.path().join("conn.log");
+        fs::write(&f, log).unwrap();
+
+        let stats = store.ingest_dir(temp_dir.path()).unwrap();
+        assert_eq!(stats.partitions_affected, 2);
+        assert_eq!(stats.files_failed, 0);
+    }
 }

crates/acaua-core/src/zeek.rs

@@ -1,6 +1,9 @@
 use crate::error::{AcauaError, Result};
 use crate::fusion::Alert;
 use crate::pillar1::{self, BeaconingInput};
+use crate::pillar2::{self, SessionCandidate, SessionInput};
+use crate::pillar3::{self, ProtocolInput};
+use crate::pillar4::HostScore;
 use crate::storage::Storage;
 use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
@@ -21,13 +24,76 @@ pub enum ZeekLogKind {
     Unknown,
 }
 
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)]
+pub struct P1ExcludedCounts {
+    pub non_tcp_udp: usize,
+    pub multicast_broadcast: usize,
+    pub infra_port: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct AnalysisReport {
     pub directory: String,
     pub logs_seen: usize,
+    pub dns_logs_seen: usize,
     pub records: u64,
     pub alerts: Vec<Alert>,
+    pub hosts: Vec<HostScore>,
+    pub evidence_schema_version: u32,
     pub status: &'static str,
+    pub p1_excluded: P1ExcludedCounts,
+}
+
+pub fn classify_log(file_name: &str) -> ZeekLogKind {
+    match file_name {
+        "conn.log" => ZeekLogKind::Conn,
+        "dns.log" => ZeekLogKind::Dns,
+        "ssl.log" => ZeekLogKind::Ssl,
+        "smb_mapping.log" => ZeekLogKind::SmbMapping,
+        "dce_rpc.log" => ZeekLogKind::DceRpc,
+        "rdp.log" => ZeekLogKind::Rdp,
+        "ssh.log" => ZeekLogKind::Ssh,
+        "kerberos.log" => ZeekLogKind::Kerberos,
+        "ntlm.log" => ZeekLogKind::Ntlm,
+        _ => ZeekLogKind::Unknown,
+    }
+}
+
+fn is_unicast(ip_str: &str) -> bool {
+    if ip_str == "255.255.255.255" {
+        return false;
+    }
+    if let Ok(ip) = ip_str.parse::<std::net::IpAddr>() {
+        !ip.is_multicast()
+    } else {
+        true
+    }
+}
+
+fn is_infra_port(proto: &str, port: u16) -> bool {
+    matches!(
+        (proto, port),
+        ("udp", 53)
+            | ("tcp", 53)
+            | ("udp", 123)
+            | ("udp", 67)
+            | ("udp", 68)
+            | ("udp", 546)
+            | ("udp", 547)
+            | ("udp", 137)
+            | ("udp", 138)
+            | ("udp", 1900)
+            | ("udp", 5353)
+            | ("udp", 5355)
+    )
+}
+
+struct AlertItem {
+    pillar: u8,
+    orig_h: String,
+    dest: String,
+    resp_p: u16,
+    alert: Alert,
 }
 
 pub fn analyze_dir(
@@ -43,22 +109,41 @@ pub fn analyze_dir(
     }
 
     let store = Storage::open(data_dir)?;
-    let stats = store.ingest_dir(directory)?;
+    let mut stats = store.ingest_dir(directory)?;
-    let tuples = store.query_7day_tuples()?;
+    let tuple_sessions = store.query_7day_sessions()?;
 
-    let mut alerts = Vec::new();
+    let mut alert_items: Vec<AlertItem> = Vec::new();
+    let mut p1_excluded = P1ExcludedCounts::default();
 
-    for tuple in tuples {
+    // Pilar 1 evaluation: low-and-slow C2 beaconing detection (formula v1.2)
-        if tuple.timestamps.len() < 20 {
+    for tuple in &tuple_sessions {
+        if tuple.proto != "tcp" && tuple.proto != "udp" {
+            p1_excluded.non_tcp_udp += 1;
+            continue;
+        }
+
+        if !is_unicast(&tuple.resp_h) {
+            p1_excluded.multicast_broadcast += 1;
+            continue;
+        }
+
+        if is_infra_port(&tuple.proto, tuple.resp_p) {
+            p1_excluded.infra_port += 1;
+            continue;
+        }
+
+        let timestamps: Vec<f64> = tuple.sessions.iter().map(|s| s.start_ts).collect();
+        if timestamps.len() < 20 {
             continue;
         }
 
         let beacon_score = pillar1::score(&BeaconingInput {
-            timestamps: tuple.timestamps.clone(),
+            timestamps: timestamps.clone(),
+            proto: tuple.proto.clone(),
         })?;
 
         if beacon_score.score >= 0.40 {
-            let latest_ts = tuple.timestamps.last().copied().unwrap_or(0.0);
+            let latest_ts = timestamps.last().copied().unwrap_or(0.0);
             let latest_date_str = match OffsetDateTime::from_unix_timestamp(latest_ts.floor() as i64)
             {
                 Ok(dt) => {
@@ -72,8 +157,8 @@ pub fn analyze_dir(
 
             let mut hasher = Sha256::new();
             let raw_id = format!(
-                "1|{}|{}|{}|{}",
+                "1|{}|{}|{}|{}|{}",
-                tuple.orig_h, tuple.resp_h, tuple.resp_p, latest_date_str
+                tuple.proto, tuple.orig_h, tuple.resp_h, tuple.resp_p, latest_date_str
             );
             hasher.update(raw_id.as_bytes());
             let hash = hasher.finalize();
@@ -84,9 +169,8 @@ pub fn analyze_dir(
             }
 
             let summary = format!(
-                "Beaconing detected: {}:{} -> {}:{} (period ~{:.0}s, score {:.3})",
+                "Beaconing detected: {}:any -> {}:{} (period ~{:.0}s, score {:.3})",
                 tuple.orig_h,
-                "any",
                 tuple.resp_h,
                 tuple.resp_p,
                 beacon_score.period_detected_seconds,
@@ -96,37 +180,202 @@ pub fn analyze_dir(
             let evidence = serde_json::to_value(&beacon_score)?;
             let created_at = OffsetDateTime::now_utc().to_string();
 
-            alerts.push(Alert {
+            alert_items.push(AlertItem {
-                id: alert_id,
                 pillar: 1,
-                score: beacon_score.score,
+                orig_h: tuple.orig_h.clone(),
-                summary,
+                dest: tuple.resp_h.clone(),
-                evidence,
+                resp_p: tuple.resp_p,
-                created_at,
+                alert: Alert {
+                    id: alert_id,
+                    pillar: 1,
+                    summary,
+                    score: beacon_score.score,
+                    evidence,
+                    created_at,
+                },
             });
         }
     }
 
+    // Pilar 2 evaluation: persistent TCP sessions
+    for tuple in &tuple_sessions {
+        if tuple.proto != "tcp" {
+            continue;
+        }
+
+        let candidates: Vec<SessionCandidate> = tuple
+            .sessions
+            .iter()
+            .map(|s| SessionCandidate {
+                start_ts: s.start_ts,
+                end_ts: s.end_ts,
+                duration: s.duration,
+                parts: s.parts,
+                orig_bytes: s.orig_bytes,
+                resp_bytes: s.resp_bytes,
+            })
+            .collect();
+
+        let session_score = pillar2::score(&SessionInput {
+            sessions: candidates,
+            failed_attempts: tuple.failed_attempts,
+        })?;
+
+        if session_score.score >= 0.2181 {
+            let mut hasher = Sha256::new();
+            let raw_id = format!(
+                "2|{}|{}|{}|{:.6}",
+                tuple.orig_h, tuple.resp_h, tuple.resp_p, session_score.session_start_ts
+            );
+            hasher.update(raw_id.as_bytes());
+            let hash = hasher.finalize();
+            let mut alert_id = String::with_capacity(16);
+            for b in &hash[..8] {
+                use std::fmt::Write;
+                let _ = write!(alert_id, "{b:02x}");
+            }
+
+            let summary = format!(
+                "Persistent session: {} -> {}:{} ({:.1}h, score {:.3})",
+                tuple.orig_h,
+                tuple.resp_h,
+                tuple.resp_p,
+                session_score.longest_session_seconds / 3600.0,
+                session_score.score
+            );
+
+            let evidence = serde_json::to_value(&session_score)?;
+            let created_at = OffsetDateTime::now_utc().to_string();
+
+            alert_items.push(AlertItem {
+                pillar: 2,
+                orig_h: tuple.orig_h.clone(),
+                dest: tuple.resp_h.clone(),
+                resp_p: tuple.resp_p,
+                alert: Alert {
+                    id: alert_id,
+                    pillar: 2,
+                    summary,
+                    score: session_score.score,
+                    evidence,
+                    created_at,
+                },
+            });
+        }
+    }
+
+    // Pilar 3 evaluation: information volume in unique DNS subdomains per
+    // (orig_h, base_domain), measured per UTC hour. JSD only travels in the
+    // alert evidence with weight 0.0; it never changes the score.
+    let dns_stats = store.ingest_dns_directory(directory)?;
+    stats.files_seen += dns_stats.files_seen;
+    stats.records_ingested += dns_stats.records_ingested;
+    stats.lines_skipped += dns_stats.lines_skipped;
+
+    let dns_queries = store.query_window_dns()?;
+    let protocol_scores = pillar3::score(&ProtocolInput {
+        queries: dns_queries,
+    })?;
+    for protocol_score in &protocol_scores {
+        if protocol_score.status == "ok" && protocol_score.score >= 0.40 {
+            alert_items.push(AlertItem {
+                pillar: 3,
+                orig_h: protocol_score.orig_h.clone(),
+                dest: protocol_score.base_domain.clone(),
+                resp_p: 0,
+                alert: pillar3::alert_for(protocol_score)?,
+            });
+        }
+    }
+
+    // Sort alerts by (pillar, orig_h, dest, resp_p, id)
+    alert_items.sort_by(|a, b| {
+        (&a.pillar, &a.orig_h, &a.dest, a.resp_p, &a.alert.id).cmp(&(
+            &b.pillar,
+            &b.orig_h,
+            &b.dest,
+            b.resp_p,
+            &b.alert.id,
+        ))
+    });
+
+    let alerts: Vec<Alert> = alert_items.into_iter().map(|item| item.alert).collect();
+    let hosts = crate::fusion::fuse(&alerts)?;
+
     Ok(AnalysisReport {
         directory: directory.display().to_string(),
         logs_seen: stats.files_seen,
+        dns_logs_seen: dns_stats.files_seen,
         records: stats.records_ingested,
         alerts,
+        hosts,
+        evidence_schema_version: 1,
         status: "ok",
+        p1_excluded,
     })
 }
 
-pub fn classify_log(name: &str) -> ZeekLogKind {
+#[cfg(test)]
-    match name {
+mod tests {
-        "conn.log" => ZeekLogKind::Conn,
+    use super::*;
-        "dns.log" => ZeekLogKind::Dns,
+    use std::fs;
-        "ssl.log" => ZeekLogKind::Ssl,
+
-        "smb_mapping.log" => ZeekLogKind::SmbMapping,
+    #[test]
-        "dce_rpc.log" => ZeekLogKind::DceRpc,
+    fn test_zeek_classify_logs() {
-        "rdp.log" => ZeekLogKind::Rdp,
+        assert_eq!(classify_log("conn.log"), ZeekLogKind::Conn);
-        "ssh.log" => ZeekLogKind::Ssh,
+        assert_eq!(classify_log("dns.log"), ZeekLogKind::Dns);
-        "kerberos.log" => ZeekLogKind::Kerberos,
+        assert_eq!(classify_log("ssl.log"), ZeekLogKind::Ssl);
-        "ntlm.log" => ZeekLogKind::Ntlm,
+        assert_eq!(classify_log("other.log"), ZeekLogKind::Unknown);
-        _ => ZeekLogKind::Unknown,
+    }
+
+    #[test]
+    fn test_p1_exclusions_integration() {
+        // Test 14: synthetic conn.log with 4 tuples:
+        // 1. udp/123 (NTP) -> infra exclusion
+        // 2. 224.0.0.251:5353 (multicast) -> multicast exclusion
+        // 3. icmp -> non-tcp/udp exclusion
+        // 4. tcp/8443 -> valid beaconing tuple (alerts)
+        let temp_dir = tempfile::tempdir().unwrap();
+        let log_dir = temp_dir.path().join("logs");
+        let data_dir = temp_dir.path().join("data");
+        fs::create_dir_all(&log_dir).unwrap();
+        fs::create_dir_all(&data_dir).unwrap();
+
+        let mut lines = Vec::new();
+        lines.push("#separator \\x09".to_string());
+        lines.push(
+            "#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto".to_string(),
+        );
+
+        let t0 = 1793400000.0;
+        for i in 0..42 {
+            let t = t0 + (i as f64 * 14400.0);
+            // 1. udp/123
+            lines.push(format!(
+                "{t:.6}\tu1_{i}\t10.0.0.1\t123\t198.51.100.1\t123\tudp"
+            ));
+            // 2. multicast
+            lines.push(format!(
+                "{t:.6}\tu2_{i}\t10.0.0.1\t5353\t224.0.0.251\t5353\tudp"
+            ));
+            // 3. icmp
+            lines.push(format!(
+                "{t:.6}\tu3_{i}\t10.0.0.1\t0\t198.51.100.2\t0\ticmp"
+            ));
+            // 4. tcp/8443
+            lines.push(format!(
+                "{t:.6}\tu4_{i}\t10.0.0.1\t40000\t198.51.100.3\t8443\ttcp"
+            ));
+        }
+
+        let conn_file = log_dir.join("conn.log");
+        fs::write(&conn_file, lines.join("\n")).unwrap();
+
+        let report = analyze_dir(&log_dir, &data_dir).unwrap();
+        assert_eq!(report.p1_excluded.infra_port, 1);
+        assert_eq!(report.p1_excluded.multicast_broadcast, 1);
+        assert_eq!(report.p1_excluded.non_tcp_udp, 1);
+        assert_eq!(report.alerts.len(), 1, "only tcp/8443 should alert");
+        assert!(report.alerts[0].summary.contains("8443"));
     }
 }

crates/acaua-core/tests/fixtures/zeek_split/SOURCE.txt

@@ -0,0 +1,13 @@
+Synthetic test fixture for Zeek idle TCP split verification.
+
+Generated with:
+1. python3 research/validation/mkpcap_idle.py /tmp/zeek_split_gen/pcap/idle.pcap
+2. docker run --rm --user "$(id -u):$(id -g)" \
+     -v /tmp/zeek_split_gen/pcap:/pcap:ro -v /tmp/zeek_split_gen/out:/out -w /out \
+     zeek/zeek:9.0.1 zeek -C -r /pcap/idle.pcap
+
+Contains one TCP session (10.0.0.5:40000 -> 203.0.113.9:443) split by Zeek 9.0.1 across a 600s idle gap into two conn.log lines:
+- First line: ts=1793400000.000000, duration=1741.050000, history=ShADd
+- Second line: ts=1793402401.000000, duration=1800.050000, history=DdFfA (no 'S' in history, gap = 659.95s <= 3600s)
+
+Expected chained session: L = 4201.05s, parts = 2.

crates/acaua-core/tests/fixtures/zeek_split/conn.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2026-11-01-00-28-56
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	local_resp	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents	ip_proto
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	bool	count	string	count	count	count	count	set[string]	count
+1793400000.000000	Cr5nhu4rvbFFhLnlC7	10.0.0.5	40000	203.0.113.9	443	tcp	-	1741.050000	600	1200	S1	T	T	0	ShADd	32	1880	31	2440	-	6
+1793402401.000000	Cn4avD2d83MCmnH4xl	10.0.0.5	40000	203.0.113.9	443	tcp	-	1800.050000	600	1200	SF	T	T	0	DdFfA	32	1880	31	2440	-	6
+#close	2026-11-01-00-28-56

crates/acaua-core/tests/pillar3_e2e.rs

@@ -0,0 +1,179 @@
+//! End-to-end Pilar 3: a synthetic Zeek `dns.log` is ingested into DuckDB,
+//! read back through the 7-day window and scored by `pillar3`. The tunnel
+//! names are synthetic (SplitMix64 base32), never real captures.
+
+use acaua_core::{pillar3, storage::Storage};
+use std::fs;
+
+struct SplitMix64 {
+    state: u64,
+}
+
+impl SplitMix64 {
+    fn new(seed: u64) -> Self {
+        Self { state: seed }
+    }
+    fn next_u64(&mut self) -> u64 {
+        self.state = self.state.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        let mut z = self.state;
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+    fn below(&mut self, n: u64) -> u64 {
+        self.next_u64() % n
+    }
+}
+
+const HEADER: &str = "#separator \\x09\n\
+#set_separator\t,\n\
+#empty_field\t(empty)\n\
+#unset_field\t-\n\
+#path\tdns\n\
+#fields\tts\tuid\tid.orig_h\tid.orig_p\tid.resp_h\tid.resp_p\tproto\ttrans_id\trtt\tquery\tqclass\tqclass_name\tqtype\tqtype_name\trcode\trcode_name\tAA\tTC\tRD\tRA\tZ\tanswers\tTTLs\trejected\topcode\topcode_name\n";
+
+fn dns_line(ts: f64, uid: &str, host: &str, port: u16, query: &str) -> String {
+    format!(
+        "{ts:.6}\t{uid}\t{host}\t{port}\t10.0.2.3\t53\tudp\t1\t0.001\t{query}\t1\tC_INET\t16\tTXT\t0\tNOERROR\tF\tF\tT\tT\t0\t-\t-\tF\t0\tQUERY\n"
+    )
+}
+
+fn base32_name(rng: &mut SplitMix64) -> String {
+    const ALPHABET: &[u8] = b"abcdefghijklmnopqrstuvwxyz234567";
+    let labels = 2 + rng.below(2) as usize;
+    let mut parts = Vec::new();
+    for _ in 0..labels {
+        let len = 50 + rng.below(14) as usize;
+        parts.push(
+            (0..len)
+                .map(|_| ALPHABET[rng.below(32) as usize] as char)
+                .collect::<String>(),
+        );
+    }
+    parts.join(".")
+}
+
+fn write_dns_log(dir: &std::path::Path, lines: &[String]) {
+    let mut log = String::from(HEADER);
+    for line in lines {
+        log.push_str(line);
+    }
+    fs::write(dir.join("dns.log"), log).unwrap();
+}
+
+#[test]
+#[allow(clippy::inconsistent_digit_grouping)]
+fn synthetic_tunnel_is_ingested_and_alerted() {
+    let temp = tempfile::tempdir().unwrap();
+    let logs = temp.path().join("logs");
+    fs::create_dir_all(&logs).unwrap();
+    let store = Storage::open(temp.path().join("data")).unwrap();
+
+    let mut rng = SplitMix64::new(2026_10_31);
+    let mut lines = Vec::new();
+    for i in 0..100 {
+        lines.push(dns_line(
+            1_800_000_000.0 + i as f64 * 30.0,
+            &format!("C{i}"),
+            "10.66.66.66",
+            50000 + i,
+            &format!("{}.t.example-tunnel.net", base32_name(&mut rng)),
+        ));
+    }
+    write_dns_log(&logs, &lines);
+
+    let stats = store.ingest_dns_directory(&logs).unwrap();
+    assert_eq!(stats.files_seen, 1);
+    assert_eq!(stats.records_ingested, 100);
+
+    let queries = store.query_window_dns().unwrap();
+    assert_eq!(queries.len(), 100);
+
+    let scores = pillar3::score(&pillar3::ProtocolInput { queries }).unwrap();
+    assert_eq!(scores.len(), 1);
+    let score = &scores[0];
+    assert_eq!(score.orig_h, "10.66.66.66");
+    assert_eq!(score.base_domain, "example-tunnel.net");
+    assert_eq!(score.status, "ok");
+    assert!(
+        score.score >= 0.40,
+        "score {} bits {}",
+        score.score,
+        score.information_bits
+    );
+    assert_eq!(score.weights.information, 1.0);
+    assert_eq!(score.weights.jsd, 0.0);
+    assert_eq!(score.jsd_reference_groups, 0);
+    assert_eq!(score.jsd, 0.0);
+
+    let alerts = pillar3::alerts(&scores).unwrap();
+    assert_eq!(alerts.len(), 1);
+    assert_eq!(alerts[0].pillar, 3);
+    assert_eq!(alerts[0].id.len(), 16);
+    assert_eq!(alerts[0].score, score.score);
+    let evidence: serde_json::Value = alerts[0].evidence.clone();
+    assert_eq!(evidence["weights"]["jsd"].as_f64(), Some(0.0));
+    assert_eq!(evidence["weights"]["information"].as_f64(), Some(1.0));
+}
+
+#[test]
+fn synthetic_cloudfront_stays_below_the_alert_threshold() {
+    let temp = tempfile::tempdir().unwrap();
+    let logs = temp.path().join("logs");
+    fs::create_dir_all(&logs).unwrap();
+    let store = Storage::open(temp.path().join("data")).unwrap();
+
+    let mut rng = SplitMix64::new(7);
+    let mut lines = Vec::new();
+    const ALPHABET: &[u8] = b"abcdefghijklmnopqrstuvwxyz0123456789";
+    for i in 0..45 {
+        let name: String = (0..14)
+            .map(|_| ALPHABET[rng.below(ALPHABET.len() as u64) as usize] as char)
+            .collect();
+        lines.push(dns_line(
+            1_800_000_000.0 + i as f64 * 60.0,
+            &format!("C{i}"),
+            "10.0.2.15",
+            41000 + i,
+            &format!("{name}.cloudfront.net"),
+        ));
+    }
+    write_dns_log(&logs, &lines);
+    store.ingest_dns_directory(&logs).unwrap();
+
+    let queries = store.query_window_dns().unwrap();
+    let scores = pillar3::score(&pillar3::ProtocolInput { queries }).unwrap();
+    let alerts = pillar3::alerts(&scores).unwrap();
+    assert!(alerts.is_empty());
+    assert_eq!(scores.len(), 1);
+    assert_eq!(scores[0].score, 0.0);
+}
+
+#[test]
+fn analyze_dir_emits_a_pillar3_alert_without_any_conn_log() {
+    let temp = tempfile::tempdir().unwrap();
+    let logs = temp.path().join("logs");
+    let data = temp.path().join("data");
+    fs::create_dir_all(&logs).unwrap();
+
+    let mut rng = SplitMix64::new(99);
+    let mut lines = Vec::new();
+    for i in 0..100 {
+        lines.push(dns_line(
+            1_800_000_000.0 + i as f64 * 30.0,
+            &format!("C{i}"),
+            "10.66.66.68",
+            52000 + i,
+            &format!("{}.t.example-tunnel.net", base32_name(&mut rng)),
+        ));
+    }
+    write_dns_log(&logs, &lines);
+
+    let report = acaua_core::zeek::analyze_dir(&logs, &data).unwrap();
+    assert_eq!(report.alerts.len(), 1);
+    assert_eq!(report.alerts[0].pillar, 3);
+    assert!(report.alerts[0]
+        .summary
+        .contains("High-information DNS subdomains"));
+    assert_eq!(report.alerts[0].id.len(), 16);
+}

crates/acaua-core/tests/smoke.rs

@@ -4,6 +4,7 @@ use acaua_core::{doctor, fusion, pillar1, pillar2, pillar3, pillar4, storage, ze
 fn pillar1_returns_a_skeleton_score() {
     let input = pillar1::BeaconingInput {
         timestamps: vec![1.0, 2.0, 3.0],
+        proto: "tcp".to_string(),
     };
     let score = pillar1::score(&input).unwrap();
     assert_eq!(score.observations, 3);
@@ -13,33 +14,49 @@ fn pillar1_returns_a_skeleton_score() {
 #[test]
 fn pillar2_counts_long_connections_without_panicking() {
     let input = pillar2::SessionInput {
-        durations_secs: vec![1.0, 400.0],
+        sessions: vec![
-        bytes: vec![10, 20],
+            pillar2::SessionCandidate {
+                start_ts: 1000.0,
+                end_ts: 1001.0,
+                duration: 1.0,
+                parts: 1,
+                orig_bytes: 10,
+                resp_bytes: 20,
+            },
+            pillar2::SessionCandidate {
+                start_ts: 2000.0,
+                end_ts: 2400.0,
+                duration: 400.0,
+                parts: 1,
+                orig_bytes: 100,
+                resp_bytes: 200,
+            },
+        ],
+        failed_attempts: 0,
     };
     let score = pillar2::score(&input).unwrap();
-    assert_eq!(score.long_connections, 1);
+    assert_eq!(score.longest_session_seconds, 400.0);
-    assert_eq!(score.status, "skeleton");
+    assert_eq!(score.status, "ok");
 }
 
 #[test]
-fn pillar3_handles_an_empty_protocol_list() {
+fn pillar3_handles_empty_dns_input() {
-    let score = pillar3::score(&pillar3::ProtocolInput::default()).unwrap();
+    let scores = pillar3::score(&pillar3::ProtocolInput::default()).unwrap();
-    assert_eq!(score.distinct_protocols, 0);
+    assert!(scores.is_empty());
-    assert_eq!(score.status, "skeleton");
 }
 
 #[test]
 fn pillar4_fuses_three_inputs() {
-    let score = pillar4::score(&pillar4::FusionInput::default()).unwrap();
+    let score = pillar4::score("10.0.0.1", &pillar4::FusionInput::default()).unwrap();
-    assert_eq!(score.inputs, 3);
+    assert_eq!(score.terms.len(), 3);
-    assert_eq!(score.status, "skeleton");
+    assert_eq!(score.formula_version, "1.0");
+    assert_eq!(score.score, 0.0);
 }
 
 #[test]
 fn fusion_returns_no_alerts_yet() {
-    let report = fusion::fuse(&[0.1, 0.2]).unwrap();
+    let hosts = fusion::fuse(&[]).unwrap();
-    assert!(report.alerts.is_empty());
+    assert!(hosts.is_empty());
-    assert_eq!(report.status, "skeleton");
 }
 
 #[test]
@@ -67,5 +84,5 @@ fn zeek_classifies_known_logs() {
 fn doctor_reports_checks() {
     let report = doctor::run();
     assert!(!report.checks.is_empty());
-    assert_eq!(report.status, "skeleton");
+    assert!(report.status == "ok" || report.status == "degraded");
 }

docs/architecture.md

@@ -80,24 +80,106 @@ It is bounded in `[-1.0, 1.0]` by definition and resistant to extreme outliers.
 When `Q3 == Q1` (concentrated data), the implementation assigns `0.0` with a
 sentinel flag.
 
-### 4. Isolate the Python research workspace
+### 4. Jensen-Shannon divergence instead of Kullback-Leibler
+
+Kullback-Leibler divergence diverges to infinity when `Q(x) = 0` on non-empty `P(x)`.
+Acauã adopts the Jensen-Shannon Divergence (JSD): bounded in `[0, 1]`, symmetric,
+and strictly finite even on disjoint supports. Measured in real DNS tunneling and
+CDN traffic, JSD alone does not separate benign CDNs from tunnels; hence in v1 it
+is recorded in alert evidence with weight 0.0 rather than driving the score.
+
+### 5. Dedicated persistent session detector (Pillar 2)
+
+A long-lived interactive session or steady C2 channel does not generate frequent
+reconnections (`Δt`), making it invisible to point-process periodicity analysis.
+Acauã implements a dedicated persistent session detector with TCP session chaining
+that re-aggregates sessions split by Zeek's 5-minute inactivity timeouts.
+
+### 6. JA4 licensing boundary
+
+Only the original TLS client fingerprint JA4 is BSD-3 licensed. Other modules
+(JA4S, JA4H, JA4X, JA4SSH, JA4L) use the proprietary FoxIO License 1.1 (non-OSI).
+Acauã does not bundle or depend on FoxIO-licensed modules by default in an
+open-source architecture.
+
+### 7. Determinism and fixed reduction order
+
+Parallel reductions (e.g. via `rayon`) reorder floating-point additions and
+multiplications across threads, breaking bit-level reproducibility. All reductions,
+periodogram summations, and host evidence combinations are computed in a fixed,
+deterministic sequential order, producing byte-for-byte identical JSON evidence.
+
+### 8. Isolate the Python research workspace
 
 `research/` has its own virtual environment and dependencies. It is excluded
 from the Cargo workspace, ignored by Docker builds and CI releases, and tested
 only when research validation scripts are invoked directly.
 
-### 5. Documented degraded mode for NetFlow/IPFIX
+### 9. Documented degraded mode for NetFlow/IPFIX
 
 NetFlow v9 and IPFIX do not carry DNS query names, TLS SNI, certificate chains
 or HTTP headers. Any analysis that depends on protocol features is unavailable.
 Acauã documents this as a degraded mode and never claims parity between flow
 records and full Zeek logs.
 
-### 6. Single HTTP and RPC boundary
+### 10. Single HTTP and RPC boundary
 
 The daemon exposes an Axum API on loopback by default. The embedded web UI
 communicates with the daemon exclusively via standard HTTP calls.
 
+## Detection (v1)
+
+### Pillar 1: Low-and-slow C2 Beaconing (v1.2)
+Evaluates session starts per `(orig_h, resp_h, resp_p, proto)` using a Rayleigh/Schuster
+periodogram over an effective grid bounded by `span / 3` (requiring at least 3 full cycles
+observed). Significance is gated by dispersion regularity (`gate = min(1, regularity / 0.25)`),
+preventing false positives on bursty benign browsing. Alert threshold: `score >= 0.40`.
+Non-TCP/UDP protocols, multicast/broadcast addresses, and infrastructure service ports
+(e.g., NTP 123, DNS 53, mDNS 5353, NetBIOS, DHCP) are audited and excluded.
+
+### Pillar 2: Persistent Sessions (v1.0)
+Chains TCP connection segments across Zeek inactivity timeouts (gap <= 3600s with no `S`
+SYN flag in Zeek history). Persistence score is logarithmic:
+`persistence = clamp(ln(L / 3600) / ln(24), 0, 1)`. Sessions with `L >= 7200s`
+(`score >= 0.2181`) trigger an alert. Producer-Consumer Ratio (PCR), byte transfer rates,
+and failed connection attempts are captured in evidence.
+
+### Pillar 3: DNS Subdomain Information Capacity (v1.0)
+Measures the information payload capacity pushed across DNS queries by aggregating unique subdomains
+per `(orig_h, base_domain, utc_hour)`. Normalizes queries to lowercase, strips trailing dots, and excludes
+reverse lookups (`.in-addr.arpa`, `.ip6.arpa`), `.local`, and discovery ports (5353, 5355, 137):
+- Information capacity: $I = H \times C$ bits, where $H = -\sum p_c \log_2 p_c$ (Shannon entropy per char)
+  and $C$ is the sum of characters across unique subdomains.
+- Evaluated at peak hour: $I_{\max} = \max_h I$. Requires $U_{\text{peak}} \ge 10$ unique subdomains;
+  otherwise status is `insufficient_data` with score 0.0.
+- Scoring: $\text{score} = \text{clamp}(\log_2(I_{\max} / 8192) / \log_2(32), 0, 1)$ ($8\text{ kbit} \to 0.0$,
+  $32\text{ kbit} \to 0.40$, $256\text{ kbit} \to 1.0$).
+- Alert threshold: $\text{score} \ge 0.40$ ($\iff I_{\max} \ge 32768$ bits/h).
+- Jensen-Shannon Divergence (JSD) against the mixture of other active domains in the window is stored in
+  evidence with weight 0.0 (tunnels measured 0.157–0.193 vs. benign CDNs 0.226–0.227, inverting rank).
+
+### Pillar 4: Per-Host Evidence Fusion (v1.0)
+Combines alerts per `orig_h` using a weighted noisy-OR model across pillars 1, 2, and 3:
+- For each host, $m_i$ is the maximum alert score from pillar $i$ ($0.0$ if no alert exists).
+- Fixed weights $w_1 = w_2 = w_3 = 0.75$.
+- Score formula: $S = 1 - \prod_{i=1}^3 (1 - w_i m_i)$ evaluated in fixed sequential order $[1, 2, 3]$.
+- Properties: $S \in [0, 1]$, monotonic in every $m_i$, single pillar capped at $0.75$, two pillars capped at
+  $0.9375$, three pillars capped at $0.984375$. Invariant under alert input permutations.
+- Fusion ranks hosts in `AnalysisReport.hosts` by $(S \text{ desc}, orig\_h \text{ asc})$; it provides investigation
+  prioritization without fabricating synthetic alert classes.
+
+### Known Limitations
+1. In live captures, open sessions remain unobserved until Zeek flushes them on closure or timeout.
+2. Benign cloud polling and persistent services (e.g., smart home devices like Philips HUE, websockets, push notifications) exhibit genuine periodicity and persistence, ranking high in fused scores without network-wide context.
+3. Pillar 3 tunnel detection sensitivity is approximately 100 unique queries per hour.
+4. Base domain extraction uses an effective heuristic without a full embedded Public Suffix List (PSL).
+
+### Validation Datasets
+Public datasets used for empirical verification (detailed in `research/validation/datasets.tsv`):
+- **IoT-23** (captures 34-1, 20-1, 8-1, 42-1, Honeypot 4-1): CC BY 2.0 ground truth for C2 beaconing, persistence, and benign IoT controls.
+- **CTU-Normal** (captures 20, 18, 26): public captures representing benign browsing and Facebook traffic.
+- **Mendeley dtqbc-m-test** (Y. Bubnov): CC BY 4.0 dataset with real DNS tunneling payloads (dns2tcp, dnscapy, tuns).
+
 ## API
 
 The HTTP API is served by `acaua serve`:

docs/cli.md

@@ -39,8 +39,21 @@ acaua analyze /var/log/zeek/current
 acaua analyze /var/log/zeek/current --json
 ```
 
-Validates the target directory, ingests `conn.log` traffic into Parquet partitions,
+Validates the target directory, ingests `conn.log` and `dns.log` traffic into Parquet partitions,
-and runs threat detection algorithms.
+and runs threat detection algorithms across pillars 1–4. The human-readable output reports summary metrics,
+individual alerts, and the `hosts:` ranking section (top 10 by priority score):
+
+```text
+diretório: /var/log/zeek/current
+logs lidos: 2
+registros: 10450
+alertas: 2
+  [0.850] Beaconing detected: 192.168.1.50:any -> 198.51.100.1:443 (period ~14400s, score 0.850)
+  [0.600] Persistent session: 192.168.1.50 -> 198.51.100.1:443 (6.0h, score 0.600)
+hosts:
+  [0.784] 192.168.1.50       0.850/0.600/0.000
+estado: ok
+```
 
 ## `acaua doctor`
 

docs/pt-BR/arquitetura.md

@@ -79,24 +79,108 @@ Ela é limitada em `[-1.0, 1.0]` por definição e resistente a outliers extremo
 Quando `Q3 == Q1` (dados concentrados), a implementação atribui `0.0` com flag
 sentinela.
 
-### 4. Isolar o espaço de pesquisa Python
+### 4. Divergência de Jensen-Shannon em vez de Kullback-Leibler
+
+A divergência de Kullback-Leibler diverge para infinito quando `Q(x) = 0` sobre
+suporte não-vazio de `P(x)`. O Acauã adota a Divergência de Jensen-Shannon (JSD):
+limitada em `[0, 1]`, simétrica e estritamente finita mesmo em suportes disjuntos.
+Medições empíricas em tráfego de túnel DNS real e CDNs benignas demonstraram que a
+JSD isolada não separa CDNs de túneis; assim, na v1 ela entra na evidência do
+alerta com peso 0.0 sem distorcer o score.
+
+### 5. Detector dedicado de sessões persistentes (Pilar 2)
+
+Sessões interativas de longa duração ou canais C2 estáveis não geram reconexões
+frequentes (`Δt`), tornando-se invisíveis para a análise pontual de periodicidade.
+O Acauã implementa um detector dedicado de sessões persistentes com encadeamento
+TCP que reagrupa segmentos divididos pelo timeout de inatividade de 5 minutos do Zeek.
+
+### 6. Fronteira de licenciamento do JA4
+
+Apenas a impressão digital original de cliente TLS JA4 está sob licença BSD-3.
+Os outros módulos (JA4S, JA4H, JA4X, JA4SSH, JA4L) utilizam a FoxIO License 1.1
+proprietária (não-OSI). O Acauã não embute nem depende de módulos sob licença FoxIO
+por padrão em sua arquitetura de código aberto.
+
+### 7. Determinismo e ordem fixa de redução
+
+Reduções paralelas (ex.: via `rayon`) alteram a ordem de somas e multiplicações
+em ponto flutuante entre threads, quebrando a reprodutibilidade bit a bit. Todas as
+reduções, somas de periodograma e combinações de evidência são executadas em ordem
+sequencial fixa e determinística, produzindo evidências JSON idênticas byte a byte.
+
+### 8. Isolar o espaço de pesquisa Python
 
 `research/` tem seu próprio ambiente virtual e dependências. Ele é excluído do
 workspace Cargo, ignorado em builds Docker e releases de CI, e testado apenas
 quando scripts de validação de pesquisa forem invocados diretamente.
 
-### 5. Modo degradado documentado para NetFlow/IPFIX
+### 9. Modo degradado documentado para NetFlow/IPFIX
 
 NetFlow v9 e IPFIX não carregam nomes de consulta DNS, SNI TLS, cadeias de
 certificados ou cabeçalhos HTTP. Qualquer análise que dependa de recursos de
 protocolo fica indisponível. O Acauã documenta isso como modo degradado e nunca
 alega paridade entre registros de fluxo e logs Zeek completos.
 
-### 6. Fronteira única HTTP e RPC
+### 10. Fronteira única HTTP e RPC
 
 O daemon expõe uma API Axum em loopback por padrão. A UI web embutida se
 comunica com o daemon exclusivamente por chamadas HTTP padrão.
 
+## Detecção (v1)
+
+### Pilar 1: Beaconing C2 Low-and-Slow (v1.2)
+Avalia inícios de sessão por `(orig_h, resp_h, resp_p, proto)` através de um periodograma
+de Rayleigh/Schuster sobre uma grade efetiva limitada a `span / 3` (exigindo no mínimo 3
+ciclos completos observados). A significância é controlada pela regularidade da dispersão
+(`gate = min(1, regularidade / 0.25)`), eliminando falsos positivos em tráfego de navegação
+em rajadas. Limiar de alerta: `score >= 0.40`. Protocolos não-TCP/UDP, endereços multicast
+e broadcast, e portas de infraestrutura (ex.: NTP 123, DNS 53, mDNS 5353, NetBIOS, DHCP)
+são auditados e excluídos com transparência.
+
+### Pilar 2: Sessões Persistentes (v1.0)
+Encadeia segmentos de conexão TCP divididos por inatividade no Zeek (intervalo <= 3600s
+sem flag SYN `S` no histórico). O score de persistência é logarítmico:
+`persistence = clamp(ln(L / 3600) / ln(24), 0, 1)`. Sessões com `L >= 7200s`
+(`score >= 0.2181`) disparam alerta. A razão produtor-consumidor (PCR), taxa de bytes
+por hora e contagem de tentativas falhas são registradas na evidência.
+
+### Pilar 3: Capacidade de Informação em Subdomínios DNS (v1.0)
+Mede a capacidade física de informação transmitida por consultas DNS agregando subdomínios únicos
+por `(orig_h, base_domain, hora_utc)`. Normaliza para minúsculas, remove ponto final e exclui consultas
+reversas (`.in-addr.arpa`, `.ip6.arpa`), `.local` e portas de descoberta local (5353, 5355, 137):
+- Capacidade de informação: $I = H \times C$ bits, onde $H = -\sum p_c \log_2 p_c$ (entropia de Shannon por caractere)
+  e $C$ é a soma dos caracteres dos subdomínios únicos sem os pontos.
+- Avaliação na hora de pico: $I_{\max} = \max_h I$. Exige $U_{\text{pico}} \ge 10$ subdomínios únicos; caso
+  contrário, o status é `insufficient_data` com score 0.0.
+- Pontuação: $\text{score} = \text{clamp}(\log_2(I_{\max} / 8192) / \log_2(32), 0, 1)$ ($8\text{ kbit} \to 0.0$,
+  $32\text{ kbit} \to 0.40$, $256\text{ kbit} \to 1.0$).
+- Limiar de alerta: $\text{score} \ge 0.40$ ($\iff I_{\max} \ge 32768$ bits/h).
+- A Divergência Jensen-Shannon (JSD) contra a mistura de outros domínios da janela entra na evidência com peso 0.0
+  (túneis reais deram 0.157–0.193 vs. CDNs benignas 0.226–0.227, o que inverteria a classificação).
+
+### Pilar 4: Fusão de Evidências por Host (v1.0)
+Combina alertas por `orig_h` usando um modelo noisy-OR ponderado sobre os pilares 1, 2 e 3:
+- Para cada host, $m_i$ é a pontuação máxima de alerta do pilar $i$ ($0.0$ caso não haja alertas do pilar).
+- Pesos fixos: $w_1 = w_2 = w_3 = 0.75$.
+- Fórmula: $S = 1 - \prod_{i=1}^3 (1 - w_i m_i)$ calculado na ordem sequencial fixa $[1, 2, 3]$.
+- Propriedades: $S \in [0, 1]$, monotônica em cada $m_i$, um pilar isolado atinge no máximo $0.75$, dois pilares
+  $0.9375$ e três pilares $0.984375$. Invariante à permutação dos alertas de entrada.
+- A fusão classifica os hosts em `AnalysisReport.hosts` por $(S \text{ desc}, orig\_h \text{ asc})$; fornece priorização
+  de investigação sem fabricar categorias sintéticas de alerta.
+
+### Limitações Conhecidas
+1. Em capturas ao vivo, sessões em aberto permanecem invisíveis até que o Zeek descarregue o registro por encerramento ou timeout.
+2. Polling legítimo de nuvem e serviços persistentes (ex.: dispositivos IoT como lâmpada Philips HUE, websockets, push) exibem periodicidade e persistência reais, ficando com pontuação fundida elevada sem contexto global da rede.
+3. A sensibilidade do Pilar 3 é de cerca de 100 consultas de túnel únicas por hora.
+4. A extração de domínio-base usa heurística eficiente sem embutir a lista completa da Public Suffix List (PSL).
+
+### Dados de Validação
+Datasets públicos utilizados para verificação empírica (detalhados em `research/validation/datasets.tsv`):
+- **IoT-23** (cenários 34-1, 20-1, 8-1, 42-1, Honeypot 4-1): verdade de referência CC BY 2.0 para beaconing C2, persistência e controles benignos IoT.
+- **CTU-Normal** (capturas 20, 18, 26): capturas públicas de tráfego benigno de navegação e Facebook.
+- **Mendeley dtqbc-m-test** (Y. Bubnov): dataset sob CC BY 4.0 contendo consultas reais de túneis DNS (dns2tcp, dnscapy, tuns).
+
 ## API
 
 A API HTTP é servida por `acaua serve`:

docs/pt-BR/cli.md

@@ -39,8 +39,21 @@ acaua analyze /var/log/zeek/current
 acaua analyze /var/log/zeek/current --json
 ```
 
-Valida o diretório alvo, ingere tráfego do `conn.log` em partições Parquet e
+Valida o diretório alvo, ingere tráfego do `conn.log` e `dns.log` em partições Parquet e
-executa os algoritmos de detecção de ameaças.
+executa os algoritmos de detecção de ameaças dos pilares 1 a 4. A saída legível para humanos relata métricas de sumário,
+alertas individuais e a seção de classificação `hosts:` (top 10 por score de prioridade):
+
+```text
+diretório: /var/log/zeek/current
+logs lidos: 2
+registros: 10450
+alertas: 2
+  [0.850] Beaconing detected: 192.168.1.50:any -> 198.51.100.1:443 (period ~14400s, score 0.850)
+  [0.600] Persistent session: 192.168.1.50 -> 198.51.100.1:443 (6.0h, score 0.600)
+hosts:
+  [0.784] 192.168.1.50       0.850/0.600/0.000
+estado: ok
+```
 
 ## `acaua doctor`
 

research/README.md

@@ -9,6 +9,13 @@ Use it to generate reproducible fixtures, compare implementations against
 reference libraries, and run notebooks for the four detection pillars. The Rust
 engine must remain the only runtime used in production.
 
+Validation scripts and dataset references are located in [`validation/`](validation/):
+- `validation/datasets.tsv`: public dataset URLs, filenames, licenses, and SHA-256 checksums.
+- `validation/p1v12sim.py`: simulation and validation for Pillar 1 v1.2 (Rayleigh/Schuster).
+- `validation/p2real.py`: persistent TCP session chaining validation.
+- `validation/p3sim.py`, `p3hourly.py`, `p3real.py`: DNS tunnel information capacity analysis.
+- `validation/mkpcap_idle.py`: synthetic fixture generator for idle TCP splitting.
+
 ```bash
 cd research
 python3 -m venv .venv

research/README.pt-BR.md

@@ -9,6 +9,13 @@ Use-o para gerar fixtures reproduzíveis, comparar implementações com bibliote
 de referência e rodar notebooks dos quatro pilares de detecção. O motor Rust
 continua sendo o único runtime usado em produção.
 
+Scripts de validação e referências de datasets estão em [`validation/`](validation/):
+- `validation/datasets.tsv`: URLs de datasets públicos, arquivos, licenças e checksums SHA-256.
+- `validation/p1v12sim.py`: simulação e validação do Pilar 1 v1.2 (Rayleigh/Schuster).
+- `validation/p2real.py`: validação do encadeamento de sessões TCP persistentes.
+- `validation/p3sim.py`, `p3hourly.py`, `p3real.py`: análise de capacidade de informação em túneis DNS.
+- `validation/mkpcap_idle.py`: gerador de fixture sintético para divisão TCP por inatividade.
+
 ```bash
 cd research
 python3 -m venv .venv

research/validation/datasets.tsv

@@ -0,0 +1,10 @@
+dataset	url	license	file	sha256
+IoT-23 34-1 (Mirai)	https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-34-1/	CC BY 2.0	2018-12-21-15-50-14-192.168.1.195.pcap	92ec7e2f6658ee4b007d0b816986c46cc0338bc5e2bec6ceaaca566c695e4699
+IoT-23 20-1 (Torii)	https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-20-1/	CC BY 2.0	2018-10-02-13-12-30-192.168.100.103.pcap	4d0a00c33a4bf11228158baddd47c38f702731ea8d75ef1201061fc8bbd878b9
+IoT-23 8-1 (Hakai)	https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-8-1/	CC BY 2.0	2018-07-31-15-15-09-192.168.100.113.pcap	80dcc2602519479ddcde889fa902fee19a76696630811452f8df38888af894f2
+IoT-23 42-1 (Trojan)	https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-IoT-Malware-Capture-42-1/	CC BY 2.0	2019-01-10-14-34-38-192.168.1.197.pcap	7573797f17aae96e3804a3cfb22d531b74c17bbae740de1e14048f89130fba37
+IoT-23 4-1 (Philips HUE)	https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/IndividualScenarios/CTU-Honeypot-Capture-4-1/	CC BY 2.0	2018-10-25-14-06-32-192.168.1.132.pcap.xz	bc5a57306d9a5f614b09d9c0976f007667fee5b6de3427fbab094fa3084b9b64
+CTU-Normal-20	https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-20/	unspecified	2017-04-30_win-normal.pcap	c78a00728f115ab8fc495fd395894c43fd90cb61054eb9ef4f2d024658d5ffc5
+CTU-Normal-18	https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-18/	unspecified	2017-09-05_capture-win4.pcap	e09df1ac62d9753ed2360373915427432292192dc06e20572e45716a2932c8ce
+CTU-Normal-26	https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-26/	unspecified	2017-04-28_normal.pcap	b66103fd89c8f0fc53b76e718f53320f3af27edd302c58695fc1870ede3663c5
+Mendeley dtqbc-m-test	https://data.mendeley.com/public-api/datasets/mzn9hvdcxg	CC BY 4.0	dtqbc-m-test.csv	48f109f501e01696e78d22623e4f7938784071219acbf8924ea2211f3473a4f8

research/validation/mkpcap_idle.py

@@ -0,0 +1,35 @@
+# Writes a pcap: one TCP session 10.0.0.5:40000 -> 203.0.113.9:443 with a 600 s idle gap
+# (longer than Zeek's tcp_inactivity_timeout = 5 min), then a fresh connection from a new port.
+import struct, sys
+out = open(sys.argv[1], 'wb')
+out.write(struct.pack('<IHHiIII', 0xa1b2c3d4, 2, 4, 0, 0, 65535, 1))
+def ip(a): return bytes(int(x) for x in a.split('.'))
+def pkt(t, src, sp, dst, dp, flags, seq, ack, payload=b''):
+    tcp = struct.pack('!HHIIBBHHH', sp, dp, seq, ack, 5 << 4, flags, 65535, 0, 0) + payload
+    iph = struct.pack('!BBHHHBBH4s4s', 0x45, 0, 20 + len(tcp), 1, 0, 64, 6, 0, ip(src), ip(dst))
+    frame = b'\x00\x11\x22\x33\x44\x55' + b'\x66\x77\x88\x99\xaa\xbb' + b'\x08\x00' + iph + tcp
+    sec = int(t); usec = int(round((t - sec) * 1e6))
+    out.write(struct.pack('<IIII', sec, usec, len(frame), len(frame)) + frame)
+SYN, ACK, PSH, FIN = 0x02, 0x10, 0x08, 0x01
+C, S = '10.0.0.5', '203.0.113.9'
+t0 = 1793400000.0
+cs, ss = 1000, 5000
+pkt(t0, C, 40000, S, 443, SYN, cs, 0); cs += 1
+pkt(t0+0.05, S, 443, C, 40000, SYN|ACK, ss, cs); ss += 1
+pkt(t0+0.10, C, 40000, S, 443, ACK, cs, ss)
+# heartbeat every 60 s for 30 min (stays alive)
+t = t0 + 1
+for i in range(30):
+    pkt(t, C, 40000, S, 443, PSH|ACK, cs, ss, b'x'*20); cs += 20
+    pkt(t+0.05, S, 443, C, 40000, PSH|ACK, ss, cs, b'y'*40); ss += 40
+    t += 60
+# idle 600 s, then resume heartbeats for 30 min more, then FIN
+t += 600
+for i in range(30):
+    pkt(t, C, 40000, S, 443, PSH|ACK, cs, ss, b'x'*20); cs += 20
+    pkt(t+0.05, S, 443, C, 40000, PSH|ACK, ss, cs, b'y'*40); ss += 40
+    t += 60
+pkt(t, C, 40000, S, 443, FIN|ACK, cs, ss); cs += 1
+pkt(t+0.05, S, 443, C, 40000, FIN|ACK, ss, cs); ss += 1
+pkt(t+0.10, C, 40000, S, 443, ACK, cs, ss)
+out.close()

research/validation/p1real.py

@@ -0,0 +1,45 @@
+# Applies Pillar 1 v1.1 and v1.2 (and ablations) to real Zeek conn.log files; tuple label from IoT-23 labeled file.
+import sys, os, collections, importlib.util
+here = os.path.dirname(os.path.abspath(__file__))
+spec = importlib.util.spec_from_file_location('sim', os.path.join(here, 'p1v12sim_lib.py')); sim = importlib.util.module_from_spec(spec); spec.loader.exec_module(sim)
+def read(path):
+    fields = None
+    for line in open(path, errors='replace'):
+        line = line.rstrip('\n')
+        if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        yield dict(zip(fields, line.split('\t')))
+def labels(path):
+    lab = {}
+    if not path or not os.path.exists(path): return lab
+    for r in read(path):
+        last = ' '.join(v for k, v in r.items() if k in ('tunnel_parents', 'label', 'detailed-label', 'det_label'))
+        mal = 'Malicious' in last or 'Malicious' in str(list(r.values())[-3:])
+        k = (r['id.orig_h'], r['id.resp_h'], r['id.resp_p'])
+        lab[k] = lab.get(k, False) or mal
+    return lab
+import ipaddress
+EXCL = {('udp','5355'),('udp','546'),('udp','547'),('udp', '123'), ('udp', '53'), ('tcp', '53'), ('udp', '5353'), ('udp', '137'), ('udp', '138'), ('udp', '67'), ('udp', '68'), ('udp', '1900')}
+conn, labf = sys.argv[1], (sys.argv[2] if len(sys.argv) > 2 else None)
+lab = labels(labf)
+tup = collections.defaultdict(list); protos = {}
+for r in read(conn):
+    k = (r['id.orig_h'], r['id.resp_h'], r['id.resp_p']); tup[k].append(float(r['ts'])); protos.setdefault(k, set()).add(r['proto'])
+rows = []
+for k, ts in sorted(tup.items()):
+    if len(ts) < 20: continue
+    a = sim.v11(ts)[0]; b = sim.v12(ts)
+    span_only = sim.v12(ts, gate=1e-12)[0]; gate_only = sim.v12(ts, cycles=1e-12)[0]
+    try:
+        a_ = ipaddress.ip_address(k[1]); mc = a_.is_multicast or k[1] == '255.255.255.255'
+    except ValueError:
+        mc = False
+    excl = mc or any(p not in ('tcp', 'udp') for p in protos[k]) or any((p, k[2]) in EXCL for p in protos[k])
+    rows.append((k, len(ts), a, span_only, gate_only, b[0], b[1], b[2], excl, lab.get(k)))
+print(f"== {conn}: tuples with n>=20: {len(rows)}")
+print("tuple n v1.1 span_only gate_only v1.2 status period excluded label")
+for r in rows:
+    if max(r[2], r[5]) >= .40:
+        print(f"  {r[0]} n={r[1]} v11={r[2]:.3f} span={r[3]:.3f} gate={r[4]:.3f} v12={r[5]:.3f} {r[6]} per={r[7] and round(r[7])} excl={r[8]} mal={r[9]}")
+c = lambda i, ex: sum(1 for r in rows if r[i] >= .4 and (not ex or not r[8]))
+print(f"  alerts>=0.40: v1.1={c(2,False)} span_only={c(3,False)} gate_only={c(4,False)} v1.2={c(5,False)} v1.2+exclusion={c(5,True)}")

research/validation/p1v12sim.py

@@ -0,0 +1,104 @@
+# Pillar 1: v1.1 (implemented) vs v1.2 (proposed) on synthetic scenarios, 300 seeds each.
+import math, random, statistics, sys
+GRID = [600 * 72 ** (j / 199) for j in range(200)]
+TWO_PI = 2 * math.pi
+
+def qcont(xs, q):
+    xs = sorted(xs); h = (len(xs) - 1) * q; lo = math.floor(h); hi = math.ceil(h)
+    return xs[lo] + (h - lo) * (xs[hi] - xs[lo])
+
+def core(ts, grid):
+    t0 = ts[0]; n = len(ts); Ps = []
+    for p in grid:
+        f = 1 / p; c = s = 0.0
+        for t in ts:
+            ph = TWO_PI * f * (t - t0); c += math.cos(ph); s += math.sin(ph)
+        Ps.append((c * c + s * s) / n)
+    return Ps
+
+def disp(ts):
+    d = [b - a for a, b in zip(ts, ts[1:])]
+    med = qcont(d, .5)
+    if med <= 1e-9: return None, med
+    madm = qcont([abs(x - med) for x in d], .5)
+    return madm / med, med
+
+def v11(ts):
+    ts = sorted(ts)
+    if len(ts) < 20: return 0.0, 'insufficient_data', None
+    Ps = core(ts, GRID); pmax = max(Ps)
+    sig = min(1, max(0, (pmax - math.log(200)) / (6 * math.log(10))))
+    dn, med = disp(ts)
+    if dn is None: return 0.5 * sig, 'degenerate_deltas', None
+    reg = min(1, max(0, 1 - 2 * dn))
+    per = max(p for p, v in zip(GRID, Ps) if v >= .5 * pmax)
+    return 0.5 * sig + 0.5 * reg, 'ok', per
+
+def v12(ts, cycles=3, gate=0.25):
+    ts = sorted(ts)
+    if len(ts) < 20: return 0.0, 'insufficient_data', None
+    span = ts[-1] - ts[0]
+    grid = [p for p in GRID if p <= span / cycles]
+    if not grid: return 0.0, 'insufficient_span', None
+    Ps = core(ts, grid); pmax = max(Ps); M = len(grid)
+    sig = min(1, max(0, (pmax - math.log(M)) / (6 * math.log(10))))
+    dn, med = disp(ts)
+    if dn is None: return 0.0, 'degenerate_deltas', None
+    reg = min(1, max(0, 1 - 2 * dn))
+    g = min(1.0, reg / gate)
+    per = max(p for p, v in zip(grid, Ps) if v >= .5 * pmax)
+    return 0.5 * sig * g + 0.5 * reg, 'ok', per
+
+B = 1_790_000_000.0; T0 = 14400
+def perfect(r): return [B + k * T0 for k in range(42)]
+def grid(r): return [B + k * T0 + T0 * r.uniform(-.18, .18) for k in range(42)]
+def walk(r):
+    t, o = B, []
+    for _ in range(42): o.append(t); t += T0 * (1 + r.uniform(-.18, .18))
+    return o
+def poisson(r):
+    t, o = B, []
+    while len(o) < 40: o.append(t); t += r.expovariate(40 / (7 * 86400))
+    return o
+def dropout(r):  # 4 h beacon, +-10% cumulative jitter, 30% of beats missed (host asleep)
+    t, o = B, []
+    for _ in range(60):
+        if r.random() > .3: o.append(t)
+        t += T0 * (1 + r.uniform(-.1, .1))
+    return o
+def bursty(r, span):  # browsing-like: Poisson cluster starts, geometric cluster size, ~2 s intra-gaps
+    o = []; t = B
+    while True:
+        t += r.expovariate(1 / 900)
+        if t > B + span: break
+        k = 1
+        while r.random() < .8: k += 1
+        u = t
+        for _ in range(k): o.append(u); u += r.expovariate(1 / 2)
+    return o
+def bursty3h(r): return bursty(r, 3 * 3600)
+def bursty24h(r): return bursty(r, 86400)
+def fast_short(r):  # 15 min beacon observed only 24 h, 10% cumulative jitter
+    t, o = B, []
+    while t < B + 86400: o.append(t); t += 900 * (1 + r.uniform(-.1, .1))
+    return o
+
+def q(xs):
+    xs = sorted(xs); n = len(xs)
+    return f"min={xs[0]:.3f} p05={xs[n//20]:.3f} med={statistics.median(xs):.3f} p95={xs[-max(1,n//20)]:.3f} max={xs[-1]:.3f}"
+seeds = range(1, int(sys.argv[1]) + 1 if len(sys.argv) > 1 else 301)
+for name, gen in [('perfect', perfect), ('grid', grid), ('walk', walk), ('dropout', dropout), ('fast_short', fast_short),
+                  ('poisson', poisson), ('bursty3h', bursty3h), ('bursty24h', bursty24h)]:
+    a, b, pa, pb, st = [], [], [], [], {}
+    for s in (seeds if name != 'perfect' else [1]):
+        ts = gen(random.Random(s))
+        x = v11(ts); y = v12(ts)
+        a.append(x[0]); b.append(y[0]); st[y[1]] = st.get(y[1], 0) + 1
+        if x[2]: pa.append(x[2])
+        if y[2]: pb.append(y[2])
+    print(f"{name:10s} v1.1 {q(a)} | >=0.40: {sum(v >= .4 for v in a)}/{len(a)}")
+    print(f"{name:10s} v1.2 {q(b)} | >=0.40: {sum(v >= .4 for v in b)}/{len(b)} status={st}")
+    if name in ('grid', 'walk', 'dropout'):
+        print(f"{'':10s} period within 10% of {T0}: v1.1 {sum(abs(p-T0)<=.1*T0 for p in pa)}/{len(pa)} v1.2 {sum(abs(p-T0)<=.1*T0 for p in pb)}/{len(pb)}")
+    if name == 'fast_short':
+        print(f"{'':10s} period within 10% of 900: v1.2 {sum(abs(p-900)<=90 for p in pb)}/{len(pb)}")

research/validation/p1v12sim_lib.py

@@ -0,0 +1,51 @@
+# Pillar 1: v1.1 (implemented) vs v1.2 (proposed) on synthetic scenarios, 300 seeds each.
+import math, random, statistics, sys
+GRID = [600 * 72 ** (j / 199) for j in range(200)]
+TWO_PI = 2 * math.pi
+
+def qcont(xs, q):
+    xs = sorted(xs); h = (len(xs) - 1) * q; lo = math.floor(h); hi = math.ceil(h)
+    return xs[lo] + (h - lo) * (xs[hi] - xs[lo])
+
+def core(ts, grid):
+    t0 = ts[0]; n = len(ts); Ps = []
+    for p in grid:
+        f = 1 / p; c = s = 0.0
+        for t in ts:
+            ph = TWO_PI * f * (t - t0); c += math.cos(ph); s += math.sin(ph)
+        Ps.append((c * c + s * s) / n)
+    return Ps
+
+def disp(ts):
+    d = [b - a for a, b in zip(ts, ts[1:])]
+    med = qcont(d, .5)
+    if med <= 1e-9: return None, med
+    madm = qcont([abs(x - med) for x in d], .5)
+    return madm / med, med
+
+def v11(ts):
+    ts = sorted(ts)
+    if len(ts) < 20: return 0.0, 'insufficient_data', None
+    Ps = core(ts, GRID); pmax = max(Ps)
+    sig = min(1, max(0, (pmax - math.log(200)) / (6 * math.log(10))))
+    dn, med = disp(ts)
+    if dn is None: return 0.5 * sig, 'degenerate_deltas', None
+    reg = min(1, max(0, 1 - 2 * dn))
+    per = max(p for p, v in zip(GRID, Ps) if v >= .5 * pmax)
+    return 0.5 * sig + 0.5 * reg, 'ok', per
+
+def v12(ts, cycles=3, gate=0.25):
+    ts = sorted(ts)
+    if len(ts) < 20: return 0.0, 'insufficient_data', None
+    span = ts[-1] - ts[0]
+    grid = [p for p in GRID if p <= span / cycles]
+    if not grid: return 0.0, 'insufficient_span', None
+    Ps = core(ts, grid); pmax = max(Ps); M = len(grid)
+    sig = min(1, max(0, (pmax - math.log(M)) / (6 * math.log(10))))
+    dn, med = disp(ts)
+    if dn is None: return 0.0, 'degenerate_deltas', None
+    reg = min(1, max(0, 1 - 2 * dn))
+    g = min(1.0, reg / gate)
+    per = max(p for p, v in zip(grid, Ps) if v >= .5 * pmax)
+    return 0.5 * sig * g + 0.5 * reg, 'ok', per
+

research/validation/p2real.py

@@ -0,0 +1,52 @@
+# Pillar 2 candidate metrics on real Zeek conn.log: chained session duration per tuple, persistence score,
+# per-host robust z of ln(duration), PCR, failed attempts.
+import sys, math, collections, statistics
+def read(path):
+    fields = None
+    for line in open(path, errors='replace'):
+        line = line.rstrip('\n')
+        if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        yield dict(zip(fields, line.split('\t')))
+num = lambda x: float(x) if x not in ('-', '', '(empty)') else 0.0
+GAP = 3600.0
+def sessions(rows):
+    by4 = collections.defaultdict(list)
+    for r in rows:
+        if r['proto'] != 'tcp': continue
+        by4[(r['id.orig_h'], r['id.orig_p'], r['id.resp_h'], r['id.resp_p'])].append(r)
+    out = []
+    for k, rs in by4.items():
+        rs.sort(key=lambda r: float(r['ts']))
+        cur = None
+        for r in rs:
+            s = float(r['ts']); e = s + num(r['duration'])
+            cont = cur is not None and 'S' not in r['history'] and s - cur['end'] <= GAP
+            if cont:
+                cur['end'] = max(cur['end'], e); cur['ob'] += num(r['orig_bytes']); cur['rb'] += num(r['resp_bytes']); cur['parts'] += 1
+            else:
+                if cur: out.append(cur)
+                cur = dict(key=k, start=s, end=e, ob=num(r['orig_bytes']), rb=num(r['resp_bytes']), parts=1)
+        out.append(cur)
+    return out
+for path in sys.argv[1:]:
+    rows = list(read(path))
+    ss = sessions(rows)
+    fails = collections.Counter((r['id.orig_h'], r['id.resp_h'], r['id.resp_p']) for r in rows if r['proto'] == 'tcp' and r['conn_state'] in ('S0', 'REJ', 'RSTOS0', 'SH'))
+    host_ld = collections.defaultdict(list)
+    for s in ss:
+        d = s['end'] - s['start']
+        if d > 0: host_ld[s['key'][0]].append(math.log(d))
+    best = {}
+    for s in ss:
+        t = (s['key'][0], s['key'][2], s['key'][3]); d = s['end'] - s['start']
+        if t not in best or d > best[t]['end'] - best[t]['start']: best[t] = s
+    ranked = sorted(best.items(), key=lambda kv: -(kv[1]['end'] - kv[1]['start']))
+    print(f"== {path}: tcp sessions={len(ss)} tuples={len(best)}")
+    for t, s in ranked[:5]:
+        d = s['end'] - s['start']; xs = host_ld[t[0]]
+        xs = xs or [0.0]; med = statistics.median(xs); mad = statistics.median(abs(x - med) for x in xs) or 1e-9
+        z = (math.log(d) - med) / (1.4826 * mad) if d > 0 else 0
+        pers = min(1, max(0, math.log(d / 3600) / math.log(24))) if d > 0 else 0
+        tot = s['ob'] + s['rb']; pcr = (s['ob'] - s['rb']) / tot if tot else 0
+        print(f"  {t} L={d:.0f}s ({d/3600:.2f}h) parts={s['parts']} persistence={pers:.3f} host_robust_z={z:.1f} ob={s['ob']:.0f} rb={s['rb']:.0f} PCR={pcr:+.2f} B/h={(tot/(d/3600)) if d>0 else 0:.0f} failed_attempts={fails[t]}")

research/validation/p3hourly.py

@@ -0,0 +1,27 @@
+# Hourly (UTC-aligned) information volume of unique subdomains per (host, base) on real dns.log files.
+import sys, os, collections, math, importlib.util
+here = os.path.dirname(os.path.abspath(__file__))
+sp = importlib.util.spec_from_file_location('p3lib', os.path.join(here, 'p3lib.py')); L = importlib.util.module_from_spec(sp); sp.loader.exec_module(L)
+def read(path):
+    fields = None
+    for line in open(path, errors='replace'):
+        line = line.rstrip('\n')
+        if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        yield dict(zip(fields, line.split('\t')))
+allmax = []
+for path in sys.argv[1:]:
+    b = collections.defaultdict(set)
+    for r in read(path):
+        q = r.get('query')
+        if q in (None, '-', ''): continue
+        base, sub = L.split(q)
+        if sub: b[(r['id.orig_h'], base, int(float(r['ts']) // 3600))].add(sub)
+    best = []
+    for (h, base, hr), subs in b.items():
+        cnt = collections.Counter(c for s in subs for c in L.chars(s))
+        bits = L.H(cnt) * sum(len(L.chars(s)) for s in subs)
+        best.append((bits, h, base, hr, len(subs)))
+    best.sort(reverse=True)
+    print(f"== {path}: (host,base,hour) groups={len(best)}; top3 hourly info bits:")
+    for x in best[:3]: print(f"   bits={x[0]:.0f} host={x[1]} base={x[2]} U={x[4]}")

research/validation/p3lib.py

@@ -0,0 +1,42 @@
+import math, collections
+SLD2 = {'co', 'com', 'net', 'org', 'gov', 'edu', 'ac'}
+def split(q):
+    q = q.rstrip('.').lower()
+    labels = q.split('.')
+    k = 3 if len(labels) >= 3 and len(labels[-1]) == 2 and labels[-2] in SLD2 else 2
+    if len(labels) <= k: return q, ''
+    return '.'.join(labels[-k:]), '.'.join(labels[:-k])
+def chars(s): return [c for c in s if c != '.']
+def H(counter):
+    n = sum(counter.values())
+    return -sum(v / n * math.log2(v / n) for v in counter.values()) if n else 0.0
+def norm(counter):
+    n = sum(counter.values()); return {k: v / n for k, v in counter.items()} if n else {}
+def jsd(p, q):
+    keys = sorted(set(p) | set(q)); s = 0.0
+    for k in keys:
+        a = p.get(k, 0.0); b = q.get(k, 0.0); m = (a + b) / 2
+        if a > 0: s += 0.5 * a * math.log2(a / m)
+        if b > 0: s += 0.5 * b * math.log2(b / m)
+    return s
+def groups_from(pairs):
+    g = collections.defaultdict(set)
+    for host, q in pairs:
+        base, sub = split(q)
+        if sub: g[(host, base)].add(sub)
+    return g
+def features(g):
+    dists = {k: norm(collections.Counter(c for s in subs for c in chars(s))) for k, subs in g.items()}
+    keys = sorted(dists); N = len(keys)
+    tot = collections.defaultdict(float)
+    for k in keys:
+        for c, v in dists[k].items(): tot[c] += v
+    out = {}
+    for k in keys:
+        subs = g[k]
+        ref = {c: (tot[c] - dists[k].get(c, 0.0)) / (N - 1) for c in tot} if N > 1 else {}
+        ref = {c: v for c, v in ref.items() if v > 1e-15}
+        cnt = collections.Counter(c for s in subs for c in chars(s))
+        out[k] = dict(U=len(subs), H=H(cnt), JSD=jsd(dists[k], ref) if ref else 0.0,
+                      L=sum(len(s) for s in subs) / len(subs), maxlab=max(max(len(x) for x in s.split('.')) for s in subs))
+    return out

research/validation/p3real.py

@@ -0,0 +1,18 @@
+import sys, os, importlib.util
+here = os.path.dirname(os.path.abspath(__file__))
+sp = importlib.util.spec_from_file_location('p3lib', os.path.join(here, 'p3lib.py')); L = importlib.util.module_from_spec(sp); sp.loader.exec_module(L)
+def read(path):
+    fields = None
+    for line in open(path, errors='replace'):
+        line = line.rstrip('\n')
+        if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        yield dict(zip(fields, line.split('\t')))
+for path in sys.argv[1:]:
+    pairs = [(r['id.orig_h'], r['query']) for r in read(path) if r.get('query') not in (None, '-', '')]
+    g = L.groups_from(pairs); f = L.features(g)
+    top = sorted(f.items(), key=lambda kv: -kv[1]['U'])[:6]
+    print(f"== {path}: queries={len(pairs)} groups={len(f)}")
+    for k, v in top: print(f"  {k} U={v['U']} H={v['H']:.2f} JSD={v['JSD']:.3f} meanlen={v['L']:.1f} maxlabel={v['maxlab']}")
+    hj = sorted(f.items(), key=lambda kv: -kv[1]['JSD'])[:3]
+    for k, v in hj: print(f"  topJSD {k} U={v['U']} H={v['H']:.2f} JSD={v['JSD']:.3f} meanlen={v['L']:.1f}")

research/validation/p3sim.py

@@ -0,0 +1,37 @@
+# Inject real tunnel qnames (Mendeley DTQBC, CC BY 4.0) at various volumes into real benign DNS (CTU-Normal-20 dns.log
+# regenerated with Zeek 9.0.1); compute per-(host, base) features. 300 seeds per (class, U).
+import sys, os, csv, random, statistics, math, importlib.util
+here = os.path.dirname(os.path.abspath(__file__))
+sp = importlib.util.spec_from_file_location('p3lib', os.path.join(here, 'p3lib.py')); L = importlib.util.module_from_spec(sp); sp.loader.exec_module(L)
+def read(path):
+    fields = None
+    for line in open(path, errors='replace'):
+        line = line.rstrip('\n')
+        if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        yield dict(zip(fields, line.split('\t')))
+benign_logs = sys.argv[1].split(',')
+mend = sys.argv[2]; seeds = int(sys.argv[3])
+rows = list(csv.reader(open(mend, encoding='utf-8', errors='replace')))[1:]
+cls = {c: [r[1] for r in rows if r[0] == c] for c in '1234'}
+names = {'1': 'dns2tcp', '2': 'dnscapy', '3': 'iodine', '4': 'tuns'}
+def info_bits(f): return f['H'] * f['L'] * f['U']
+for bl in benign_logs:
+    base_pairs = [(r['id.orig_h'], r['query']) for r in read(bl) if r.get('query') not in (None, '-', '')]
+    fb = L.features(L.groups_from(base_pairs))
+    ib = sorted((info_bits(v), k, v) for k, v in fb.items())
+    print(f"== benign {bl}: groups={len(fb)}; top info_bits:")
+    for i, k, v in ib[-4:]: print(f"   {k} U={v['U']} H={v['H']:.2f} meanlen={v['L']:.1f} JSD={v['JSD']:.3f} info_bits={i:.0f}")
+    for c in '124':
+        for U in (30, 60, 100, 300):
+            I, J, Hs, Ls = [], [], [], []
+            for s in range(1, seeds + 1):
+                r = random.Random(s * 1000 + U)
+                pool = list(dict.fromkeys(cls[c])); sample = r.sample(pool, min(U, len(pool)))
+                pairs = base_pairs + [('10.66.66.66', q) for q in sample]
+                f = L.features(L.groups_from(pairs))
+                tun = [v for k, v in f.items() if k[0] == '10.66.66.66']
+                v = max(tun, key=lambda x: x['U'])
+                I.append(info_bits(v)); J.append(v['JSD']); Hs.append(v['H']); Ls.append(v['L'])
+            q = lambda xs: f"min={min(xs):.3g} med={statistics.median(xs):.3g} max={max(xs):.3g}"
+            print(f"  {names[c]:8s} U={U:4d} info_bits {q(I)} | JSD {q(J)} | H {q(Hs)} | meanlen {q(Ls)}")

research/validation/zeekstat.py

@@ -0,0 +1,38 @@
+import sys, collections
+def read(path):
+    fields=None; rows=[]
+    for line in open(path, errors='replace'):
+        line=line.rstrip('\n')
+        if line.startswith('#fields'): fields=line.split('\t')[1:]; continue
+        if line.startswith('#') or not line: continue
+        p=line.split('\t')
+        rows.append(dict(zip(fields,p)))
+    return rows
+def f(x):
+    try: return float(x)
+    except: return None
+for path in sys.argv[1:]:
+    rows=read(path)
+    tup=collections.defaultdict(list)
+    for r in rows:
+        if r.get('proto')!='tcp': continue
+        tup[(r['id.orig_h'],r['id.resp_h'],r['id.resp_p'])].append(r)
+    out=[]
+    for k,rs in tup.items():
+        ivs=sorted((f(r['ts']), f(r['ts'])+(f(r['duration']) or 0)) for r in rs)
+        # union length
+        tot=0; cs,ce=ivs[0]
+        for s,e in ivs[1:]:
+            if s<=ce: ce=max(ce,e)
+            else: tot+=ce-cs; cs,ce=s,e
+        tot+=ce-cs
+        mx=max((f(r['duration']) or 0) for r in rs)
+        ob=sum(int(r['orig_bytes']) if r['orig_bytes'].isdigit() else 0 for r in rs)
+        rb=sum(int(r['resp_bytes']) if r['resp_bytes'].isdigit() else 0 for r in rs)
+        states=collections.Counter(r['conn_state'] for r in rs)
+        out.append((tot,mx,len(rs),k,ob,rb,dict(states.most_common(3))))
+    out.sort(reverse=True)
+    allts=[f(r['ts']) for r in rows]
+    print('==',path, 'tcp tuples',len(out),'span_h %.2f'%((max(allts)-min(allts))/3600))
+    for o in out[:8]:
+        print('  union=%.0fs max=%.0fs n=%d %s ob=%d rb=%d %s'%o)