Lucas RayanandClaude Sonnet 5
c64bbbf775
Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as beaconing: on short captures, grid periods longer than the observed span never complete a cycle, so significance saturates to 1 and the score sticks at exactly 0.5. v1.2 caps the grid at a third of the observed span, requires a minimum number of completed cycles, and excludes protocols that are periodic by nature (NTP, DNS, mDNS, ICMP). Validated against 300 seeds per scenario: false positives on bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300, with every beacon-detection scenario scoring unchanged. Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling via Shannon-entropy information volume per subdomain per hour; the Jensen-Shannon divergence was measured and does not separate tunnels from benign CDN subdomains, so it stays in the evidence at zero weight instead of in the score), and Pillar 4 (deterministic noisy-OR fusion across the three pillars). The fusion formula was validated against 8 labeled real-traffic vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known limitation is kept, not hidden: a benign Philips HUE bulb's strict periodic polling and a 21.9h session score it above three real C2 hosts. No weight or threshold was tuned to mask this; it is documented in the architecture docs instead. Also fixes real bugs found along the way: ingest_dir's partition count summed file names instead of distinct dates, DuckDB errors were counted as bad rows, /api/alerts always returned an empty list regardless of the last analysis, and the dashboard still claimed every pillar was unimplemented. Restores four design corrections that were missing from docs/architecture.md (KL to Jensen-Shannon, the persistent-session detector, the JA4 licensing boundary, and deterministic reduction order), and catalogs the public datasets used for validation (IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256 checksums under research/validation/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD
43 lines
1.8 KiB
Python
43 lines
1.8 KiB
Python
import math, collections
|
|
SLD2 = {'co', 'com', 'net', 'org', 'gov', 'edu', 'ac'}
|
|
def split(q):
|
|
q = q.rstrip('.').lower()
|
|
labels = q.split('.')
|
|
k = 3 if len(labels) >= 3 and len(labels[-1]) == 2 and labels[-2] in SLD2 else 2
|
|
if len(labels) <= k: return q, ''
|
|
return '.'.join(labels[-k:]), '.'.join(labels[:-k])
|
|
def chars(s): return [c for c in s if c != '.']
|
|
def H(counter):
|
|
n = sum(counter.values())
|
|
return -sum(v / n * math.log2(v / n) for v in counter.values()) if n else 0.0
|
|
def norm(counter):
|
|
n = sum(counter.values()); return {k: v / n for k, v in counter.items()} if n else {}
|
|
def jsd(p, q):
|
|
keys = sorted(set(p) | set(q)); s = 0.0
|
|
for k in keys:
|
|
a = p.get(k, 0.0); b = q.get(k, 0.0); m = (a + b) / 2
|
|
if a > 0: s += 0.5 * a * math.log2(a / m)
|
|
if b > 0: s += 0.5 * b * math.log2(b / m)
|
|
return s
|
|
def groups_from(pairs):
|
|
g = collections.defaultdict(set)
|
|
for host, q in pairs:
|
|
base, sub = split(q)
|
|
if sub: g[(host, base)].add(sub)
|
|
return g
|
|
def features(g):
|
|
dists = {k: norm(collections.Counter(c for s in subs for c in chars(s))) for k, subs in g.items()}
|
|
keys = sorted(dists); N = len(keys)
|
|
tot = collections.defaultdict(float)
|
|
for k in keys:
|
|
for c, v in dists[k].items(): tot[c] += v
|
|
out = {}
|
|
for k in keys:
|
|
subs = g[k]
|
|
ref = {c: (tot[c] - dists[k].get(c, 0.0)) / (N - 1) for c in tot} if N > 1 else {}
|
|
ref = {c: v for c, v in ref.items() if v > 1e-15}
|
|
cnt = collections.Counter(c for s in subs for c in chars(s))
|
|
out[k] = dict(U=len(subs), H=H(cnt), JSD=jsd(dists[k], ref) if ref else 0.0,
|
|
L=sum(len(s) for s in subs) / len(subs), maxlab=max(max(len(x) for x in s.split('.')) for s in subs))
|
|
return out
|