Lucas RayanandClaude Sonnet 5
c64bbbf775
Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as beaconing: on short captures, grid periods longer than the observed span never complete a cycle, so significance saturates to 1 and the score sticks at exactly 0.5. v1.2 caps the grid at a third of the observed span, requires a minimum number of completed cycles, and excludes protocols that are periodic by nature (NTP, DNS, mDNS, ICMP). Validated against 300 seeds per scenario: false positives on bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300, with every beacon-detection scenario scoring unchanged. Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling via Shannon-entropy information volume per subdomain per hour; the Jensen-Shannon divergence was measured and does not separate tunnels from benign CDN subdomains, so it stays in the evidence at zero weight instead of in the score), and Pillar 4 (deterministic noisy-OR fusion across the three pillars). The fusion formula was validated against 8 labeled real-traffic vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known limitation is kept, not hidden: a benign Philips HUE bulb's strict periodic polling and a 21.9h session score it above three real C2 hosts. No weight or threshold was tuned to mask this; it is documented in the architecture docs instead. Also fixes real bugs found along the way: ingest_dir's partition count summed file names instead of distinct dates, DuckDB errors were counted as bad rows, /api/alerts always returned an empty list regardless of the last analysis, and the dashboard still claimed every pillar was unimplemented. Restores four design corrections that were missing from docs/architecture.md (KL to Jensen-Shannon, the persistent-session detector, the JA4 licensing boundary, and deterministic reduction order), and catalogs the public datasets used for validation (IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256 checksums under research/validation/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD
53 lines
2.8 KiB
Python
53 lines
2.8 KiB
Python
# Pillar 2 candidate metrics on real Zeek conn.log: chained session duration per tuple, persistence score,
|
|
# per-host robust z of ln(duration), PCR, failed attempts.
|
|
import sys, math, collections, statistics
|
|
def read(path):
|
|
fields = None
|
|
for line in open(path, errors='replace'):
|
|
line = line.rstrip('\n')
|
|
if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
|
|
if line.startswith('#') or not line: continue
|
|
yield dict(zip(fields, line.split('\t')))
|
|
num = lambda x: float(x) if x not in ('-', '', '(empty)') else 0.0
|
|
GAP = 3600.0
|
|
def sessions(rows):
|
|
by4 = collections.defaultdict(list)
|
|
for r in rows:
|
|
if r['proto'] != 'tcp': continue
|
|
by4[(r['id.orig_h'], r['id.orig_p'], r['id.resp_h'], r['id.resp_p'])].append(r)
|
|
out = []
|
|
for k, rs in by4.items():
|
|
rs.sort(key=lambda r: float(r['ts']))
|
|
cur = None
|
|
for r in rs:
|
|
s = float(r['ts']); e = s + num(r['duration'])
|
|
cont = cur is not None and 'S' not in r['history'] and s - cur['end'] <= GAP
|
|
if cont:
|
|
cur['end'] = max(cur['end'], e); cur['ob'] += num(r['orig_bytes']); cur['rb'] += num(r['resp_bytes']); cur['parts'] += 1
|
|
else:
|
|
if cur: out.append(cur)
|
|
cur = dict(key=k, start=s, end=e, ob=num(r['orig_bytes']), rb=num(r['resp_bytes']), parts=1)
|
|
out.append(cur)
|
|
return out
|
|
for path in sys.argv[1:]:
|
|
rows = list(read(path))
|
|
ss = sessions(rows)
|
|
fails = collections.Counter((r['id.orig_h'], r['id.resp_h'], r['id.resp_p']) for r in rows if r['proto'] == 'tcp' and r['conn_state'] in ('S0', 'REJ', 'RSTOS0', 'SH'))
|
|
host_ld = collections.defaultdict(list)
|
|
for s in ss:
|
|
d = s['end'] - s['start']
|
|
if d > 0: host_ld[s['key'][0]].append(math.log(d))
|
|
best = {}
|
|
for s in ss:
|
|
t = (s['key'][0], s['key'][2], s['key'][3]); d = s['end'] - s['start']
|
|
if t not in best or d > best[t]['end'] - best[t]['start']: best[t] = s
|
|
ranked = sorted(best.items(), key=lambda kv: -(kv[1]['end'] - kv[1]['start']))
|
|
print(f"== {path}: tcp sessions={len(ss)} tuples={len(best)}")
|
|
for t, s in ranked[:5]:
|
|
d = s['end'] - s['start']; xs = host_ld[t[0]]
|
|
xs = xs or [0.0]; med = statistics.median(xs); mad = statistics.median(abs(x - med) for x in xs) or 1e-9
|
|
z = (math.log(d) - med) / (1.4826 * mad) if d > 0 else 0
|
|
pers = min(1, max(0, math.log(d / 3600) / math.log(24))) if d > 0 else 0
|
|
tot = s['ob'] + s['rb']; pcr = (s['ob'] - s['rb']) / tot if tot else 0
|
|
print(f" {t} L={d:.0f}s ({d/3600:.2f}h) parts={s['parts']} persistence={pers:.3f} host_robust_z={z:.1f} ob={s['ob']:.0f} rb={s['rb']:.0f} PCR={pcr:+.2f} B/h={(tot/(d/3600)) if d>0 else 0:.0f} failed_attempts={fails[t]}")
|