Lucas RayanandClaude Sonnet 5
c64bbbf775
Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as beaconing: on short captures, grid periods longer than the observed span never complete a cycle, so significance saturates to 1 and the score sticks at exactly 0.5. v1.2 caps the grid at a third of the observed span, requires a minimum number of completed cycles, and excludes protocols that are periodic by nature (NTP, DNS, mDNS, ICMP). Validated against 300 seeds per scenario: false positives on bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300, with every beacon-detection scenario scoring unchanged. Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling via Shannon-entropy information volume per subdomain per hour; the Jensen-Shannon divergence was measured and does not separate tunnels from benign CDN subdomains, so it stays in the evidence at zero weight instead of in the score), and Pillar 4 (deterministic noisy-OR fusion across the three pillars). The fusion formula was validated against 8 labeled real-traffic vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known limitation is kept, not hidden: a benign Philips HUE bulb's strict periodic polling and a 21.9h session score it above three real C2 hosts. No weight or threshold was tuned to mask this; it is documented in the architecture docs instead. Also fixes real bugs found along the way: ingest_dir's partition count summed file names instead of distinct dates, DuckDB errors were counted as bad rows, /api/alerts always returned an empty list regardless of the last analysis, and the dashboard still claimed every pillar was unimplemented. Restores four design corrections that were missing from docs/architecture.md (KL to Jensen-Shannon, the persistent-session detector, the JA4 licensing boundary, and deterministic reduction order), and catalogs the public datasets used for validation (IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256 checksums under research/validation/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD
38 lines
2.3 KiB
Python
38 lines
2.3 KiB
Python
# Inject real tunnel qnames (Mendeley DTQBC, CC BY 4.0) at various volumes into real benign DNS (CTU-Normal-20 dns.log
|
|
# regenerated with Zeek 9.0.1); compute per-(host, base) features. 300 seeds per (class, U).
|
|
import sys, os, csv, random, statistics, math, importlib.util
|
|
here = os.path.dirname(os.path.abspath(__file__))
|
|
sp = importlib.util.spec_from_file_location('p3lib', os.path.join(here, 'p3lib.py')); L = importlib.util.module_from_spec(sp); sp.loader.exec_module(L)
|
|
def read(path):
|
|
fields = None
|
|
for line in open(path, errors='replace'):
|
|
line = line.rstrip('\n')
|
|
if line.startswith('#fields'): fields = line.split('\t')[1:]; continue
|
|
if line.startswith('#') or not line: continue
|
|
yield dict(zip(fields, line.split('\t')))
|
|
benign_logs = sys.argv[1].split(',')
|
|
mend = sys.argv[2]; seeds = int(sys.argv[3])
|
|
rows = list(csv.reader(open(mend, encoding='utf-8', errors='replace')))[1:]
|
|
cls = {c: [r[1] for r in rows if r[0] == c] for c in '1234'}
|
|
names = {'1': 'dns2tcp', '2': 'dnscapy', '3': 'iodine', '4': 'tuns'}
|
|
def info_bits(f): return f['H'] * f['L'] * f['U']
|
|
for bl in benign_logs:
|
|
base_pairs = [(r['id.orig_h'], r['query']) for r in read(bl) if r.get('query') not in (None, '-', '')]
|
|
fb = L.features(L.groups_from(base_pairs))
|
|
ib = sorted((info_bits(v), k, v) for k, v in fb.items())
|
|
print(f"== benign {bl}: groups={len(fb)}; top info_bits:")
|
|
for i, k, v in ib[-4:]: print(f" {k} U={v['U']} H={v['H']:.2f} meanlen={v['L']:.1f} JSD={v['JSD']:.3f} info_bits={i:.0f}")
|
|
for c in '124':
|
|
for U in (30, 60, 100, 300):
|
|
I, J, Hs, Ls = [], [], [], []
|
|
for s in range(1, seeds + 1):
|
|
r = random.Random(s * 1000 + U)
|
|
pool = list(dict.fromkeys(cls[c])); sample = r.sample(pool, min(U, len(pool)))
|
|
pairs = base_pairs + [('10.66.66.66', q) for q in sample]
|
|
f = L.features(L.groups_from(pairs))
|
|
tun = [v for k, v in f.items() if k[0] == '10.66.66.66']
|
|
v = max(tun, key=lambda x: x['U'])
|
|
I.append(info_bits(v)); J.append(v['JSD']); Hs.append(v['H']); Ls.append(v['L'])
|
|
q = lambda xs: f"min={min(xs):.3g} med={statistics.median(xs):.3g} max={max(xs):.3g}"
|
|
print(f" {names[c]:8s} U={U:4d} info_bits {q(I)} | JSD {q(J)} | H {q(Hs)} | meanlen {q(Ls)}")
|