Lucas RayanandClaude Sonnet 5
c64bbbf775
Pillar 1 v1.1 (shipped earlier) flagged benign browsing bursts as beaconing: on short captures, grid periods longer than the observed span never complete a cycle, so significance saturates to 1 and the score sticks at exactly 0.5. v1.2 caps the grid at a third of the observed span, requires a minimum number of completed cycles, and excludes protocols that are periodic by nature (NTP, DNS, mDNS, ICMP). Validated against 300 seeds per scenario: false positives on bursty traffic drop from the 290-300/300 that v1.1 produced to 0/300, with every beacon-detection scenario scoring unchanged. Add Pillar 2 (persistent/long-lived sessions, with Zeek's fragmented TCP sessions rejoined after a 5-minute gap), Pillar 3 (DNS tunneling via Shannon-entropy information volume per subdomain per hour; the Jensen-Shannon divergence was measured and does not separate tunnels from benign CDN subdomains, so it stays in the evidence at zero weight instead of in the score), and Pillar 4 (deterministic noisy-OR fusion across the three pillars). The fusion formula was validated against 8 labeled real-traffic vectors (IoT-23, CTU-Normal) with tolerance <= 1e-4. A known limitation is kept, not hidden: a benign Philips HUE bulb's strict periodic polling and a 21.9h session score it above three real C2 hosts. No weight or threshold was tuned to mask this; it is documented in the architecture docs instead. Also fixes real bugs found along the way: ingest_dir's partition count summed file names instead of distinct dates, DuckDB errors were counted as bad rows, /api/alerts always returned an empty list regardless of the last analysis, and the dashboard still claimed every pillar was unimplemented. Restores four design corrections that were missing from docs/architecture.md (KL to Jensen-Shannon, the persistent-session detector, the JA4 licensing boundary, and deterministic reduction order), and catalogs the public datasets used for validation (IoT-23, CTU-Normal, Mendeley DTQBC) with their licenses and SHA-256 checksums under research/validation/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAyr8d7HuAdUGB6aA5xmDD
39 lines
1.5 KiB
Python
39 lines
1.5 KiB
Python
import sys, collections
|
|
def read(path):
|
|
fields=None; rows=[]
|
|
for line in open(path, errors='replace'):
|
|
line=line.rstrip('\n')
|
|
if line.startswith('#fields'): fields=line.split('\t')[1:]; continue
|
|
if line.startswith('#') or not line: continue
|
|
p=line.split('\t')
|
|
rows.append(dict(zip(fields,p)))
|
|
return rows
|
|
def f(x):
|
|
try: return float(x)
|
|
except: return None
|
|
for path in sys.argv[1:]:
|
|
rows=read(path)
|
|
tup=collections.defaultdict(list)
|
|
for r in rows:
|
|
if r.get('proto')!='tcp': continue
|
|
tup[(r['id.orig_h'],r['id.resp_h'],r['id.resp_p'])].append(r)
|
|
out=[]
|
|
for k,rs in tup.items():
|
|
ivs=sorted((f(r['ts']), f(r['ts'])+(f(r['duration']) or 0)) for r in rs)
|
|
# union length
|
|
tot=0; cs,ce=ivs[0]
|
|
for s,e in ivs[1:]:
|
|
if s<=ce: ce=max(ce,e)
|
|
else: tot+=ce-cs; cs,ce=s,e
|
|
tot+=ce-cs
|
|
mx=max((f(r['duration']) or 0) for r in rs)
|
|
ob=sum(int(r['orig_bytes']) if r['orig_bytes'].isdigit() else 0 for r in rs)
|
|
rb=sum(int(r['resp_bytes']) if r['resp_bytes'].isdigit() else 0 for r in rs)
|
|
states=collections.Counter(r['conn_state'] for r in rs)
|
|
out.append((tot,mx,len(rs),k,ob,rb,dict(states.most_common(3))))
|
|
out.sort(reverse=True)
|
|
allts=[f(r['ts']) for r in rows]
|
|
print('==',path, 'tcp tuples',len(out),'span_h %.2f'%((max(allts)-min(allts))/3600))
|
|
for o in out[:8]:
|
|
print(' union=%.0fs max=%.0fs n=%d %s ob=%d rb=%d %s'%o)
|