How much can network traffic reveal about LLM agent tasks and operations for auditing?

ANT pairs traffic with three levels of agent behavior

The paper asks how much an auditor can learn about an LLM agent from network traffic alone, without inspecting private user content. To answer that, the authors built ANT (Agent Network Traffic), a dataset that places network traffic alongside agent behavior information at three granularities: risk, scenario, and behavior primitive.

The scale is specific. ANT contains 3,114 execution episodes collected under five agent combinations across 20 tasks in five scenarios, corresponding to 659 distinct task instances and 104.07 GB of traffic. It comprises 276,417 bidirectional flows and 40,049 behavior primitive segments organized into 47 macro groups. Collection ran on geographically distributed cloud servers at a total cost of US$2,176.50.

The five scenarios are four common agent use cases plus one high-risk case: Computational Artifact Engineering, General Knowledge Question Answering, Scientific Evidence Assessment, Interactive Application Operation, and Cybersecurity Misuse. The authors state this is, to their knowledge, the first dataset dedicated to network traffic from client-side LLM agent execution.

Existing traffic datasets lack joint task and stage labels

The gap the paper identifies is annotation, not raw traffic. Website and application corpora label sites or apps, but not complete agent executions with synchronized stages. AgentPrint identifies agents and coarse activities, but the authors say it lacks joint annotations for episode risk, task scenarios, and behavior primitives aligned with execution stages.

That missing combination is what makes the auditing question testable. Without labels that tie a packet trace to both the task being run and the stage of execution, a benchmark cannot ask whether traffic distinguishes a benign workflow from a malicious one, or one scenario from another. ANT aligns packet traces with task and stage semantics so that common splits, objectives, and metrics can be shared across three objectives.

Thirteen baselines share one split and header-only inputs

The benchmark evaluates 13 encrypted traffic classification and website fingerprinting baselines covering convolutional networks, Transformers, state-space models, and learned traffic representations. Encrypted traffic classification baselines include FlowLens, YaTC, NetMamba, TFusion, and TrafficFormer; website fingerprinting baselines include DF, AWF, Tik-Tok, Var-CNN, NetCLR, TMWF, ARES, and BAPM.

Episodes are randomly split within each task into fixed training, validation, and test sets shared by all benchmarks. Behavior primitive segments inherit their source episode's split, so segments from one episode never cross sets. The episode test set contains 209 episodes, of which 189 are benign and 20 are malicious.

Inputs use only packet headers and side-channel features such as packet direction, timing, and length. Identifying fields such as source and destination IP addresses are masked to mitigate shortcut learning. Encrypted traffic classification baselines operate on flows, while website fingerprinting baselines operate on episodes; for episode-level evaluation, flow predictions are aggregated using the strategy with the highest validation accuracy among four candidates.

Risk identification collapses when malicious tasks look benign

The risk results are the sharpest warning in the paper. Under full traffic, several baselines reach high accuracy while barely detecting the malicious class. FlowLens, YaTC, and NetMamba all report accuracy of 0.923 to 0.928, but their F1 scores are only 0.333 to 0.400 and recall is 0.200 to 0.250, with false positive rates near zero.

The pattern is consistent across most baselines: accuracy stays high because the test set is dominated by benign episodes, while recall on the 20 malicious episodes stays low. TFusion and TrafficFormer reach accuracy of 0.933 with F1 of 0.462 and recall of 0.300.

Two baselines break the pattern. DF reaches accuracy 0.976, F1 0.872, and recall 0.850 with a false positive rate of 0.011. Tik-Tok reaches accuracy 0.981, F1 0.905, and recall 0.950, though its false positive rate is 0.016. The authors' explanation is that methods struggle to identify risk when malicious workflows resemble benign tasks, so the headline is not that traffic cannot flag risk, but that most tested methods do not do so reliably on this benchmark.

Scenario recognition weakens when traffic patterns look alike

The second objective asks whether traffic can tell which of the five scenarios an episode belongs to. The paper reports that existing methods struggle to distinguish scenarios with similar traffic patterns, which is a narrower failure than failing at scenario recognition in general.

The dataset itself shows why some scenarios are separable and others are not. Comparing distinct tasks within a scenario against cross-scenario pairs, primitive embedding MMD distances are 15.46% smaller within scenario, and Hellinger distances between primitive macro group proportions are 14.79% smaller. Both associations remain significant after Holm correction, with p = 0.0006 and p = 0.0050.

Ordering adds structure beyond composition. For semantic directed areas and primitive macro group transition residuals, the distance reductions are 6.74% and 5.70%, and their cohesion gains exceed the means under composition-preserving random reorderings by 5.77 and 5.24 percentage points, with Holm-adjusted p = 0.0050 for both. The authors note these order effects persist when role positions are fixed, so observed ordering contributes scenario structure beyond which primitives appear.

Primitive classification favors frequent and distinctive groups

The third objective, behavior primitive classification, is the most reliable of the three, but reliability is conditional. The paper reports that primitive classification is more reliable for frequent macro groups and those with distinctive traffic patterns than for rare or semantically similar groups.

The 47 macro groups are highly uneven. In the primitive type counts, groups such as search, browser fetch, code creation, conversation input, and task prompt contain thousands of segments, while groups such as http header probe, data decoding, xss testing, and bypass attempt contain only tens to low hundreds. The smallest macro group contains 14 primitive types.

Clustering diagnostics show the grouping is usable but not crisp. At K = 47, the cosine silhouette is approximately 0.157 and seed stability is approximately 0.51 ARI. For primitive identification, training classes containing fewer than 200 segments are oversampled to 200 by random sampling with replacement within each class, while validation and test sets retain their original distributions, so rare-group performance is measured on the real, imbalanced distribution.

The benchmark measures detection, not attribution or intent

Several boundaries qualify how far these results travel. The episode test set contains only 20 malicious episodes, so risk metrics such as F1 and recall rest on a small positive class, and the near-zero false positive rates reported by several baselines should be read against that base rate.

The benchmark also uses header and side-channel features only, with identifying fields masked. That design isolates what traffic shape reveals, but it means the results do not describe what an auditor could achieve with additional telemetry, endpoint logs, or payload access. The authors frame ANT as a common basis for developing more precise auditing and forensic analysis, not as a finished auditing system.

Collection used locally executed agents, locally simulated services for state-changing benign interactions, and isolated Docker environments for attack execution, with no real-user accounts or personally identifiable information. The malicious scenario is therefore a contained Cybersecurity Misuse setting, and the risk results describe that setting rather than live organizational traffic.

Prioritize primitive-level signals over episode-level risk scores

The paper's own comparison points to a practical ordering. Episode-level risk identification is where most baselines fail: high accuracy with F1 between 0.231 and 0.462 and recall between 0.150 and 0.300 for most methods, even though DF and Tik-Tok reach F1 of 0.872 and 0.905. Scenario recognition degrades specifically when traffic patterns are similar. Primitive classification is the objective the authors describe as more reliable, with the caveat that reliability tracks group frequency and traffic distinctiveness.

That ordering matters because it changes what an auditor should trust. A high episode-level accuracy number on this benchmark can coexist with missing most malicious episodes, so accuracy alone is not the decision-relevant metric; F1 and recall on the malicious class are. And because primitive reliability depends on how frequent and how distinctive a macro group is, the 47-group structure gives a concrete way to separate behaviors that traffic can support from those it cannot.

The most useful reading of ANT is therefore as a map of where traffic-based auditing is strong and where it is thin: primitive-level signals for frequent, distinctive behaviors, with risk and scenario judgments treated as the harder and less settled targets.

Sources used in this answer