TILT-Bench

We train one DeepLog model and evaluate it three ways. It detects real anomalies well, but performs worse than chance on injected timing faults.

With fixed-count windows, PCA, Isolation Forest, LogAnomaly-style, and Transformer scores didn't move at all. The change was exactly 0.0 in all 172 settings.

Datasets

We injected timing faults into seven datasets. We also looked at four others but had to leave them out.

Placebo-controlled detection

Summary

The usual log anomaly detection benchmarks (BGL, Thunderbird, Spirit) are almost entirely made of content-based anomalies and have no timing faults, so they can't tell you whether a detector catches stalls or bursts. TILT-Bench fixes that by adding controlled timing faults, changing only the timestamps and leaving the log content and order alone. We prove that with fixed-count windows, content-based sequence models like DeepLog and LogBERT can't see these faults at all. We also show that raw AUC-ROC on injected data gets skewed by where faults happen to land, which is why we use a placebo-controlled evaluation.

Benchmark features

Methodology

First, we check that existing datasets really don't have timing anomalies. Then we add timing faults scaled to each node. The event sequences stay exactly the same and only the timestamps change, so we can see how windowing (fixed-time vs. fixed-count) affects what a model is able to notice.

Premise audit

We went through every labeled anomaly in BGL, Thunderbird, and Spirit and sorted it as a new/rare template, a pure-order anomaly, or a timing-gap anomaly. Between 99.31% and 99.99% come down to new or rare templates. Across all three datasets, only one anomalous row was purely about timing.

Fault injection model

We inject faults one node at a time, scaled to that node’s median absolute deviation of inter-arrival gaps. A stall adds a long delay and pushes every later timestamp forward. A burst squeezes a run of gaps together and pulls them back. A slowdown gradually stretches the gaps over a run.

Physical-plausibility gating

We only inject into nodes that have enough events at a reasonable rate, and no transient fault lasts longer than an hour.

Placebo-controlled evaluation

Faults land on specific nodes, and some of those nodes might already look a bit unusual. That means raw AUC mixes real detection with where the fault happened to land. To fix this, we match each injected row to its clean version to get a placebo score. The part of detection that actually comes from the fault is δ = AUC(injected) − AUC(placebo).

Results

Windowing decides what a model can see

With fixed-count windows, template-based detectors (PCA, DeepLog, LogBERT) completely miss timing faults that keep event order intact. The score difference between clean and injected logs was exactly 0.0 in all 172 cells.

Good at one thing, blind to another

DeepLog scores an AUC-ROC of 0.965 on Thunderbird and 0.746 on BGL’s real anomalies, but the exact same model only gets 0.464 on injected BGL timing faults. So this isn’t a training problem. The model just never receives the timing information.

Fixed-time leakage

With fixed-time windows, shifting timestamps moves events into different windows and changes the counts. That lets content-only detectors “catch” bursts without ever reading time (PCA even reaches δ = +0.30 on OpenStack bursts).

AUC can be misleading

On Spirit bursts, a LogAnomaly-style detector gets a raw AUC of 0.997, but its placebo AUC is also 0.997, so δ is only 0.00002. On BGL stalls, correcting for placement completely flips the detector ranking.

HDFS works differently

HDFS anomalies are about missing steps in a sequence (like a missing acknowledgment), not new content or timing faults. That makes it a poor fit for this kind of injection.