Can AI Investigate an Incident Hidden in a Day of Enterprise Logs?
Finding a suspicious event is easy.
Figuring out whether it is actually part of an attack is harder.
Now connect that event across identity, SaaS, cloud, Kubernetes, and network activity. Separate it from a full day of normal employee behavior. Reconstruct what happened. Then prove every major claim with evidence.
That is what we built IncidentBench to test.
We don't give models a clean set of suspicious events and ask them to explain what happened.
We give them a noisy day inside a synthetic enterprise and tell them to investigate.
The model has to search the evidence, decide what matters, follow the attack across systems, and submit a structured report with citations that resolve back to the underlying records.
The question is not whether AI can tell a convincing incident story.
It is whether the evidence supports it.
The incident
The scenario starts with a compromised employee account.
From there, the attacker:
- Uses single sign-on access.
- Downloads a sensitive document.
- Pivots into the cloud environment.
- Retrieves credentials from a container platform.
- Launches a workload.
- Sends data to an external destination.
We gave the models a day of evidence, not a handful of clues
The dataset contains roughly 4.5 GB of uncompressed logs across 2,022 artifacts, including a packet capture.
Evidence includes:
- Identity and SaaS audit logs
- Cloud activity
- Application telemetry
- Host authentication
- Kubernetes audit logs
- Network traffic
- Packet capture data
We intentionally keep the provider-specific formats.
No clean, normalized list of suspicious events waits for the model.
There is also plenty of background activity and benign behavior that looks suspicious at first glance.
That matters.
A security benchmark can become surprisingly easy if every event provided to the model is relevant. At that point, the task is mostly summarization.
Incident response does not work like that.
An investigator has to decide what to search for, what to ignore, which leads to follow, and when there is enough evidence to make a claim.
The model still has to investigate
The model does not get the answer key.
It cannot access the scorer.
Instead, it works through a limited investigation interface that allows it to:
- List available evidence.
- Search records and text.
- Read bounded sections of files.
- Resolve specific JSON pointers.
- Inspect packets from an offline capture.
- Submit a structured investigation report.
What we saw
Across retained sets of five completed runs, investigation quality varied considerably.
These are results from one scenario, not a general-purpose model leaderboard.
| Model | Recorded Effort | Mean Score / 100 | Mean Estimated Cost |
|---|---|---|---|
| Gemini 3.8 Flash | Medium | 90.39 | $0.576 |
| GPT-6 Astra | Medium | 85.22 | $0.449 |
| GPT-5.6 Sol | Medium | 77.58 | $0.313 |
| Grok 4.6 | Medium | 69.02 | $0.448 |
| Qwen3.8 Max 0902 | Medium | 67.33 | $0.325 |
| Claude Fable 5.1 | Medium | 65.32 | $3.782 |
| GLM 5.3 Flash | High | 64.00 | $0.020 |
| Fugu Max | High | 60.77 | $0.335 |
| Muse Spark 1.3, contributor route | High | 59.26 | $0.025 |
There are some important caveats here.
Provider effort settings are not equivalent. Medium from one provider does not mean the same thing as medium somewhere else.
One good run does not tell you much
This was one of the clearest lessons.
Grok scored 87.56 on its first investigation.
Across five completed runs, its average fell to 69.02, with scores ranging from 48.13 to 87.56.
Qwen started at 81.17 and finished with a five-run average of 67.33.
Fugu Max went from an initial 70.90 to a mean of 60.77.
Gemini held up much better, finishing with a five-run mean of 90.39.
Cheap and fast are not the same thing
Cost also told only part of the story.
Muse averaged roughly $0.025 per completed investigation, but took around 602 seconds of end-to-end wall time.
GLM averaged about $0.020 and 208 seconds.
Astra averaged roughly $0.449 and 124 seconds.
That creates different options depending on the job.
A very cheap model might make sense for broad triage.
A faster model might be worth considerably more when an analyst is waiting for an answer during an active incident.
What this benchmark does not prove
This is still one fixed synthetic incident.
We are not claiming these numbers tell us how every model will perform across:
- Different attack families
- Different companies
- Missing telemetry
- Poor-quality logs
- Adversarial log manipulation
- Different security tools
- Different investigation budgets
It also does not tell us whether a model should be allowed to contain an incident autonomously.
Where we go next
The next version needs broader coverage:
- More incident types.
- More dataset seeds.
- Explicit negative controls.
- Consistently pinned routing.
- Consistent resource budgets.
- More models.
- Reporting on failed attempts alongside completed runs.
One scenario gives us a useful instrument.
More scenarios tell us whether we are actually measuring a capability.
The takeaway
AI can investigate surprisingly large, noisy security datasets.
But being able to describe an attack is not the same thing as being able to investigate one.
The stronger runs did something very familiar to good human investigators.
They followed the evidence.
They connected activity across systems.
They ignored plausible distractions.
They kept digging when the first answer was not enough.
And when they finally made a claim, they could point back to the record that supported it.
The weaker investigations often failed much earlier.
