On Sunday evening I sat down planning to install an attack emulation tool on my Linux endpoint. Instead I spent the night finding two hosts that had been quietly broken since the night before.
The tell was small enough that I nearly missed it. I had Fleet open for an unrelated reason, checking what integrations one of my endpoints was running, and noticed that my linux-endpoint was showing yellow — degraded. The Proxmox dashboard wasn't showing anything alarming yet so I assumed it would resolve itself. But it did not. It turned out a single log file grew to 12 GB and took the file system with it, and this was just the first host.
The second (ghosts) one took longer for me to realize since it wasn't being tracked by Elastic. The only reason I knew something was off with it was because it was taking significantly longer to respond to the commands I was typing in the SSH session. It was the same issue, it kept writing until the container log file reached 23 GB.
The underlying reasons for both were the same, and the issue on ghosts was what led to linux-endpoint also capping its storage.
Same failures for both, but both hosts still up. This is the story of the platform that broke, how it's built, and what a self-inflicted outage taught me about the difference between a system that is running and a system that is working.
What the platform is
The lab exists so I can practice detection engineering end to end: infrastructure as code, multi-source telemetry, detection-as-code with tests and CI, adversary emulation, and a response path. I wanted to try to capture the whole lifecycle, as I wanted to get into detection engineering after years in incident response and malware analysis.
Two decisions shaped this moment.
Initially, I had the SIEM hosted in AWS in EC2. I had actually wanted to host everything that I could in the cloud, barring the macOS endpoint. However, I miscalculated the running costs due to not accounting for the NAT gateway and so the costs ended up breaking around $95 a month. I wanted this to be a long term project, so I figured that just buying used micro form factor desktops (Optiplex Micro, ThinkCentre Tiny) would save me over the whole course of the project. The main thing was I wanted to get hands on with infrastructure as code (IaC) aka Terraform which is why I initially had it on AWS, but it turned out Proxmox had support for it too, so my main driver for it being in the cloud was gone.
I also wanted the lab to be noisy, and this is the decision that would lead to it breaking. I didn't want to write rules towards a silent environment, where the only events being ingested in the SIEM were the events conducting the activity I am writing rules for. So I decided to try to use CMU-SEI's GHOSTS for NPC emulation.
Briefly going over the rest of the environment: a two-node Proxmox cluster, Elastic with Fleet-managed agents, endpoints across Windows, Linux, and macOS, a Zeek sensor on a port mirror, Atomic Red Team and Stratus Red Team for emulation, Tines as the response layer, and S3 snapshots for retention. Everything the Proxmox provider can manage is in Terraform, and for the host layer that it can't — the cluster itself, the quorum weighting, the network bridge conversion, the index lifecycle policies — those live in architecture decision records instead, which is its own kind of documentation debt I've made peace with.
Now here's how two logging defaults took out the activity-generation layer and one endpoint.
The cascade
It started with Docker.
The GHOSTS API stack runs as containers. Docker's json-file log driver has no size limit by default and nothing in my compose file overrode it. One container's log grew to 23 GB on a 39 GB disk, and the host filled.
At this point the API stopped serving requests, but it did not stop running. docker ps reported Up the entire time, because from Docker's perspective the process was alive and healthy, it simply couldn't write anything to a full filesystem. A database and an API that can't write look exactly like a database and an API that are fine, if the only question you ask is whether the process exists.
Then the interesting part. The clients, the agents running simulated activity on the endpoints, could no longer post their results back to the API. They did not crash. They did not stop. They buffered, spooling results to local disk, full HTTP response bodies included, waiting for the API to come back.
Meanwhile those same clients had their own problem. GHOSTS ships with logging at TRACE, and one of its targets is the console. On a systemd service, console output goes to the journal, and the journal goes to syslog. So every action the generator simulated produced log lines on the way to disk, meanwhile logrotate was configured weekly with no size cap.
I thought this was fine. These logs are written in KBs, there's no way they'd fill up storage measured in GBs, right? Well, turns out I was wrong. Between the 12 GB of TRACE-level syslog and the 2.4 GB spool of undeliverable results, the endpoint's 20 GB disk filled by Sunday afternoon, about fifteen hours after the API host did.
Two unbounded log defaults on two hosts, and one dead host led to another. The API had been down for roughly twenty-one hours before I noticed.
What it actually taught me
The obvious lesson is "set log rotation limits," which is true, boring, and not why I'm writing this. Here's what was worth the Sunday.
Resilience hides failure
The single most uncomfortable detail in this incident is that the clients kept working. They had cached timelines to execute, so they kept generating activity on the endpoints. Elastic Agent kept shipping that activity. Telemetry never stopped arriving, so nothing downstream looked wrong.
Spooling into disk when an upstream dependency is unavailable provides a layer of graceful degradation, but every layer of graceful degradation in a pipeline is also a layer of concealment. The system's ability to survive a partial failure is precisely what let the partial failure run all day when I left it unattended.
My lesson was this: the more resilient a pipeline is, the longer it can continue producing plausible output after something inside it has actually died. The hosts stay up, the SIEM is still ingesting, and everything looks fine. If your health model is "am I receiving events," a sufficiently well-engineered system will lie to you for as long as its buffers hold.
The future canary
Fleet reported an agent as degraded, and that was correct and timely. The failure was interpretive, not instrumental. A component being unwell doesn't tell me what's broken downstream — nothing connected "one agent is degraded" to "a component on a different host has been down since last night and part of my telemetry is stale."
This is the argument for canaries. Instead of relying on a dashboard to report component health, I can check that the entire chain runs end-to-end, and in a timely manner. Generate a known event, confirm it lands in the index, confirm the detection fires, and alarm loudly on the absence of that chain. Once the detection pipeline is running it's something I intend to implement.
The fix uncovered another one
But that wasn't the end. I checked Fleet the next day and found linux-endpoint's agent flapping between healthy and degraded. There were over two thousand state changes in a day, against a baseline of maybe five. It wasn't crashing, it was missing check-ins and recovering, doing this over and over again.
This time the culprit was memory. GHOSTS was expanding to about 2.3 GB of RAM and the kernel had OOM-killed it four separate times that day. The agent's components were missing heartbeats because they couldn't get scheduled while GHOSTS was taking the RAM.
The reason it only started after I fixed the storage issue was because when the filesystem was full, GHOSTS couldn't allocate freely since it was stalling on writes instead. The disk-full condition was masking the memory problem. Fixing the first failure is what exposed the second.
So the same lesson fired twice in one system, in different directions. Resilience hid the outage from me, and then one failure hid another failure from me. Neither of those announced itself, and both were only found because I went looking at something adjacent.
The platform is the beginning
With all that being said, the infrastructure is built. Terraform-managed, multi-source telemetry from cloud and endpoint, three operating systems, emulation tooling on all of them, a response path, retention and backups, and, after this little fiasco, log size caps on the hosts that actually fill, written into the provisioning code so a rebuild can't quietly reintroduce the problem.
Now I can finally get to the point on why this project exists: detections. Rules written against real telemetry, validated against real emulated attacks, tested in CI, documented with their own false-positive profiles and blind spots, and deployed through a pipeline. All of that will be in part two, but I wanted to write this down to record and share my experiences in starting this lab.