Centralising Fleet Logs in Graylog Instead of Per-Host Files
Context
Logs lived on each machine in local files under their own rotation, read only reactively after something had already gone wrong. Correlating an issue across more than one host meant tailing files in parallel and reconstructing the timeline by hand, which does not scale past a couple of boxes.
Decision
Ship logs from every machine (rsyslog over a syslog input, GELF for structured sources) to a single Graylog Open instance backed by OpenSearch for search and storage and MongoDB for configuration, reached privately over WireGuard and the LAN.
Alternatives Considered
Keep per-host log files and SSH in to read them
- Nothing new to run or maintain
- Reactive only; no cross-host correlation
- Scattered logs are an archive you must know to visit, not observability
Use a hosted log-management SaaS
- No infrastructure to maintain and managed retention
- Logs—and whatever sensitive content they carry—leave to a third party
- Ongoing per-ingest cost and another external dependency
Reasoning
One destination turns "where do I look?" into a single answer and makes fleet-wide queries possible instead of host-by-host spelunking. Keeping ingest on the same LAN segment makes onboarding a new host friction-free, so machines actually get pointed at the collector. Self-hosting keeps the log data in-house, and the instance stays private (WireGuard, DNS-01 TLS) so centralising never means exposing a searchable index of the infrastructure.