High Spec Logging on Low Spec Hardware

Being a server admin with a ramen noodle budget my servers aren’t super fancy however, I still need logging to make sure I know what’s going on to fix things. This means that if I want logging I have to use things that are as minimal as possible. I’ve used journald logs for a while now and frankly I’m completely over reading non-essential[1] logs via ssh, especially when there’s huge dumps of data to sift through.

So first here are the server specs I have to work with:

2 Lightsail instances: 2 Cores w/ 4GB RAM (each)
3 VPS’s(IONOS): 1 core w/ 2GB RAM (each)

Between these I’ve got dev databases, websites, and game servers. The game servers are the ones I want to look at today since those produce logs when I’m not looking(i.e. when other people are playing/using commands on the chatbot).

What I use and why

In short: vector and honeycomb

I would normally go for something self-hosted and FOSS with an aggregate like honeycomb but honestly I just needed something that worked and honeycomb is exactly that. It comes with a web UI which is nice because I don’t always have access to a terminal nor do I want to ssh into burning servers anyway.

Why Vector over other options

It’s lightweight
Super easy to setup

Vector is statically compiled so its a single binary(around 40 MEG). Configuration consists of a single config file for multiple sinks(in my case).

Ready to send to basically any data sink I want.

No really look at how many sinks it supports: ref. I settled on honeycomb personally because its one less thing to host and because none of the logs I’m sending actually contain sensitive data so I’m not really worried about it.

It’s easy to fan out data

The ease of creating data sinks its trivial to take your data inputs and fan them out to basically anywhere to have lots of backups(not that I really need it buy heywhynot).

Where I run it

Vector lives right next to my chat-bot which people control to turn game servers on/off. I don’t run vector on the websites I run are behind Nginx which hasn’t failed me in a few months and the occasional restart seems to remedy all things. The Mysql database I have is literally for running tests so I’m basically staring at it when it does produce logs and they have entirely to do with the tests I’m running at the time.