Server Monitoring & Visibility

I'm currently only exploring these concepts using NixOS. Ideally, I think this should be more related to somwhere else in my braindump, but I can't find a better place.

Grafana

Query, visualize, alert on, and explore metrics, logs and traces wherever they are stored.

Prometheus

Collects and stores metrics via time series data. Used as data source by Grafana. The node exporter should be installed on every server which wants their metrics collected. Prometheus is only installed on an aggregation server.

Loki + promtrail

Collects and stores logs. Used as a datasource by Grafana. The promtrail should be installed on every server to collect it's data. Loki is only installed on an aggregation server.

References