2018-09-15 - Setting up Prometheus (in progress)
I will be following the course:
https://app.pluralsight.com/player?course=monitoring-containerized-app-health-docker
and creating a "Workspace" Container: https://bitbucket.org/geircode/setting_up_prometheus
This figure shows that we are collecting metrics not only from the containers, but also from the Docker Hosts.
The Prometheus server must be running in the same network as the Containers are in, and it uses DNS to resolve the IP.
I created a docker-compose file to setup the container instead of running it from "docker run". A docker-compose file is more declarative and effective way to start and manage containers.
version: '3.5'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus-1
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
networks:
- setting_up_prometheus-network
networks:
setting_up_prometheus-network:
external: true
Start the container with:
docker-compose -f docker-compose.prometheus.yml up -d docker exec -it prometheus-1 /bin/sh #start an interactive terminal into the container
The volume from the "docker-compose" file makes it possible to edit the prometheus config on the fly:
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
But I don't know yet if Prometheus can reload on change, but I guess I will find that out shortly.
According to this article, it can be configure to reload on change: https://www.robustperception.io/reloading-prometheus-configuration
Right now the network looks like this:
Meaning, that from inside the workspace container, I can reach the Prometheus container from the DNS name and run this command to reload configuration:
curl -X POST http://prometheus:9090/-/reload
The DNS name "prometheus" comes from service definition in the docker-compose" file:
But when I tried to "reload" I got this error:
root@19d52739a0f2:/app# curl -X POST http://prometheus:9090/-/reload Lifecycle APIs are not enabled root@19d52739a0f2:/app#
"Lifecycle APIs are not enabled"? Ok, how do I enable them?
Hmm, do enable this I need to add:
--web.enable-lifecycle
to something somewhere. It seems that Prometheus is based only on using "Command-Line Flags" on startup. Why not just use Environment variables like everyone else? This should have been solved by using Environment variables, which is a more Container friendly way of doing stuff. Because of this, I had to create my own Dockerfile overriding the default settings from Prometheus Container Image, and build a new Container Image based on my new configurations.
The Prometheus docs explains why they turned it off by default:
Yes, I understand, but live reloading is very nice when figuring stuff out. Right, moving on.
Next challenge was to insert my own "prometheus.yml" configuration into the container in order for it to reload on this one. But first I had to move the config file to another folder inside "/etc/prometheus/config" in order for the volume to work:
volumes:
- ./config:/etc/prometheus/config
and update the "–config.file" parameter to point to this one:
FROM prom/prometheus
CMD [ "--config.file=/etc/prometheus/config/prometheus.yml", \
"--storage.tsdb.path=/prometheus", \
"--web.console.libraries=/usr/share/prometheus/console_libraries", \
"--web.enable-lifecycle", \
"--web.console.templates=/usr/share/prometheus/consoles" ]
Here I have changed the default value of "config.file" and added "web.enable-lifecycle".
Ok, now I have a working Prometheus container with the possibility to reload the "prometheus.yml" configuration on the fly. Next step is to actually collect some metrics from a Container.