Monitoring Docker Containers with Prometheus, Windows Exporter, Cadvisor and Grafana

Monitoring Docker Containers with Prometheus, Windows Exporter, Cadvisor and Grafana

·

7 min read

Introduction

In this blog post, we delve into the comprehensive monitoring of Docker containers and Windows host machines using Prometheus and Grafana. Through the utilization of cAdvisor, administrators gain insights into container metrics including CPU, memory, and network usage. The integration of Windows Exporter broadens monitoring capabilities to Windows servers, capturing vital system-level data such as CPU utilization and disk activity.

Prometheus acts as the central hub, aggregating metrics from both cAdvisor and Windows Exporter, facilitating in-depth analysis and timely alerting. Grafana enhances Prometheus by providing customizable dashboards for visualizing metrics, enabling real-time monitoring and effective troubleshooting. This cohesive solution optimizes resource utilization, proactively identifies issues, and simplifies management across Dockerized environments and Windows servers.

The Setup

In our monitoring setup, we leverage the power of Docker containers to run Grafana, cAdvisor, and Prometheus seamlessly across our environment. Docker provides an efficient and portable way to deploy these monitoring tools, ensuring consistency and scalability. Here in this setup, to enable containerization, we'll be using Docker Desktop for Windows.

Docker Desktop

In this setup, enabling Docker Desktop to expose metrics is essential. Enabling Docker Desktop to expose metrics through the Prometheus endpoint offers valuable insights into the performance and health of your Docker environment. By collecting and monitoring these metrics, you gain visibility into resource utilization, container health, and system behavior. To do this, follow the following steps:

  1. Open Docker Desktop Settings: Start by opening Docker Desktop on your machine.

  2. Access Settings: Click on the Docker Desktop icon in your system tray (Windows) or menu bar (macOS), then select "Settings."

  3. Enable Metrics: In the Settings window, navigate to the "Docker Engine" tab.

  4. Add Prometheus Configuration: In the "Docker Engine" tab, you can add the following configuration to enable the Prometheus endpoint:

{
  "builder": {
    "features": {
      "buildkit": true
    },
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": true, // Enable experimental features
  "metrics-addr": "0.0.0.0:9323" // Expose metrics on port 9323
}

That should look something like this. Finally, hit "Apply & Restart".

Running Grafana

To deploy Grafana locally as a Docker container, we pull the Grafana image from Docker Hub and run it using the following command:

docker run -d --name=grafana -p 3000:3000 grafana/grafana

This command pulls the Grafana image and starts the container, exposing port 3000 for accessing the Grafana web interface. You can access Grafana by navigating to http://localhost:3000 in your web browser.

Deploying Prometheus

Prometheus serves as the cornerstone of our monitoring infrastructure, aggregating and storing metrics from various endpoints. Prometheus is configured to scrape metrics from our Docker containers and Windows hosts, facilitating comprehensive monitoring across our entire infrastructure.

Prepare a directory to centralize your configuration files. Initially, establish a folder and within it, create a file named prometheus.yaml, which will serve as the configuration file for launching the Prometheus container. Populate your prometheus.yaml with the following settings:

# prometheus.yaml
# my global config
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "windows-exporter"
    static_configs:
      - targets: ["host.docker.internal:9182"]

  - job_name: 'docker-containers'
    static_configs:
      - targets: ["host.docker.internal:9323"]

  - job_name: 'cadvisor'
    static_configs:
      - targets: ["host.docker.internal:8080"]

Here you can notice the targets are set to ["host.docker.internal:<port>"] in some cases. In Docker Desktop, "host.docker.internal" resolves to the internal IP address of the host machine from within the Docker container. This allows Docker containers to access services running on the host machine, such as applications running on localhost or other services exposed by the host.

So, when you specify "host.docker.internal:<port>" as a target in your Prometheus configuration within a Docker container, it enables Prometheus to scrape metrics from services running on the host machine, such as the Windows exporter, Docker containers, or cAdvisor.

Open a terminal from your present directory and execute the command below to initiate Prometheus as a Docker container:

docker run --name prometheus -d -p 127.0.0.1:9090:9090 -v "$(pwd)/prometheus.yaml:/etc/prometheus/prometheus.yml" prom/prometheus

This command pulls the Prometheus image and starts the container, binding port 9090 locally for accessing the Prometheus web UI and API. You can access Prometheus by navigating to http://localhost:9090 in your web browser. Just to check if everything is running properly, you can head to http://localhost:9090 and run the up command and check if all the Prometheus jobs are running properly (Check for "1" corresponding to each job.)

Deploying Cadvisor

cAdvisor plays a crucial role in monitoring container resource utilization and performance metrics. We deploy cAdvisor as a Docker container with the following command:

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:$VERSION

This command pulls the cAdvisor image and starts the container with necessary volume mounts and privileges to collect detailed container metrics. cAdvisor exposes port 8080 for accessing its web interface. You can access cAdvisor by navigating to http://localhost:8080 in your web browser.

Setting Up Grafana

First, ensure that Grafana is running. You can access the Grafana dashboard by navigating to localhost:3000 in your web browser.

Adding Prometheus as a Data Source

To begin integrating Prometheus with Grafana, navigate to the "Connections" menu located in the left sidebar. Within the "Connections" menu, select the "Data Sources" option. This will present you with a list of available data sources. From this list, choose Prometheus.

Upon selecting Prometheus, you will be presented with a section labeled "HTTP." In this section, enter the URL of your Prometheus server. The default URL is http://localhost:9090, assuming Prometheus is running on the same machine as Grafana. However, if your Prometheus server resides on a different machine, you will need to provide the appropriate URL (if localhost doesn’t work, use IPv4 instead like “http://IPv4:9090”)

Once you have successfully added Prometheus as a data source, you can start creating dashboards to visualize your metrics. To do so, locate the "+" icon in the left sidebar and click on it to reveal the "Create" menu. From the "Create" menu, select "Dashboard" to initiate the dashboard creation process.

Adding Panels

After creating a new dashboard, you can start adding panels to display the metrics you want to monitor. To add a panel, simply click on the "Add Panel" button. For each panel, you will need to configure the metric query using PromQL (Prometheus Query Language). PromQL provides a powerful way to filter and aggregate metrics from your Prometheus server.

You can find the metrics that I have used for building the dashboard in my git repo - https://github.com/aftab-s/HomeLab-Observability.git

Customizing your Dashboard for Optimal Viewing

Once you have added panels to your dashboard, it's time to craft an aesthetically pleasing layout. Grafana's intuitive drag-and-drop interface allows you to arrange and resize panels to suit your preferences. Experiment with different arrangements to find a layout that effectively conveys your metrics and enhances visual clarity. To further customize your dashboard, click on the gear icon located at the top of the dashboard.

This will reveal a panel of settings where you can modify the dashboard's overall appearance. Adjust the title to accurately reflect the metrics being displayed. Additionally, you can customize the time range to focus on a specific period of time or set it to auto-update to display real-time data.

Saving the Dashboard

After meticulously crafting your dashboard, don't forget to save it for future reference. Click on the disk icon in the top menu, provide a descriptive name for your dashboard, and click "Save." This will ensure that your personalized dashboard layout and settings are preserved for future access.

Conclusion

In summary, the integration of Prometheus, Grafana, cAdvisor, and Windows Exporter provides a comprehensive solution for monitoring Docker containers and Windows servers. With cAdvisor and Windows Exporter capturing metrics and Prometheus enabling analysis and alerting, administrators gain valuable insights into system health and resource utilization. Grafana's customizable dashboards enhance visualization and troubleshooting efforts. Leveraging Docker Desktop's metric-exposing capabilities further simplifies monitoring tasks. By adopting this setup, users ensure efficient management and proactive issue resolution across their Dockerized environments and Windows servers, enhancing overall system reliability and performance.

Did you find this article valuable?

Support Aftab S by becoming a sponsor. Any amount is appreciated!