Smart Monitoring with Prometheus & Grafana

Grafana is an open-source analytics and visualization platform that connects to data sources, such as Prometheus, to create dashboards, alerts, and visualize real-time performance metrics.

Prometheus is an open-source monitoring and alerting toolkit that collects time-series metrics from targets like Node Exporter, stores them locally, supports PromQL for querying, and integrates with tools like Grafana for visualization

Node Exporter is a Prometheus exporter that exposes hardware and operating system metrics, including CPU usage, memory, disk and filesystem statistics, network I/O, and system load.

Prerequisite

Ensure that the Grafana server is up and running. If it's not, refer to this article

Now, set up Prometheus and Node Exporter on another node and visualize the data on the Grafana dashboard.

Create a Prometheus System User

Prometheus should run as a non-login system user:

sudo useradd --system --no-create-home --shell /bin/false Prometheus

Then create a directory

sudo mkdir prometheus

Download and install the Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz

Extract Prometheus files, move them, and create directories:

tar -xvf prometheus-2.47.1.linux-amd64.tar.gz
cd prometheus-2.47.1.linux-amd64/
sudo mkdir -p /data /etc/prometheus
sudo mv prometheus promtool /usr/local/bin/
sudo mv consoles/ console_libraries/ /etc/prometheus/
sudo mv prometheus.yml /etc/prometheus/prometheus.yml

Ensure Prometheus owns its files:

sudo chown -R prometheus:prometheus /etc/prometheus/ /data/

Create a systemd unit configuration file for Prometheus:

vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/data \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle

[Install]
WantedBy=multi-user.target

Reload systemd and restart the service:

sudo systemctl enable prometheus
sudo systemctl start Prometheus
sudo systemctl status Prometheus

You can access Prometheus in a web browser using your server’s IP and port 9090

http://your-server-ip:9090

Install Node Exporter

Create User

sudo useradd --system --no-create-home --shell /bin/false node_exporter

Create a directory

sudo mkdir node-exporter

Download and install the Node-Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

Extract Prometheus files, move them, and create directories:

tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/

Create a systemd unit configuration file for Node-exporter

vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter --collector.logind

[Install]
WantedBy=multi-user.target

Reload systemd and restart the service:

sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

You can access Node exporter in a web browser using your server’s IP and port 9100

http://your-server-ip:9100

Prometheus Configuration:

vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

systemctl restart prometheous

To install Grafana, follow this link.

Now open the Grafana dashboard

http://your-grafana-server-ip:3000

Add Prometheus Data Source:

To visualize metrics, you need to add a data source. Follow these steps:

Click on the Add new data source

Now select the Prometheus

Set the name of your server, enter your worker's IP address with the corresponding port, and then click Save and Test.

Now, go back to the dashboard section and add a new dashboard.

For the node exporter ID, navigate to the node exporter website

We are using the ID:- 1860

To import a dashboard, fill in the details such as the server name, select the dashboard folder, enter the UID, and choose Prometheus as the data source type. Click on the import