Monitoring Linux and Windows Server using Prometheus and Grafana

12 min readMay 31, 2021

Prometheus is a time series database, created in 2012 and part of the Cloud Native Computing Foundation, that exposes dozens of exporters for you to monitor anything.
On the other hand, Grafana is probably one of the most popular monitoring tools.
In Grafana, you create dashboards that bind to datasources (such as Prometheus) in order to visualize your metrics in near real-time.
Grafana & Prometheus natively bind together, so today we are going to see how you can setup Prometheus and Grafana on your Linux system.

A) Create a VM for installing Prometheus and Grafana:

Choose a VM with at least 2 vCPU and 4GB RAM for smooth functioning of the monitoring services initially. It can be increased as per requirement. In our case we will choose standard B2s having Ubuntu 18.04 LTS.

B) Installing Prometheus:

First, we are going to see how you can install the latest version of Prometheus and configure for your Linux server.

1) Download Prometheus :

First, head over to the Prometheus downloads page.

Make sure to filter for your operating system and your CPU architecture (in my case Linux and amd64)

Make sure to select the latest stable version, and not the “rc” one, as it is not considered stable enough for now.

Download the archive on your system by clicking on the archive, or by running a wget command if you are using the Terminal.

wget https://github.com/prometheus/prometheus/releases/download/v2.27.0/prometheus- 2.27.1.linux-amd64.tar.gz

You should now have the tar.gz file on your system.

Untar it to extract the files in the archive.

tar xvzf prometheus-2.27.1.linux-amd64.tar.gz

The archive contains many important files, but here is the main ones you need to know.

prometheus.yml: the configuration file for Prometheus. This is the file that you are going to modify in order to tweak your Prometheus server, for example to change the scraping interval or to configure custom alerts;

prometheus: the binary for your Prometheus server. This is the command that you are going to execute to launch a Prometheus instance on your Linux box;

promtool: this is a command that you can run to verify your Prometheus configuration.

We are not going to execute directly the Prometheus, instead we are going to configure it as a service.

It will bring more robustness and reliability in case our Prometheus server were to stop suddenly.

2) Configuring Prometheus as a service:

Note: Make sure you have port 9090 open in security group for Prometheus on instance it is installed.

First of all, for security purposes, you are going to create a Prometheus user with a Prometheus group.

$ sudo useradd -rs /bin/false prometheus

Make sure to move the binaries to your local bin directory.

I stored my binaries in a Prometheus folder, located on my home directory. Here’s the command to move them to the bin directory.

$ cd prometheus/prometheus-2.27.0.linux-amd64/
$ sudo cp prometheus promtool /usr/local/bin

Give permissions to the Prometheus user for the prometheus binary.

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

Create a folder in the /etc folder for Prometheus and move the console files, console libraries and the prometheus configuration file to this newly created folder.

$ sudo mkdir /etc/prometheus
$ sudo cp -R consoles/ console_libraries/ prometheus.yml /etc/prometheus

Create a folder in /var/lib folder for the data storage of Prometheus.

$ sudo mkdir -p var/lib/prometheus
$ sudo chown -R prometheus:prometheus /var/lib/prometheus

You are now set to create a Prometheus service.

Head over to the /lib/systemd/system folder and create a new file named prometheus.service

$ cd /lib/systemd/system
$ sudo vi prometheus.service

Press I and paste the following content inside.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
— config.file=/etc/prometheus/prometheus.yml \
— storage.tsdb.path=/var/lib/prometheus \
— web.console.templates=/etc/prometheus/consoles \
— web.console.libraries=/etc/prometheus/console_libraries \
— web.listen-address=0.0.0.0:9090
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target

Save your file, enable your service at startup, and start your service.

$ sudo systemctl enable prometheus
$ sudo systemctl daemon-reload
$ sudo systemctl start prometheus

Now that we have our Prometheus server running, let’s connect to the Web UI to make sure that everything is okay.

Unless you modified it in the service file, your Prometheus server should be running at http://localhost:9090.

By default, this is what you should see on your screen.

By default, Prometheus should start monitoring itself.

Click on Status, then Targets in top bar menu, and verify that you have one target: the Prometheus server itself.

C) Installing Grafana:

In this first step, you will install Grafana onto your Ubuntu 18.04 server. You can install Grafana either by downloading it directly from its official website or by going through an APT repository. Because an APT repository makes it easier to install and manage Grafana’s updates, you’ll use that method in this tutorial.

Although Grafana is available in the official Ubuntu 18.04 packages repository, the version of Grafana there may not be the latest, so use Grafana’s official repository.

1) Download Grafana:

Download the Grafana GPG key with wget, then pipe the output to apt-key. This will add the key to your APT installation’s list of trusted keys, which will allow you to download and verify the GPG- signed Grafana package.

$ wget -q -O — https://packages.grafana.com/gpg.key | sudo apt-key add -

In this command, the option -q turns off the status update message for wget, and -O outputs the file that you downloaded to the terminal. These two options ensure that only the contents of the downloaded file are pipelined to apt-key.

Next, add the Grafana repository to your APT sources:

$ sudo add-apt-repository “deb https://packages.grafana.com/oss/deb stable main”

Refresh your APT cache to update your package lists:

$ sudo apt update

You can now proceed with the installation:

$ sudo apt install grafana

Once Grafana is installed, use systemctl to start the Grafana server:

$ sudo systemctl start grafana-server

Next, verify that Grafana is running by checking the service’s status:

$ sudo systemctl status grafana-server

Lastly, enable the service to automatically start Grafana on boot:

$ sudo systemctl enable grafana-server

2) Updating Credentials:

Because every Grafana installation uses the same administrative credentials by default, it is best practice to change your login information as soon as possible. In this step, you’ll update the credentials to improve security.

Start by navigating to http://your_ip:3000 from your web browser. This will bring up the default login screen where you’ll see the Grafana logo, a form asking you to enter an email or username and password, a Log in button, and a Forgot your password? link.

Enter admin into both the User and Password fields and then click on the Log in button.

Enter the password you’d like to start using into the New password and Confirm new password fields.

From here, you can click Save to save the new information or press Skip to skip this step. If you skip, you will be prompted to change the password next time you login.

In order to increase the security of your Grafana setup, click Save. You’ll return to the Home Dashboard page:

3) Configure Prometheus as a Grafana datasource:

First, head to the datasources panel by clicking on Configuration > Data sources via the left menu.

Click on “Add a datasource”

Select a Prometheus data source on the next window.

Here is the configuration for Prometheus. Make sure to skip the TLS verification as you are using a self-signed certificate.

Click on “Save and Test” at the bottom of your configuration window, and make sure that your data source is working properly.

D) Installing the Node Exporter to monitor Linux metrics:

Now that your Prometheus data source is working, it is time to install your exporter.

As a reminder, exporters are standalone entities that regularly aggregate metrics for a wide variety of targets: operating systems, databases, websites.

The Node Exporter is an exporter that gathers metrics about your Linux system : the CPU usage, the memory usage as well as various statistics on your filesystems.

Note: Make sure you have port 9100 open in security group for node exporter on instance it is installed.

1) Downloading the Node Exporter

First of all, we are going to download the Node exporter on our system.

Head over to https://prometheus.io/download/ and select Linux operating systems for amd64 CPU architectures.

Scroll down, and find the section dedicated to the node exporter.

Simply click on it, or copy the link and run a wget command.

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter- 1.1.2.linux-amd64.tar.gz

Extract the archive to access the files inside it.

$ tar xvzf node_exporter-1.1.2.linux-amd64.tar.gz

Inside your newly created directory, you should now see the node_exporter binary ready for use.

Again, move this executable to your /usr/local/bin folder.

$ cd node_exporter-1.1.2.linux-amd64.tar.gz
$ sudo cp node_exporter /usr/local/bin

Create a new user for the node exporter, and change the permissions for the node exporter binary.

$ sudo useradd -rs /bin/false node_exporter
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

2) Create a Node Exporter service

Again, we are going to run the node exporter as a service.

Go to the /lib/systemd/system folder and create a new file named node_exporter.service.

$ cd /lib/systemd/system
$ sudo vi node_exporter.service

Paste the following content in your node exporter service.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
— collector.mountstats \
— collector.logind \
— collector.processes \
— collector.ntp \
— collector.systemd \
— collector.tcpstat \
— collector.wifi
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target

By default, a lot of modules (cpu, mem, disk) are already enabled by default, so we only need to enable a few more.

When you are done, enable your service, and start it.

$ sudo systemctl daemon-reload
$ sudo systemctl enable node_exporter
$ sudo systemctl start node_exporter

(Make sure that everything is running smoothly)

$ sudo journalctl -f -u node_exporter.service

3) Configure the Node Exporter as a Prometheus target

Now that the node exporter is up and running, we need to tell Prometheus to scrape it periodically.

To do that, go to your Prometheus configuration file located at /etc/prometheus/prometheus.yml.

$ cd /etc/prometheus
$ sudo vi prometheus.yml

In the static_configs part of your configuration file, add a new entry for the node exporter.

static_configs:
- targets: [‘localhost:9090’, ‘localhost:9100’]

Restart Prometheus for your changes to be applied.

$ sudo systemctl restart prometheus

Head over to https://localhost:9100/targets and make sure that your target is correctly scraped.

4) Building a Grafana dashboard to monitor Linux metrics

The last step will be to import a Grafana dashboard that reflects the metrics that we are gathering with Prometheus.

The “Linux Exporter Node” dashboard seems good fit. Use 14513 to import from Grafana.com

a) Importing dashboard into Grafana

We don’t have to design the entire dashboard by ourselves, we can import dashboard right into our Grafana server.

To do so, click on “Import” by hovering the “Plus” icon, and clicking on “Import.

On the next window, simply enter the dashboard ID in the corresponding field (14513 in our case)

The dashboard should be automatically detected by Grafana.

On the next window, select your Prometheus data source, and click on Import.

You are done! Your dashboard is now up and running.

E) Installing the Windows Exporter to monitor Windows metrics:

1) Windows Server Monitoring Architecture

For Windows hosts, you are going to use the Windows exporter.

The Windows exporter will run as a Windows service and it will be responsible for gathering metrics about your system.

In short, here is the final architecture that you are going to build.

2) Installing the Windows Exporter

The Windows exporter is an awesome exporter for Windows Servers.

It will export metrics such as the CPU usage, the memory and the disk I/O usage.

The windows exporter can also be used to monitor IIS sites and applications, the network interfaces, and the services.

In order to install the windows exporter, head over to the Windows exporter releases page on GitHub.

a) Downloading the Windows Exporter MSI

As of May 2021, the latest version of the windows exporter is v0.16.0.

On the releases page, download the MSI file corresponding to your CPU architecture. In my case, I am going to download the windows_exporter-0.16.0-amd64.msi file.

b) Running the Windows exporter installer:

Note: Make sure you have port 9182 open in security group for windows exporter on which it is installed.

When the download is done, simply click on the MSI file and start running the installer.

Windows should now start configuring your windows exporter.

You should be prompted with a firewall exception. Make sure to accept it for the windows exporter to run properly.

The MSI installation should exit without any confirmation box. However, the Windows exporter should now run as a Windows service on your host.

To verify it, head over to the Services panel of Windows (by typing Services in the Windows search menu).

In the Services panel, search for the “Windows exporter” entry in the list. Make sure that your service is running properly.

c) Observing Windows Server metrics

Now that your exporter is running, it should start exposing metrics on http://localhost:9182/metrics Open your web browser and navigate to the Windows exporter URL. This is what you should see in your web browser.

Great!

Windows Server monitoring is now active using the windows exporter. If you remember correctly, Prometheus scrapes targets

As a consequence, we have to configure our Windows Server as a Prometheus target.

This is done in Prometheus configuration file.

d) Binding Prometheus to the Windows exporter

Head over to your configuration file (mine is located at /etc/prometheus/prometheus.yml) and edit the following changes to your file.

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: ‘prometheus’

# Careful, the scrape timeout has to be lower than the scrape interval.

scrape_interval: 6s
scrape_timeout: 5s
static_configs:
- targets: [‘localhost:9090’, ‘localhost:9182’]

Save your file, and restart your Prometheus service.

$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus

Head back to the Prometheus UI, and select the “Targets” tab to make sure that Prometheus is correctly connected to the Windows exporter.

If you are getting the following error, “context deadline exceeded”, make sure that the scrape timeout is set in your configuration file.

Great! Our Windows Server monitoring is almost ready.

e) Building an Awesome Grafana Dashboard