Setting up Zabbix monitoring

Zabbix monitoring
This document will help you install Zabbix on a Debian server. On my end, I did this on an LXC container. The agents are connecting over a secure connection with certificates. We also have a look at monitoring a remote VPS server over a Tinc VPN connection. You can skip that part if you are trying to get the monitoring going without to much fuss. There is also an example ansible playbook to install the Zabbix agent and the certificates needed for to communicate securily.

Zabbix server

Install Zabbix server on a machine. I installed it in a LXC container. Choose the version with the database backend support you like.:

apt install zabbix-server-pgsql

apt install postgresql

Create a new database and access user (let's call both "zabbix"). By default on Debian systems you need to do this as the "postgres" user, and create a database and database user. By default the options --no-superuser, --no-createrole and --no-createdb are set so there is only need to specify --pwprompt. If given, createuser will issue a prompt for the password of the new user.

su - postgres

createdb zabbix
createuser -P zabbix

You will need to enter a password for the new postgres user account. Then send the necessary SQL files to the database to create the required database schema:

zcat /usr/share/zabbix-server-pgsql/{schema,images,data}.sql.gz \
   | psql -h localhost zabbix zabbix

exit

More info on the Zabbix database scripts Next we need to install a front end. We choose the php frint end.:

apt install zabbix-frontend-php

This installs apache. You could also use nginx which is my favourite but to get this tutorial going, I'll go with Apache. Apache2 may need package "libapache2-mod-php" for PHP support:

sudo apt-get install libapache2-mod-php

An URL alias will make Zabbix frontend available from:

https://yourserver/zabbix

To enable the zabbix frontend:

a2enconf zabbix-frontend-php

Or with:

ln -s /usr/share/doc/zabbix-frontend-php/examples/apache.conf /etc/apache2/conf-available/zabbix.conf
a2enconf zabbix

Reload the Apache configuration:

sudo service apache2 reload

or restarting Apache:

sudo apache2ctl restart

Then navigate to https://server/zabbix/ In my case, I'm installing zabbix on a lxc container without a gui. In order to locally access the site, I use SSH and forward a local port to the http port so I can manage the site from my local machine.:

ssh -L 8080:localhost:80 <zabbix_server>

Then open a browser and navigate to https://localhost:8080/zabbix for the configuration:

Starting Zabbix configuration

Check of pre-requisites

The default username is "Admin" with password "zabbix". To prevent other users from viewing your data, change this password immediately after logging in. If you are using PostgreSQL as a database backend please make sure that the 'php-pgsql' package is installed.

Zabbix also requires minimum resource settings for PHP to work properly. During the first-time setup it will complain if these are not set. When starting the configuration, I got this error:

Error

PHP option "date.timezone" unknown Fail

Go to https://php.net/date.timezone and select your timezone. Set these parameters in your "/etc/php/7.0/apache2/php.ini" file for Apache2 or in "/etc/php/7.0/fpm/php.ini" for nginx/php-fpm.:

post_max_size = 16M
max_execution_time = 300
max_input_time = 300
always_populate_raw_post_data = -1
date.timezone = ...

Reload apache:

service apache2 reload

Configure DB connection

Enter details of the database created earlier on.

Zabbix server details

After going throught the next configuration steps, the config couldn't be saved:

Cannot create the configuration file.

The webinterface allows you to download the file and put it where it belongs. No, not where the fa fa-sun fa-2x yellow doesn't shine :). Copy the config file to /etc/zabbix:

/etc/zabbix/zabbix.conf.php

If you revisit the webinterface, you are greated by the login page. Login with the above Admin user and change the password. Also, if you see on the overview page, the zabbix-server isn't running, start it:

service zabbix-server start

Zabbix Agent

Since we also want to monitor the monitoring host, we also install the Zabbix Agent.:

apt-get install zabbix-agent

Change the folliwing settings of the Zabbix agent config:

vi /etc/zabbix/zabbix_agentd.conf
...
Server=<your_zabbix_server_ip_address>
ServerActive=<your_zabbix_server_ip_address>
ListenIP=<ip for the agent to listen on>
Hostname=<client_hostname>

Hostname should be the same as specified in the host information on the Zabbix server. If you don't, you will see the following message:

148:20180206:160008.872 no active checks on server [<ip>:10051]: host [<hostname>] not found

Check the config:

grep ^[^#] /etc/zabbix/zabbix_agentd.conf

If you are working with a lot of systems that need to be added to the monitoring system, ansible or a similar software is a great tool to automate this.

Anyway, if you are manually doing the install over ssh, you can easy the installation by using bigger one liners to update, install and configure the agent:

apt update; apt -y upgrade; apt -y install zabbix-agent

sed -i 's/Server=127.0.0.1/Server=<zabbix_serverip>/g' /etc/zabbix/zabbix_agentd.conf;sed -i 's/ServerActive=127.0.0.1/ServerActive=<zabbix_serverip>/g' /etc/zabbix/zabbix_agentd.conf;sed -i 's/Hostname=Zabbix server/Hostname=<myhostname>/g' /etc/zabbix/zabbix_agentd.conf

service zabbix-agent restart

Next restart the client:

systemctl restart zabbix-agent
or
service zabbix-agent restart
or
/etc/init.d/zabbix-agent restart

Check if the agent is running:

netstat -tulp
...
tcp        0      0 0.0.0.0:zabbix-agent    0.0.0.0:*               LISTEN      30441/zabbix_agentd
tcp6       0      0 [::]:zabbix-agent       [::]:*                  LISTEN      30441/zabbix_agentd

Check the status of the agent:

systemctl status zabbix-agent
or
service zabbix-agent status

You will need to do this for all the machines you want to add to the monitoring system.

Adding adding the Zabbix server to Zabbix monitoring

Adding hosts

To add hosts, you need to install the zabbix agent on it, and edit the /etc/zabbix/zabbix_agentd file. Log in to the Zabbix Server web interface by navigating to the address https://your_zabbix_server_ip_address/zabbix/. Go to Configuration > Hosts > on the right choose "Create host".

Adding hosts to Zabbix

When adding hosts, you should give at least:

  • a hostname: use the exact hostname as you will use in the agent host configuration
  • visible name
  • a group where the machine will be in (or create a new group)
  • an ip address (Agent interfaces)
  • 1 or more templates (for instance Template OS Linux and Template ICMP Ping). Template OS Linux already is linked to the Template Zabbix Agent.

Adding a remote server with Tinc

Note

This part is only usefull if you want to add a remote host to the monitoring. If that's not the case for you, skip to the next part.

Warning

If you want to monitor a remote machine, first setup your monitoring before starting to monitor a machine over a Tinc VPN. Even then, it pays off to first try a simple Tinc VPN connection as described here before doing the adjustments below.

If you have a remote server that's not part of your network, you won't be able to monitor it or at least not without some additional steps. For this to work, we will use Tinc VPN. See this article on Tinc on how to setup Tinc.

After we've connected the network of the monitoring system to the network of the remote server, we can start monitoring. When connecting 2 networks of hosts, it's wise to plan ahead. In this case there might be some other containers you want to connect to the Tinc VPN network. This is made easier by using bridges on both ends. By doing so, you can have a LXC container be part of a management network and a Tinc VPN network, greatly increasing the usability of your solution.

On both servers we add a network where the Tinc VPN is connected to. As an example, on server 1 our Tinc VPN network 10.0.0.0/24, on server 2 10.0.1.0/24. To create a bridge:

server 1:

brctl addbr br0
ifconfig br0 10.0.0.254 netmask 0.0.0.0
brctl stp br0 on

server 2:

brctl addbr br0
ifconfig br0 10.0.1.254 netmask 0.0.0.0
brctl stp br0 on

Note

An interface that is a member of a bridge can't have an IP address. You will need to assign an IP address to the bridge instead.

Since we want this change to be persistent, and the bridge creating is needed when the Tinc VPN is up and running, we will add a tinc-up script to do the heavy lifting for us, and create the bridge and add the network interfaces to the bridge.

On server 1, where the Zabbix server LXC container is running, we will add an network interface to the container in the Tinc VPN range:

vi /var/lib/lxc/<zabbix server>/config
...
lxc.network.type = veth
lxc.network.name = eth2
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = 00:AA:BB:FF:10:51
lxc.network.ipv4 = 10.0.0.252/24

Reboot the system. After the Tinc VPN with the correct tinc_up script is running, the br0 bridge will also be up and running and eth2 of this zabbix server container will be added to the Tinc VPN network.

Tinc setup on server 1

On server 1, make sure the Tinc VPN is working (see all needed config files below), as well as the bridge and LXC container interfaces if needed.

bridge

br0 has both the Tinc VPN connection as well as 1 interface of the Zabbix server container:

bridge name bridge id               STP enabled     interfaces
br0             8000.2ad346ef9337   yes     <tinc-vpn>
                                        vethMFAKYY
virbr0              8000.fe27c9b1973d       yes     veth0IRBX2
                                        veth3FIDLL
                                        ...

route

Relevant routes:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         <gateway>       0.0.0.0         UG    0      0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 br0
10.0.1.0        10.0.1.254      255.255.255.0   UG    0      0        0 br0
10.0.1.254      *               255.255.255.255 UH    0      0        0 br0

To accomplish this, I added a route to the bridge on the other remote server 10.0.1.254. Then I added a net via this gateway. And finally I deleted a wrong route. This is what I had to do on my end, you situation might be different so don't do this unless it's necessary:

route add -host 10.0.1.254 dev br0
route add -net 10.0.1.0 netmask 255.255.255.0 gw 10.0.1.254
route del -net 10.0.0.0 netmask 255.255.255.0 br0

tinc.conf

Name = server1
AddressFamily = ipv4
Mode = switch
Device = /dev/net/tun

tinc-up

This is a script meant to setup the bridging and routing for the Tinc VPN, the bridges and the LXC container interfaces. The script can be finetuned more and I added a lot of echo statements in there as I usually run it from the command line and the echo's are helpful. Feel free to change it how you want it to run. You will need to adjust the settings below:

#!/bin/bash
ifconfig $INTERFACE 0.0.0.0

# Edit this variable: the 'other' host.
REMOTE_IP=10.0.1.251

# Edit this variable: the bridge address on 'this' host.
BRIDGE_ADDRESS=10.0.0.254/24

# Edit this variable: the bridge address on the 'other' host.
OTHER_BRIDGE_ADDRESS=10.0.1.254

# Name of the bridge (should match /etc/default/docker and /etc/tinc/).
BRIDGE_NAME=br0

# LXC subnet of the other host
LXC_SUBNET=10.0.0.0
OTHER_LXC_SUBNET=10.0.1.0

# LXC container interfaces to be added to the bridge
LXC_INTERFACES=( 'vethMFAKYY'  )

# First remove LXC interfaces from bridge
echo "Removing LXC interfaces from $BRIDGE_NAME"
for net in ${LXC_INTERFACES[@]}
do
  echo "  brctl delif $BRIDGE_NAME $net"
  brctl delif $BRIDGE_NAME $net
  sleep 1
done

# Deactivate the bridge
echo "Deactivating bridge"
ip link set $BRIDGE_NAME down

# Remove the bridge
brctl delbr $BRIDGE_NAME

# Add the bridge
echo "Creating bridge"
brctl addbr $BRIDGE_NAME
brctl stp $BRIDGE_NAME on

# Set up the IP for the bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME

# Activate the bridge
ip link set $BRIDGE_NAME up

# Show bridge
echo "brctl show $BRIDGE_NAME"
brctl show $BRIDGE_NAME

# Make the other bridge routable.
echo "- Add route to $OTHER_BRIDGE_ADDRESS"
route add -host $OTHER_BRIDGE_ADDRESS dev $BRIDGE_NAME

# Add routes to the LXC subnets
route add -net $LXC_SUBNET netmask 255.255.255.0 dev $BRIDGE_NAME
echo "- Add route to $LXC_SUBNET"
route add -net $OTHER_LXC_SUBNET netmask 255.255.255.0 dev $BRIDGE_NAME
echo "- Add route to $OTHER_LXC_SUBNET"

# Might need to restart the lxc container.
# Preferably scripts is running before starting the containers.
# Try to add the container veth interfaces to the newly setup bridge
echo "Adding LXC interfaces to the bridge $BRIDGE_NAME"
for net in ${LXC_INTERFACES[@]}
do
  echo "  brctl addif $BRIDGE_NAME $net"
  brctl addif $BRIDGE_NAME $net
done

# Add Tinc-VPN to bridge
# If the interface is already mentioned in LXC_INTERFACES
# you can comment the following line.
brctl addif br0 $INTERFACE
ifconfig $INTERFACE up

# Meant for testing, comment this line if the script runs ok
echo "tinc-up script finished" > /root/tincstartup.log

tinc-down

#!/bin/bash
ifconfig $INTERFACE down
brctl delif br0 $INTERFACE

To enable the Tinc VPN to be started automatically:

vi /etc/tinc/nets.boot
...
<tinc-vpn>

Enable tinc in systemd:

systemctl enable tinc
systemctl enable tinc@<tinc-vpn>

Tinc setup on server 2

On server 2, the setup is basically the same. We still need to setup up another interface to the server. On Debian this is done via /etc/network/interfaces where we define a virtual interface on the existing interface:

vi /etc/network/interfaces
...
auto ens18
allow-hotplug ens18
iface ens18 inet static
        address <external_ip>
        netmask 255.255.255.0
        network <network_ip>
        broadcast <broadcast_ip>
        gateway <gateway_ip>
    up   ip addr add 10.0.1.251/24 dev $IFACE label $IFACE:0
    down ip addr del 10.0.1.251/24 dev $IFACE label $IFACE:0

Restart networking:

/etc/init.d/networking restart

Check if the device is ready:

ens18:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.1.251  netmask 255.255.255.0  broadcast 10.0.1.255

This IP should be reachable from the monitoring container. Try to ping it. Next, edit the zabbix agent configuration file:

...
ListenIP=10.0.1.251

If the agent still doesn't register in the zabbix, have a look at the zabbix agent log file:

tail /var/log/zabbix/zabbix_agentd.log

...failed to accept an incoming connection: connection from "10.0.0.254" rejected, allowed hosts: "10.0.0.252"

I had to allow connections from the bridge ip on server 1.:

...
Server=10.0.0.252, 10.0.0.254
ListenIP=10.0.1.251
ServerActive=10.0.0.252, 10.0.0.254

And restart the zabbix agent:

systemctl restart zabbix-agent

Now check the zabbix server and you should see the agent can now connect.

To enable the Tinc VPN to be started automatically:

vi /etc/tinc/nets.boot
...
<tinc-vpn>

Enable tinc in systemd:

systemctl enable tinc
systemctl enable tinc@<tinc-vpn>

bridge

br0 has both the Tinc VPN connection as well as 1 interface of the Zabbix server container:

bridge name bridge id               STP enabled     interfaces
br0         8000.ea8199c6d2b6       yes         <tinc-vpn>

route

Relevant routes:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         <external ip>   0.0.0.0         UG    0      0        0 ens18
10.0.0.0        10.0.0.254      255.255.255.0   UG    0      0        0 br0
10.0.0.254      0.0.0.0         255.255.255.255 UH    0      0        0 br0
10.0.1.0        0.0.0.0         255.255.255.0   U     0      0        0 br0

I had to do these route adjustments:

route add -host 10.0.0.254 dev br0
route add -net 10.0.0.0 netmask 255.255.255.0 gw 10.0.0.254
route del -net 10.0.1.0 netmask 255.255.255.0 br0

tinc.conf

Name = server2
AddressFamily = ipv4
Mode = switch
Device = /dev/net/tun
ConnectTo = server1

tinc-up script

You will need to adjust the settings below:

#!/bin/bash

ifconfig $INTERFACE 0.0.0.0

# Edit this variable: the 'other' host.
# Doesn't matter if the IP isn't yet available
REMOTE_IP=10.0.0.250

# Edit this variable: the bridge address on 'this' host.
BRIDGE_ADDRESS=10.0.1.254/24

# Edit this variable: the bridge address on the 'other' host.
OTHER_BRIDGE_ADDRESS=10.0.0.254

# Name of the bridge.
BRIDGE_NAME=br0

# LXC subnet of the other host
LXC_SUBNET=10.0.0.0
OTHER_LXC_SUBNET=10.0.1.0

# LXC container interfaces to be added to the Tinc bridge
LXC_INTERFACES=( '<tinc-vpn>'  )

# First remove LXC interfaces from bridge
echo "Removing LXC interfaces from $BRIDGE_NAME"
for net in ${LXC_INTERFACES[@]}
do
  echo "  brctl delif $BRIDGE_NAME $net"
  brctl delif $BRIDGE_NAME $net
  sleep 1
done

# Deactivate the bridge
echo "Deactivating bridge"
ip link set $BRIDGE_NAME down

# Remove the bridge
brctl delbr $BRIDGE_NAME

# Add the bridge
echo "Creating bridge"
brctl addbr $BRIDGE_NAME
brctl stp $BRIDGE_NAME on

# Set up the IP for the bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME

# Activate the bridge
ip link set $BRIDGE_NAME up

# Show bridge
echo "brctl show $BRIDGE_NAME"
brctl show $BRIDGE_NAME

# Make the other bridge routable.
echo "- Add route to $OTHER_BRIDGE_ADDRESS"
route add -net $OTHER_BRIDGE_ADDRESS netmask 255.255.255.0 dev $BRIDGE_NAME

# Add routes to the LXC subnets
route add -net $LXC_SUBNET netmask 255.255.255.0 dev $BRIDGE_NAME
echo "- Add route to $LXC_SUBNET"
route add -net $OTHER_LXC_SUBNET netmask 255.255.255.0 dev $BRIDGE_NAME
echo "- Add route to $OTHER_LXC_SUBNET"

# Might need to restart the lxc container.
# Preferably scripts is running before starting the containers.
# Try to add the container veth interfaces to the newly setup bridge
echo "Adding LXC interfaces to the bridge $BRIDGE_NAME"
for net in ${LXC_INTERFACES[@]}
do
  echo "  brctl addif $BRIDGE_NAME $net"
  brctl addif $BRIDGE_NAME $net
done

# Add Tinc-VPN to bridge
# If the interface is already mentioned in LXC_INTERFACES
# you can comment the following line.
brctl addif br0 $INTERFACE
ifconfig $INTERFACE up

# Meant for testing, comment this line if the script runs ok
echo "tinc-up script finished" > /root/tincstartup.log

tinc-down

#!/bin/bash
ifconfig $INTERFACE down
brctl delif br0 $INTERFACE

route LXC container

Relevant routes on the LXC containers where I installed Zabbix server:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         <lxc net gw>    0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth2
<lxc net>       0.0.0.0         255.255.255.0   U     0      0        0 eth0

The 2nd interface is the one we added earlier and that's connected to the br0 bridge. The Tinc VPN is also connected to this bridge.:

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.252  netmask 255.255.255.0  broadcast 10.0.0.255

In order to get everything working, I need to adjust my iptables firewall to allow forwarding between the brO bridges. Also make sure you allow port 655 for Tinc VPN.

It might be necessary to allow forwarding:

echo 1 > /proc/sys/net/ipv4/ip_forward

Secure the agent connection

We will generate certificates for the Zabbix server and agent and require encrypted connections on the Zabbix agent side for passive items. To do this, we will setup our own Certificate Authority (CA). As root, on your local or management machine:

$ mkdir zabbix_ca
$ chmod 700 zabbix_ca
$ cd zabbix_ca

Generate the root CA key

Generate the root CA key:

openssl genrsa -aes256 -out zabbix_ca.key 4096

When prompted, enter a password twice to protect the key. Generate and self-sign the root certificate like this:

openssl req -x509 -new -key zabbix_ca.key -sha256 -days 3560 -out zabbix_ca.crt

Generate a Zabbix server key

We'll generate a server key and a Certificate Signing Request (CSR).:

openssl genrsa -out zabbix_server.key 2048
openssl req -new -key zabbix_server.key -out zabbix_server.csr

No need for a password here. Next we need to sign the key as CA. That's why we also needed the csr.:

openssl x509 -req -in zabbix_server.csr -CA zabbix_ca.crt -CAkey zabbix_ca.key -CAcreateserial -out zabbix_server.crt -days 10000 -sha256

You will be prompted to enter the CA password.

Generate a Zabbix agent key

Generate certificate request:

openssl genrsa -out zabbix_agent.key 2048
openssl req -new -key zabbix_agent.key -out zabbix_agent.csr

No need for a password here. Next we need to sign the key as CA:

openssl x509 -req -in zabbix_agent.csr -CA zabbix_ca.crt -CAkey zabbix_ca.key -CAcreateserial -out zabbix_agent.crt -days 10000 -sha256

Now we have a key for the root CA, a key for the Zabbix agents, and a key for the Zabbix server. The agent and server keys are distributed respectively to the server and the client.

Setting up the certificates for the Zabbix agent

For the systems where you have an agent:

mkdir /etc/zabbix/zabbix_agent_certs

Copy the zabbix agent certificate to this directory. In this example our CA system is also where we do our first test.:

cd /root/zabbix_ca
cp zabbix_ca.crt /etc/zabbix/zabbix_agent_certs/
cp zabbix_agent.crt /etc/zabbix/zabbix_agent_certs/
cp zabbix_agent.key /etc/zabbix/zabbix_agent_certs/

chown zabbix zabbix_agent_certs
chmod 500 zabbix_agent_certs

Edit the zabbix agent config:

vi /etc/zabbix/zabbix_agentd.conf
....
TLSAccept=cert, unencrypted
TLSConnect=unencrypted
TLSCAFile=/etc/zabbix/zabbix_agent_certs/zabbix_ca.crt
TLSCertFile=/etc/zabbix/zabbix_agent_certs/zabbix_agent.crt
TLSKeyFile=/etc/zabbix/zabbix_agent_certs/zabbix_agent.key

Restart the agent:

service zabbix-agent restart

Check the config:

[root@buri]$ tail /var/log/zabbix-agent/zabbix_agentd.log
20843:20180201:173714.745 agent #1 started [collector]
20845:20180201:173714.746 agent #2 started [listener #1]
20846:20180201:173714.746 agent #3 started [listener #2]
20847:20180201:173714.746 agent #4 started [listener #3]
20848:20180201:173714.746 agent #5 started [active checks #1]
20850:20180201:173714.747 agent #6 started [active checks #2]
20848:20180201:173714.756 no active checks on server [127.0.0.1:10051]: host [Zabbix server] not found
20850:20180201:173714.759 no active checks on server [192.168.122.50:10051]: host [Zabbix server] not found
20845:20180201:173715.196 failed to accept an incoming connection: from 127.0.0.1: unencrypted connections are not allowed
20846:20180201:173730.508 failed to accept an incoming connection: from 127.0.0.1: unencrypted connections are not allowed

Note

This could fail on systems with a Zabbix cliënt with a lower version as lower versions don't always support the TLS options.

Setting up the certificates for the Zabbix server

On the Zabbix server:

cd /etc/zabbix
mkdir zabbix_server_certs

Copy the zabbix server certificates from the CA machine to the zabbix server.:

-r-x------ 1 root   root 2171 feb  1 17:24 zabbix_ca.crt
-r-x------ 1 root   root 1712 feb  1 17:24 zabbix_server.crt
-r-x------ 1 root   root 1675 feb  1 17:24 zabbix_server.key

Set the permissions and owner:

chmod 500 zabbix_server_certs
chown -R zabbix zabbix_server_certs/

Edit zabbix_server.conf and modify these parameters:

TLSCAFile=/etc/zabbix/zabbix_server_certs/zabbix_ca.crt
TLSCertFile=/etc/zabbix/zabbix_server_certs/zabbix_server.crt
TLSKeyFile=/etc/zabbix/zabbix_server_certs/zabbix_server.key

Restart the server and check the logs:

/etc/init.d/zabbix-server restart
or
systemctl restart zabbix-server

tail /var/log/zabbix-server/zabbix_server.log

To test it:

zabbix_get -s 127.0.0.1 -k system.cpu.load --tls-connect cert --tls-ca-file /etc/zabbix/zabbix_server_certs/zabbix_ca.crt --tls-cert-file /etc/zabbix/zabbix_server_certs/zabbix_server.crt --tls-key-file /etc/zabbix/zabbix_server_certs/zabbix_server.key

0.100000

We get an answer, the encrypted connection works.

Zabbix Agent playbook

A ansible playbook that updats the system and installs Zabbix agent and copies the necessary files to connect with a certificate. The hosts are defined in /etc/ansible/hosts:

vi /etc/ansible/hosts
...
[test]
<hostname>

The playbook:

---
- hosts: test
  tasks:
  - name: Update the system
    apt:
      update_cache: yes
      upgrade: yes
      cache_valid_time: 3600
  - name: Creates the zabbix agent directory
    file:
      path: /etc/zabbix/zabbix_agent_certs
      state: directory
      owner: zabbix
      group: root
      mode: 0500
  - name: Copy own root CA certificate
    copy:
      src: distribute_zabbix_keys/zabbix_ca.crt
      dest: /etc/zabbix/zabbix_agent_certs
      owner: zabbix
      group: root
      mode: 0500
  - name: Copy Zabbix agent certificate
    copy:
      src: distribute_zabbix_keys/zabbix_agent.crt
      dest: /etc/zabbix/zabbix_agent_certs
      owner: zabbix
      group: root
      mode: 0644
  - name: Copy Zabbix agent key
    copy:
      src: distribute_zabbix_keys/zabbix_agent.key
      dest: /etc/zabbix/zabbix_agent_certs
      owner: zabbix
      group: root
      mode: 0644
  - name: Copy the zabbix config to old
    shell:
        cmd: cp /etc/zabbix/zabbix_agentd.conf /etc/zabbix/zabbix_agentd.conf.old
  - name: Add TLS info to zabbix agent config
    shell:
      cmd: |
        cat <<EOT >> /etc/zabbix/zabbix_agentd.conf
        TLSAccept=cert, unencrypted
        TLSConnect=unencrypted
        TLSCAFile=/etc/zabbix/zabbix_agent_certs/zabbix_ca.crt
        TLSCertFile=/etc/zabbix/zabbix_agent_certs/zabbix_agent.crt
        TLSKeyFile=/etc/zabbix/zabbix_agent_certs/zabbix_agent.key
        EOT
  - name: Restart Zabbix agent
    service:
      name: zabbix-agent
      state: restarted

To run the playbook, I use this command:

ansible-playbook setup_zabbix_agent.yml -b --ask-sudo-pass

We have only touched the surface of monitoring: we have a Zabbix server securely communicating with a Zabbix cliënt. We are monitoring a remote system over a Tinc VPN. Next steps could be the tuning of the tresholds specified, having a closer look at templates and so on.