induction-monitoring

Induction to monitoring

View the Project on GitHub infra-helpers/induction-monitoring

ElasticSearch (ES) Cluster Setup on Proxmox LXC Containers

Beige, yellow, and blue loom bands, by Michael Walter on Unsplash

Overview

That README is part of a broader tutorial about monitoring, and gives details on how to setup a two-node ElasticSearch (ES) cluster on LXC containers of a Proxmox host, secured by an SSH gateway, also acting as an Nginx-based reverse proxy.

For the installation of the Proxmox host and LXC containers themselves, refer to the dedicated tutorial on GitHub, itself a full tutorial on Kubernetes (k8s). Only a summary is given here for the differences with Kubernetes clusters.

Table of Content (ToC)

Table of contents generated with markdown-toc

References

LXC Containers on Proxmox

ElasticSearch (ES)

Security

The following documentations are mentioned for reference only. In this repository, the ES cluster is secured through a reverse proxy/SSH gateway mechanism only.

Host preparation

In that section, it is assumed that we are logged on the Proxmox host as root.

The following parameters are used in the remaining of the guide, and may be adapted according to your configuration:

VM ID Private IP Host name (full) Short name
104 10.30.2.4 proxy8.example.com proxy8
191 10.30.2.191 es-in1.example.com es-int1
192 10.30.2.192 es-in2.example.com es-int2
194 10.30.2.194 kfk-in1.example.com kfk-int1
195 10.30.2.195 kfk-in2.example.com kfk-int2

The loopback network interface

auto lo iface lo inet loopback

auto eno1 iface eno1 inet manual

auto eno2 iface eno2 inet manual

auto bond0 iface bond0 inet manual bond-slaves eno1 eno2 bond-miimon 100 bond-mode active-backup

vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.

auto vmbr0 iface vmbr0 inet static address ${HST_IP} netmask 255.255.255.0 gateway ${HST_GTW_IP} bridge_ports bond0 bridge_stp off bridge_fd 0

auto vmbr2 iface vmbr2 inet static address 10.30.2.2 netmask 255.255.255.0 bridge-ports none bridge-stp off bridge-fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s ‘10.30.2.0/24’ -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s ‘10.30.2.0/24’ -o vmbr0 -j MASQUERADE

root@proxmox:~$ cat /etc/systemd/network/50-default.network

This file sets the IP configuration of the primary (public) network device.

You can also see this as “OSI Layer 3” config.

It was created by the OVH installer, please be careful with modifications.

Documentation: man systemd.network or https://www.freedesktop.org/software/systemd/man/systemd.network.html

[Match] Name=vmbr0

[Network] Description=network interface on public network, with default route DHCP=no Address=${HST_IP}/24 Gateway=${HST_GTW_IP} IPv6AcceptRA=no NTP=ntp.ovh.net DNS=127.0.0.1 DNS=8.8.8.8

[Address] Address=${HST_IPv6}

[Route] Destination=2001:0000:0000:34ff:ff:ff:ff:ff Scope=link

root@proxmox:~$ cat /etc/systemd/network/50-public-interface.link

This file configures the relation between network device and device name.

You can also see this as “OSI Layer 2” config.

It was created by the OVH installer, please be careful with modifications.

Documentation: man systemd.link or https://www.freedesktop.org/software/systemd/man/systemd.link.html

[Match] Name=vmbr0

[Link] Description=network interface on public network, with default route MACAddressPolicy=persistent NamePolicy=kernel database onboard slot path mac #Name=eth0 # name under which this interface is known under OVH rescue system #Name=eno1 # name under which this interface is probably known by systemd


* The maximal virtual memory needs to be increased on the host:
```bash
$ sysctl -w vm.max_map_count=262144
$ cat >> /etc/sysctl.conf << _EOF

###########################
# Elasticsearch in VM
vm.max_map_count = 262144

_EOF

Get the latest CentOS templates

Kernel modules

Overlay module

root@proxmox:~$ modprobe overlay && \
  cat > /etc/modules-load.d/docker-overlay.conf << _EOF
overlay
_EOF

nf_conntrack

SSH gateway and reverse proxy

ip addr add ${HST_GTW_IP}/5 dev eth0 ip link set eth0 up ip route add default via ${GTW_IP} dev eth0

ip addr add 10.30.2.4/24 dev eth1 ip link set eth1 up

_EOF [root@proxy8]# chmod 755 ~/bin/netup.sh [root@proxy8]# ~/bin/netup.sh [root@proxy8]# dnf -y upgrade [root@proxy8]# dnf -y install epel-release [root@proxy8]# dnf -y install NetworkManager-tui [root@proxy8]# systemctl start NetworkManager.service
&& systemctl status NetworkManager.service
&& systemctl enable NetworkManager.service [root@proxy8]# nmcli con # to check the name of the connection [root@proxy8]# nmcli con up “System eth0” [root@proxy8]# exit


* Complement the installation on the SSH gateway/reverse proxy container.
  For security reason, it may be a good idea to change the SSH port
  from `22` to, say `7022`:
```bash
root@proxmox:~$ pct enter 104
[root@proxy8]# dnf -y install hostname rpmconf dnf-utils wget curl net-tools tar
[root@proxy8]# hostnamectl set-hostname proxy8.example.com
[root@proxy8]# dnf -y install htop less screen bzip2 dos2unix man man-pages
[root@proxy8]# dnf -y install sudo whois ftp rsync vim git-all patch mutt
[root@proxy8]# dnf -y install java-11-openjdk-headless
[root@proxy8]# dnf -y install nginx python3-pip
[root@proxy8]# pip-3 install certbot-nginx
[root@proxy8]# rpmconf -a
[root@proxy8]# ln -sf /usr/share/zoneinfo/Europe/Paris /etc/localtime
[root@proxy8]# setenforce 0
[root@proxy8]# dnf -y install openssh-server
root@proxy8# sed -i -e 's/#Port 22/Port 7022/g' /etc/ssh/sshd_config
[root@proxy8]# systemctl start sshd.service \
	&& systemctl status sshd.service \
	&& systemctl enable sshd.service
[root@proxy8]# mkdir ~/.ssh && chmod 700 ~/.ssh
[root@proxy8]# cat > ~/.ssh/authorized_keys << _EOF
ssh-rsa AAAA<Add-Your-own-SSH-public-key>BLAgU first.last@example.com
_EOF
[root@proxy8]# chmod 600 ~/.ssh/authorized_keys
[root@proxy8]# passwd -d root
[root@proxy8]# rpm --import http://wiki.psychotic.ninja/RPM-GPG-KEY-psychotic
[root@proxy8]# rpm -ivh http://packages.psychotic.ninja/7/base/x86_64/RPMS/keychain-2.8.0-3.el7.psychotic.noarch.rpm

ES cluster

10.30.2.191 es-int1.example.com es-int1 10.30.2.192 es-int2.example.com es-int2

_EOF


* A few handy aliases:
```bash
root@proxy8:~# cat >> ~/.bashrc << _EOF

# Source aliases
if [ -f ~/.bash_aliases ]
then
        . ~/.bash_aliases
fi

_EOF
root@proxy8:~$ cat ~/.bash_aliases << _EOF
# User specific aliases and functions
alias dir='ls -laFh --color'
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

_EOF
root@proxy8:~# . ~/.bashrc
root@proxy8:~# exit

_EOF [root@proxy8]# htpasswd -c /etc/nginx/.kibana-user New password: Re-type new password: Adding password for user > [root@proxy8]# /usr/local/bin/certbot --nginx [root@proxy8]# nginx -t [root@proxy8]# nginx -s reload [root@proxy8]# exit


# Elasticsearch (ES) cluster
* Create the container for node 1:
```bash
root@proxmox:~$ pct create 191 local:vztmpl/centos-8-default_20191016_amd64.tar.xz --arch amd64 --cores 2 --hostname es-int1.example.com --memory 16134 --swap 32268 --net0 name=eth0,bridge=vmbr2,gw=10.30.2.2,ip=10.30.2.191/24,type=veth --onboot 1 --ostype centos
root@proxmox:~$ pct resize 191 rootfs 50G
root@proxmox:~$ ls -laFh /var/lib/vz/images/191/vm-191-disk-0.raw
-rw-r----- 1 root root 50G Dec 19 22:27 /var/lib/vz/images/191/vm-191-disk-0.raw
root@proxmox:~$ cat /etc/pve/lxc/191.conf
arch: amd64
cores: 2
hostname: es-int1.example.com
memory: 16134
net0: name=eth0,bridge=vmbr2,gw=10.30.2.2,hwaddr=1A:EC:7F:9E:90:34,ip=10.30.2.191/24,type=veth
onboot: 1
ostype: centos
rootfs: local:191/vm-191-disk-0.raw,size=50G
swap: 32268

ip addr add 10.30.2.191/24 dev eth0 ip link set eth0 up ip route add default via 10.30.2.2 dev eth0

_EOF [root@es-int1]# chmod 755 ~/bin/netup.sh [root@es-int1]# ~/bin/netup.sh [root@es-int1]# dnf -y upgrade [root@es-int1]# dnf -y install epel-release [root@es-int1]# dnf -y install NetworkManager-tui [root@es-int1]# systemctl start NetworkManager.service
&& systemctl status NetworkManager.service
&& systemctl enable NetworkManager.service [root@es-int1]# nmcli con # to check the name of the connection [root@es-int1]# nmcli con up “System eth0” [root@es-int1]# exit


* Manual setup of the network for node 2:
```bash
root@proxmox:~$ pct start 192 && pct enter 192
[root@es-int2]# mkdir ~/bin && cat > ~/bin/netup.sh << _EOF
#!/bin/sh

ip addr add 10.30.2.192/24 dev eth0
ip link set eth0 up
ip route add default via 10.30.2.2 dev eth0

_EOF
[root@es-int2]# chmod 755 ~/bin/netup.sh
[root@es-int2]# ~/bin/netup.sh
[root@es-int2]# dnf -y upgrade
[root@es-int2]# dnf -y install epel-release
[root@es-int2]# dnf -y install NetworkManager-tui
[root@es-int2]# systemctl start NetworkManager.service \
	&& systemctl status NetworkManager.service \
	&& systemctl enable NetworkManager.service
[root@es-int2]# nmcli con # to check the name of the connection
[root@es-int2]# nmcli con up "System eth0"
[root@es-int2]# exit

Elastic

ES_HOME=”/usr/share/elasticsearch” export PATH=”${PATH}:${ES_HOME}/bin”

_EOF [root@es-int]# . ~/.bashrc


## Kibana
* Install Kibana on a single node, `es-int1.example.com`
  (`10.30.2.191`). Edit `/etc/kibana/kibana.yml` and change the
  `server.host` value from `localhost` into `0.0.0.0`:
```bash
[root@es-int1]$ sed -i -e 's/#server.host: "localhost"/server.host: "0.0.0.0"/g' /etc/kibana/kibana.yml
[root@es-int1]$ systemctl enable kibana \
	&& systemctl restart kibana && systemctl status kibana

Access to ES from outside through proxy8 with SSH

Kafka

Overview

A Kafka cluster is made of so-called broker nodes coordinated by Zookeeper. Zookeeper may be a cluster itself, which has to contain an odd number of nodes (in order that a leader can always be elected). As the infrastructure is expected to be used just for small data and traffic, it will be fine to have Zookeeper be made of a single node cluster.

The Kafka cluster will be made of two broker nodes, as to allow for some replication.

Kafka releases are depending on the Scala version. It is relevant only for the Scala-based bindings.

Setup of the containers

ip addr add 10.30.2.194/24 dev eth0 ip link set eth0 up ip route add default via 10.30.2.2 dev eth0

_EOF [root@kfk-int1]# chmod 755 ~/bin/netup.sh [root@kfk-int1]# ~/bin/netup.sh [root@kfk-int1]# dnf -y upgrade [root@kfk-int1]# dnf -y install epel-release [root@kfk-int1]# dnf -y install NetworkManager-tui [root@kfk-int1]# systemctl start NetworkManager.service && systemctl status NetworkManager.service && systemctl enable NetworkManager.service [root@kfk-int1]# nmcli con # to check the name of the connection [root@kfk-int1]# nmcli con up “System eth0” [root@kfk-int1]# exit


* Manual setup of the network for node 2:
```bash
root@proxmox:~$ pct start 195 && pct enter 195
[root@kfk-int2]# mkdir ~/bin && cat > ~/bin/netup.sh << _EOF
#!/bin/sh

ip addr add 10.30.2.195/24 dev eth0
ip link set eth0 up
ip route add default via 10.30.2.2 dev eth0

_EOF
[root@kfk-int2]# chmod 755 ~/bin/netup.sh
[root@kfk-int2]# ~/bin/netup.sh
[root@kfk-int2]# dnf -y upgrade
[root@kfk-int2]# dnf -y install epel-release
[root@kfk-int2]# dnf -y install NetworkManager-tui
[root@kfk-int2]# systemctl start NetworkManager.service && systemctl status NetworkManager.service && systemctl enable NetworkManager.service
[root@kfk-int2]# nmcli con # to check the name of the connection
[root@kfk-int2]# nmcli con up "System eth0"
[root@kfk-int2]# exit

Kafka

Host tikfk1 HostName kfk-int1.example.com ProxyCommand ssh -W %h:22 root@tiproxy8 ForwardAgent yes Host tikfk2 HostName kfk-int2.example.com ProxyCommand ssh -W %h:22 root@tiproxy8 ForwardAgent yes

_EOF


## Kafka specificities
The following tasks have to be performed on every node.

Zookeeper is bundled with the Kafka tar-ball.

* Install Java:
```bash
root@proxmox:~$ pct enter 194 # pct enter 195
[root@kfk-intN]# dnf -y install java-11-openjdk-headless

[Confluent] name=Confluent repository baseurl=https://packages.confluent.io/rpm/5.4 gpgcheck=1 gpgkey=https://packages.confluent.io/rpm/5.4/archive.key enabled=1 _EOF [root@kfk-intN]# dnf -y install confluent-community-2.12 [root@kfk-intN]# chown cp-kafka:confluent /var/log/kafka
&& chmod u+wx,g+r,o= /var/log/kafka
&& chown cp-kafka:confluent /var/log/confluent
&& chmod u+wx,g+wx,o= /var/log/confluent
&& chown cp-kafka:confluent /var/lib/kafka
&& chmod u+wx,g+r,o= /var/lib/kafka
&& chown cp-kafka:confluent /var/lib/zookeeper
&& chmod u+wx,g+r,o= /var/lib/zookeeper


* Setup Zookeeper on a single node. Zookeeper needs to be installed
  on an odd number of nodes:
```bash
[root@kfk-int1]# cat >> /etc/kafka/zookeeper.properties << _EOF
tickTime=2000
clientPort=2181
initLimit=5
syncLimit=2
server.1=10.30.2.194:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
_EOF
[root@kfk-int1]# systemctl start confluent-zookeeper && systemctl status confluent-zookeeper && systemctl enable confluent-zookeeper

WATCHER::

WatchedEvent state:SyncConnected type:None path:null Node does not exist: /brokers/ids


* Setup Kafka on each node:
```bash
[root@kfk-intN]# sed -i -e 's|broker.id=0|broker.id=N|g' /etc/kafka/server.properties
[root@kfk-intN]# sed -i -e 's|offsets.topic.replication.factor=1|offsets.topic.replication.factor=2|g' /etc/kafka/server.properties
[root@kfk-intN]# sed -i -e 's|transaction.state.log.replication.factor=1|transaction.state.log.replication.factor=2|g' /etc/kafka/server.properties
[root@kfk-intN]# sed -i -e 's|zookeeper.connect=localhost:2181|zookeeper.connect=10.30.2.194:2181|g' /etc/kafka/server.properties
[root@kfk-intN]# systemctl start confluent-kafka && systemctl status confluent-kafka && systemctl enable confluent-kafka

WATCHER::

WatchedEvent state:SyncConnected type:None path:null [1, 2]


* Setup Kafka REST proxy and start it (configure Zookeeper):
```bash
[root@kfk-intN]# sed -e 's|#zookeeper.connect=localhost:2181|zookeeper.connect=10.30.2.194:2181|g' /etc/kafka-rest/kafka-rest.properties
[root@kfk-intN]# systemctl start confluent-kafka-rest && systemctl status confluent-kafka-rest && systemctl enable confluent-kafka-rest

Access to Kafka and Zookeeper through proxy8 with SSH

WATCHER::

WatchedEvent state:SyncConnected type:None path:null [1, 2]


* From Travis CI, list the Kafka topics:
```bash
build@travis-ci$ ssh root@tiproxy8 kafka-topics.sh --list --zookeeper 10.30.2.194:2181
__consumer_offsets
test