Category Archives: Script

Deploy Elasticsearch stack with podman and Ansible

Deploy Elasticsearch stack with podman and Ansible

Leave a reply

Deploy Elasticsearch stack with podman and Ansible. Halfway on the road towards complete automation. But without the necessity of a complex orchestration tool. Somewhere between pets and cattles.

There is an existing Ansible collection containers.podman to handle podman pods and containers. Although Elastic the company already maintains an Ansible playbook for Elasticsearch, it uses regular Linux packages and not container images. Meet abalage.elasticstack_podman a collection of Ansible roles to deploy and handle an Elasticsearch cluster and its components like Kibana, Filebeat, Metricbeat and Logstash.

You can download it from galaxy.ansible.com or from github.

Requirements

Any operating system which supports a relatively recent version of podman (>=3.0) is required. Beware that CentOS 7 is not among them. The playbooks were tested on AlmaLinux 8.4 and OpenSUSE Leap 15.3. However on OpenSUSE you need to use a third party repository (Virtualization_containers).

The collection does not contain a reverse proxy for Kibana. You can use either Traefik of NGINX. The Kibana container is already provides labels for Traefik.

Features

I implemented the following features.

  • It deploys an Elasticsearch cluster. Works with single node deployments. However you can build a cluster of multiple nodes as well. You can even run multiple nodes on the same host OS.
  • Use Kibana for visualization.
  • Metricbeat automatically collects and stores all components metrics in the cluster. Use Kibana’s Stack Monitoring app to access the metrics.
  • Filebeat sends the components logs to Elasticsearch. Use Kibana’s Logs app to access the logs.
  • Optionally you can set up Logstash containers too. Although there are not many pipeline templates available.
  • Automatically populates built-in and custom users, passwords and roles. It does not support AD integration yet.
  • Pods and containers are automatically started upon reboot by using systemd units.
  • Supports host firewalld. Disabled by default.
  • Works best with host networking. Support for bridge networking is best effort and has scalability limitations. It does not support rootless networking at the moment.

Usage of the collection

I expect you already have an Ansible control node and several managed hosts. The collection was developed and tested with Ansible 2.9.

Create your deployment playbook

A playbook defines which play, roles and tasks of the collection are executed on which hosts. There is an existing playbook called elk-podman-deployment you can use. For example there is an example playbook in the repository too.

Create your deployment inventory

The deployment inventory describes how your cluster looks like. You can use the variables from the role’s defaults to create an inventory form scratch. However I provide a example inventory that you can customize.

Do not forget to encrypt sensitive data with ansible-vault.

I highly recommend to create proper X.509 certificates for TLS for security reasons. Make sure to follow the Securing Elasticsearch cluster guide to create such certificates.

Run the playbook

Once the inventory is complete, you can run the playbook like this tot deploy Elasticsearch stack with podman and Ansible.

$ ansible-playbook -i /path/to/production.ini playbook.yml --vault-password-file /path/to/vault-secret

It is a good idea to run in check mode on the first run to see whether is there anything missing from the inventory.

Reverse proxy

The collection itself does not provide any reverse proxy.

You can use any kind of reverse proxy to provide access to Kibana or any other components. I suggest to use Traefik for auto-discovery.

Conclusion

Developing all these roles and task were fun. I could learn a lot about Ansible. Therefore I can recommend this collection to anyone who would need such a setup but without the requirement of having a complex orchestration platform. I am aware of production systems deployed by this playbook.

However. I think this approach on the long run is not feasible. The architecture can grow to became uncontrolled pretty easily, unless someone constantly maintains the collection and provides support.

I could think of better alternatives like incorporating the container parts into Elastic’s official Ansible playbook. So the support would come from the vendor and not from the community. It might also worth to try some Edge/IoT oriented Kubernetes distribution like K3s which is lightweight but also supports Helm charts or better Operators.

What do you think?

Managing Podman pods with pods-compose

Leave a reply

Managing Podman pods with pods-compose makes your move to Podman easier. I already converted my docker-compose services to pods with Podman, however I really missed some features, up until now. Let’s meet pods-compose.

Missing features of Podman

As soon as I managed to convert docker-compose services to pods I realized that managing them will be not as easy as it was with docker-compose. I faced with the following problems.

  1. There is no autostart for pods and containers. Of course you can generate systemd service units for all components but managing them is not easy without an automation tool. There are separate service files for each pod and container.
  2. For updates you might have to stop, remove, recreate everything from scratch, unless you script it. There is no ‘up‘, ‘down‘ or ‘build‘ features like you have with docker-compose.
  3. There is no single point of configuration which I could use to describe all the pods and containers. I could write and maintain Kubernetes YAML files, but that’s even harder than using the CLI syntax I am already familiar with.

I needed a tool which makes managing Podman pods easier. podman-compose looked promising but it did not really work for me and I also did not like its CLI interface. So I decided to write my own tool.

Design goals of pods-compose

I did not want to put a lot of effort into this. I only wanted the following additional abilities.

  • Be able to automatically start and stop all pods and containers upon reboot.
  • Tear down existing pods at once.
  • Create pods and containers from a description at once.
  • Build all the images I define with a single command.

I did not want to rely on docker-compose’s YAML format. Intentionally there is no support for using an existing compose configuration. Although I was already familiar with that format, I wanted a complete migration not just a partial one.

Managing Podman pods with pods-compose

As a kickstart, let’s get a glimpse into the similarities between pods-compose and docker-compose.

pods-composedocker-compose
Deploy pod(s)–up [POD]up [SERVICE]
Tear down pod(s)–down [POD]down [SERVICE]
Start pod(s)–start [POD]start [SERVICE]
Stop pod(s)–stop [POD]stop [SERVICE]
Restart pod(s)–restart [POD]restart [SERVICE]
Build all container images–buildbuild
Status of pods and containers–psps
Generate Kubernetes Pod YAML(s)–generate

Autostart pods and containers

Podman can generate systemd units for pods and containers. However there will be many of them, making it hard to overview and maintain it. Because pods-compose takes care of starting and stopping pods with a single command line option, I could create a single systemd service file instead of many.

The install script will deploy that systemd service file for pods-compose. Enabling it makes your pods and containers to start automatically upon reboot. And of course gracefully stop before the system halts.

Further details can be found in the readme of pods-compose.

Things you still have to do manually

Creating containers at least first time is a manual procedure. People usually start with ‘docker run’ commands then once the result looks okay then will create a docker-compose YAML.

This will not change with pods-compose. You still have to create your pods and containers with ‘podman run’. However you do not have to create any YAML files. The tool will create them for you.
Luckily podman CLI syntax is almost the same as docker’s, so it is easy to make progress fast.

The other part is defining which image should be built by pods-compose. Because this information cannot be set in Kubernetes YAML files, you can use pods-compose‘s INI formatted configuration file to define the TAG and the CONTEXT of images. As a result, pods-compose will build all the images for you.

Final words

Let me know if you are still missing some features you would love to see implemented in pods-compose. Also please share if you liked it.