Category Archives: Tutorials

Tutorials and guides about what you can do on a *nix server. The tutorials will guide you how to build and maintain a home server. It teaches you how to create a playground where you can try out new and fancy software. Some of these guides are from systems running businesses in production.

openSUSE Linux gecko | CC-BY-SA 3.0, https://github.com/openSUSE/artwork/

Install OpenSUSE MicroOS in KVM with Ignition

Install OpenSUSE MicroOS in KVM with Ignition. A step-by-step guide to provision container specific OS instances really fast.

About OpenSUSE MicroOS

I needed a container specific OS since I converted my docker-compose services to pods with Podman. Fedora CoreOS looks promising. However I have been using OpenSUSE for years, so it was convenient for me to try MicroOS which is derived from Tumbleweed.

These are the features which I like the most.

Install OpenSUSE MicroOS in KVM with Ignition

There are documentations about MicroOS, but I could not find a complete guide about how to install OpenSUSE MicroOS in KVM. Also its Ignition guide directly redirects to CoreOS’s git repo for documentation. The information is there but putting it together takes time.
As I already managed to do it then why not to share it? 🙂

In my guide I will use virsh (libvirt) and virt-install (virt-manager) to provision headless MicroOS VMs based on the downloadable KVM images they made available. Both tools are higher lever APIs to KVM / QEMU.

Installing libvirt to make using QEMU easy

OpenSUSE has a package pattern for turning your OS into a virtualization host. Following the Virtualization Guide will definitely help. But if you do not want to read it all then just run the following command.

zypper in -t pattern kvm_server kvm_tools

Should you want to read more about managing virtual machines with libvirt then there is a documentation for that too. I will not go into details this time.

Downloading and verify installation media of MicroOS

Once the tools are ready then download the installation media and verify the checksum and signature.

# cd /var/lib/libvirt/images
# curl -LO http://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2
# curl -LO http://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2.sha256
# curl -LO http://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2.sha256.asc

Note that sha256 checksum and signature was made for a snapshot whose name is different from the file we downloaded, though the content is the same. Probably the other files on their webserver are just symbolic links.

# sha256sum openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2
# gpg --search-keys B88B2FD43DBDC284
# gpg --recv-keys 0x22C07BA534178CD02EFE22AAB88B2FD43DBDC284
# gpg --verify openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2.sha256.asc

Creating Ignition configuration for VM

Ignition expect its configuration to be in JSON. However one just not create a JSON file by hand. But creates a YML file and convert it with semantic checks (and some boilerplate) to JSON by using CoreOS’s fcct. Here is my example. It is pretty straight forward.

variant: fcos
version: 1.0.0
passwd:
  users:
    - name: root
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHj2D8PAxz0MKV9UJ8dxavlVzdAvMBrfGp38sj4q/aRbkcuYVNHAQh+xXHI0VcPEtu9rqZbvqfmQt0DFhsdf938W6r3y6mLp4+6KIDgb4Jj2B3zlzBIF0haAFi/GZAp4dh4uuhHsVvZGqsdqCglxUnIPb+i8IyYA8GGU+3IOgRfjjtpMfDJcWZTzGm56yDsBYORX3EckkGcWTN4/oW0SKWoO9zf/887/CvVZF/0V7corEAdMyTCiSSqqUjIDLAZpCMU4czadZop7cvVjGT6WLmyGDuTBruvjsMwxYA/OMAZrUuOEoAW0bf/QZRZ4tO7ku+o0oqwca5uwVbuouAFovJ [email protected]
      password_hash: "$1$salt$qJH7.N4xYta3aEG/dfqo/0"
storage:
  files:
    - path: /etc/sysconfig/network/ifcfg-eth0
      mode: 0600
      overwrite: true
      contents:
        inline: |
          BOOTPROTO='static'
          STARTMODE='auto'
          BROADCAST=''
          ETHTOOL_OPTIONS=''
          IPADDR='192.168.0.10/24'
          MTU=''
          NAME=''
          NETWORK=''
          REMOTE_IPADDR=''
          ZONE=public
    - path: /etc/sysconfig/network/routes
      mode: 0644
      overwrite: true
      contents:
        inline: |
          default 192.168.0.254 - -
    - path: /etc/hostname
      mode: 0644
      overwrite: true
      contents:
        inline: |
          example.com

Please note that on OpenSUSE you cannot just provide static DNS information by overwriting /etc/resolv.conf. Because the content of /etc/resolv.conf is managed by netconfig. And it configuration file bigger than optimal to include it in a YML file. Though you can configure it manually after the first run.

Once you are ready, put the contents into config.fcc and convert it to JSON by using fcct.

# podman run -i --rm quay.io/coreos/fcct:release --pretty --strict < config.fcc > config.ign

Create a KVM host with virt-install

The Ignition file can be specified via QEMU command line. Adjust the specification of the VM according to your needs.

# virt-install --import --connect qemu:///system --name example \
    --ram 1024 --vcpus 1 \
    --disk size=20,backing_store=/var/lib/libvirt/images/openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2,bus=virtio \
    --os-variant=opensusetumbleweed \
    --network bridge=br0,model=virtio \
    --noautoconsole \
    --graphics spice,listen=127.0.0.1 \
    --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=/path/to/config.ign"

No you have just installed OpenSUSE MicroOS in KVM with Ignition. Enjoy.

Managing VM with Virsh

Virsh is another tool to manage your VMs. Here I provide some examples I frequently use. They could be useful.

Console access

You can attach to the serial console of the VM with the following command.

# virsh console example

Disconnecting from Virsh console is possible with SHIFT+5.

Remote console access

Remote access to the console is possible with Spice. If you are not in production then you can easily access the remote Spice port without TLS via SSH port forward.

Delete virtual machines

This can be useful when you are not satisfied with the result and want to start over from scratch. Storage files are not automatically deleted.

# virsh dumpxml --domain example | grep 'source file'
      <source file='/var/lib/libvirt/images/example.qcow2'/>
        <source file='/var/lib/libvirt/images/openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2'/>
# virsh destroy example
# virsh undefine example
# rm -f /var/lib/libvirt/images/example.qcow2

I hope you will enjoy your shiny new container host. 🙂

Managing Podman pods with pods-compose

Managing Podman pods with pods-compose makes your move to Podman easier. I already converted my docker-compose services to pods with Podman, however I really missed some features, up until now. Let’s meet pods-compose.

Missing features of Podman

As soon as I managed to convert docker-compose services to pods I realized that managing them will be not as easy as it was with docker-compose. I faced with the following problems.

  1. There is no autostart for pods and containers. Of course you can generate systemd service units for all components but managing them is not easy without an automation tool. There are separate service files for each pod and container.
  2. For updates you might have to stop, remove, recreate everything from scratch, unless you script it. There is no ‘up‘, ‘down‘ or ‘build‘ features like you have with docker-compose.
  3. There is no single point of configuration which I could use to describe all the pods and containers. I could write and maintain Kubernetes YAML files, but that’s even harder than using the CLI syntax I am already familiar with.

I needed a tool which makes managing Podman pods easier. podman-compose looked promising but it did not really work for me and I also did not like its CLI interface. So I decided to write my own tool.

Design goals of pods-compose

I did not want to put a lot of effort into this. I only wanted the following additional abilities.

  • Be able to automatically start and stop all pods and containers upon reboot.
  • Tear down existing pods at once.
  • Create pods and containers from a description at once.
  • Build all the images I define with a single command.

I did not want to rely on docker-compose’s YAML format. Intentionally there is no support for using an existing compose configuration. Although I was already familiar with that format, I wanted a complete migration not just a partial one.

Managing Podman pods with pods-compose

As a kickstart, let’s get a glimpse into the similarities between pods-compose and docker-compose.

pods-composedocker-compose
Deploy pod(s)–up [POD]up [SERVICE]
Tear down pod(s)–down [POD]down [SERVICE]
Start pod(s)–start [POD]start [SERVICE]
Stop pod(s)–stop [POD]stop [SERVICE]
Restart pod(s)–restart [POD]restart [SERVICE]
Build all container images–buildbuild
Status of pods and containers–psps
Generate Kubernetes Pod YAML(s)–generate

Autostart pods and containers

Podman can generate systemd units for pods and containers. However there will be many of them, making it hard to overview and maintain it. Because pods-compose takes care of starting and stopping pods with a single command line option, I could create a single systemd service file instead of many.

The install script will deploy that systemd service file for pods-compose. Enabling it makes your pods and containers to start automatically upon reboot. And of course gracefully stop before the system halts.

Further details can be found in the readme of pods-compose.

Things you still have to do manually

Creating containers at least first time is a manual procedure. People usually start with ‘docker run’ commands then once the result looks okay then will create a docker-compose YAML.

This will not change with pods-compose. You still have to create your pods and containers with ‘podman run’. However you do not have to create any YAML files. The tool will create them for you.
Luckily podman CLI syntax is almost the same as docker’s, so it is easy to make progress fast.

The other part is defining which image should be built by pods-compose. Because this information cannot be set in Kubernetes YAML files, you can use pods-compose‘s INI formatted configuration file to define the TAG and the CONTEXT of images. As a result, pods-compose will build all the images for you.

Final words

Let me know if you are still missing some features you would love to see implemented in pods-compose. Also please share if you liked it.

Docker and Podman logo

Convert docker-compose services to pods with Podman

How to deploy pods with Podman when you only need a single-host system and not a complex Kubernetes. Convert your docker-compose services to pods with Podman.

For a single host setup or even for a now officially dead Docker Swarm setup using docker-compose is pretty convenient. But I wanted to get rid of Docker completely and migrate my docker-compose services to pods with Podman.

The reasons why I convert docker-compose services to pods

I have been using Docker’s container technology for about 4-5 years. Both in production and in different labs. Call me an old fashioned but I always managed to set up systems either with pure Docker containers or with docker-compose. However there are things I cannot easily forget.

Here are the top reasons why I decided to convert my docker-compose services to pods with Podman and get rid of Docker completely.

  • Closing issues with an attitude of ‘we don’t really care’. Like #4513, or #22920#issuecomment-264036710
  • Leaving important security requests open for years. Like #3480#issuecomment-482587531.
  • Recurring errors like failing to create many bridged network at once on a clean system, claiming ‘ERROR: Pool overlaps with other one on this address space‘.
  • Too many fiddling with iptables rules on a system using firewalld. This may not be a problem where a host OS’ only role is to run containers. But there are legit cases where containers may run on a host serving other purposes as well.
  • Daemon changes causing data losses. I learned the hard way why putting a production SQL database (state-full) into a container is a NO GO.
  • Inconsistency between recommendations and real life experience. Like “Don’t run more than one process in a single container” – Have you seen GitLab’s official Docker image?
  • Various issues with logging. Some of them are already explained in posts like top 10 Docker logging gotchas.

I know some of these reasons may not apply to recent versions of Docker. And I am also aware that some issues are container technology related, so they may apply to Podman containers as well.

The basis of migration

Any migration requires planning and testing. So I started off with my home lab which hosts different systems. In my lab docker-compose took care of composing all services with a single YAML file. The following simplified figure shows a high level overview of the network architecture. Although the picture may indicate, the reverse proxy is not the gateway for the containers.

Network architecture of services orchestrated by docker-compose
Figure 1: Network architecture of services orchestrated by docker-compose

Although this system worked pretty well, I have some issues with it.

  1. All networks use a bridge network driver to provide network isolation of service groups. Therefore you have to create many networks, which in turn improves complexity.
  2. The network of Reverse Proxy has to be literally connected to all other bridges to have access to the web servers. However, this way the proxy container could access all exposed ports of all containers on any networks the proxy container is attached to. It provides a bigger attack surface.
  3. Docker makes these networking possible with lots of iptables rules (so as Podman) which are hard to overview and pollute the iptables rules you may already have.

Planning the conversion of docker-compose services to pods

There is a very fundamental difference between Docker and Podman. Podman supports the concepts of pods for instance. This is intentionally very similar to Kubernetes’ pods. Containers in a pod shares the same namespace, like network. So all containers in the same pod looks like sharing the same localhost network. And each pod has its own localhost.

With Docker (Figure 1) there are 5 networks for 9 containers. With Podman by using pods there is only 1 network for 5 pods (Figure 2).

Network architecture of services orchestrated by podman
Figure 2: Network architecture of services orchestrated by Podman

Pods provide another layer of isolation I really like. This way containers of any pods could only access ports published by other pods and not the containers themselves.

Challenges with Podman

Migrating to a new technology is not without compromises or challenges. Podman is around for a while and is rapidly evolving. Here are the challenges I had to handle.

Assign IP addresses to pods and not to containers

You can join a container to any networks. But a pod can be only joined to the default network. According to my understanding this will be changed later. This is the reason why I stick to the default network in my setup.

“Most of the attributes that make up the Pod are actually assigned to the “infra” container.  Port bindings, cgroup-parent values, and kernel namespaces are all assigned to the “infra” container. “

https://developers.redhat.com/blog/2019/01/15/podman-managing-containers-pods/

By default pods will connect to network labeled cni_default_network in libpod.conf. If you join the pod’s containers to other networks, the pod will still have its IP assigned from the default network. However containers will have IPs assigned from the specified networks. As far as I know this symptom looks like a bug.

DNS name resolution between containers and pods

By using the container plugin dnsname you can get name resolution between containers on the same network. However at the moment you cannot have DNS between pods. That feature is under development.

While I am waiting for support of DNS on pod level, I worked around this limitation. I publish the exposed ports of pods to their gateway’s IP address 10.88.0.1 and not the IP address of their infra container itself. As long as the gateway’s IP address static this will work.

Replacing functionalities of docker-compose

The YAML format of docker-compose uses an abstraction above ‘docker run‘ command. However I realized that all the hard work docker-compose did to me was to create networks and assign container’s to them. And of course deploying services.

Networking: Rootfull container networking (CNI)

Luckily describing how a network should look like is not the role of Podman but CNI and its plugins. You can see the layout of the default network in Figure 3.

topology of default network of podman
Figure 3. Low level network topology of Podman’s default network, called ‘podman’.

The published ports are not visible from the outside network unless you set up routes externally. Or you can simply set the IP address of the host for serving published ports (--publish 192.168.122.253:80:80). Effectively it will be another DNAT rule. For my simple case it is enough.

P.s. Do not forget to enable IP forwarding with sysctl to persist across reboots.

P.s. 2: You may need to change the default firewall backend from iptables to firewalld in CNI configuration. So you will have a cleaner overview of your chains and rules.

Rootless containers uses slirp4netns instead of CNI. You can read more about the differences in this post configuring container networking with Podman.

Build images: use Dockefile to build an image

I really like that you do not have to learn another language to build an image. Use the same Dockerfile format you are already familiar with.

Building an image is not the task of Podman but another tool called buildah. Although you can even use podman build, it will actually use Buildah in the background. Assuming you have your Dockerfile in the current working directory, it will look like this. It can even publish the image to a Docker repository.

buildah pull docker.io/opensuse/leap:15.1
buildah bud -t example/example-container:latest .

Deployment: Managing container life-cycles

Docker-compose made it possible to deploy all services at once with docker-compose up -d. Achieving the same with Podman is possibly by using its support for Kubernetes YAML and some shell scripting.

  1. Set up the pods and containers with all the settings you need with plain podman run commands. Here is a shell script example for gitea.
    #!/bin/bash
    podname="gitea"
    version="1.9"
    publish_ip="10.88.0.1"
    
    podman pod rm -f ${podname} podman pod create --name ${podname} --hostname ${podname} -p ${publish_ip}:3000:3000
    
    podman run -d --name ${podname}-svc --hostname ${podname}-svc --expose 3000 --pod ${podname} -e TZ="Europe/Budapest" \
     -v /srv/${podname}:/data \
     gitea/gitea:${version}
  2. Generate a Kubernetes compatible YAML file.
    podman generate kube -f gitea.yml gitea
  3. Replay it.
    podman pod rm -f gitea podman play kube gitea.yml

I wrote a wrapper script called pods-compose for imitating docker-compose start/stop/up/down but for pods to easy my life. There is a already post about managing Podman pods with pods compose.

Final thoughts

I did a lot of testing, so I managed to convert all docker-compose services to pods with Podman and with some shell scripting. I still have to figure out how to auto start of pods. There are shareable systemd devices for containers, but I want to test it for pods. See you next time.

OBD2 car metrics in Elasticsearch

Display OBD2 car metrics in Elasticsearch

It is an experiment about displaying OBD2 car metrics in Elasticsearch. To see how fuel economy changes by changing your driving habits.
With the help of an ELM327 compatible OBD2 reader and a Android app Dashcommand I could export OBD2 car metrics in Elasticsearch. The app also added GPS data. It was not as easy as it looked like at the beginning.

What is OBD2 car metrics?

OBD and OBD2 are standards for on-board diagnostics of various vehicles like cars and trucks. Basically it is an interface for the car’s brain. With an OBD2 adapter you can query the status and metrics of various subsystems of a vehicle. You can find this port in almost any car manufactured in the last three decades.

This interface usually is not used by customers but by experienced car mechanics or technicians. There are many apps which can use an ELM327 adapter to read data,  some even claim to clear Check Engine light too. However these cheap adapters are not comparable to professional devices which in are usually much more expensive and can do a lot more.

I have seen such professional devices which can even draw diagrams of metrics in real time. I though somehow the basics should be possible with Elasticsearch as well. Maybe not in real time. 🙂

What OBD2 car metrics can we get?

I drive a lot. Therefore one of my main motivation is to improve my fuel economy to simply save costs. I wanted to read the following metrics from my car.

  • Speed
  • Acceleration
  • Fuel consumption (actual, average)
  • Engine RPM, power and torque (the last two are calculated values)
  • GPS latitude, longitude and altitude

Different apps were tested before but most of them either did not support exporting OBD2 car metrics nor could add GPS data to the metrics. I even put some effort to use Carscanner and Trackbook GPS together but although correlating data from two different apps was fun, but the results was not satisfying for at all. Creating an app from scratch was not an option as I wanted to focus my resources on visualization and understanding the data and not on developing an app.

After some trials I found two possible solutions. One of them was AutoPi, which looked very promising but it was too expensive (275 EUR) for the task. The other one was Dashcommand which turned out absolutely worth the money (~ 7 EUR) for such a hobby project. It is capable to record much more data and metrics than I needed.

(I am really sorry AutoPi. I admit that I found you cooler than Dashcommand.)

Dashcommand

Basic usage

After you completed the first run wizard, specified your car’s details, you should go to Settings and check “Enable GPS”. On the main screen tap on DATA GRID and click on the computer icon to start recording metrics.

Once you stopped recording you should check LOG FILES on the main screen and export the logs as CSV files (spreadsheet file). Depending on the length of your track it will take some time to prepare the data and save it for you. Transfer it to your computer.

Processing the exported data

We have the CSV file but in its current format it is just a really huge mass of numeric data. Therefore I needed a tool to convert it into a format what Elasticsearch can ingest. I have developed a script in Perl using the Search::Elasticsearch library to send the records in the appropriate format to Elasticsearch.

Before you would use the script the index template mapping needs to be set in Elasticsearch. The procedure has been already described in my previous posts. You can download the index template mapping from GitHub. After the mapping is added you can use the script.

The script requires a “-d” switch pointing to the CSV file you want to process and an “-e” switch pointing your Elasticsearch ingest node.

$ perl dashcommand-csv-parser.pl -d DataLog.csv -e 1.2.3.4:9200 2>/dev/null

In the example above I muted stderr as it gets polluted by the trace messages of Search::Elasticsearch library that I could not turn off.

Please note, the resulted documents will not be consistent in terms of available key-value pairs. The reason is that not all metrics are available in every frame. This is expected. A sample document looks like this.

{
  "_index": "dashcommand-2019-06-19",
  "_type": "_doc",
  "_id": "pQfBk2sB4oJcycVygM3v",
  "_version": 1,
  "_score": null,
  "_source": {
    "calc.fuel_flow_avg l/h": "4.306",
    "calc.acceleration_g g": "-0.000",
    "@timestamp": "2019-06-19T07:19:17",
    "sae.maf g/s": "2.06",
    "calc.fc.ifc_avg l/100km": "5.9",
    "sae.vss km/h": "0",
    "calc.map kpa": "31.6",
    "frame number": "8134",
    "calc.fuel_flow l/h": "0.704",
    "calc.engine_power hp": "2",
    "calc.distance km": "242.4",
    "calc.acceleration m/s²": "-0.000",
    "sae.iat °c": "38",
    "calc.engine_torque n·m | kg-f·m": "25",
    "sae.rpm rpm": "716",
    "calc.boost_pressure kpa | bar | kg-f/cm²": "-0.674",
    "sae.sparkadv °": "3.0"
  },
  "fields": {
    "@timestamp": [
      "2019-06-19T07:19:17.000Z"
    ]
  }
}

Further readings about OBD data and metrics

If you are interested in about what these metrics mean. I highly suggest to check the following links. I found them really helpful during my work.

Elasticsearch and Kibana

All these stuff means nothing without shiny visualizations right?

Dashboard of OBD2 car metrics in ElasticsearchDashboard showing all the metrics I mentioned at the beginning.

Visualizing GPS altitude in Kibana

Here is a tricky one. In my previous posts like in NGINX visualization, or Fail2ban visualization I have shown that in Kibana you can easily visualize locations by the latitude and longitude data. But what about altitude?

Change the aggregation type from the default Count to Max and the field to aux.gps.altitude. It will effectively show the maximum value of the altitude. The value actually reflects the geological altitude on the map. It is sad that the map visualization is only 2D.

OBD Altitude data in Geo MapTip: Do the same trick with line charts with chart type set to area. It will look like a cross-section of a map.

Where is North?

The GPS records contain the actual GPS course too. By using the Gauge visualization and a narrow enough time interval you can actually see your direction. You have to set up at least 4 ranges according to a 360 degrees circle. It is a bit quirky I know. If you set up more ranges, then it will look like as a compass.GPS Course in Gauge Elasticsearch

How my fuel consumption looked like?

As we have almost all OBD2 car metrics in Elasticsearch, we can put both average and instantaneous fuel consumption on a stacked bar chart.

Fuel consumption ElasticsearchLooks like I had a moment of full throttle at the end of my trip. 🙂

Verdict

Should you like to see your car’s data in Elasticsearch, then download all visualizations and dashboards from the projects GitHub repository. I only tested the script under Linux. It may or may not work under Windows. It is time to give your car a ride. 🙂

China traffic details

Which Android app phones home to China?

In my previous blog post I created a system to monitor my network traffic. This system is capable to visualize connections even in geographic manner. Checking the data I found two network devices who phones home to servers located in China. What can I find out about those connections. What are they? Do they pose any security threat to me?

My sole purpose was to experiment and learn new things. Please mind that I could pick any other countries the same way as I chose China. Although I have security concerns, I want to learn and not to make statements over any countries or vendors.

Continue reading