Generic Operation Guide

Starting a Microservice

Prerequisites

To start a Microservice, UVAP needs to be installed first. For detailed instructions and more information, see the Installation guide.

Docker

A Microservice can be run in a Docker container. Familiarity with Docker is required to properly set up production-ready UVAP component instances. This guide describes how to create and start a container running an UVAP component, and what UVAP-specific information is required to do so. However, any other features of Docker can be applied, and the resulting container may be handled as Docker makes it possible.

Docker images

To create a Docker container, the first thing needed is a Docker image, and to know the name/tag of that image. In the case of UVAP components, the name of the Docker image can be determined based on Git tags present in the UVAP Git repository. These Git tags each represent a given release version of UVAP, and each UVAP version have specific Docker images of UVAP components. Determining versions and corresponding Docker image names can be done as follows:

The following command prints the most recent release Git tag:

$ git -C "${UVAP_HOME}" tag \
    --list \
    --sort=-creatordate \
    --merged HEAD \
    'release/*' \
  | head -n1

Example output: release/uvap-2.680.1

The following command prints the most recent version postfix of the name of the Docker images of all UVAP components, by taking the most recent release Git tag without the release/ part:
```
$ git -C "${UVAP_HOME}" tag \
    --list \
    --sort=-creatordate \
    --merged HEAD \
    'release/*' \
  | head -n1 | cut -f2 -d/
```
Example output: uvap-2.680.1
If an older version of UVAP is needed instead of the most recent one, all the previous release tags can be found in the history of the Git repository (which can be viewed with thegit -C "${UVAP_HOME}" log command). Checking out to the desired version makes the above commands print information about that release.
Given a certain UVAP version, the Docker images of UVAP components look like ultinous/uvap:[component name]_[version], where:
- [component_name] is the name of the UVAP component - it is written in the Operation Guide of that component
- [version] is the above selected UVAP version
Example: the Docker image of the Detection Filter Microservice for the uvap-2.680.1 version is: ultinous/uvap:kafka_detection_filter_uvap-2.680.1.
Also, the Docker images of the actual most recent version of UVAP components are tagged with the _latest name postfix as well aside of the actual version name postfix.
Example: the Docker image of the Detection Filter Microservice for the actual most recent version is: ultinous/uvap:kafka_detection_filter_latest.

Docker containers

To run a Microservice, the next step after the Docker image is determined, is to create its Docker container. In the following section the set of UVAP-specific information is described, which needs to be given in the form of arguments for the docker container create command to create a container for the Microservice in general. Certain UVAP components, like MGR require more arguments to create the container. See the Operation Guide of the given Microservice for details.

For each running instance of a Microservice, their own Docker container is advised to be created as follows:

Let's refer to the name of the given UVAP component as [component].
Determine the Docker image of [component] as written above, and let's refer to it as [image].
Make sure, that the Docker image ([image]) of the container is present by downloading it from DockerHub:
```
$ docker pull [image]
```
A name is suggested to be given to the container, for example uvap_[component], or if multiple instances of the component will be created, then uvap_[component]_[instance number]. Let's refer to the given name as [name]. If a name is not specified for the container manually, then the Docker engine will generate a random name for it.

Some example names:
- for one Reidentifier instance: uvap_reid
- for two MGR instances: uvap_mgr_01 and uvap_mgr_02
It is advised not run the Microservice as root, but as a less privileged Unix user, for example a service user specifically created for the purpose of running UVAP services in general. Find a suitable Unix user (let's refer to it as [user]), and determine the Unix user ID and the Unix group ID of it with the id command line utility:
```
$ id -u [user] && id -g [user]
```
Example output:
```
1000
1000
```
Let's refer to the user ID (the first line of the output) as [uid] and to the group ID (the second line of the output) as [gid].
The configuration files of the Microservice have to be mounted into the container, as these are not present in the Docker image, but are present on the host computer. All Microservices require a properties configuration file, and aside that, some need a prototxt or a JSON configuration file as well. See the configuration documentation of the given Microservice in the Developer Guide for details. It is advised to create a separate subdirectory for the configuration files of the Microservice, and place all the necessary configuration files there, so mounting this one subdirectory will make all the configuration files present in the container. Let's refer to this directory on the host as [conf_on_host], and as [conf_in_container] in the container.
An environment variable has to be present, which specifies, where the main configuration file of the Microservice in the container will be. The name of this environment variable depends on the UVAP component, see it in the Microservice Configuration sections in the Developer Guide of the component itself - let's refer to it as [properties_env_var_name]. The value of this environment variable depends on where the configuration files are mounted into the container by the previous step. Let's refer to it as [properties_env_var_value].

Example: when creating a container for a Tracker Microservice, the name of the environment variable has to be KAFKA_TRACKER_MS_PROPERTY_FILE_PATHS, and the value of it can be e.g. /etc/uvap/tracker/01/tracker.properties, the only constraint is that this configuration file of properties of the Microservice has to be in the directory (or in a subdirectory of) [conf_in_container].
The Docker network in which the Microservice will take place has to be given. Any network is suitable, from which the configured Kafka cluster can be reached. Let's refer to this network as [network]. To quickly test, whether the configured Kafka cluster is reachable from [network] with the Kafka broker [broker], run a produce-and-consume kafkacat routine with the following test containers:
```
$ echo "test message" \
  | docker run -i --rm --net [network] edenhill/kafkacat:1.5.0 kafkacat \
    -b [broker] \
    -P \
    -t docker-network-test
$ docker run --rm --net [network] edenhill/kafkacat:1.5.0 kafkacat \
    -b [broker] \
    -C \
    -t docker-network-test \
    -o -1 -e
```
The expected output is the "test message" string, and that the end of the test topic is reached.
The configuration of the Microservice specifies some port numbers for certain network services of the Microservice. These ports may be needed to be opened on the host as well, not only in the container (as done by default). This can be done by publishing these ports. It is not necessary if the host network is chosen in the previous step. Let's refer to these port numbers as [port_1], [port_2], [...] and [port_n].

Creating and starting containers

Given all the above information, containers for all Microservices can be created with any of the Docker-supporting orchestration systems. A few examples are present in the following sections, but plenty of other solutions are out there, or if you want, you may create your own container-management system.

Docker CLI

Using the command-line-interface of the Docker engine is the most basic way to create a Docker container. It's suitable for practicing and smoke-testing UVAP, but less usable for managing a reasonably big production environment. A container can be created with the following example command:

$ docker container create \
  --name [name] \
  --user [uid]:[gid] \
  -v "[conf_on_host]:[conf_in_container]:ro" \
  --env [properties_env_var_name]="[properties_env_var_value]" \
  --net [network] \
  --publish [port_1]:[port_1] \
  --publish [port_2]:[port_2] \
  [...] \
  --publish [port_n]:[port_n] \
  [image]

After the container is created, the Microservice is not yet running, but can be started in two different ways:

Foreground mode: the console is blocked until the Microservice finishes running, and the output of the container in printed on the console. This is suitable for batch-processing data with the Microservice. To start the Microservice in foreground mode, run the following command:
```
$ docker container start --attach [name]
```
Background mode: the Microservice runs as a daemon, does not affect the console, and the output of the container is handled by Docker as the log of the container. This is suitable for live analysis. To start the Microservice in background mode, run the following command:
```
$ docker container start [name]
```

Docker Compose

Docker Compose is a more advanced tool than the Docker CLI. Using it makes your UVAP Microservices manageable in a more convenient way. Docker Compose makes it possible to describe all your containers in one or more configuration files (called Compose files), defining a whole application by doing so, and then use the Docker Compose CLI anytime to manage the application easily. An example Compose file for creating a container follows:

version: '2.3'
services:
  [name]:
    image: '[image]'
    user: '[uid]:[gid]'
    volumes:
      - type: bind
        source: '[conf_on_host]'
        target: '[conf_in_container]'
        read_only: true
    environment:
      - '[properties_env_var_name]=[properties_env_var_value]'
    ports:
      - [port_1]:[port_1]
      - [port_2]:[port_2]
      [...]
      - [port_n]:[port_n]
networks:
  default:
    external:
      name: [network]

After creating such Compose file(s), the Microservices can be started up in background mode with the following command:

docker-compose up --detach

As can be seen, issuing one CLI command is enough at a time, no matter how much Microservices there are defined.

Docker Compose not only helps a lot with creating and starting many containers, but also has several other useful features.

Other container tricks and tips

As mentioned earlier, creating such Docker containers to run UVAP Microservices results in completely conventional Docker containers, and any features of the Docker engine not mentioned above can be applied to these containers. That being said, all the above information are really just examples, and only point to the UVAP-specific good-to-knows.

In a production environment the following additional container configurations might be useful to take into consideration:

Setting a restart policy
Using a different logging engine, e.g. syslog
Limiting resource usages
Modifying health-check options
Managing namespaces
Overriding security options