Additional ClickHouse clusters

More than one ClickHouse storage cluster can be added to KUMA. The process of adding a new ClickHouse cluster consists of several steps:

1. Preparing the target machine

   On the target machine, specify the FQDN of the server with the KUMA Core in the /etc/hosts directory.

2. Preparing cluster inventory file

   Depending on the type of installation – local or remote – the inventory file is prepared on the target machine or on KUMA Core machine.

3. Installing additional cluster
4. Creating a storage

When creating storage cluster nodes, verify the network connectivity of the system and open the ports used by the components.

Preparing cluster inventory file

Installation, update, and removal of KUMA components is performed from the directory containing the unpacked installer by using the Ansible tool and the user-created inventory file containing a list of the hosts of KUMA components and other parameters. The inventory file is in the YAML format.

To create an inventory file:

1. Go to the KUMA unarchived installer directory by executing the following command:

   cd kuma-ansible-installer

2. Create an inventory file by copying additional-storage-cluster.inventory.yml.template:

   cp additional-storage-cluster.inventory.yml.template additional-storage-cluster.inventory.yml

3. Edit the inventory file parameters:
   • If you want demonstration services to be created during the installation, set the deploy_example_services parameter value to true.

     deploy_example_services: true

     Demonstration services can only be created during the initial installation of KUMA. When updating the system using the same inventory file, no demonstration services will be created.

   • If the machines are not registered in your organization's DNS zone, set the generate_etc_hosts parameter to true, and for each machine in the inventory, replace the ip (0.0.0.0) parameter values with the actual IP addresses.

     generate_etc_hosts: true

     When using this parameter, the installer will automatically add the IP addresses of the machines from the inventory file to the /etc/hosts files on the machines where KUMA components are installed.

   • Set the ansible_connection parameter:
     • Specify local if you want to install the cluster locally:

       ansible_connection: local

     • Specify ssh if you want to install the cluster remotely, from a server with KUMA Core installed:

       ansible_connection: ssh

4. In the storage section, specify the full names of the domains of the hosts on which you want to install the cluster nodes in the inventory file. If the machines are not registered in the DNS zone of your organization, replace the parameter values ip (0.0.0.0) with the actual IP addresses.

   Storage components: clusters, shards, replicas, and keepers.

   A ClickHouse cluster is a logical group of machines that possess all accumulated normalized KUMA events. It consists of one or more logical shards.

   A shard is a logical group of machines that possess a specific portion of all normalized events accumulated in the cluster. It consists of one or more replicas. Increasing the number of shards lets you do the following:

   • Accumulate more events by increasing the total number of servers and disk space.
   • Absorb a larger stream of events by distributing the load associated with an influx of new events.
   • Reduce the time taken to search for events by distributing search areas among multiple machines.

   A replica is a machine that is a member of the logical shard and possesses a copy of the data of this shard. If there are multiple replicas, there are multiple copies (data is replicated). Increasing the number of replicas lets you do the following:

   • Improve fault tolerance.
   • Distribute the total load related to data searches among multiple machines (although it's best to increase the number of shards for this purpose).

   A keeper is a machine that coordinates data replication at the cluster level. There must be at least one machine with this role for the entire cluster. The recommended number of the machines with this role is 3. The number of machines involved in coordinating replication must be an odd number. The keeper and replica roles can be combined in one machine.

   Each machine in the storage section can have the following parameter combinations:

   • shard + replica + keeper
   • shard + replica
   • keeper

   If the shard and replica parameters are specified, the machine is a part of a cluster and helps accumulate and search for normalized KUMA events. If the keeper parameter is additionally specified, the machine also helps coordinate data replication at the cluster-wide level.

   If only keeper is specified, the machine will not accumulate normalized events, but it will participate in coordinating data replication at the cluster-wide level. The keeper parameter values must be unique.

   If several replicas are defined within the same shard, the value of the replica parameter must be unique within this shard.

The inventory file is created. It can be used to create a ClickHouse cluster.

It is recommended that you not remove the inventory file after installing KUMA:

• If you change this file (for example, add information about a new server for the collector), you can reuse it to update the system with a new component.
• You can use this inventory file to delete KUMA.

Installing additional cluster

KUMA is installed using the Ansible tool and the YML inventory file.

To install an additional KUMA cluster:

1. On a preconfigured target machine or a machine with the KUMA Core installed (depending on the ansible_connection setting), open the folder with an unpacked installer file.
2. Launch the installer by executing the following command:

   PYTHONPATH="$(pwd)/ansible/site-packages:${PYTHONPATH}" python3 ./ansible/bin/ansible-playbook -i additional-storage-cluster.inventory.yml additional-storage-cluster.playbook.yml

The additional ClickHouse cluster is installed. To write data to the cluster using KUMA, you need to create a storage.

Deleting a cluster

To delete a ClickHouse cluster,

execute the following command:

systemctl stop kuma-storage-<storage ID> && systemctl stop kuma-clickhouse && systemctl disable kuma-storage-<storage ID> && systemctl disable kuma-clickhouse && rm -rf /usr/lib/systemd/system/kuma-storage-<storage ID>.service && rm -rf /usr/lib/systemd/system/kuma-clickhouse.service && systemctl daemon-reload && rm -rf /opt/kaspersky/kuma

The KUMA storage and ClickHouse cluster services are stopped and deleted.