Ensuring uninterrupted collector operation

An uninterrupted event stream from the event source to KUMA is important for protecting the network infrastructure. Continuity can be ensured though automatic forwarding of the event stream to a larger number of collectors:

• On the KUMA side, two or more identical collectors must be installed.
• On the event source side, you must configure control of event streams between collectors using third-party server load management tools, such as rsyslog or nginx.

With this configuration of the collectors in place, no incoming events will be lost if the collector server is unavailable for any reason.

Please keep in mind that when the event stream switches between collectors, each collector will aggregate events separately.

If the KUMA collector fails to start, and its log includes the "panic: runtime error: slice bounds out of range [8:0]" error:

1. Stop the collector.

   sudo systemctl stop kuma-collector-<collector ID>

2. Delete the DNS enrichment cache files.

   sudo rm -rf /opt/kaspersky/kuma/collector/<collector ID>/cache/enrichment/DNS-*

3. Delete the event cache files (disk buffer). Run the command only if you can afford to jettison the events in the disk buffers of the collector.

   sudo rm -rf /opt/kaspersky/kuma/collector/<collector ID>/buffers/*

4. Start the collector service.

   sudo systemctl start kuma-collector-<collector ID>

Event stream control using rsyslog

To enable rsyslog event stream control on the event source server:

1. Create two or more identical collectors that you want to use to ensure uninterrupted reception of events.
2. Install rsyslog on the event source server (see the rsyslog documentation).
3. Add rules for forwarding the event stream between collectors to the configuration file /etc/rsyslog.conf:

   *. * @@ <main collector server FQDN>: <port for incoming events> $ActionExecOnlyWhenPreviousIsSuspended on *. * @@ <backup collector server FQDN>: <port for incoming events> $ActionExecOnlyWhenPreviousIsSuspended off

   Example configuration file

   Example configuration file specifying one primary and two backup collectors. The collectors are configured to receive events on TCP port 5140.

   *.* @@kuma-collector-01.example.com:5140 $ActionExecOnlyWhenPreviousIsSuspended on & @@kuma-collector-02.example.com:5140 & @@kuma-collector-03.example.com:5140 $ActionExecOnlyWhenPreviousIsSuspended off

4. Restart rsyslog by running the following command:

   systemctl restart rsyslog.

Event stream control is now enabled on the event source server.

Event stream control using nginx

To control event stream using nginx, you need to create and configure a ngnix server to receive events from the event source and then forward these to collectors.

To enable nginx event stream control on the event source server:

1. Create two or more identical collectors that you want to use to ensure uninterrupted reception of events.
2. Install nginx on the server intended for event stream control.
   • Installation command in Oracle Linux 8.6:

     $sudo dnf install nginx

   • Installation command in Ubuntu 20.4:

     $sudo apt-get install nginx

     When installing from sources, you must compile with the parameter -with-stream option:
     $ sudo ./configure -with-stream -without-http_rewrite_module -without-http_gzip_module

3. On the nginx server, add the stream module to the nginx.conf configuration file that contains the rules for forwarding the stream of events between collectors.

   Example stream module

   Example module in which event stream is distributed between the collectors kuma-collector-01.example.com and kuma-collector-02.example.com, which receive events via TCP on port 5140 and via UDP on port 5141. Balancing uses the nginx.example.com ngnix server.

   stream {  upstream syslog_tcp { server kuma-collector-1.example.com:5140; server kuma-collector-2.example.com:5140; } upstream syslog_udp { server kuma-collector-1.example.com:5141; server kuma-collector-2.example.com:5141; }  server { listen nginx.example.com:5140; proxy_pass syslog_tcp; } server { listen nginx.example.com:5141 udp; proxy_pass syslog_udp; proxy_responses 0; } }  worker_rlimit_nofile 1000000; events { worker_connections 20000; } # worker_rlimit_nofile is the limit on the number of open files (RLIMIT_NOFILE) for workers. This is used to raise the limit without restarting the main process. # worker_connections is the maximum number of connections that a worker can open simultaneously.

4. Restart nginx by running the following command:

   systemctl restart nginx

5. On the event source server, forward events to the ngnix server.

Event stream control is now enabled on the event source server.

Nginx Plus may be required to fine-tune balancing, but certain balancing methods, such as Round Robin and Least Connections, are available in the base version of ngnix.

For more details on configuring nginx, please refer to the nginx documentation.