Using external ClickHouse DBMS

In addition to the ClickHouse DBMS, which is a component of Kaspersky Container Security and is included in the distribution kit, the solution can also work with the resources of the external ClickHouse DBMS. To do this, you must do the following:

• Create a database for Kaspersky Container Security, add and configure users, and define a disk policy if different disks are used for long-term and short-term data storage.
• Specify the required variables in the values.yaml configuration file for the external ClickHouse DBMS.

Kaspersky Container Security works with ClickHouse 22.6 or later.

Creating a database for Kaspersky Container Security

To create a database for Kaspersky Container Security,

In ClickHouse on your workstation, run the following command:

CREATE DATABASE IF NOT EXISTS kcs

where kcs is the name of the database for Kaspersky Container Security.

To configure the settings of the created database for Kaspersky Container Security:

1. Add users and define their authorization method. To do this, you must do the following:
   1. Add the following users:
      • a user with rights to read data received by the Kaspersky Container Security core (reader).

        <roles>

        <kcs_reader_role>

        <grants>

        <query>GRANT SELECT ON kcs.*</query>

        </grants>

        </kcs_reader_role>

      • a user with rights to write data from external agent requests (writer).

        <roles>

        <kcs_writer_role>

        <grants>

        <query>GRANT CREATE TABLE, INSERT, ALTER, UPDATE ON kcs.*</query>

        <query>GRANT SELECT (source_ip, source_port, source_alias, dest_ip, dest_port, dest_alias, protocol, severity, action, event_time, count, type) ON kcs.node_agent_events</query>

        </grants>

        </kcs_writer_role>

   2. Specify the user authorization method: with a password or with a certificate.

      Example of configuring users with password authentication

      <clickhouse>

      ...

      <kcsuser-write>

      <password>*********</password>

      <networks>

      <ip>::/0</ip>

      </networks>

      ...

      <grants>

      <query>GRANT kcs_writer_role</query>

      </grants>

      </kcsuser-write>

      <kcsuser-read>

      <password>*********</password>

      <networks>

      <ip>::/0</ip>

      </networks>

      ...

      <grants>

      <query>GRANT kcs_reader_role</query>

      </grants>

      </kcsuser-read>

      ...

      <roles>

      <kcs_reader_role>

      <grants>

      <query>GRANT SELECT ON kcs.*</query>

      </grants>

      </kcs_reader_role>

      <kcs_writer_role>

      <grants>

      <query>GRANT CREATE TABLE, INSERT, ALTER, UPDATE ON kcs.*</query>

      <query>GRANT SELECT (source_ip, source_port, source_alias, dest_ip, dest_port, dest_alias, protocol, severity, action, event_time, count, type) ON kcs.node_agent_events</query>

      </grants>

      </kcs_writer_role>

      ...

      </roles>

      ...

      </clickhouse>

      Example of configuring users with certificate authentication

      <clickhouse>

      ...

      <kcsuser-write>

      <ssl_certificates>

      <common_name>kcsuser-write</common_name>

      </ssl_certificates>

      <networks>

      <ip>::/0</ip>

      </networks>

      ...

      <grants>

      <query>GRANT kcs_writer_role</query>

      </grants>

      </kcsuser-write>

      <kcsuser-read>

      <ssl_certificates>

      <common_name>kcsuser-read</common_name>

      </ssl_certificates>

      <networks>

      <ip>::/0</ip>

      </networks>

      ...

      <grants>

      <query>GRANT kcs_reader_role</query>

      </grants>

      </kcsuser-read>

      ...

      <roles>

      <kcs_reader_role>

      <grants>

      <query>GRANT SELECT ON kcs.*</query>

      </grants>

      </kcs_reader_role>

      <kcs_writer_role>

      <grants>

      <query>GRANT CREATE TABLE, INSERT, ALTER, UPDATE ON kcs.*</query>

      <query>GRANT SELECT (source_ip, source_port, source_alias, dest_ip, dest_port, dest_alias, protocol, severity, action, event_time, count, type) ON kcs.node_agent_events</query>

      </grants>

      </kcs_writer_role>

      ...

      </roles>

      ...

      </clickhouse>

2. Specify disks for short-term and long-term data storage. When working with ClickHouse, Kaspersky Container Security can store large amounts of data with various retention periods. By default, the major part of events is stored for a maximum of 30 minutes, whereas information about incidents is stored for up to 90 days. Since event recording requires a considerable resources to ensure high recording speed and disk space provision, it is recommended to use different disks for short-term and long-term data storage.

   Example of configuring data storage settings

   <clickhouse>

   ...

   <storage_configuration>

   <disks>

   <kcs_disk_hot>

   <path>/etc/clickhouse/hot/</path>

   </kcs_disk_hot>

   <kcs_disk_cold>

   <path>/etc/clickhouse/cold/</path>

   </kcs_disk_cold>

   </disks>

   <policies>

   <kcs_default>

   <volumes>

   <default>

   <disk>kcs_disk_hot</disk>

   </default>

   <cold>

   <disk>kcs_disk_cold</disk>

   </cold>

   </volumes>

   </kcs_default>

   </policies>

   </storage_configuration>

   ...

   </clickhouse>

Page top

Configuring the external ClickHouse DBMS settings

To configure the Kaspersky Container Security settings to use the external ClickHouse DBMS:

1. In the values.yaml configuration file, specify that the solution uses the external ClickHouse DBMS:

   default:

   kcs-clickhouse:

   external: true

2. Specify the variables for using the external ClickHouse DBMS:

   configmap:

   infraconfig:

   type: fromEnvs

   envs:

   ...<ariables for using the external ClickHouse DBMS>

   In this section you must specify the following variables:

   • EXT_CLICKHOUSE_PROTOCOL is the protocol for connection to the external ClickHouse DBMS.
   • EXT_CLICKHOUSE_HOST is the host for connection to the external ClickHouse DBMS.
   • EXT_CLICKHOUSE_PORT is the port for connection to the external ClickHouse DBMS.
   • EXT_CLICKHOUSE_DB_NAME is the name of the database prepared for using with Kaspersky Container Security.
   • EXT_CLICKHOUSE_COLD_STORAGE_NAME is the name of the disk, where ClickHouse will long term store data about incidents.
   • EXT_CLICKHOUSE_STORAGE_POLICY_NAME is the name of the data storage policy according to which ClickHouse will transfer the data about incidents to the disk for long-term storage.

     If you use the same disk for short-term and long-term data storage, the EXT_CLICKHOUSE_COLD_STORAGE_NAME and EXT_CLICKHOUSE_STORAGE_POLICY_NAME values are not specified.

   • EXT_CLICKHOUSE_SSL_AUTH is the variable for SSL authorization of ClickHouse users. If the true value is specified, authorization is performed without passwords using client certificates.

     If TLS_INTERNAL is false, EXT_CLICKHOUSE_SSL_AUTH must also be false.

   • EXT_CLICKHOUSE_ROOT_CA_PATH is the path to the CA certificate, which is specified if the https protocol is used to connect to ClickHouse ( EXT_CLICKHOUSE_PROTOCOL: https). You can specify the path in one of the following ways:
     • Put the ClickHouse CA certificate in the directory specified by the path. In this case, you must uncomment the secret.cert-kcs-clickhouse-ca block.
     • Use Vault to store certificate data. In this case, you must uncomment the cert-kcs-clickhouse-ca block in the vault.certificate section.
3. Specify values of secrets for using the external ClickHouse DBMS:

   configmap:

   secret:

   infracreds:

   type: fromEnvs

   envs:

   ...<secrets for using the external ClickHouse DBMS>

   In this section you must specify the following:

   • EXT_CLICKHOUSE_WRITE_USER is the name of a user with permissions to write created for using with Kaspersky Container Security.
   • CLICKHOUSE_WRITE_PASSWORD is the password of a user with permissions to write created for using with Kaspersky Container Security.
   • EXT_CLICKHOUSE_READ_USER is the name of a user with read rights prepared for use with Kaspersky Container Security.
   • CLICKHOUSE_READ_PASSWORD is the password of a user with permissions to read created for using with Kaspersky Container Security.

     CLICKHOUSE_READ_PASSWORD and CLICKHOUSE_WRITE_PASSWORD are not used if EXT_CLICKHOUSE_SSL_AUTH is set to true.

   Usernames and passwords can also be specified using the Vault secret storage.

   Example of configuring the external ClickHouse DBMS settings

   kcs-clickhouse:

   external: true

   persistent: true

   ...

   configmap:

   infraconfig:

   type: fromEnvs

   envs:

   ...

   EXT_CLICKHOUSE_PROTOCOL: https

   EXT_CLICKHOUSE_HOST: clickhouse.ns.svc.cluster.local

   EXT_CLICKHOUSE_PORT: 8443

   EXT_CLICKHOUSE_DB_NAME: kcs

   EXT_CLICKHOUSE_COLD_STORAGE_NAME: cold

   EXT_CLICKHOUSE_STORAGE_POLICY_NAME: kcs_default

   EXT_CLICKHOUSE_SSL_AUTH: false

   EXT_CLICKHOUSE_ROOT_CA_PATH: /etc/ssl/certs/kcs-clickhouse-ca.crt

   ...

   secret:

   ...

   infracreds:

   type: fromEnvs

   envs:

   ...

   EXT_CLICKHOUSE_WRITE_USER: kcsuser-write

   EXT_CLICKHOUSE_READ_USER: kcsuser-read

   CLICKHOUSE_WRITE_PASSWORD: **************

   CLICKHOUSE_READ_PASSWORD: ***********

   ...

   When using Vault:

   vault:

   ...

   secret:

   type: managedByVault

   ...

   EXT_CLICKHOUSE_WRITE_USER: kv/secret/kcs/clickhouse@EXT_CLICKHOUSE_WRITE_USER

   EXT_CLICKHOUSE_READ_USER: kv/secret/kcs/clickhouse@EXT_CLICKHOUSE_READ_USER

   CLICKHOUSE_WRITE_PASSWORD: kv/secret/kcs/clickhouse@CLICKHOUSE_WRITE_PASSWORD

   CLICKHOUSE_READ_PASSWORD: kv/secret/kcs/clickhouse@CLICKHOUSE_READ_PASSWORD

   ...