Normalizers

Normalizer resources are used to convert raw events of various formats so that they conform to the KUMA event data model. This turns them into normalized events that can be processed by other KUMA resources and services.

A normalizer resource consists of the main normalizer and optional extra normalizers. Data is transmitted through a tree-like structure of normalizers depending on the defined conditions, which lets you configure complex logic for processing events.

A normalizer resource is created in several steps:

1. Creating the main normalizer

   The main normalizer is created by using the Add event parsing button. Entry of normalizer settings is finished by clicking OK.

   The main normalizer that you created will be displayed as a dark circle. Clicking on the circle will open the normalizer options for editing. When you hover over the circle, a plus sign is displayed. Click it to add more normalizers.

2. Creating conditions for using an extra normalizer

   Clicking on the normalizer plus sign opens the Add normalizer to normalization scheme window in which you can specify the conditions that will cause data to be forwarded to the new normalizer.

3. Creating an extra normalizer

   When the previous step is finished, a window will open for creating an extra normalizer. Entry of normalizer settings is finished by clicking OK.

   The extra normalizer you created is displayed as a dark block that indicates the conditions under which this normalizer will be used (see step 2). The conditions can be changed by moving your mouse cursor over the extra normalizer and clicking the button showing the pencil image.

   If you hover the mouse pointer over the extra normalizer, a plus button appears, which you can use to create a new extra normalizer. To delete a normalizer, use the button with the trash icon.

   If you need to create more normalizers, repeat steps 2 and 3.

4. Completing creation of a normalizer resource

   Normalizer resource creation is finished by clicking the Save button.

Normalizer settings

The normalizer window contains two tabs: Normalization scheme and Enrichment.

Normalization scheme

This tab is used to specify the main settings of the normalizer and to define the rules for converting events into KUMA format.

Available settings:

• Name (required)—the name of the normalizer. Must contain from 1 to 128 Unicode characters. The name of the main normalizer will be used as the name of the normalizer resource.
• Tenant (required)—name of the tenant that owns the resource.

  This setting is not available for extra normalizers.

• Parsing method (required)—drop-down list for selecting the type of incoming events. Depending on your choice, you can use the preconfigured rules for matching event fields or set your own rules. When you select some parsing methods, additional parameter fields required for filling in may become available.

  Available parsing methods:

  • json

    This parsing method is used to process JSON data.

  • cef

    This parsing method is used to process CEF data.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

  • regexp

    This parsing method is used to create custom rules for processing JSON data.

    In the Normalization parameter block field, add a regular expression (RE2 syntax) with named capture groups. The name of a group and its value will be interpreted as the field and the value of the raw event, which can be converted into an event field in KUMA format.

    To add event handling rules:

    1. Copy an example of the data you want to process to the Event examples field. This is an optional but recommended step.
    2. In the Normalization parameter block field add a regular expression with named capture groups in RE2 syntax, for example "(?P<name>regexp)".

       You can add multiple regular expressions by using the Add regular expression button. If you need to remove the regular expression, use the button.

    3. Click the Copy field names to the mapping table button.

       Capture group names are displayed in the KUMA field column of the Mapping table. Now you can select the corresponding KUMA field in the column next to each capture group. Otherwise, if you named the capture groups in accordance with the CEF format, you can use the automatic CEF mapping by selecting the Use CEF syntax for normalization check box.

    Event handling rules were added.

  • syslog

    This parsing method is used to process data in syslog format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

  • csv

    This parsing method is used to create custom rules for processing CSV data.

    When choosing this method, you must specify one of the possible delimiters for values in the Delimiter field:

    • \ n (used by default)
    • \t
    • \0
  • kv

    This parsing method is used to process data in key-value pair format.

    If you select this method, you must provide values in the following required fields:

    • Pair delimiter—specify a character that will serve as a delimiter for key-value pairs. By default, the line feed character is used, although you can specify any one-character (1 byte) value, provided that the character is not the same as the value delimiter.
    • Value delimiter—specify a character that will serve as a delimiter between the key and the value. By default, the "=" character is used, however, you can specify any one-character (1 byte) value, provided that the character is not the same as the delimiter of key-value pairs.
  • xml

    This parsing method is used to process XML data.

    When this method is selected in the parameter block XML Attributes you can specify the key attributes to be extracted from tags. If an XML structure has several attributes with different values in the same tag, you can indicate the necessary value by specifying its key in the Source column of the Mapping table.

    To add key XML attributes,

    Click the Add field button, and in the window that appears, specify the path to the required attribute.

    You can add more than one attribute. Attributes can be removed one at a time using the cross icon or all at once using the Reset button.

    If XML key attributes are not specified, then in the course of field mapping the unique path to the XML value will be represented by a sequence of tags.

  • netflow5

    This parsing method is used to process data in the NetFlow v5 format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

    In mapping rules, the protocol type for netflow is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format on the Enrichment normalizer tab, you should create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

  • netflow9

    This parsing method is used to process data in the NetFlow v9 format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

    In mapping rules, the protocol type for netflow is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format on the Enrichment normalizer tab, you should create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

  • ipfix

    This parsing method is used to process IPFIX data.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

    In mapping rules, the protocol type for netflow is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format on the Enrichment normalizer tab, you should create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

  • sql

    This parsing method is used to process SQL data.

• Keep raw log (required)—in this drop-down list, you can indicate whether you need to store the original raw event in the newly created normalized event. Available values:
  • Never—do not save the raw event This is the default setting.
  • Only errors—save the raw event in the Raw field of the normalized event if errors occurred when parsing it. This value is convenient to use when debugging a service: in this case, every time an event has a non-empty Raw field, you know there was a problem.

    If fields containing the names *Address or *Date* do not comply with normalization rules, these fields are ignored. No normalization error will occur, and the values of the fields will not show up in the Raw field of the normalized event even if Keep raw log → Only errors was indicated.

  • Always—always save the raw event in the Raw field of the normalized event.

  This setting is not available for extra normalizers.

• Save extra fields (required)—in this drop-down list, you can choose whether you need to save fields of the original event in the normalized event if no mapping rules have been configured for them (see below). The data is stored in the Extra event field. By default, fields are not saved.
• Description—up to 256 Unicode characters describing the resource.

  This setting is not available for extra normalizers.

• Event examples—in this field, you can provide an example of data that you want to process. Event examples can also be loaded from a TSV, CSV, or TXT file by using the Load from file button.
• Mapping settings block—here you can configure mapping of original event fields to fields of the event in KUMA format:
  • Source—column for the names of the original event fields that you want to convert into KUMA event fields.

    Clicking the button next to the field names in the Source column opens the Conversion window, in which you can use the Add conversion button to create rules for modifying the original data before they are written to the KUMA event fields.

    Available conversions

    Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

    Available conversions:

    • lower—is used to make all characters of the value lowercase
    • upper—is used to make all characters of the value uppercase
    • regexp—is used to apply a RE2 regular expression to the value. When this conversion type is selected, the field appears where regular expression should be added.
    • substring—is used to delete characters in the position range specified in the Start and the End fields. These fields appear when this conversion type is selected.
    • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
      • Replace chars—in this field you can specify the character sequence that should be replaced.
      • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
    • trim is used to remove the characters specified in the Chars field from trailing positions of the value. The field appears when this type of conversion is selected.
    • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
    • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
    • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
      • Expression—in this field you can specify the regular expression which results that should be replaced.
      • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
  • KUMA field—drop-down list for selecting the required fields of KUMA events. You can search for fields by entering their names in the field.
  • Label—in this column, you can add a unique custom label to event fields that begin with DeviceCustom*.

  New table rows can be added by using the Add row button. Rows can be deleted individually using the button or all at once using the Clear all button.

  If you have loaded data into the Event examples field, the table will have an Examples column containing examples of values carried over from the raw event field to the KUMA event field.

Enrichment

This tab is used to add additional data to fields of a normalized event by using enrichment rules similar to the rules in enrichment rule resources. These enrichment rules are stored in the normalizer resource where they were created. There can be more than one enrichment rule. Enrichments are created by using the Add enrichment button.

Settings available in the enrichment rule settings block:

• Source kind (required)—drop-down list for selecting the type of enrichment. Depending on the selected type, you may see advanced settings that will also need to be completed.

  Available Enrichment rule source types:

  • constant

    This type of enrichment is used when a constant needs to be added to an event field.

    When choosing this type, you must specify the value to add to the event field in the Constant field. The value should not be longer than 255 Unicode characters. If you leave this field blank, the existing event field value will be cleared.

  • dictionary

    This type of enrichment is used if you need to add a value from dictionary.

    When this type is selected in the Dictionary name drop-down list, you must select the dictionary that will provide the values. In the Key fields settings block, you must use the Add field button to select the event fields whose values will be used for dictionary entry selection.

  • event

    This type of enrichment is used when you need to write a value from another event field to the current event field.

    When this type is selected in the Source field drop-down list, you must select the event field from where the value will be copied to the target field. Clicking the button opens the Conversion window in which you can, using the Add conversion button, create rules for modifying the original data before writing them to the KUMA event fields.

    Available conversions

    Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

    Available conversions:

    • lower—is used to make all characters of the value lowercase
    • upper—is used to make all characters of the value uppercase
    • regexp—is used to apply a RE2 regular expression to the value. When this conversion type is selected, the field appears where regular expression should be added.
    • substring—is used to delete characters in the position range specified in the Start and the End fields. These fields appear when this conversion type is selected.
    • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
      • Replace chars—in this field you can specify the character sequence that should be replaced.
      • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
    • trim is used to remove the characters specified in the Chars field from trailing positions of the value. The field appears when this type of conversion is selected.
    • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
    • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
    • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
      • Expression—in this field you can specify the regular expression which results that should be replaced.
      • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
  • template

    This type of enrichment is used when you need to write a value obtained by processing Go templates into the event field.

    When this type is selected, a Go template must be specified in the Template field.

    Event field names are passed in the {{.EventField}} format, where EventField is the name of the event field from which the value must be passed to the script.

    Example: Attack on {{.DestinationAddress}} from {{.SourceAddress}}

• Target field (required)—drop-down list for selecting the KUMA event field that should receive the data.

Condition for forwarding data to an extra normalizer

The Add normalizer to normalization scheme window is used to specify the conditions under which the data will be sent to an extra normalizer.

Available settings:

• Fields to pass into normalizer—used to indicate event fields in case you want to send only events with specific fields to the extra normalizer. Leave the field empty if you want to send all data to the extra normalizer.
• Use normalizer for events with specific event field values—used to indicate event fields if you want the extra normalizer to receive only events in which specific values are assigned to certain fields. The value is specified in the Condition value field.

  The data processed by these conditions can be preconverted by clicking the button. This opens the Conversion window, in which you can use the Add conversion button to create rules for modifying the original data before it is written to the KUMA event fields.

  Available conversions

  Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

  Available conversions:

  • lower—is used to make all characters of the value lowercase
  • upper—is used to make all characters of the value uppercase
  • regexp—is used to apply a RE2 regular expression to the value. When this conversion type is selected, the field appears where regular expression should be added.
  • substring—is used to delete characters in the position range specified in the Start and the End fields. These fields appear when this conversion type is selected.
  • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
    • Replace chars—in this field you can specify the character sequence that should be replaced.
    • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
  • trim is used to remove the characters specified in the Chars field from trailing positions of the value. The field appears when this type of conversion is selected.
  • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
  • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
  • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
    • Expression—in this field you can specify the regular expression which results that should be replaced.
    • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.

Preset normalizers

The normalizers listed in the table below are included in the KUMA kit.