Setting Adapter Ingestion Rules
  • 17 Jul 2022
  • 6 Minutes to read
  • Dark
    Light
  • PDF

Setting Adapter Ingestion Rules

  • Dark
    Light
  • PDF

Use Ingestion Rules to decide which entities to ingest from the data fetched from adapters.

Prerequisites

In order to use this feature:

  • You have to have the required permissions.

Using Ingestion Rules

  1. Click the image.png icon on the left navigation panel to open the Adapters page.

  2. Search for and select the relevant adapter. The Adapter Connections page opens displaying the list of connected connections.

  3. Click Advanced Settings. The Adapter Advanced Settings pane appears.

  4. Click Ingestion Rules Configuration. The Ingestion Rules Configuration tab is displayed.

  5. Toggle on Enable ingestion rules to configure which data to ingest from the adapter into the system; the Ingestion Rules pane is displayed.
    IngestionRulesPane

  6. If you want to add more than one rule, from the drop-down select the operator to apply between the rules:

    1. Choose OR to ingest an entity once one of the rules apply (this is the default),
    2. Choose AND to ingest an entity only if all of the rules apply.
  7. Enter the Ingestion Rule. Click the + button to add another Ingestion Rule.

  8. Once you have entered all the rules you need, click Save Config to save your settings. An entity is ingested according to the boolean logic of the operator you set.

Creating the Statement

To use Ingestion Rules you need to create a statement that describes the data to ingest.
The statements are built with the following syntax:

{device | user }.{flattened key path of the entity} {operator} {rule values} then {action} {post rule values}

To better understand this, it can be broken down to:
Pre-Ingestion:

  1. {device | user }.{flattened key path of the entity} {operator} {rule values}
  2. {device | user} {operator} {rule values}
    Post-Ingestion:
    then {action} {post rule values}
  • {device | user } - defines whether this rule will be applied on a device entity or user entity

  • {flattened key path of the entity} - absolute flattened path of the requested entity key. This is any column displayed on an asset page (including the various date columns), except True/False values. See Obtaining the Flattened Key Path of the Entity for an explanation of how to get the values.

    For example:

    • device.ad_da_source - will check the ad_da_source value under the device entity
    • device.network_interfaces.ips - will check the value of the IP addresses under the network_interfaces nested dictionary in the device entity.
  • {operator} - The condition operator for the statement; for example ‘==’ which is Equal. See Supported Condition Operators for a full list of conditions.

  • {rule values} - A single value or a list of values to check against the device | user entity value.

  • then {action} - the action to apply if the Ingestion Rule applies;. for example, ‘skip_field’ removes fields from the entity and does not insert them into the database. 'Remove_values' removes values in a list from the field.

  • {post rule values} - a single value or a list of values to input on the action. The values must be input in a comma separated list format, that is ["single_value"]

You can also build a statement for KeyExists.
The syntax for key_exists is {device | user} {operator} {pre_value}

Ingestion Rules for Dates
You can use a date in an Ingestion Rule

  1. Obtain the flattened key path of the entity for the date column as explained below.
  2. Use the relevant < or > operator as required.
  3. Add the date in the {rule values} field in the format date(yyyy-mm-dd), or use the following syntax to express x days ago date(now - xd).

Obtaining the Flattened Key Path of the Entity

The flattened key path of the entity is the name of the key path as it is displayed in the Axonius database.
To obtain this value:

  1. Go to the Devices or Users page.
  2. Select the adapter you are interested in.
  3. Use Edit Columns to make sure the columns you want are displayed in the Devices/Users table.
  4. Create a query that will result in the entity you are interested in, for instance the query below will display AD DC Source.

IngestionQuery

  1. From the resulting AQL query, copy the entity name and use it in the statement.

Ingestion2

Supported Condition Operators

The following operators for conditions are supported:

  • Equal\Not Equal (== | !=) - checks whether the value from the entity is equal\not equal to the rule value\s.

    For example:

    • device.ad_dc_source != "10.10.11.5". Means that the source value is not 10.10.11.5
    • device.ad_dc_source != ["10.10.11.0/24"] (will check if value equal to str(list))
  • Smaller than\Greater than (< |>) - (for dates only) checks whether the value from the entity is smaller or greater than the date in the rule value.

    For example:

    • user.termination_date > date(2202-05-31). Means that the user termination date is greater than May 31st 2021
    • user.termination_date < date(now - 5d) Means that the user termination date is less than 5 days ago
  • Value In\Value Not In (in | not_in) - checks whether the value from the entity is in\not in the list of rule values
    Examples

    device.ad_name in ["EC2AMAZ-71GIQSBBB", "EC2AMAZ-71GIQSORRRR"]
    
    device.ad_name not_in ["EC2AMAZ-71GIQSBBB", "EC2AMAZ-71GIQSORRRR", "EC2AMAZ-71GIQSO"]
    
  • IP address in or not in (in_net | not_in_net) - checks whether the IP address\network value from the entity is in\not in the rule_values IP network. This means that you can check if an IP\Subnet is in a CIDR. This is applicable both for IPv4 and for IPv6.

    Examples

    device.network_interfaces.ips in_net ["10.10.11.0/24"]
    
    device.network_interfaces.ips not_in_net ["192.168.11.0/24"]
    
  • Starts or Does not Start with (starts_with | not_starts_with) - check whether the value from the entity starts with a required pattern that was entered.
    Examples

    device.ad_usn_changed starts_with ["91"]
    
    device.ad_name starts_with ["EC2"]
    
  • Ends or Does not End With (ends_with | not_ends_with) - check whether the value from the entity ends with a required pattern that was entered.
    Examples

    user.mail ends_with ["@gmail.com", "@yahoo.com"]
    
    user.username not_ends_with ["_test"]
    
  • Key exists or does not exist (key_exists | key_not_exists) - check whether a key value exists or not.
    Examples

     device key_exists ["network_interfaces.ip", "ad_name"]
    
    user key_not_exists ["database"] 
    
  • Field equal\not equal (field_equal |field_not_equal) - checks whether the value from the value of one field in an entity is equal or not equal to the value of another field in that entity.

Examples

   device.hostname field_not_equal ["ad_name"].   

Means that the value in the device hostname field is not equal to the value in the ad_name field.

Action

skip_field - removes the required fields from the entity and does not insert them into the database.
Examples

device.ad_dc_source == "10.10.11.5" then skip_field ["network_interfaces.ips", "ad_name"]
device.ad_usn_changed starts_with ["91"] then skip_field ["ad_dc_source"]

Remove_values - removes values (from a list) from a field. This action can only be run on fields that contain lists of values. Please note, if you already fetched these values, they will still appear in the asset table for the amount of time set in Delete devices/users that have not been returned from the source in the last X hours. To only see the latest values, from the Query Wizard create a query that shows “from last fetch” = true.

Examples

user key_exists ["last_used_users"] then remove_values last_used_user starts_with ["man"]


Remove_items - removes items (from a list). Some fields contain lists of items, for example network_interfaces which contains sub fields such as MAC addresses, IP addresses, etc. If a condition is applied to one of the sub fields, then when it is true, the complete item is removed and is not ingested.

Example

device key_exists ["id"] then remove_items network_interfaces.ip4 in_net ["10.10.11.44"]


trim_suffix / trim_prefix - If a field has a certain suffix or prefix then remove the suffix or prefix from the field.

Examples

device key_exists ["id"] then trim_prefix hostname ["domain.local"]
user key_exists ["id"] then trim_suffix hostname [".com"]


Save Config to save the configuration.

Implementing the Ingestion Rules

From the Add Connection page click Save and Fetch
The system then performs a fetch. Each entity brought from the adapter goes through the Ingestion Rule process and sees that it meets one of the conditions. If it does the entity will be ingested into Axonius, otherwise not. After the Fetch using Ingestion Rules is completed, open the Devices or Users page in order to see your input. If required, you can fine tune the rules.



Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.