Setting Adapter Ingestion Rules
  • 27 Mar 2024
  • 11 Minutes to read
  • Dark
    Light
  • PDF

Setting Adapter Ingestion Rules

  • Dark
    Light
  • PDF

Article Summary

Use Ingestion Rules to decide which entities to ingest from the data fetched from adapters.

Prerequisites

In order to use this feature:

  • You have to have the required permissions.

Using Ingestion Rules

  1. Click the image.png icon on the left navigation panel to open the Adapters page.
  2. Search for and select the relevant adapter. The Adapter Profile page opens displaying the list of connected connections.
  3. Under Advanced Settings, choose Ingestion Rules Configuration. The Ingestion Rules Configuration page is displayed.

IngestionPaneNEw

  1. Toggle on Enable ingestion rules to configure which data to ingest from the adapter into the system.

IngestionRules1

Ingestion Rules Structure:
Ingestion rules work using boolean operator conditions. Ingestion rules support multiple statements. These are either handled using OR or AND as follows:

  • Choose OR to ingest an entity once one of the rules applies (this is the default).
  • Choose AND to ingest an entity only if all of the rules apply.

Ingestiondiagram

  1. If you want to add more than one rule, from the drop-down select the operator to apply between the rules OR or AND

  2. Enter one Ingestion Rule statement in the input box. Click the + button to add another Ingestion Rule.

  3. Once you have entered all the rules you need, click Save Config to save your settings. An entity is ingested according to the boolean logic of the operator you set.

Creating the Ingestion Rule Statement

To use Ingestion Rules you need to create a statement that describes the data to ingest. A statement is made up of a primary statement and an optional post action.

Primary Statement

An entity is considered for ingestion if the pre-ingestion statement is true. A pre-ingestion statement can be built in one of two ways:

  • {device | user }.{flattened key path of the entity} {operator} {rule values}
  • {device | user } {operator} {rule values}

Post Action

Post actions are optional actions performed on the entity before it is ingested into Axonius. Common use cases for post statements are to filter out poor data or to drop PII.

The syntax is as follows:
{primary statement} then {action} {post rule values}

Ingestion Rules Components

Entities

Defines whether this rule will be applied on a device entity or user entity

Flattened Key Path

The flattened key path of the entity is the name of the key path as presented in the Axonius database. The following example can explain how to obtain this value. “I want to only ingest device from the cmdb_ci_server table in ServiceNow”. You can translate this to a Query Wizard statement.

IngestionFlattenedEg

This statement is automatically translated into an Axonius statement in the Devices search bar.

("adapters_data.service_now_adapter.class_name" == "cmdb_ci_server")

Because Ingestion rules are already being performed at an adapter level, we can ignore adapters_data.service_now_adapter.

That leaves the flattened key path to simply be class_name

IngestionPathMain

When working with complex objects such as network interface IP addresses, you need to copy the complete complex object, for instance network_interfaces.ips.

IngestionMore

Operators Supported

The following operators are supported:

Equal\Not Equal (== | !=) - checks whether the value from the entity is equal\not equal to the rule value\s.

device.class_name == "cmdb_ci_computer"
device.class_name != "cmdb_ci_computer"

Do not use equals with lists, as they will be converted to strings

 'device.ad_dc_source != ["10.10.11.0/24"]` ( checks if value equal to str(list)

Dates Smaller Than\Greater Than (< |>)

Checks whether the value from the entity is smaller or greater than the date in the rule value.

user.termination_date > date(2202-05-31)

Ingest users whose termination date is greater than May 31st 2021

user.last_logon > date(now-10d)

Ingest all users with last_login in the last 10 days

Note: Dates currently do not support greater than or equal to >= OR less than or equal to <=

Numbers

Checks whether the value from the entity is greater than or less than the value provided

device.confidence_level < 65 then skip_field ["os.type", "os.distribution", "os.type_distribution", "os.os_str"]

If confidence level for devices is below 65, then do not ingest OS level information. This rule all excludes all devices with a confidence level >=65. Using this rule in combination with device key_exists ["id"] will ensure that all devices are still ingested from that adapter.

device.policies_compliance_count >= 1

ingests all devices with at least one compliance policy present.

Value In\Value Not In (in | not_in)

Checks whether the value from the entity is in\not in the list of rule values

device.ad_name in ["EC2AMAZ-71GIQSBBB", "EC2AMAZ-71GIQSORRRR"]
device.ad_name not_in ["EC2AMAZ-71GIQSBBB", "EC2AMAZ-71GIQSORRRR", "EC2AMAZ-71GIQSO"]

IP Address In or Not In (in_net | not_in_net)

Checks whether the IP address\network value from the entity is in\not in the rule_values IP network. This means that you can check if an IP\Subnet is in a CIDR. This is applicable both for IPv4 and for IPv6.

Examples

device.network_interfaces.ips in_net ["10.10.11.0/24"]
device.network_interfaces.ips not_in_net ["192.168.11.0/24"]

Starts or Does not Start with (starts_with | not_starts_with)

Checks whether the value from the entity starts with a required pattern that was entered.
Examples

device.ad_usn_changed starts_with ["91"]
device.ad_name starts_with ["EC2", "S3"]

Ends or Does not End With (ends_with | not_ends_with)

Checks whether the value from the entity ends with a required pattern that was entered.
Examples

user.mail ends_with ["@gmail.com", "@yahoo.com"]
user.username not_ends_with ["_test"]

Key Exists or Does Not Exist (key_exists | key_not_exists)

Checks whether all supplied key values exist or not.

device key_exists ["network_interfaces.ip", "ad_name"]
user key_not_exists ["database"]

Field Equal\Not Equal (field_equal |field_not_equal)

Checks whether the value from the value of one field in an entity is equal or not equal to the value of another field in that entity.

Examples

device.hostname field_not_equal ["ad_name"]

Only ingest devices where the hostname field is not equal to the value in the ad_name field.

Field Starts With (field_starts_with)

Checks whether the value from the value of one field in an entity starts with the value of another field in that entity.

Examples

device.ad_sAMAccountName field_starts_with ["ad_name"]

Ingest the device only if its sAMAccountName name begins with the ad_name.

Field Not Starts With ( field_not_starts_with)

Checks whether the value from the value of one field in an entity does not start with the value of another field in that entity.

Examples

device.hostname field_not_starts_with ["name"] then skip_field ["hostname", "fqdn"]

The device will only be ingested if its hostname does not start with its asset name. Those devices ingested will also have their hostname and FQDN fields dropped. Consider adding device key_exists ["id"] to this rule (or'd together) if you would still like to ingest devices where their hostname begins with their asset name.

Contains (contains) and Not Contains (not_contains)

Checks whether the value from the entity does/does not contain any of the rule values. This is case insensitive.

Examples

device.ad_sAMAccountName contains ["axonius"]
device.ad_sAMAccountName not_contains ["check1", "check2"]

Post Actions

Post actions can perform additional operations on the device before it is ingested. A post-action is not required for an Ingestion Rule statement. The following post action operators are available.

Skip Field (skip_field)

Removes the required fields from the entity and does not insert them into the database.

device.ad_dc_source == "10.10.11.5" then skip_field ["network_interfaces.ips", "ad_name"]

This rule also excludes all devices with an ad_dc_source not equal to 10.10.11.5

Therefore, if you want to include ALL devices/users, but only skip fields on a subset of devices, include the following rule (in conjunction with the OR operator) to ensure that all devices/users are still ingested from that adapter.

 device.ad_usn_changed starts_with ["91"] then skip_field ["ad_dc_source"]
 device key_exists ["id"] 

If you want to skip a field(s) on all devices coming from an adapter, the ingestion rule would be the following:

device key_exists ["id"] then skip_field ["physical_location", "qualys_agnet_ports"]
  • device key_exists ["id"] ensures that all devices are ingested from that adapter
  • then skip_field ["physical_location", "qualys_agnet_ports"] allows you to specify which field(s) to not ingest on those devices.

Remove Values (remove_values)

Removes values (from a list) from a field. This action can only be run on fields that contain lists of values. Please note, if you already fetched these values, they will still appear in the asset table for the amount of time set in Delete devices/users that have not been returned from the source in the last X hours. To only see the latest values, from the Query Wizard create a query that shows “from last fetch” = true. Use remove_values on fields that contain a list of simple objects such as strings (Last Used Users from WMI as seen above) or numbers.

IngestionremoveValue

Examples

device key_exists ["last_used_users"] then remove_values last_used_user starts_with ["Admin"]

will ingest all devices with a last used user, but will only include last used users that do not start with "Admin"

Remove Items (remove_items)

Removes items (from a list). Some fields contain lists of items, for example network_interfaces which contains sub fields such as MAC addresses, IP addresses, etc. If a condition is applied to one of the sub fields, then when it is true, the complete item is removed and is not ingested. remove_items should be used for lists of complex objects, such as Network Interfaces

egNetworkIn

Example

device key_exists ["id"] then remove_items network_interfaces.ips_raw_in_net ["10.0.48.0/24"]

This rule will ingest all devices, but will remove all associated network interface information for IPs between 10.10.48.0 - 10.10.48.255


Trim Prefix/Suffix (trim_suffix / trim_prefix)

Removes a certain prefix or suffix from the field if it exists.

Examples

device key_exists ["id"] then trim_prefix hostname ["domain.local"]
device key_exists ["id"] then trim_prefix network_interfaces.ips ["(IP Address)", "IP Prefix"]
user key_exists ["id"] then trim_suffix hostname [".com"]


Note:

If you want to run a post action on only a subset of entities and still ingest all other entities, you will need to include the following rule:
device|user key_exists ["id"]

This is useful for scenarios when you would like to remove employee IDs from an MSSP that collides with your primary email.

The Ingestion Rules would look like the following (joined with 'OR')

user.email ends_with ["@mssp.com"] then skip_field ["employee_id"]

user key_exists ["id"]

Testing Ingestion Rules

From the Add Connection page, click Save and Fetch. You can always wait until the adapter fetches in accordance with your discovery cycle.
The system then performs a fetch. Each entity brought from the adapter goes through the Ingestion Rule process.

After the fetch is complete, open the Devices or Users page in order to see your results. In the query wizard, specify your adapter and select “From Last Fetch - yes”

TestIngestoion

You can also see how many devices or users were ignored before and after the Ingestion Rules were implemented in the Adapter Fetch History.

IngestionAdapterFetch

If required, you can fine-tune the rules and re-run a fetch.

Entities not ingested will still appear in Axonius for the amount of time set in Delete devices/users that have not been returned from the source in the last X hours setting.

Ingestion Rules Examples

A problematic MAC address is being added to my devices after using an ethernet connection.

Perhaps the same ethernet connection is used when imaging company workstations. This might cause the same MAC address to appear on multiple devices. This may cause unintended correlation issues.

device key_exists ["id"] then remove_items network_interfaces.mac == "00:00:5E:00:53:AF"

This will find any network interface records containing this MAC address and remove it from the device

Exclude device records that have neither open ports NOR hostname.

Whenever we want to “exclude” a device, we need to think about the other side of the coin. What type of devices do we want to “ingest”?

It would be equivalent to say the following, “I want to ingest devices that have an open port OR hostname”

device key_exists ["hostname"]
device key_exists ["open_ports"]

These rules can be used together with OR.

IngestionEG1

Why can’t we combine these statements into a single rule?

device key_exists ["hostname", "open_ports"]

That is because BOTH keys need to exist on the device when we insert both fields into the list. Again, key_exists checks whether all supplied key values exist or not. If any key does not exist, then the device will not be ingested.

Exclude data from an acquired company in your CMDB

Perhaps you acquire another company and begin putting their devices into your CMDB. However, perhaps we do not want to ingest those devices into Axonius. You could use the following ingestion rule:

device.company != "Child Company"

Working with Boolean Values

IngestionEG2

While has_agent is a boolean field (True|False), the UI shows Yes|No. What do we do if we want to only ingest devices with an agent?

While the field in the UI shows, “Yes” or “No”, we want to use the actual value, “True” or “False”. Our rule would be the following:

device.has_agent == "True"

Exclude specific classes from the CMDB

Let’s filter the following classes from the ServiceNow fetch list: cmdb_ci_win_server, cmdb_ci_linux_server, cmdb_ci_unix_server, etc.

Before building the rule, we can build a field segmentation to get a rough idea of the devices we want to exclude

IngestionEG4

8 Win Servers + 7 Unix Servers + 5 Linux Servers = 20 total devices to exclude

The ingestion rule should ignore 20 additional devices after implementation.

device.class_name not_in ["cmdb_ci_win_server", "cmdb_ci_linux_server", "cmdb_ci_unix_server"]

IngestionCMDB

Success. Note the adapter was already ignoring 5 devices prior to the ingestion rule

Filtering AWS Resources Coming from CrowdStrike

Maybe your security controls are installed on ephemeral EC2 instances in AWS that constantly get spun up and spun down. While we have filtered these devices from the AWS adapter, they are still appearing in Axonius from the other adapters that see them, such as CrowdStrike.

Ingestionnext

To start we might want to remove based on cloud_provider.

device.cloud_provider != "AWS"

With this rule, we should expect 3 devices to be ignored.

IngestinEG6

However, this rule excluded our non-AWS devices too. Because those devices don’t have a cloud_provider field, the ingestion rule failed on those devices. So, we need another check to bring in devices that don’t have a cloud_provider.

device key_not_exists ["cloud_provider"]

IngestionEG7

These rules would be used together with OR

Excluding Expired Certificates

Perhaps we are pulling in certificate data from a Certificate Lifecycle Management system such as Digicert, Sectigo, or Venafi, and we want to filter out expired certificates that are considered stale. In this scenario, we want to exclude all certs that expired more than 30 days ago.

device.certificate.cert_expiry_date > date(now - 30d)

This statement will look back 30 days and ingest all certificates that have expired in the last 30 days or have yet to expire.



Was this article helpful?

Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.