- 08 Oct 2024
- 20 Minutes to read
- Print
- DarkLight
- PDF
CSV
- Updated on 08 Oct 2024
- 20 Minutes to read
- Print
- DarkLight
- PDF
The CSV adapter imports .csv files with inventory information including devices and serial numbers, users, and installed software data.
It is possible to configure the CSV adapter to fetch files from various storage places including: Microsoft OneDrive, Azure, and Amazon S3.
The CSV adapter parameters and functionality are common to all adapters that import files:
To see current configuration parameters for the following adapters refer to CSV Legacy Remote File Configuration
- CSV - Applications
- CSV - DNS Records
- CSV-Expenses
- CSV-Licenses
- F-Secure Policy Manager - imports .csv files.
- Forcepoint Web Security Endpoint CSV File - imports .csv files.
- L0phtCrack 7 - imports .csv files.
- Masscan - imports .json files.
- Nmap Security Scanner - imports .xml files.
- Tenable Nessus CSV File - imports .csv files.
- Varonis CSV - imports .csv files.
Parameters
File contains users information - Select this option to imports the file as a list of users instead of devices. See the below section for fields required in a Users Information File.
File contains installed software information - Select this option to import an installed software list instead of devices. See the below section for fields required in a Software Applications File.
File contains database information - Select this option to import the file as a list of databases instead of devices. See the below section for fields required in a Databases File.
File name (required) - Provide a unique name for the adapter connection. The value supplied here is populated in the File name field for the data supplied by a specific adapter connection.
Note:The specified File Name is not required to be the actual imported file name. This field is an identifier for use in the Query Wizard.
Select file source - The CSV adapter supports upload of files from a variety of file sources. The parameters you need to enter change according to the file source that you select. The default is Upload File. Refer to File Sources for full details.
Encoding (optional) - Specify a specific file encoding or let Axonius decode it. When you enter an encoding type Axonius tries to decode the CSV file based on the specified file encoding type (for example, utf-8) for this connection. Otherwise Axonius tries to decode the CSV file based on common file encoding types for this connection.
Base64 encoding type is also supported for this adapter.
Ignore illegal characters - Select this option to ignore illegal characters during the data import. An illegal character is any character that cannot be translated in the specified file encoding. If you do not select this option, and an illegal character is found, the entire data import fails.
Verify SSL - Select whether to verify the SSL certificate of the server against the CA database inside of Axonius. For more details, see SSL Trust & CA Settings.
HTTPS Proxy (optional) - Connect the adapter to a proxy instead of directly connecting it to the domain.
HTTPS Proxy User Name (optional) - The user name to use when connecting to the value supplied in Host Name or IP Address via the value supplied in HTTPS Proxy.
HTTPS Proxy Password (optional) - The password to use when connecting to the server using the HTTPS Proxy.
Custom prefix for dynamic fields (optional) - Specify a prefix to be added for dynamic fields. A dynamic field refers to any field that is not part of an asset default field. This can assist you in identifying such fields.
Multi-value fields delimiter (optional) - Specify a delimiter to separate between values within the same field in the imported CSV file. When you specify a delimiter Axonius considers fields that contain the specified delimiter as multi-value fields. For example, ';'. Otherwise Axonius considers all imported fields as single-value fields.
File Type - Select the type of file uploaded, either CSV, or Excel Spreadsheet. When you select "Excel Spreadsheet", the adapter supports .xls , and.xlsx files, and pulls in the entirety of the first Worksheet as if it were a CSV table. Functionality for tables uploaded from Excel is the same as for CSV files.
Allow empty values - Select this option to allow the system to support assets with empty fields. If an asset was created with a field that contained a value, when the CSV file subsequently contains an empty field with the same name, the device or user asset will display that field without a value in it.
To learn more about common adapter connection parameters and buttons, see Adding a New Adapter Connection.
File Sources
The CSV adapter supports upload of files from a variety of file sources. The parameters you need to enter change according to the file source that you select. The default is Upload File.
If you are uploading a file from an online storage location and you want to use this file only for custom enrichment, you must disable the Active connection setting on the CSV adapter connection. In this case, the CSV adapter connection will not fetch new assets.
Upload File
Use Upload File to import a local CSV, JSON, or XML. file.
When you select Upload File the following parameters are available:
Upload File (required) - select a local CSV file to import.
- When using this option, the data imported from the CSV will never be fetched again, as the file is static.
- Each adapter that imports files supports a different file type: CSV, JSON, or XML.
Microsoft OneDrive
Upload a file from Microsoft OneDrive.
- Azure Cloud Environment (optional) - Select your Microsoft Azure or Microsoft Entra ID (Azure AD) cloud environment type.
- Microsoft OneDrive Tenant ID (optional) - Microsoft Entra ID (Azure AD) ID. Used to authenticate Microsoft OneDrive through Microsoft Entra ID (Azure AD) and use the Graph API.
- Microsoft OneDrive Client ID - The Application ID of the Axonius application.
- Microsoft OneDrive Client Secret, Microsoft OneDrive OAuth Authorization Code, Microsoft OneDrive Redirect URI - Enter these parameters when multi-factor authentication is used in the Azure/Microsoft Office account. To use this option an OAuth Authorization Code must be created. The OAuth Token/Code Procedure will use the following URL
https://login.microsoftonline.com/[[TENANT_ID]]/oauth2/v2.0/authorize?client_id=[[CLIENT_ID]]&scope=https://graph.microsoft.com/.default&response_type=code&redirect_uri=[[REDIRECT_URI]]&response_mode=query
5. Path to resource (URL/Folder path) (optional)
- If a folder path is supplied:
- The remaining necessary parameters for Microsoft OneDrive must be provided.
- The path must be separated by forward slashes.
- For personal files on Microsoft OneDrive, use the path relative to 'My files'. For example, the folder path for "My files > Documents > Axonius > file.csv" should be Documents/Axonius/file.csv).
- Use the URL of the file when opened in the browser (recommended). Alternatively, use the Copy Link from the file (this method will work, but, the link will expire and will need to be regularly updated).
- Username for remote resource and Password for remote resource - Username and password for the remote resource. You cannot use these parameters when multi-factor authentication is used in the Azure/Microsoft Office account.
When uploading files from Microsoft OneDrive, and using OAuth authentication the value supplied in Microsoft OneDrive Client ID must have Files.Read.All delegated permissions in the Azure application in order to to fetch files.
When uploading files from Microsoft OneDrive, the value supplied in Username for remote resource (Share/URL) must have Files.Read.All permissions to fetch files.
Microsoft Azure
Upload a file from a container on Azure.
- Azure Storage Container Name (required) - The name of the container on Azure where the CSV file is located.
- Azure Connection String (required) - The connection string that includes the authorization information required to access data in the Azure Storage account. You can find the connection string in the Azure portal under Storage Accounts -> [Account Name] -> Access Keys, where [Account Name] is the specific storage account that contains the CSV files to be ingested into Axonius. Copy the entire connection string and paste it into this field.
- Azure Blob Name (required) - The Azure Blob Name.
Microsoft SharePoint
Upload a file from Microsoft SharePoint.
Authentication Method - Select the authentication method, either User Credentials, Client Credentials, or Client REST Credentials.
User Credentials
- Enter a user name and password
- Tenant - Microsoft Entra ID (Azure AD) ID. Used to authenticate Microsoft SharePoint through Microsoft Entra ID (Azure AD) and use the Graph API.
- Client ID - The Application ID of the Axonius application.
Client Credentials -
- Enter a Client ID and Secret
- Tenant - Microsoft Entra ID (Azure AD) ID. Used to authenticate Microsoft SharePoint through Microsoft Entra ID (Azure AD) and use the Graph API.
- Domain - The domain
Client REST Credentials - Configure the following:
- Host Name or IP Address (default: graph.microsoft.com)- The hostname or IP address of the SharePoint server.
- Tenant - Microsoft Entra ID (Azure AD) ID. Used to authenticate Microsoft SharePoint through Microsoft Entra ID (Azure AD) and use the Graph API.
- Client ID - The Application ID of the Axonius application.
- Client Secret - Specify a non-expired key, generated from the new client secret.
SharePoint Host Name - Domain/Host Name of the SharePoint site i.e. “companyname.sharepoint.com”
SharePoint Site Name - Name of the SharePoint site the file is located in
SharePoint Folder Path - Relative path to the file from the “Documents” location i.e. “Documents/Path/To/File”
SharePoint File Name - Name of the file
Permissions required: Note The Sharepoint account needs the "Sites.Read.All" permission, either assigned to a user or client application through Entra ID.
Amazon S3 Bucket
Upload a file from an Amazon S3 Bucket.
- Amazon S3 bucket name (required) - The name of the S3 bucket from which to fetch the file.
- Amazon S3 object location (key) (required) - The location within the S3 bucket from where the file can be fetched.
- Amazon S3 Use EC2 Attached Instance Profile
- If enabled, Axonius uses the EC2 instance (Axonius installed on) attached IAM role / instance profile.
- If disabled, Axonius uses the supplied account details in the AWS Access Key ID and AWS Access Key Secret.
- Amazon S3 Access Key ID and Amazon S3 Secret Access Key (optional) - The credentials used to access the S3 object.
- Amazon S3 Region - The Amazon region on which the S3 bucket is located.
- Amazon S3 Interface VPC Endpoint - Custom VPC endpoint.
- Amazon S3 directory (default: false). - Select this option to show that the object specified in object loction is a directory.
Box Platform
Upload a file from Box Platform.
- Box Platform private key configuration file (required) - The private key configuration file that provides the Required Permissions to fetch assets. This JSON authentication file must have permission to read/download the specified File ID. In order to fetch files from Box Platform both of these settings must be configured.
- Box File ID (required) - The ID of the Box file. The ID for any file can be determined by visiting a file in the web application and copying the ID from the URL. For example, for the URL https://*.app.box.com/files/123 the file_id is 123. Refer to Box Platform documentation for more information
URL/FTP
Supply a URL or FTP from which to fetch the file.
- Path to resource (URL) (required) - Specify an HTTP(S) URL or FTP URLwhere a file can be fetched for this connection.
- For HTTP(S) URL
- The endpoint must support the HTTP GET method.
- All URLs must start with HTTP:// or with HTTPS://
- For FTP URL
- All URLs must start with FTP:// or with SFTP:// or with FTPS://
- The default port for each method is as follows:
- FTP: 21
- SFTP: 22
- FTPS: 990
- A custom port can be specified in the supplied path, for example: ftps://my.host.in.axonius.com:21/path/to/file.ext
Username for remote resource and Password for remote resource - Username and password for the URL. These settings may be required if the "ubuntu" user on the Axonius system does not have access to the URL. The username and password are used for BASIC authentication of this connection.
Additional HTTP headers (optional) - Specify additional information to pass with the HTTP request (for example {"Accept": "text/csv"}).
Content key in JSON (If returned as JSON) - When fetching the CSV file via URL, if the URL returns the response as a JSON expression, set this field to the JSON key that represents the actual content of the CSV file. For example, if the URL returns the following JSON expression:
{'content': 'column1,column2\na,b', 'name': 'file.csv'}
, set this fields as "content".
SMB Share
Upload a file from an SMB share path.
Path to resource (SMB) (required) - Specify an SMB share path where a file can be fetched. This path must include the file name and must start with double-backslashes ("\\").
Username for remote resource and Password for remote resource (optional) - Username and password for the SMB share. These settings may be required if the "ubuntu" user on the Axonius system does not have access to the SMB share. The username and password are used for authentication of this connection.
Suppress NetBIOS name lookup - Select this option so that Axonius does not verify the server's name via NetBios. This option must be enabled in order to use wildcards in SMB file names. When this setting is enabled, the path must include the server's full NetBIOS name in the following format:
\\<full_server_NetBIOS_name>\path\to\file.ext
- Wildcards are supported in file names as follows:
- Suppress NetBIOS name lookup must be enabled.
- The asterisk * wildcard matches any sequence of characters (0 or more, including NULL characters).
- The ? wildcard matches a single character (or a NULL at the end of a file name).
- The matching file names are sorted by file creation time.
- If multiple files match the wildcard search, the most recently created file is selected.
Google Sheets
You can upload a file that was previously fetched by the Google Sheets adapter.
- Service account JSON credentials (required) - A JSON Key Pair associated with a service account that has the Required Permissions to fetch assets. Click Upload to upload a file containing the binary contents of the keypair file (JSON) generated for the service account credentials.
- Spreadsheet ID (required) - The Spreadsheet ID (gathered from the link to the spreadsheet)
- Data range (A1 or R1C1 notation) (required) - A1 or R1C1 notation of the data range to read. Example: 'My Worksheet'!A1:Z99 to pull cells A1 to Z99 from the worksheet “My Worksheet”. Always use single-quotes when specifying a worksheet that contains spaces in the name. A data range and a worksheet range must be specified as an absolute path.
Advanced Settings
Advanced settings can either apply for all connections for this adapter, or you can set different advanced settings and/or different scheduling for a specific connection. Refer to Advanced Configuration for Adapters.
- Set Time Zone - Set the time zone of date fields fetched with this adapter. Default is UTC.
- Use fetch time for Last Seen - Select this option to set that all entities (devices and users) fetched by this adapter have their Last Seen set to the time the entity was fetched (fetch_time). When you choose this option, the system does not look in the CSV file for a date and only uses the value set here.
- Additional primary keys for software files - Enter additional primary keys to use when correlating installed software CSV files. By default, only "hostname" is used as a primary key. Contact Axonius support before you use this option.
- Do not add filename to entity IDs - By default Axonius adds the filename to the ID of the entities created by the CSV file (device ID etc). Select this option to not add the filename to the entity ID.
- Parse double quotes as escaping - Some CSV formats use double quotes as escaping for a single " sign. Select this option to explicitly parse double quotes as escaping, to prevent errors that might occur by incorrect dynamic parsing of double quotes.
APIs
When uploading files from Microsoft OneDrive, Axonius uses the List FIles Shared With Me - OneDrive API - OneDrive dev center.
If multiple file options are added to a 'file-based' adapter, what file is imported?
If multiple import file import types are provided in the CSV adapter (or for other file-based adapter), they are imported in the following order:
- Path to resource (SMB/URL/Folder path)
- Amazon S3 Bucket
- Uploaded file
The file content is determined based on the following order:
- User assets - If File contains users information is selected
- Installed software list - If File contains installed software information field is enabled.
- Device assets - If none of the options above is selected, the CSV is assumed to be a device CSV. If a device exists more than once in a CSV file, only the first device found is used.
Which fields are required for each Import Type?
For single value fields the following delimiters are supported:s , , ; and \t .
Import Type | *KEY fields | Optional fields |
---|---|---|
From a File | - File name - Upload file | - File contains users information - File contains installed software information - Encoding - Ignore illegal characters - Custom prefix for dynamic fields - Multi-value fields delimiter - Connection Label |
From an HTTP(S) URL | - File name - Path to resource (SMB/URL/Folder path) - Verify SSL - Choose Instance (on multi-node Axonius environment) | - File contains users information - File contains installed software information - Username for online resource (Share/URL) - Password for online resource (Share/URL) - Encoding - HTTP proxy - HTTP proxy - HTTPS proxy - Additional HTTP headers - Ignore illegal characters - Custom prefix for dynamic fields - Multi-value fields delimiter - Connection Label |
From an FTP URL | - File name - Path to resource (SMB/URL/Folder path) - Choose Instance (on multi-node Axonius environment) | - File contains users information - File contains installed software information - Username for online resource (Share/URL) - Password for online resource (Share/URL) - Encoding - Ignore illegal characters - Custom prefix for dynamic fields - Multi-value fields delimiter - Connection Label |
From a file share | - File name - Path to resource (SMB/URL/Folder path) - Choose Instance (on multi-node Axonius environment) - Suppress NetBIOS name lookup | - File contains users information - File contains installed software information - Username for online resource (Share/URL) - Password for online resource (Share/URL) - Encoding - Ignore illegal characters - Custom prefix for dynamic fields - Multi-value fields delimiter - Connection Label |
From Microsoft OneDrive | - File name - Path to resource (SMB/URL/Folder path) - Choose Instance (on multi-node Axonius environment) | - File contains users information - File contains installed software information - Username for online resource (Share/URL) - Password for online resource (Share/URL) - Microsoft OneDrive Tenant ID - Microsoft OneDrive Client ID - Azure Cloud Environment - Encoding - Ignore illegal characters - Custom prefix for dynamic fields - Multi-value fields delimiter - Connection Label |
Which fields are imported with a devices file?
The following data is imported as common data fields while any other data in the CSV/JSON/XML is exclusively be Adapter Specific data.
UI Field Name | Accepted CSV Field Name(s) | Notes |
---|---|---|
Architecture | architecture | |
Asset Name | name, vmname, displayname, assetname, machinename, instancename, samaccountname, endpointname, machine | If no hostname is configured, the Asset Name value is used for the Host Name. |
Device Manufacturer Serial | serial, serialnumber, sn, hostserialnumber, deviceserialnumber, serial#, endpointserialnumber | *KEY |
Device Manufacturer | manufacturer, devicemanufacturer | |
Device Model | model, modelid, endpointmodel | |
Domain | domain, domainname, endpointdomain | If this value is not specified AND the device is specified in DOMAIN\Name format, Axonius replaces the Domain value with the parsed out DOMAIN. |
Host Name | hostname, host, fqdn, fullyqualifieddomainname, compname, computername, servername, dnsname, hosthostname, endpointfqdn | *KEY - If the device is specified in DOMAIN\Name, Axonius parses the DOMAIN value out. If the CSV field is set to "unknown", Axonius sets the Host Name to blank. |
ID | id, identifier, serialnumber, assetid, resourceid | *KEY - The ID field is a combination of the "CSV File Name" value and the specified field names. |
IPs | ipaddresstext, ip, ipaddress, ipaddresses, ips, primaryip, endpointipaddress, , ipaddresstext, ip, ipaddress, ipaddresses, ips, primaryip, endpointipaddress, registerip, sourceip, managementip, privateip, allips, lastip, address, ipaddresslist, ipaddri, ipaddrs, ipaddr, localip, privateipaddresses, ipfirst | This field accepts a comma separated set of IP addresses. |
Last Seen | lastmessagetime, lastdiscoveredtime, lastseen, lastcheckin | If this value is not specified, enter the time that the CSV was last imported. |
Last Used Users | username | This appends to the existing Last Used Users list if the device already exists. |
MAC | mac, macaddress, macaddresses, macs | *KEY - This field accepts a comma separated set of MAC addresses. |
Machine | name | |
Network Interfaces | networkinterfaces | Axonius attempts to parse IP address(es), MAC address(es), and network interface cards from this field. |
OS (see Notes) | os, osname, osversion, operatingsystem, osmode, uname, endpointos | This field is parsed out into multiple properties within the OS field. Not all OSes are parsed properly. Please reach out to Axonius if an OS is not parsing as expected. |
OS: Kernel Version | kernel, kernelversion | |
Software Name | packages | This is delimited by spaces. |
Which fields are imported with a users file?
The following data is imported as common data fields while any other data in the CSV/JSON is exclusively Adapter Specific data.
UI Field Name | Accepted CSV Field Name(s) | Notes |
---|---|---|
Domain | domain, domainname, endpointdomain | |
First Name | firstname, givenname | |
ID | id, identifier, serialnumber, assetid, resourceid | *KEY - The ID field is a combination of the "CSV File Name" value and the specified field names. |
Last Name | lastname, surname, sn | |
mail, email, usermail, mailaddress, email address, emailprimarywork | *KEY | |
Name | name, vmname, displayname, assetname, machinename, instancename, samaccountname, endpointname | *KEY |
User Name | username | *KEY |
Which fields are imported with a software applications file?
The following data is imported as common data fields while any other data in the CSV/JSON is exclusively Adapter Specific data.
In order for vulnerabilities to be parsed from the CSV adapter, the MINIMUM requirements are:
- The adapter is configured with "File contains installed software"
- The file has at least the following headers:
- Hostname (or any of the headers supported as hostname)
- Software Name (header must be present, though may be empty on a row)
- CVE ID
The other headers (or data in a row for those headers) are optional for the purposes of parsing vulnerabilities
UI Field Name | Accepted CSV Field Name(s) | Notes |
---|---|---|
Host Name | hostname, host, fqdn, fullyqualifieddomainname, compname, computername, servername, dnsname, hosthostname, endpointfqdn | *KEY - This field is required as the software list is imported to each individual device. |
Software Name | softwarename, swname | *KEY this field is required in order to parse installed software. This field may be left empty on a row with CVE ID. |
Software Path | softwarepath, swpath | |
Software Vendor | softwarevendor, swvendor | |
Software Version | softwareversion, swversion | |
CVE ID | cve, cveid, cvelist, grypecve | If present, a row featuring a CVE ID is parsed as vulnerable software in addition to installed software. |
CVE Description | cvedescription | This field will be ignored if CVE ID is empty or not present. |
CVE Severity | cveseverity | CVE Severity needs to be one of the values listed here. An invalid CVE Severity value is ignored. This field will be ignored if CVE ID is empty or not present. 'NONE', 'LOW', 'MEDIUM', 'MODERATE', 'SEVERE', 'SERIOUS', 'HIGH', 'CRITICAL', 'URGENT', 'INFO', 'UNTRIAGED', 'NEGLIGIBLE' |
CVE Status | cvestatus | CVE Status needs to be one of the values listed here. An invalid CVE Status value is ignored. This field will be ignored if CVE ID is empty or not present. 'open', 'closed', 'reopen', 'expired', 'done', 'valid', 'obsolete', 'pending' |
Which fields are imported with a databases file?
The following data is imported as common data fields while any other data in the CSV/JSON is exclusively Adapter Specific data.
UI Field Name | Accepted CSV Field Name(s) | Notes |
---|---|---|
ID | id, identifier, serialnumber, assetid, recid, deviceid, objectid, hostid, databaseid | *KEY - This field is required (database ID) |
Name | name, diaplayname, assetname, instancename, databasename | *KEY - This field is required (database name) |
Instance Type | instance, instancetype | |
Status | status, assetstatus | |
Creation Date | creationdatetime, datecreation | |
Port | port | |
IP | ip, ipaddress, ipaddresses, ips, sourceip, ipaddresstext |
Example CSV File
For an example of a CSV file, download the following zipped csv: