Overview
The File Extractor ingests documents from local files, fileshares, S3 repositories, and Infinit.e shares (eg uploaded via the file uploader). It can also be used to ingest the output of custom analytic plugins.
...
| Field | Description | Data Type | ||
|---|---|---|---|---|
username | Username for file share authentication | string | ||
password | Password for file share authentication | string | ||
domain | Domain location of the file share | string | ||
pathInclude | Optional - regex, only files with complete paths matching the regular expression are processed further | string | ||
pathExclude | Optional - regex, files with complete paths matching the regular expression are ignored (and matching directories are not traversed) | string | ||
renameAfterParse | Optional, renames files after they have been ingested - the substitution variables "$name" and "$path" are supported; or "" or "." deletes the file // (eg "$path/processed/$name") | string | ||
type | One of "json", "xml", "tika", "*sv", or null to auto decide | string | ||
mode | "normal" (defaults if mode not present), "streaming", see below "mode" (from v0.3) is only applied in JSON/XML/*sv modes
| string | ||
XMLRootLevelValues | The root level value of XML to which parsing should begin // also currently used as an optional field for JSON, if present will create a document each time that field is encountered // (if left blank for JSON, assumes the file consists of a list of concatenated JSON objects and creates a document from each one) // (Also reused with completely different meaning for CSV - see below) // (In office mode, can be used to configure Tika - see below) | string | ||
XmlIgnoreValues | XML values that, when parsed, will be ignored - child elements will still be part of the document metadata, just promoted to the parent level. // (Also reused with completely different meaning for CSV) | string | ||
XmlSourceName | If present, and a primary key specified below is also found then the URL gets built as XmlSourceName + xml[XmlPrimaryKey], Also supported for JSON and CSV. | string | ||
XmlPrimaryKey | Parent to XmlRootLevelValues. This key is used to build the URL as described above. Also supported for JSON and CSV. | string | ||
XmlAttributePrefix |
For XML only, this string is pre-pended to XML attributes before they become JSON fields. |
Connecting to File Locations
...
You can use XmlRootLevelValues to set the field names.
When you do this, CSV parsing occurs automatically and the records are mapped into a metadata object called "csv" with the field names corresponding to the values of this array.
In the source example below, the field names will correspond to the included array: "device","date", "srcIP" etc.
...
| Code Block |
|---|
"processingPipeline": [ {
"file": {
"XmlRootLevelValues": [
"device",
"date",
"srcIP",
"dstIP",
"alert",
"country"
],
"XmlIgnoreValues": [
"device,date,srcIP"
],
"domain": "DOMAIN",
"password": "PASSWORD",
"type": "csv",
"username": "USER",
"url": "smb://FILESHARE:139/cyber_logs/"
}
}, |
When you do this, CSV parsing occurs automatically and the records are mapped into a metadata object called "csv" with the field names corresponding to the values of this array.
For example, here is the metadata that is generated using the above source
| Code Block |
|---|
"fullText": "SCANNER_1 , 2012-01-01T13:43:00 , 10.0.0.1 , 66.66.66.66 , DUMMY_ALERT_TYPE_1 , United States", "mediaType": ["Log"],
"metadata": {"info": [{
"alert": "DUMMY_ALERT_TYPE_1 ",
"country": "United States",
"date": "2012-01-01T13:43:00",
"device": "SCANNER_1 ",
"dstIP": "66.66.66.66",
"srcIP": " 10.0.0.1" |
Deriving Field Names Automatically
...