The file extractor ingests various file types from their locations and performs processing based on the configuration.
The File Extractor is capable of ingesting files from the following locations:
- Windows/Samba shares
- harvester's local filesystem
- Amazon S3
The File Extractor supports the following file types
- Office documents (Word, Powerpoint etc.)
- text-based documents (emails)
- CSV
- XML and JSON
- Infinit.e shares
- The results of Infinit.e plugins
The following table describes the parameters of the file extractor configuration.
Field | Description | Data Type |
---|
username | Username for file share authentication | string |
password | Password for file share authentication | string |
domain | Domain location of the file share | string |
pathInclude | Optional - regex, only files with complete paths matching the regular expression are processed further | string |
pathExclude | Optional - regex, files with complete paths matching the regular expression are ignored (and matching directories are not traversed) | string |
renameAfterParse | Optional, renames files after they have been ingested - the substitution variables "$name" and "$path" are supported; or "" or "." deletes the file // (eg "$path/processed/$name") | string |
type | One of "json", "xml", "tika", "*sv", or null to auto decide | string |
mode | "normal" (defaults if mode not present), "streaming", see below "mode" (from v0.3) is only applied in JSON/XML/*sv modes | string |
XMLRootLevelValues | The root level value of XML to which parsing should begin // also currently used as an optional field for JSON, if present will create a document each time that field is encountered // (if left blank for JSON, assumes the file consists of a list of concatenated JSON objects and creates a document from each one) // (Also reused with completely different meaning for CSV - see below) // (In office mode, can be used to configure Tika - see below) Info |
---|
If any of the root level (or ignore) values contain a prefix (eg "stix:IpAddress"), then prefixes are used to compare against all fields. Eg: - "<test:IpAddress></test:IpAddress>" will match "XmlRootLevelValues": [ "IpAddress" ] but not [ "other:IpAddress" ]
- "<test:EmailAddress></test:EmailAddress>" will match [ "EmailAddress" ] but not [ "EmailAddress", "other:IpAddress" ]
- (because specifying "other:IpAddress" means that it will be in "prefix mode" - you'd then need to specify "test:EmailAddress" instead of "EmailAddress")
Note also that (currently) the resulting JSON does not include the prefix, even in "prefix mode", eg "<test:Attribute>blah</test:Attribute>" will map to "Attribute": "blah". |
| string |
XmlIgnoreValues | XML values that, when parsed, will be ignored - child elements will still be part of the document metadata, just promoted to the parent level. (Also reused with completely different meaning for CSV - see below) (Note that if the ignore value contains a prefix, then all XML processing is performed in "prefix mode" - see above under "XmlRootLevelValues") | string |
XmlSourceName | If present, and a primary key specified below is also found then the URL gets built as XmlSourceName + xml[XmlPrimaryKey], Also supported for JSON and CSV. | string |
XmlPrimaryKey | This key is used to build the URL as described above. Also supported for JSON and CSV. | string |
XmlPreserveCase | By default, XML field names are lower case, if this is false, then the case of the fieldnames is preserved | |
XmlAttributePrefix | - For "*sv" files when XmlRootLevelValues is set controls the separators as follows: the first char in the string is the separator, the (optional) second char in the string is the quote, and the (optional) third char in the string is the escape character (eg the default is ",\"\\")
For XML only, this string is pre-pended to XML attributes before they become JSON fields. (otherwise attributes are ignored) - Eg <test id="alex"></test> would map to '{ "test": { "PREFIXid": "alex" } }' with XmlAttributePrefix: "", or just an empty object if not specified (the default)
| |
The configuration will depend on the locations of the files you are trying to extract.
To connect to the text extractor's local filesystem the following url format must be used:
"file://<path including leading '/'>"
Info |
---|
"file://" sources can only be run by administrators if secure mode is enabled (harvest.secure_mode=true in the configuration). Local filesystem usage is mostly intended for testing, debugging, and "micro installations". The "tomcat" user must have read access to the directories and files on the path. |
You can connect the File extractor to Infinit.e shares and the results of custom jobs.
To connect to an Infinit.e share, the following url format must be used:
"inf://share/<shareid>/<ignored>"
The share id can be obtained in the url of the file uploader.
After the "<shareid>/" portion of the URL, any arbitrary text can be added to make the role of the share clearer. This text is ignored by the file harvester.
The source must share at least one community with the share in order to be processed.
Warning |
---|
The uploaded share must be a zip file containing the JSON/XML/CSV files to be imported. |
To connect to an Infinit.e custom job, the following url format must be used:
"inf://custom/<customid-or-jobtitle>"
custom id and title can be obtained in the URL field of the plugin manager.
After the "<customid-or-jobtitle>/" portion of the URL, any arbitrary text can be added to make the role of the share clearer. This text is ignored by the file harvester.
The source must share at least one community with the custom plugin in order to be processed.
To connect to a Windows/Samba share, the following url format must be used:
"smb://server:port/path"
To connect to an Amazon S3 location, the following url format must be used:
"s3://bucket_name>/" or "s3://<bucket_name>/path/"
The files in the S3 bucket should be readable by the account specified by the access key.
Info |
---|
S3 is not supported by default, the AWS SDK JAR must be copied into the classpath as described here. The S3 harvester will only read 1000 docs per directory. To handle larger numbers of documents, they should either be split into smaller subdirectories, or "renameAfterParse" should be used to delete/move them after processing. |
A username/password is required to connect to your Amazon S3 environment.
For S3, the Access ID should be entered into the "username", and the Secret Key into the "password"
Info |
---|
It is recommended for security that you create a separate AWS user with no permissions other than S3 read/list on the directories. |
Domain
This field can be left blank for Amazon S3 environments.
This section describes the configurations for the various supported file types.
To connect to Office files, the following url format must be used:
- the path of the file (ie file.url + path-relative-to-url)
Example:
Connects to an "office" document on a samba drive.
"smb://modus:139/enron/enron_mail_20110402/maildir/"
You can use the XmlRootlevelValues
parameter to configure Apache Tika for parsing of Office-type files.
XmlRootlevelValues will accept a sting value which can be used to pass configuration values to Apache's Tika module. Currently there are the following supported configurations:
You can include the string "output:xml" or "output:html" to change the output of Tika from raw text to XML or HTML.
You can configure Tika Elements by using paramName
and paramValue
to send functions and arguments to Tika.
The string must be in the following format
- bypass:<MEDIATYPE>
- (will just return the raw text - binary not current supported - for the corresponding mediatype, eg "bypass:message/rfc822" will just return the raw text from an email)
- Strings of the format "<MEDIATYPE>:{ paramName: paramValue, ...}"
<MEDIATYPE> is in standard MIME format and determines which Tika element to configure. The paramNames and paramValues correspond to functions and arguments.
Example:
"application/pdf:{'setEnableAutoSpace':false}"
where application/pdf will calll PDFParser. and setEnableAutoSpace(false)corresponds to a paramName and paramValue.
To connect to these file types, the following url format must be used:
- path-of-file (as above) + <hash of object> + ".csv"/.json/.xml
Example:
To connect to a samba fileshare
"url": "smb://FILESHARE:139/cyber_logs/"
If xmlsourcename and xmlprimarykey are specified, the following url format must be used
- xmlsourcename + object,get(xmlprimarykey)
Example:
"url": "smb://HOST:139/SHARE/PATH/TO/"
You can use the file extractor to configure the root JSON object for parsing.
In the example below, the parameter XMLRootlevelValues
is used to set the root object.
In addition, you can use XmlSourceName
to build the document URL. If specified, the document URL is built as "XmlSourceName" + xml("XmlPrimaryKey").
You can use the parameter XmlPrimaryKey
to help identify whether a record is new or previously harvested.
Info |
---|
For JSON files, the parameter XmlIgnoreValues is not applicable. |
Code Block |
---|
"description": "A large set of tweets related to Super Storm Sandy",
"isApproved": true,
"isPublic": false,
"mediaType": "Social",
"tags": [
"twitter",
"gnip"
],
"title": "Super Storm Sandy - Twitter: SANDY_SUBSTRING",
"processingPipeline": [
{
"file": {
"XmlPrimaryKey": "link",
"XmlSourceName": "",
"XmlRootLevelValues": [],
"domain": "XXX",
"password": "XXX",
"username": "XXX",
"url": "smb://HOST:139/SHARE/PATH/TO/"
}
}, |
You can use XmlRootLevelValues
to set the root object for xml file parsing.
In the example below, the field "Incident" is set as the root object.
In addition, the parameter XmlIgnoreValues
is used to ignore certain xml nodes in the xml document.
XmlPrimaryKey
identifies the primary key in the data set, and is use to help identify whether a record is new or previously harvested
XmlSourcename is used to build the new document url of the document that will be generated by the file extraction.
Code Block |
---|
{
"description": "wits test",
"isPublic": true,
"mediaType": "Report",
"searchCycle_secs": -1,
"tags": [
"incidents",
"nctc",
"terrorism",
"wits",
"events",
"worldwide"
],
"title": "wits test",
"processingPipeline": [
{
"file": {
"XmlIgnoreValues": [
"DefiningCharacteristicList",
"TargetedCharacteristicList",
"WeaponTypeList",
"PerpetratorList",
"VictimList",
"EventTypeList",
"CityStateProvinceList",
"FacilityList"
],
"XmlPrimaryKey": "icn",
"XmlRootLevelValues": [
"Incident"
],
"XmlSourceName": "https://wits.nctc.gov/FederalDiscoverWITS/index.do?N=0&Ntk=ICN&Ntx=mode%20match&Ntt=",
"domain": "XXX",
"password": "XXX",
"username": "XXX",
"url": "smb://modus:139/wits/allfiles/"
}
}, |
Using the File Harvester