Overview

The File Extractor ingests documents from local files, fileshares, S3 repositories, and Infinit.e shares (eg uploaded via the file uploader). It can also be used to ingest the output of custom analytic plugins.

...

Connecting to File Locations

...

"description": "A large set of tweets related to Super Storm Sandy",
    "isApproved": true,
    "isPublic": false,
    "mediaType": "Social",
    "tags": [
        "twitter",
        "gnip"
    ],
    "title": "Super Storm Sandy - Twitter: SANDY_SUBSTRING",
    "processingPipeline": [
        {
            "file": {
                "XmlPrimaryKey": "link",
                "XmlSourceName": "",
                "XmlRootLevelValues": [],
                "domain": "XXX",
                "password": "XXX",
                "username": "XXX",
                "url": "smb://HOST:139/SHARE/PATH/TO/"
            }
        },

Configuring XML

You can use XmlRootLevelValues to set the root object for xml file parsing.

In the example below, the field "Incident" is set as the root object.

In addition, the parameter XmlIgnoreValues is used to ignore certain xml nodes in the xml document.

XmlPrimaryKey identifies the primary key in the data set, and is use to help identify whether a record is new or previously harvested

XmlSourcename is used to build the new document url of the document that will be generated by the file extraction.

...

Configuring XML

You can use XmlRootLevelValues to set the root object for xml file parsing.

In the example below, the field "Incident" is set as the root object.

In addition, the parameter XmlIgnoreValues is used to ignore certain xml nodes in the xml document.

XmlPrimaryKey identifies the primary key in the data set, and is use to help identify whether a record is new or previously harvested

XmlSourcename is used to build the new document url of the document that will be generated by the file extraction.

{
    "description": "wits test",
    "isPublic": true,
    "mediaType": "Report",
    "searchCycle_secs": -1,
    "tags": [
        "incidents",
        "nctc",
        "terrorism",
        "wits",
        "events",
        "worldwide"
    ],
    "title": "wits test",
    "processingPipeline": [
        {
            "file": {
                "XmlIgnoreValues": [
                    "DefiningCharacteristicList",
                    "TargetedCharacteristicList",
                    "WeaponTypeList",
                    "PerpetratorList",
                    "VictimList",
                    "EventTypeList",
                    "CityStateProvinceList",
                    "FacilityList"
                ],
                "XmlPrimaryKey": "icn",
                "XmlRootLevelValues": [
                    "Incident"
                ],
                "XmlSourceName": "https://wits.nctc.gov/FederalDiscoverWITS/index.do?N=0&Ntk=ICN&Ntx=mode%20match&Ntt=",
                "domain": "XXX",
                "password": "XXX",
                "username": "XXX",
                "url": "smb://modus:139/wits/allfiles/"
            }
        },

Configuring

...

SV

For SV type files, the root level values and field names can be set manually or automatically.

...

You can use XmlRootLevelValues to set the root level values/field names.In the source example below, the field names   When you do this, CSV parsing occurs automatically and the records are mapped into a metadata object called "csv" with the field names corresponding to the values of this array.

In the source example below, the field names will correspond to the included array: "device","date", "srcIP" etc.

...

:

...

"device","date", "srcIP" etc.

 "processingPipeline": [        {
            "dstIPfile",: {
                   "alertXmlRootLevelValues",: [
                    "countrydevice"
 ,
              ],                 "XmlIgnoreValuesdate": [,
                    "device,date,srcIP",
                ],    "dstIP",
                 "domain":   "DOMAINalert",
                "password":    "PASSWORDcountry",
                "type": "csv"],
                "usernameXmlIgnoreValues": "USER",
       [
        "url": "smb://FILESHARE:139/cyber_logs/"           "device,date,srcIP"
 }            },

When you do this, CSV parsing occurs automatically and the records are mapped into a metadata object called "csv" with the field names corresponding to the values of this array.

For example, here is the metadata that is generated using the above source

 "fullText": "SCANNER_1 ], 2012-01-01T13:43:00 , 10.0.0.1 , 66.66.66.66 , DUMMY_ALERT_TYPE_1 , United States",
                "mediaTypedomain": ["LogDOMAIN"],,
                "metadatapassword": {"info": [{PASSWORD",
                "alerttype": "DUMMY_ALERT_TYPE_1 csv",
                "countryusername": "United StatesUSER",
        "date": "2012-01-01T13:43:00",         "deviceurl": "SCANNER_1 ",smb://FILESHARE:139/cyber_logs/"
        "dstIP": "66.66.66.66",    }
      "srcIP": " 10.0.0.1" },

Deriving Field Names Automatically

...

Header Prefixed by String:

If " XmlIgnoreValues is set to: "#", and the first three lines are "#", "#header", and "#field1,field2,field3" then the processing will assume the 3 fields are field1, field2, and field3.

...

In the example source below, XmlIgnoreValues automatically identifies the header using "#".  This also identifies the field names using the separator ",".

...

In the case where the first header is not prefixed by a string, it is still necessary to identify it as the header row.  

For example, consider a header row formated formatted as follows:

"field1,field2,field3"

In this case, XmlIgnorevalues should be set to the following: [ "\"field1\" ]

...

For SV files, the XmlAttributePrefix is used to establish the default setting for separator, quote and escape characters: ",\"\\")

You can change the values of XmlAttributePrefix to change any of these default settings.

For example, to change the separator from "," to ";" the following configuration is required: ";'\\"

Specifying Network Resources

...

For

...

CSV*, JSON and XML files,  it is possible to use XmlSourceName and XmlPrimaryKey to identify the unique network resources that link to the documents. 

The document's unique network resource must be in the following format:  "CONSTANT_URL_PATH + VARIABLE_ID"

...

Also, the "VARIABLE_ID"

...

must be one of the fields in the XML/JSON object.

*XmlRootLevelValues must be set

...