Overview

Once data is ingested into Infint.e from the various extractors it is stored in JSON format including its metadata fields and content.  It also contains sub-objects such as entities and associations.

...

onUpdateScript

Use Cases

The fields of the Document storage settings configuration can be used to support the following use cases

...

Anchor
storage storage

Document Storage Settings

Metadata Field Storage

...

Consider this document:

Code Block
{ //... "metadata": { "field1": { //... }, "field2": { "field2.1": "test", "field2.2": "object" } } }

Here are some example metadataFieldStorage fields, and the resulting documents after the pipeline element is complete.

Code Block

"metadataFieldStorage": "+" { //... "metadata": { } }   "metadataFieldStorage": "field1" { //... "metadata": { "field1": { //... } } }   "metadataFieldStorage": "-field2.2" { //... "metadata": { "field1": { //... } "field2": { "field2.1": "test" } } }   "metadataFieldStorage": "field2.2" // NOT ALLOWED   "metadataFieldStorage": "-field1,field2.2" { //... "metadata": { "field2": { "field2.1": "test" } } }

Filtering Creation of Entities and Associations

rejectDocCriteria provides a way to evaluate some data for a specific set of criteria.  If the return value is non-null (ie. the criteria has matched on some of the data) this returned data the document will not be used to generate entities, or associations. This parameter is a good way to create filters on the creation of metadata entities and associations.

The example source below shows how to populate this parameter with a script for filtering purposes.

be discarded.

In the example , used to analyze some data in a Twitter feed, the JSON object "link" would not be used as a metadata object for the creation of any entities or associations.  The object would simply be logged as part of the discovered metadata.

below, if the JSON field obtained from a twitter aggregation service didn't contain one of the two fields "link" or "object", then it would be discarded.

},        {
            "storageSettings": {
                "rejectDocCriteria": "$SCRIPT( if (null == _doc.metadata.json[0].link || null == _doc.metadata.json[0].object) return 'reject'; )"
            }
        }
    ]
}

...

Anchor
updating updating

Updating Documents

...

...

This script has access to the following Javascript objects:

...

The last evaluated expression in the script (eg you don't "return val;" you just end the script "val;"), which can be a string, an object, or an array of objects is placed in a metadata field called "_PERSISTENT_".

Preserving Metadata From Old Versions Of Documents

The following code saves the entirety of the old document's metadata:

...

 "onUpdateScript": "var retVal = _old_doc.metadata; retVal;"}
// RESULT (IN THE CASE OF A DOCUMENT THAT DOESN'T CHANGE):
{
    // Usual document fields
    "metadata": {
        "test1": "test",
        "test2": { "field": "value" },
        "_PERSISTENT_": [{
            "test1": "test",
            "test2": { "field": "value" },
        }]
    }
}

Generating New Metadata Based On Both New And Old Versions Of Documents

In this example, the return value will represent the delta of the two documents under comparison.

...