Overview

Once data is ingested into Infint.e from the various extractors it is stored in JSON format including its metadata fields and content.  It also contains sub-objects such as entities and associations.

Infinit.e provides a variety of mechanisms by which documents can be updated over time.  For example, the updateCycle_secs field can be set on RSS sources to periodically update documents based on RSS feeds.  You can configure the behavior of how Infinit.e stores documents and updates existing documents by using Document storage settings.  Persistent fields can be called out which will remain intact across document updates.

Format

{
	"display": string,
	"storageSettings": {
		"rejectDocCriteria":string,//OPTIONAL: If populated, runs a user script function and if return value is non-null doesn't create the object and logs the output.  *Not* wrapped in $SCRIPT().
		"onUpdateScript":string,//OPTIONAL: Used to preserve existing metadata when documents are updated, and also to generate new metadata based on the differences between old and new documents. *Not* wrapped in $SCRIPT().
		"metadataFieldStorage"string,//OPTIONAL: A comma-separated list of top-level metadata fields to either exclude (if "metadataFields" starts with '-'), or only include (starts with '+', default) - the fields are deleted at that point in the pipeline.
	}
} 

Description

The following table describes the parameters of the document storage settings configuration.

onUpdateScript

Use Cases

The fields of the Document storage settings configuration can be used to support the following use cases

• Determine which metadata fields will be stored and used for creation of entities/associations

See examples below.

• Determine how documents will be updated
  • Retain existing metadata/entities/associations
  • Build new metadata/entities/associations

See examples below.

Examples

Document Storage Settings

Metadata Field Storage

You can use metadataFieldStorage to either include or exclude metadata fields from document creation.  To use this field you must specify a comma-separated list of top-level metadata fields to either exclude (if "metadataFields" starts with '-'), or only include (starts with '+', default).

Filtering Creation of Entities and Associations

rejectDocCriteria provides a way to evaluate some data for a specific set of criteria.  If the return value is non-null (ie. the criteria has matched on some of the data) this returned data will not be used to generate entities, or associations. This parameter is a good way to create filters on the creation of metadata entities and associations.

The example source below shows how to populate this parameter with a script for filtering purposes.

In the example, used to analyze some data in a Twitter feed, the JSON object "link" would not be used as a metadata object for the creation of any entities or associations.  The object would simply be logged as part of the discovered metadata.

},        {
            "storageSettings": {
                "rejectDocCriteria": "$SCRIPT( if (null == _doc.metadata.json[0].link || null == _doc.metadata.json[0].object) return 'reject'; )"
            }
        }
    ]
}

Updating Documents

It is possible to use onUpdateScript to configure the behavior of how documents will be updated.

Existing documents can be updated in a number of different cases:

• Files can be updated (changing their "modified time")
• For RSS feeds/URLs, the source parameter "updateCycle_secs" will periodically update the file.
• Database sources can be updated as the result of a SQL call.

When a document is updated it is essentially equivalent to deleting and the re-creating it, except that its "_id" field is preserved).

Document storage settings provides a mechanism to do the following useful activities:

• Preserve metadata from the old document (eg so the entities/associations can be recreated)
• Generate new metadata (and thence entities/associations) based on the differences between successive documents.

onUpdateScript can be configured with a script, that will either preserve metadata from the old document, or create new metadata.

This script has access to the following Javascript objects:

• "_old_doc": The document object that is about to be deleted
• "_doc": The newly created document object after all metadata/entity/association creation.

The last evaluated expression in the script (eg you don't "return val;" you just end the script "val;"), which can be a string, an object, or an array of objects is placed in a metadata field called "_PERSISTENT_".

Preserving Metadata

The following code saves the entirety of the old document's metadata:

 "onUpdateScript": "var retVal = _old_doc.metadata; retVal;"}
// RESULT (IN THE CASE OF A DOCUMENT THAT DOESN'T CHANGE):
{
    // Usual document fields
    "metadata": {
        "test1": "test",
        "test2": { "field": "value" },
        "_PERSISTENT_": [{
            "test1": "test",
            "test2": { "field": "value" },
        }]
    }
}

Generating New Metadata

In this example, the return value will represent the delta of the two documents under comparison.

"onUpdateScript": "var delta = _old_doc.metadata.length - _doc.metadata.length; var retVal = { 'delta': delta }; retVal;"}