...

iterateOver can be used in more advanced cases where you need to iterate over entities to derive associations.

There are three different approaches to using iterateOver for association of entities

...

.

Iterate Over a Metadata Field:

Specify a metadata field, in order to generate associations.

eg.  "iterateOver": "json.twitter_entities.user_mentions"

This is the same as iterating over a metadata array to obtain entities using Manual entities.

See detailed example below

Iterate Over a Single Entity Type:

Use "dummy" to iterate over only one entity type,

eg "entity1/dummy" or "entity2,dummy"

See detailed example below

Multiplicative:

Create one association for every combination of entities of specified types

...

Multiplicative

Multiplicative association are associations that are created by "multiplying" a combination of entities, locations, and times together to determine the number of associations to extract from the source data.

In the example, a perpetrator (Sunni Islamic Extremist) attacked multiple types of victims (an adult and a child) in Sri Lanka.

The association specification uses the multiplicative format to create events using the following math to determine the total number of associations: Entity1 (Person Perpetrator) * Entity2 (Victim Type) * Geo_index (Location) = Total Number of Associations.

...

eg. "iterateOver": "entity1/entity2/geo_index",

See detailed example below

Associative:

Create one association for every pair (/set) of entities of specified types

See detailed example below

creationCriteriaScripts

Association fields generated from the entity loop are placed in "_iterator". For example, for "iterateOver": "entity1/entity2/geo_index", an _iterator object with the following fields is available in the Javascript: "_iterator.entity1_index", "_iterator.entity2_index", "_iterator.geo_index".

These fields can be usefully used together with "creationCriteriaScript" scriptlets to filter out unwanted associations, eg when looping over entity1 and entity2 with the same entity type, the following script would ensure the association didn't involve the same entity:

"creationCriteriaScript": "$SCRIPT( return _iterator.entity1_index != _iterator.entity2_index; )", "iterateOver": "entity1/entity2", "entity1": "EmailAddress", "entity2": "EmailAddress", //etc

The creationCriteriaScript runs before the association is generated (so can be safely used to remove items that would return errors).

See detailed example below

Examples

iterateOver

Anchor
metadata metadata

iterateOver a Metadata Field

The following code example is used to process email communication between two parties.  The association between the two entities is formed by iterating over the metadata object: "email_meta.Message-To"

{
            "associations": [
           "creationCriteriaScript": "$FUNC( isOrganizationSpecified(); )",     {
                    "entity1assoc_type": "OrganizationEvent",
                    "entity2entity1": "FacilityType$SCRIPT( return _doc.metadata._FILE_METADATA_[0].metadata.Author[0];)",
                    "geo_indexentity2": "Location$SCRIPT(return _value;)",
                    "iterateOver": "entity1/entity2/geo_indexemail_meta.Message-To",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; publishedDate;)",
                    "verb": "attackedemailed",
                    "verb_category": "assaultemailed/attackcommunicated"
                },
            ]
   {     }

Output:

The output displays the association between the sender and receiver in the email correspondence.

 ],
    "associations": [
        {
  "creationCriteriaScript": "$FUNC( isOrganizationSpecified(); )",       "entity1": "cara.semperger@enron.com",
            "entity1_index": "Organizationcara.semperger@enron.com/account",
              "verb": "emailed",
     "entity2": "VictimType",        "verb_category": "emailed/communicated",
            "geo_indexentity2": "Locationwill.smith@enron.com",
       
            "iterateOverentity2_index": "entity1/entity2/geo_indexwill.smith@enron.com/account",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attacked2001-07-09T14:33:32",
                    "verb_category"assoc_type": "assault/attackEvent"
        }
       },
                {
                    "creationCriteriaScript": "$FUNC( isOrganizationSpecified(); )",
        ],

Anchor
multi multi

Multiplicative

Multiplicative association are associations that are created by "multiplying" a combination of entities, locations, and times together to determine the number of associations to extract from the source data.

In the example, a perpetrator (Sunni Islamic Extremist) attacked multiple types of victims (an adult and a child) in Sri Lanka.

The association specification uses the multiplicative format to create events using the following math to determine the total number of associations: Entity1 (Person Perpetrator) * Entity2 (Victim Type) * Geo_index (Location) = Total Number of Associations.

},
             "entity1": "Organization",   {
                    "entity2creationCriteriaScript": "HostageType$FUNC( isOrganizationSpecified(); )",
                    "geoentity1": "Organization",
                    "entity2": "FacilityType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "took hostageattacked",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$SCRIPT$FUNC( if (isOrganizationSpecified() == false) return true; ); )",
                    "entity1": "PersonPerpetratorOrganization",
                    "entity2": "FacilityTypeVictimType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attacked",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$SCRIPT$FUNC( if (isOrganizationSpecified() == false) return true; )",
                    "entity1": "PersonPerpetratorOrganization",
                    "entity2": "VictimTypeHostageType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attackedtook hostage",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$SCRIPT( if (isOrganizationSpecified() == false) return true; )",
                    "entity1": "PersonPerpetrator",
                    "entity2": "HostageTypeFacilityType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "took hostageattacked",
                    "verb_category": "assault/attack"
                }

Sample Output:

TODO add sample output.

Associative

Additive associations cover the less common case where (eg) 2 entity types have the same number of elements and are ordered "in lock step". For example:

"entities": [
	{ "index": "alex/person", ... },
	{ "index": "craig/person", ... },
	{ "index": "baltimore/city", ... },
	{ "index": "washington dc/city", ...},
	...
]

In this case the additive association specification:

{
	"iterateOver": "entity1,entity2", // note "," instead of "/"
	,
                {
                    "creationCriteriaScript": "$SCRIPT( if (isOrganizationSpecified() == false) return true; )",
                    "entity1": "PersonPersonPerpetrator",
	
                    "entity2": "CityVictimType",
	"verb_category
                    "geo_index": "lives inLocation",
	...
}

Would generate the 2 associations "alex/person lives in baltimore/city" and "craig/person lives in washintgon dc/city".

JSON

In the example, the associations code block has been configured to act on the entity that represents the username of the Twitter user.  It also takes into account a secondary entity representing the hashtag of the user's tweet.  It acts on the document metadata values that have been provided.

The expected output will report what the tweets were about for the usernames.

 {
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "assoc_typeverb": "Eventattacked",
                    "entity1verb_indexcategory": "$SCRIPT( return _doc.metadata.json[0].actor.preferredUsername + '/twitterhandle';)",assault/attack"
                },
          "entity2_index": "$SCRIPT( return _iterator.text + '/hashtag'; )", {
                    "iterateOvercreationCriteriaScript": "json.twitter_entities.hashtags$SCRIPT( if (isOrganizationSpecified() == false) return true; )",
                    "verbentity1": "tweets_aboutPersonPerpetrator",
                    "verb_categoryentity2": "tweets_about"
                }HostageType",
                },

Output:

In the sample output, an association has been created based on the two entities that were configured in the associations block.

{    "geo_index": "associations": [
Location",
       {             "assoc_typeiterateOver": "Evententity1/entity2/geo_index",
                    "entity1time_start": "focalcrm$SCRIPT( return _doc.metadata.incidentdate[0]; )",
             "entity1_index": "focalcrm/twitterhandle",       "verb": "took hostage",
      "entity2": "crm",             "entity2verb_indexcategory": "crmassault/hashtag",attack"
                 "verb": "tweets_about",
            "verb_category": "tweets_about"
        },

Best Practices for Using iterateOver for Associations

Association fields generated from the entity loop are placed in "_iterator". For example, for "iterateOver": "entity1/entity2/geo_index", an _iterator object with the following fields is available in the Javascript: "_iterator.entity1_index", "_iterator.entity2_index", "_iterator.geo_index".

These fields can be usefully used together with "creationCriteriaScript" scriptlets to filter out unwanted associations, eg when looping over entity1 and entity2 with the same entity type, the following script would ensure the association didn't involve the same entity:

"creationCriteriaScript": "$SCRIPT( return _iterator.entity1_index != _iterator.entity2_index; )", "iterateOver": "entity1/entity2", "entity1": "EmailAddress", "entity2": "EmailAddress", //etc

...

}

Sample Output:

TODO add sample output.

Anchor
associative associative

Associative

Additive associations cover the less common case where (eg) 2 entity types have the same number of elements and are ordered "in lock step". For example:

"entities": [
	{ "index": "alex/person", ... },
	{ "index": "craig/person", ... },
	{ "index": "baltimore/city", ... },
	{ "index": "washington dc/city", ...},
	...
]

In this case the additive association specification:

{
	"iterateOver": "entity1,entity2", // note "," instead of "/"
	"entity1": "Person",
	"entity2": "City",
	"verb_category": "lives in",
	...
}

Would generate the 2 associations "alex/person lives in baltimore/city" and "craig/person lives in washintgon dc/city".

...