Versions Compared

Old Version 7

changes.mady.by.user andrew johnston

Saved on

compared with

New Version 8

changes.mady.by.user andrew johnston

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Examples

Basic Association

In the basic association example, the associations code block has been configured to act on the entity that represents the username of the Twitter user.  It also takes into account a secondary entity representing the hashtag of the user's tweet.  It acts on the document metadata values that have been provided.

The expected output will report what the tweets were about for the usernames.

...

code specifies an entity from the document metadata.  The association specifies the relationship between the type of offense and the date and time at which it was reported.

Code Block
  {
            "associations": [
               "assoc_type": "Event", {
                    "entity1_index": "$SCRIPT( return _doc.metadata.json[0].actor.preferredUsername + '/twitterhandle';)$metadata.offense,$metadata.method",
                    "entity2_indexgeotag": "$SCRIPT( return _iterator.text + '/hashtag'; )"{},
                    "iterateOvertime_start": "json.twitter_entities.hashtags"$metadata.reportdatetime",
                    "verb": "tweets_aboutreported",
                    "verb_category": "tweets_aboutcrime"
                },
            ]
        },

 

Output

In the sample output, an association has been created based on the two entities that were configured in the associations blocksample output:

The resulting output displays the details of the date and time, and the specific type of offense reported.

Code Block
{     ""associations": [
        {
            "assoc_typeentity1": "Eventtheft,2",
            "entity1_index": "focalcrmtheft,2/criminalactivity",
            "entity1_indexverb": "focalcrm/twitterhandlereported",
            "entity2verb_category": "crmcrime",
            "entity2time_indexstart": "crm/hashtag2011-01-29T00:00:00",
            "verbgeotag": "tweets_about", {
             "verb_category": "tweets_about"   "lat": 38.9099278028729,
                "lon": -77.0436067765966
            },

 

Multiplicative Associations

Multiplicative association are associations that are created by "multiplying" a combination of entities, locations, and times together to determine the number of associations to extract from the source data.

Multiplicative Associations are created by specifying which entity types to populate the entity1, entity2, geo_index, time_start, and time_end fields of the events created. The iterateOver field is used to specify the order in which the entity types are multiplied to determine the total number of associations.

In the associations block, two entities are specified for association using $SCRIPT.  In order to obtain the association, it will iterateOver the returned metadata which represents the Message-To field of the email message.

The specified scripts return metadata values for the associations.

 

Code Block
 {            "associations": [
            "assoc_type": "Summary"
        }

 

iterateOver

iterateOver can be used in more advanced cases where you need to iterate over entities to derive associations.

There are three different approaches to using iterateOver for association of entities

  • Multiplicative
  • Associative
  • JSON

Multiplicative

Multiplicative association are associations that are created by "multiplying" a combination of entities, locations, and times together to determine the number of associations to extract from the source data.

In the example, a perpetrator (Sunni Islamic Extremist) attacked multiple types of victims (an adult and a child) in Sri Lanka.

The association specification uses the multiplicative format to create events using the following math to determine the total number of associations: Entity1 (Person Perpetrator) * Entity2 (Victim Type) * Geo_index (Location) = Total Number of Associations.

Info
  • If the "iterateOver" field contains neither "," or "/" ("," is for additive associations, see below) then it is treated as an iterator over a metadata field, just as described under Specifying Entities, section "Create Entities from Arrays of Items".
  • To iterate just over a single entity type, use "dummy", eg "entity1/dummy" or "entity2,dummy" (The '/' vs ',' are equivalent in this case).
Code Block
},
                {
                    "creationCriteriaScript": "$FUNC( isOrganizationSpecified(); )",
                    "entity1": "Organization",
                    "entity2": "FacilityType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attacked",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$FUNC( isOrganizationSpecified(); )",
                    "entity1": "Organization",
                    "entity2": "VictimType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attacked",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$FUNC( isOrganizationSpecified(); )",
                    "entity1": "Organization",
                    "entity2": "HostageType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "took hostage",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$SCRIPT( if (isOrganizationSpecified() == false) return true; )",
                    "entity1": "PersonPerpetrator",
                    "entity2": "FacilityType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attacked",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$SCRIPT( if (isOrganizationSpecified() == false) return true; )",
                    "entity1": "PersonPerpetrator",
                    "entity2": "VictimType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "attacked",
                    "verb_category": "assault/attack"
                },
                {
                    "creationCriteriaScript": "$SCRIPT( if (isOrganizationSpecified() == false) return true; )",
                    "entity1": "PersonPerpetrator",
                    "entity2": "HostageType",
                    "geo_index": "Location",
                    "iterateOver": "entity1/entity2/geo_index",
                    "time_start": "$SCRIPT( return _doc.metadata.incidentdate[0]; )",
                    "verb": "took hostage",
                    "verb_category": "assault/attack"
                }

 

Sample Output:

TODO add sample output.

Associative

Additive associations cover the less common case where (eg) 2 entity types have the same number of elements and are ordered "in lock step". For example:

Code Block
"entities": [
	{ "index": "alex/person", ... },
	{ "index": "craig/person", ... },
	{ "index": "baltimore/city", ... },
	{ "index": "washington dc/city", ...},
	...
]

In this case the additive association specification:

Code Block
{
	"iterateOver": "entity1,entity2", // note "," instead of "/"
	"entity1": "Person",
	"entity2": "City",
	"verb_category": "lives in",
	...
}

Would generate the 2 associations "alex/person lives in baltimore/city" and "craig/person lives in washintgon dc/city".

JSON

In the example, the associations code block has been configured to act on the entity that represents the username of the Twitter user.  It also takes into account a secondary entity representing the hashtag of the user's tweet.  It acts on the document metadata values that have been provided.

The expected output will report what the tweets were about for the usernames.

Code Block
 {                    "assoc_type": "Event",
                    "entity1_index": "$SCRIPT( return _doc.metadata.json[0].actor.preferredUsername + '/twitterhandle';)",
                    "entity2_index": "$SCRIPT( return _iterator.text + '/hashtag'; )",
                    "iterateOver": "json.twitter_entities.hashtags",
 {                     "assoc_typeverb": "Eventtweets_about",
                    "entity1verb_category": "$SCRIPT( return _doc.metadata._FILE_METADATA_[0].metadata.Author[0];)"tweets_about"
                },
                },

 

Output:

In the sample output, an association has been created based on the two entities that were configured in the associations block.

Code Block
{
    "entity2associations": "$SCRIPT(return _value;)",[
        {
            "iterateOverassoc_type": "email_meta.Message-ToEvent",
            "entity1": "focalcrm",
      "time_start": "$SCRIPT( return _doc.publishedDate;)",        "entity1_index": "focalcrm/twitterhandle",
            "verbentity2": "emailedcrm",
      
             "verbentity2_categoryindex": "emailedcrm/communicated"
        hashtag",
       }

 

 

Output

In the sample output, the association displays the sender and receiver of the email message.

Code Block
 ],    "associationsverb": ["tweets_about",
        {    "verb_category": "tweets_about"
        "entity1": "cara.semperger@enron.com",
            "entity1_index": "cara.semperger@enron.com/account",
            "verb": "emailed",
            "verb_category": "emailed/communicated",
            "entity2": "will.smith@enron.com",
            "entity2_index": "will.smith@enron.com/account",
            "time_start": "2001-07-09T14:33:32",
            "assoc_type": "Event"
        }
    ],},

creationCriteriaScript

 

Info

Association fields generated from the entity loop are placed in "_iterator". For example, for "iterateOver": "entity1/entity2/geo_index", an _iterator object with the following fields is available in the Javascript: "_iterator.entity1_index", "_iterator.entity2_index", "_iterator.geo_index".

These fields can be usefully used together with "creationCriteriaScript" scriptlets to filter out unwanted associations, eg when looping over entity1 and entity2 with the same entity type, the following script would ensure the association didn't involve the same entity:

"creationCriteriaScript": "$SCRIPT( return _iterator.entity1_index != _iterator.entity2_index; )", "iterateOver": "entity1/entity2", "entity1": "EmailAddress", "entity2": "EmailAddress", //etc

The creationCriteriaScript runs before the association is generated (so can be safely used to remove items that would return errors).

 

IN PROGRESS

 

Panel

Footnotes:

Legacy documentation:

Legacy documentation:

 

...