Overview
This toolkit element enables the generation of one or more types of entity based on the document or content metadata. The expressions default to replacement strings, or $SCRIPT(...) can be used to return a string using javascript.
...
| Parameter | Description |
|---|---|
iterateOver | OPTIONAL: If specified, a metadata field (nesting supported using dot notation) which is looped over to generate calls with _value/_iterator/_index |
disambiguated_name | MANDATORY: String/script, the disambiguated name of the entity |
actual_name | OPTIONAL: String/script, the actual name of the entity if different to the disambiguated name |
dimension | MANDATORY: String/script: Must be/return one of "Who", "What", "Where" |
type | MANDATORY: String/script: It is recommended to use a type from the // OpenCyc, AlchemyAPI, or OpenCalais ontologies, for compatibility with future Infinit.e features |
linkdata | OPTIONAL: if present should return a comma-separated list of URLs (commas should be URL-encoded) |
relevance | OPTIONAL: String/script: Must specify/return a double/string-parsable-into-a-double |
sentiment | OPTIONAL: String/script: Must specify/return a double/string-parsable-into-a-double, by convention this is between -1.0 and 1.0. |
frequency | OPTIONAL: Must specify/return a long/string-parsable-into-a-long |
geotag | OPTIONAL: Format is identical to the docGeo format specified above "lat": "string", "lon": "string, "city": "string", "stateProvince": "string, "country": "string", "countryCode": "string |
ontology_type | OPTIONAL: String/script: Only used if geotag is specified: // allows specification of the scale of the geographic entity (see below for useful link), defaults to "point" |
useDocGeo | OPTIONAL: If true, uses any lat/long generated from the top level "docGeo" specification, defaults to false |
creationCriteriaScript | OPTIONAL: script: If populated, runs a user script function and if return value is false doesn't create the object |
Examples
...
Basic
In the example source, the entities block has been configured to set actual_name as a value which will be provided by the document metadata. Entities will expect a JSON object which is a representation of the Twitter user's display name.
...
| Code Block |
|---|
], "mediaType": ["Social"],
"metadata": {"json": [{
"actor": {
"displayName": "CRM Buddy",
"followersCount": "245",
"friendsCount": "0",
"id": "id:twitter.com:835627776",
"image": "http://a0.twimg.com/profile_images/2630355549/8cad59efaddd57283dbb159332336744_normal.jpeg",
"languages": ["en"],
"link": "http://www.twitter.com/FocalCRM",
"links": [{"rel": "me"}],
"listedCount": "6",
"objectType": "person",
"postedTime": "2012-09-20T13:59:56.000Z",
"preferredUsername": "FocalCRM",
"statusesCount": "3688",
"summary": "",
"verified": "false"
}, |
...
Extracting Entities From Arrays with iterateOver
You can use iterateOver to create an entity for each object in an array.
...
Basic Example
In the example below, the source for Entities has been configured to iterateOver the document metadata "victim."
In the example above the iterateOver value is set to "victim" meaning that the Structured Analysis Harvester will iterate (or loop) over the metadata.victim[] objects and create an entity for each object in the array.
In the example, frequency uses $FUNC to call the function getVictimCount, which was previously imported into the script engine. For more information about calling imported javascript functions, see Using Javascript.
...
| Code Block |
|---|
},
{
"dimension": "Who",
"disambiguated_name": "$FUNC( getVictim(); )",
"frequency": "$FUNC( getVictimCount(); )",
"type": "VictimType",
"useDocGeo": false,
"iterateOver": "victim"
},
{
"dimension": "Who",
"disambiguated_name": "$FUNC( getVictim(); )",
"frequency": "$hostagecount",
"type": "HostageType",
"useDocGeo": false,
"iterateOver": "victim"
}
]
}, |
Sample Output:
The sample output displays the entities created based on the array that was returned for the metadata field "victim."
| Code Block |
|---|
},
{
"actual_name": "Targeted, Civilian, Adult from Afghanistan",
"dimension": "Who",
"disambiguated_name": "Targeted, Civilian, Adult from Afghanistan",
"doccount": 0,
"frequency": 5,
"index": "targeted, civilian, adult from afghanistan/victimtype",
"relevance": 0,
"totalfrequency": -1,
"type": "VictimType"
},
{
"actual_name": "Targeted, Civilian, Child from Afghanistan",
"dimension": "Who",
"disambiguated_name": "Targeted, Civilian, Child from Afghanistan",
"doccount": 0,
"frequency": 2,
"index": "targeted, civilian, child from afghanistan/victimtype",
"relevance": 0,
"totalfrequency": -1,
"type": "VictimType"
}, |
Metadata:
The original document metadata used to create the "victim" entities.
| Code Block |
|---|
"victim": [
{
"child": "No",
"combatant": "No",
"deadcount": "1",
"definingcharacteristic": "Unknown",
"hostagecount": "0",
"indicator": "Targeted",
"nationality": "Afghanistan",
"targetedcharacteristic": "Unknown",
"victimtype": "Civilian",
"woundedcount": "4"
},
{
"child": "Yes",
"combatant": "No",
"deadcount": "2",
"definingcharacteristic": "Unknown",
"hostagecount": "0",
"indicator": "Targeted",
"nationality": "Afghanistan",
"targetedcharacteristic": "Unknown",
"victimtype": "Civilian",
"woundedcount": "0"
}
], |
Specifying Entity Location
...
iterateOver Best Practices
This section contains a few useful tips for using iterateOver.
For non-nested entity specification objects, the first field in the iterateOver field refers to the metadata object, eg "iterateOver": "location" refers to "_doc.metadata.location".
For nested objects, the first field refers to the "parent" object (but you shouldn't be using nesting now that dot notation is available!).
- Reminder:
Nesting:
Nesting is supported using the "dot notation" eg if in the above instance, the victim was inside an object (or array of objects) called "more_information", then the "iterateOver" field would be set to "more_information.victim".
(This would be equivalent to the less tidy technique of nesting the Entity Specification JSON object, the first having "iterateOver": "more_information", and containing a second Entity Specification JSON object identical to the original example).
About Arrays and Objects:
Arrays and objects are treated equally in the dot-notation (ie an object is just treated like an array of size 1)
eg. for both "{ A: { B: { C: "value" } } }" and "{ A: [ B: [ { C: [ value ] } ] ] }", you would use iterateOver: "A.B.C" to get to "value"
If you are iterating over an object, then use "_iterator.FIELD" in scripts, "$FIELD" for normal strings.
| Info |
|---|
Note that "$metadata.X" won't work inside "iterateOver" clauses, you have to use constructs like "$SCRIPT( return _doc.metadata.X[0]; )" to get at the top-level fields. We will probably fix this at some point.) |
If you are iterating over a value then use "_value" in scripts, "$" for normal strings.
Primitives:
If any of the fields point to primitives (eg B: [ "val1", "val2" ] in the example above) then an error is thrown unless the "creation criteria" script for the nested object is specified (C in this example).
You can still throw errors from the script by checking if "(_iterator==null)" if you want to) This enables writing objects that will handle fields being either primitives or objects (eg by checking vs _iterator and _value).
...
Specifying Entity Geographic Location
It is also possible to set values for entity parameters using Javascript functions declared in Globals.
In the example source, the entity block is configured to output an entity with dimension "where." $SCRIPT is used to call functions already declared in globalsGlobals.
In the example, inline javascript Javascript is used by enclosing the javascript in the $SCRIPT() block. For more information about inline javascript, see Using Javascript.
...
In some cases, it will not be clear what geographical type a field is (eg. a freeform field that might be city, state, or country). The geographical specification allows you to specify alternatives.
...