Overview
This toolkit element enables the generation of one or more types of entity based on the document or content metadata. The expressions default to replacement strings, or $SCRIPT(...) can be used to return a string using javascript.
Format
{ "display": string, "entities": [ { "iterateOver" : "string", // OPTIONAL: If specified, a metadata field (nesting supported using dot notation) which is looped over to generate calls with _value/_iterator/_index "disambiguated_name" : "string", // MANDATORY: String/script, the disambiguated name of the entity "actual_name" : "string", // OPTIONAL: String/script, the actual name of the entity if different to the disambiguated name "dimension" : "string", // MANDATORY: String/script: Must be/return one of "Who", "What", "Where" "type" : "string", // MANDATORY: String/script: It is recommended to use a type from the // OpenCyc, AlchemyAPI, or OpenCalais ontologies, for compatibility with future Infinit.e features "linkdata": "string", // OPTIONAL: if present should return a comma-separated list of URLs (commas should be URL-encoded) "relevance" : "string", // OPTIONAL: String/script: Must specify/return a double/string-parsable-into-a-double "sentiment" : "string", // OPTIONAL: String/script: Must specify/return a double/string-parsable-into-a-double, by convention this is between -1.0 and 1.0. "frequency" : "string", // OPTIONAL: Must specify/return a long/string-parsable-into-a-long "geotag" : { // OPTIONAL: Format is identical to the docGeo format specified above "lat": "string", "lon": "string, "city": "string", "stateProvince": "string, "country": "string", "countryCode": "string }, "ontology_type": "string", // OPTIONAL: String/script: Only used if geotag is specified: // allows specification of the scale of the geographic entity (see below for useful link), defaults to "point" "useDocGeo": "boolean", // OPTIONAL: If true, uses any lat/long generated from the top level "docGeo" specification, defaults to false "creationCriteriaScript" : "string", // OPTIONAL: script: If populated, runs a user script function and if return value is false doesn't create the object } ] }
Description
Entities are the who, what, and where's contained within a record (i.e. people, places, and things).
The following table describes the parameters of the manual entities configuration.
Parameter | Description |
---|---|
iterateOver | OPTIONAL: If specified, a metadata field (nesting supported using dot notation) which is looped over to generate calls with _value/_iterator/_index |
disambiguated_name | MANDATORY: String/script, the disambiguated name of the entity |
actual_name | OPTIONAL: String/script, the actual name of the entity if different to the disambiguated name |
dimension | MANDATORY: String/script: Must be/return one of "Who", "What", "Where" |
type | MANDATORY: String/script: It is recommended to use a type from the // OpenCyc, AlchemyAPI, or OpenCalais ontologies, for compatibility with future Infinit.e features |
linkdata | OPTIONAL: if present should return a comma-separated list of URLs (commas should be URL-encoded) |
relevance | OPTIONAL: String/script: Must specify/return a double/string-parsable-into-a-double |
sentiment | OPTIONAL: String/script: Must specify/return a double/string-parsable-into-a-double, by convention this is between -1.0 and 1.0. |
frequency | OPTIONAL: Must specify/return a long/string-parsable-into-a-long |
geotag | OPTIONAL: Format is identical to the docGeo format specified above "lat": "string", "lon": "string, "city": "string", "stateProvince": "string, "country": "string", "countryCode": "string |
ontology_type | OPTIONAL: String/script: Only used if geotag is specified: // allows specification of the scale of the geographic entity (see below for useful link), defaults to "point" |
useDocGeo | OPTIONAL: If true, uses any lat/long generated from the top level "docGeo" specification, defaults to false |
creationCriteriaScript | OPTIONAL: script: If populated, runs a user script function and if return value is false doesn't create the object |
Examples
Entities
In the example source, the entities block has been configured to set actual_name
as a value which will be provided by the document metadata. Entities will expect a JSON object which is a representation of the Twitter user's display name.
Data is extracted from the source using the $ operator.
}, { "entities": [ { "actual_name": "$metadata.json.actor.displayName", "dimension": "Who", "disambiguated_name": "$metadata.json.actor.preferredUsername", "linkdata": "$metadata.json.actor.link", "type": "TwitterHandle" },
Sample Output
In the output we see that an entity has been created based on the Twitter user's Twitter handle.
{ "actual_name": "CRM Buddy", "dimension": "Who", "disambiguated_name": "FocalCRM", "doccount": 0, "frequency": 1, "index": "focalcrm/twitterhandle", "linkdata": ["http://www.twitter.com/FocalCRM"], "relevance": 0, "totalfrequency": -1, "type": "TwitterHandle" },
Metadata
The metadata that document metadata used to create the JSON object.
], "mediaType": ["Social"], "metadata": {"json": [{ "actor": { "displayName": "CRM Buddy", "followersCount": "245", "friendsCount": "0", "id": "id:twitter.com:835627776", "image": "http://a0.twimg.com/profile_images/2630355549/8cad59efaddd57283dbb159332336744_normal.jpeg", "languages": ["en"], "link": "http://www.twitter.com/FocalCRM", "links": [{"rel": "me"}], "listedCount": "6", "objectType": "person", "postedTime": "2012-09-20T13:59:56.000Z", "preferredUsername": "FocalCRM", "statusesCount": "3688", "summary": "", "verified": "false" },
Extracting Entities From Arrays
You can use manual entities with iterateOver
to create an entity for each object in the array, when the document metadata contains an array for a specific metadata field.
Source
In the example below, the source for entities has been configured to iterateOver
the document metadata "victims."
In the example, frequency
calls the function getVictimCount
, previously imported into the script engine, using the $FUNC() block. For more information about calling imported javascript functions, see Javascript globals.
}, { "dimension": "Who", "disambiguated_name": "$FUNC( getVictim(); )", "frequency": "$FUNC( getVictimCount(); )", "type": "VictimType", "useDocGeo": false, "iterateOver": "victim" }, { "dimension": "Who", "disambiguated_name": "$FUNC( getVictim(); )", "frequency": "$hostagecount", "type": "HostageType", "useDocGeo": false, "iterateOver": "victim" } ] },
Sample Output
In the sample output you can see some of the entities outputted based on the example source. The entities have been created based on the array that was returned for the metadata field "victim."
}, { "actual_name": "Targeted, Civilian, Adult from Afghanistan", "dimension": "Who", "disambiguated_name": "Targeted, Civilian, Adult from Afghanistan", "doccount": 0, "frequency": 5, "index": "targeted, civilian, adult from afghanistan/victimtype", "relevance": 0, "totalfrequency": -1, "type": "VictimType" }, { "actual_name": "Targeted, Civilian, Child from Afghanistan", "dimension": "Who", "disambiguated_name": "Targeted, Civilian, Child from Afghanistan", "doccount": 0, "frequency": 2, "index": "targeted, civilian, child from afghanistan/victimtype", "relevance": 0, "totalfrequency": -1, "type": "VictimType" },
Metadata
The metadata was used by Manual entities to create the two entities for metadata value "victim."
"victim": [ { "child": "No", "combatant": "No", "deadcount": "1", "definingcharacteristic": "Unknown", "hostagecount": "0", "indicator": "Targeted", "nationality": "Afghanistan", "targetedcharacteristic": "Unknown", "victimtype": "Civilian", "woundedcount": "4" }, { "child": "Yes", "combatant": "No", "deadcount": "2", "definingcharacteristic": "Unknown", "hostagecount": "0", "indicator": "Targeted", "nationality": "Afghanistan", "targetedcharacteristic": "Unknown", "victimtype": "Civilian", "woundedcount": "0" } ],
Specifying Entity Location
In the example source, the entity block is configured to output an entity using the dimension "where." The location is determined using javascript, and the scripts are calling on function methods defined in the javascript Globals section.
In the example, inline javascript is used by enclosing the javascript in the $SCRIPT() block. For more information about inline javascript, see Javascript globals.
}, { "dimension": "Where", "disambiguated_name": "$metadata.json.actor.location.displayName", "geotag": { "city": "$SCRIPT( return getAddressVal( _doc.metadata.json[0].actor.location.displayName, 0 ) )", "stateProvince": "$SCRIPT( return getRegion(getAddressVal( _doc.metadata.json[0].actor.location.displayName, 1 )) )", "countryCode": "US", "alternatives": [ { "stateProvince": "$SCRIPT( return getRegion(getAddressVal( _doc.metadata.json[0].actor.location.displayName, 1 )) )", "countryCode": "US" } ] },
The example source will return output with the dimension of "Where" if the location can be determined by the metadata.
In some cases, it will not be clear what geographical type a field is (eg a freeform field that might be city, state, or country). The geographical specification allows you to specify alternatives.
The alternatives are tried in order until one of them works or there are no more to try.