Overview

This toolkit element passes the document full text to an external (or embedded) extraction engine to return entities and associations (and optionally metadata).

Format

{
	"display": string,
	"featureEngine": {
		"criteria":string,// A javascript expression that is passed the document as _doc - if returns false then this pipeline element is bypassed
		"enginename":string,// The name of the text engine to use (can be fully qualified (eg "com.ikanow.infinit.e.harvest.boilerpipe"), or just the name (eg "boilerpipe") if the engine is registered in the Infinit.e system configuration)
		"engineConfig":{"config_param_name",string,...},// The configuration object to be passed to the engine
		"entityFilter":string,// (regex applied to entity indexes, starts with "+" or "-" to indicate inclusion/exclusion, defaults to include-only)
		"assocFilter":string,// (regex applied to new-line separated association indexes, starts with "+" or "-" to indicate inclusion/exclusion, defaults to include-only) 
	}
}

Description

Feature extraction uses text obtained from the text extraction stage to generate entities, associations, and potentially metadata.  Text extraction is a separate stage in the pipeline with different extraction engines.

For example Alchemy API can perform both text extraction using the Alchempy API, and feature extraction using the Alchemy metadata API.

The following table describes the parameters of the feature extraction configuration.

criteria

enginename

engineConfig

entityFilter

assocFilter

IKANOWS supports the following feature extraction engines:

• Textrank*
• OpenCalais*
• AlchemyAPI-metadata**
• salience*

*requires a text extractor beforehand.

*includes its own built-in text extractor.

Examples

This section describes the configuration details for the supported extractors, and provides examples where applicable.

OpenCalais

The following custom configuration parameters are possible for Open Calais and can be set using the engineConfig parameter.

Examples

The following example source uses Alchemy API as the text engine, and OpenCalais as the feature engine.  In both cases, the default configuration of these engines is used to output entities and associations for the ingested RSS data.

{
    "description": "Article on Medical Issues",
    "harvestBadSource": false,
    "isApproved": true,
    "isPublic": true,
    "key": "http.www.mayoclinic.com.rss.blog.xml",
    "mediaType": "News",
    "modified": "Oct 19, 2010 11:31:59 AM",
    "tags": [
        "topic:healthcare",
        "industry:healthcare",
        "mayo clinic",
        "health"
    ],
    "title": "MayoClinic: General Topics",
    "processingPipeline": [
        {
            "feed": {
                "extraUrls": [
                    {
                        "url": "http://www.mayoclinic.com/rss/blog.xml"
                    }
                ]
            }
        },
        {
            "textEngine": {
                "engineName": "AlchemyAPI"
            }
        },
        {
            "featureEngine": {
                "engineName": "OpenCalais"
            }
        }
    ]
}

Salience

The following custom configuration parameters are possible for Salience and can be set using the engineConfig parameter.

Examples

Setting the data path:

The data path for salience should be set using the following format:

"<BASEDIR>/data"

where BASEDIR is the environment variable "lxainstall" - will normally be set to "/opt/lexalytics/salience-5.x/"

Setting the data path for language packs:

You can use the data_path parameter to switch between different language packs.

To set the language pack for Spanish

 "/opt/lexalytics/salience-5.x/spanish".

Setting the license_path:

You can use the license_path parameter to specify the license for salience.

For example

"<BASEDIR>/license.v5".

Should only need to be changed when running the harvest engine locally. Paths that don't start with "/" are assumed to be relative to BASEDIR.