Format

{
	"display": string,
	"featureEngine": {
		"criteria":string,// A javascript expression that is passed the document as _doc - if returns false then this pipeline element is bypassed
		"enginename":string,// The name of the text engine to use (can be fully qualified (eg "com.ikanow.infinit.e.harvest.boilerpipe"), or just the name (eg "boilerpipe") if the engine is registered in the Infinit.e system configuration)
		"engineConfig":{"config_param_name",string,...},// The configuration object to be passed to the engine
		"entityFilter":string,// (regex applied to entity indexes, starts with "+" or "-" to indicate inclusion/exclusion, defaults to include-only)
		"assocFilter":string,// (regex applied to new-line separated association indexes, starts with "+" or "-" to indicate inclusion/exclusion, defaults to include-only),
		"exitOnError": boolean // if true (default) true then errors during featureExtraction will cause the doc to be removed from the pipeline. If false, the processing will continue.
	}
}

Description

Feature extraction uses text obtained from the text extraction stage to generate entities, associations, and potentially metadata.  Text extraction is a separate stage in the pipeline with different extraction engines.

The following table describes the parameters of the feature extraction configuration.

criteria

engineName

engineConfig

entityFilter

assocFilter

exitOnError

engineConfig

The "engineConfig" configuration object is a set of key/value pairs of strings that depends on the extractor type, "pre-integrated" configurations are described below, eg:

{
	"featureEngine": {
		"engineName": "salience",
		"engineConfig": {
			"salience.shortFormContent": "true",
			"salience.kw_score_threshold": "0.5"
		}
	}
}

Supported Feature Extraction Engines

IKANOWS supports the following feature extraction engines:

• Textrank* ("textrank")
• OpenCalais* ("opencalais")
• AlchemyAPI** ("alchemyapi")
• AlchemyAPI-metadata** ("alchemyapi-metadata")
• salience* ("salience")
• regex* - a mechanism for converting regexes into entities from text or metadata ("regex")

*requires a text extractor beforehand.

**includes its own built-in text extractor, though can run behind an alternative text extractor also.

Textrank

A "behind-the-firewall" OSS solution that uses TextRank (with some OpenNLP-based pre-processing) to extract key phrases from the text.

Open Calais

Extracts entities and associations using the free OpenCalais service. No sentiment analysis function is available at this time.

OpenCalais will truncate text that is larger than 99KB.

The following custom configuration parameters are possible for Open Calais and can be set using the engineConfig parameter.

The parameters should all be prefixed by "app.opencalais."

Alchemy API

Uses the Named Entity Extraction and Sentiment Analysis functions of the commercial AlchemyAPI service (there is a free tier for AlchemyAPI but it is very restrictive). AlchemyAPI has the ability to extract associations (as well as much more), but this feature has not yet been integrated into the tool.

There are two Alchemy services that can be called:

• Alchemy API
• Alchemy API-metadata*

*includes many of the same features of Alchemy API but also allows more advanced batching of documents and keyword control.

AlchemyAPI will truncate text longer than 145KB.

Both of these services can support both text extraction and feature extraction.  However, if you only need to perform text extraction Alchemy API should be used.

You can use engineConfig to pass configuration parameters to the Alchemy API service as follows:

AlchemyAPI-metadata

Uses the Keyword Extraction and Sentiment Analysis functions of the commercial AlchemyAPI service (there is a free tier for AlchemyAPI but it is very restrictive). This service also tags AlchemyAPI Concepts to documents as metadata.

You can use engineConfig to pass the parameters of the Alchemy API metadata configuration, as described below.

Salience

Salience* is an embedded solution requiring no internet connectivity. Generates reliable topics, keywords, entities, and associations from text. Named entities and topics are customizable by users.

*Enterprise only

The following custom configuration parameters are possible for Salience and can be set using the engineConfig parameter.

The following parameters should all be prefixed by "salience." (no app unlike the others)

Setting the data path:

The data path for salience should be set using the following format:

"<BASEDIR>/data"

where BASEDIR is the environment variable "lxainstall" - will normally be set to "/opt/lexalytics/salience-5.x/"

Setting the data path for language packs:

You can use the data_path parameter to switch between different language packs.

To set the language pack for Spanish

 "/opt/lexalytics/salience-5.x/spanish".

Setting the license_path:

You can use the license_path parameter to specify the license for salience.

For example

"<BASEDIR>/license.v5".

Should only need to be changed when running the harvest engine locally. Paths that don't start with "/" are assumed to be relative to BASEDIR.

date_extractor

The Date Extractor* esigned to find the published date posted in a document's full text and set it as the document's published date.

*Enterprise Only

The following custom configuration parameters are possible for the Date Extractor and can be set using the engineConfig parameter.

The following parameters should all be prefixed by "date."

**Processing Order Refers to the order that the values are checked (if enabled.) Logic with the processing order of 1 will be checked first if it is enabled, other wise 2 will be checked, and so on. Once a value is found, the searching will not continue and that value will become the document published date.

Regex

Unlike many other extractors, the regex extractor does nothing by default - the configuration defines its functionality, and can vary from simple to more sophisticated.

When a regex is specified for engineConfig, you are not simply sending the fields of the configuration to an external service.  Instead, you can specify regexes in order to create entities based on a document's fullText, or other fields.

The following use cases are supported:

• By default, run a regex against the document fullText to create an Entity using entity type/dimension.
• Use $ to define the fields which will be configured by the regex, and then specify entity using entity type/dimension.
• Specify different regexes for different fields, and for each generated entity specify entity type/dimension
• build up complex sets of fields using field variables, and specify multiple regexes to act on specific variables.

For more information, see detailed examples below.

Examples

This section provides detailed examples for the supported feature extractors.

Alchemy API

Using Alchemy API As A Text Extractor

In the example below, Alchemy API is only used as a text extractor.  As such most of the configuration parameters are not applicable and the default settings can be taken.  In this specific example, featureEngine uses OpenCalais.

Source Configuration:

{
    "description": "Article on Medical Issues",
    "harvestBadSource": false,
    "isApproved": true,
    "isPublic": true,
    "key": "http.www.mayoclinic.com.rss.blog.xml",
    "mediaType": "News",
    "modified": "Oct 19, 2010 11:31:59 AM",
    "tags": [
        "topic:healthcare",
        "industry:healthcare",
        "mayo clinic",
        "health"
    ],
    "title": "MayoClinic: General Topics",
    "processingPipeline": [
        {
            "feed": {
                "extraUrls": [
                    {
                        "url": "http://www.mayoclinic.com/rss/blog.xml"
                    }
                ]
            }
        },
        {
            "textEngine": {
                "engineName": "AlchemyAPI"
            }
        },
        {
            "featureEngine": {
                "engineName": "OpenCalais"
            }
        }
    ]
}

Output:

The output contains the "description" and entities resulting from the textEngine and featureEngine settings.

{
    "_id" : "4e1c8afa7d56bb818ed10f76",
    "created" : "1310493434159",
    "description" : "Clarify the role of carbohydrates in the Dr. Bernstein diet and find a 
         healthy eating plan that works for you.",
    "entities" : [
    {
        "actual_name" : "certified diabetes",
        "dimension" : "What",
        "disambiguous_name" : "certified diabetes",
        "doccount" : NumberLong(38),
        "frequency" : 3,
        "gazateer_index" : "certified diabetes/medicalcondition",
        "relevance" : "0.711",
        "totalfrequency" : NumberLong(114),
        "type" : "MedicalCondition"
    },
    {
        "actual_name" : "Diabetes Unit",
        "dimension" : "Who",
        "disambiguous_name" : "Diabetes Unit",
        "doccount" : NumberLong(38),
        "frequency" : 1,
        "gazateer_index" : "diabetes unit/organization",
        "relevance" : "0.235",
        "totalfrequency" : NumberLong(38),
        "type" : "Organization"
    },
    {
        "actual_name" : "Mayo Clinic",
        "dimension" : "What",
        "disambiguous_name" : "Mayo Clinic",
        "doccount" : NumberLong(514),
        "frequency" : 2,
        "gazateer_index" : "mayo clinic/facility",
        "relevance" : "0.305",
        "totalfrequency" : NumberLong(1033),
        "type" : "Facility"
    },

Using Alchemy API-metadata for Feature Extraction

In this example, Alchemy API metadata is used for feature extraction.  It is configured to act on a batch of documents (100) and to return a maximum of 5 keywords per document.   The strict setting will return more high quality keywords, and less keywords overall.

Source Configuration:

The source configuration shows how Alchemy API Metadata parameters can be used to set batch sizing and keywords settings.  In addition, the beginning of the entities block is included to show how automatic feature extraction and manual entities can be combined to achieve highly customizable results.

 },
        {
            "featureEngine": {
                "engineName": "AlchemyAPI-metadata",
                "engineConfig": {
                    "app.alchemyapi-metadata.batchSize": 100,
                    "app.alchemyapi-metadata.numKeywords": 5,
                    "app.alchemyapi-metadata.strict": "true"
                }
            }
        },
        {
            "entities": [
                {
                    "actual_name": "$metadata.json.actor.displayName",
                    "dimension": "Who",
                    "disambiguated_name": "$metadata.json.actor.preferredUsername",
                    "linkdata": "$metadata.json.actor.link",
                    "type": "TwitterHandle"
                },

Output:

The output reveals the results of featureEngine and entities.  The entities are returned indexed by keyword.

   },
        {
            "actual_name": "Amex Teams",
            "dimension": "What",
            "disambiguated_name": "Amex Teams",
            "doccount": -1,
            "frequency": 1,
            "index": "amex teams/keyword",
            "relevance": 0.758636,
            "sentiment": 0.160753,
            "totalfrequency": -1,
            "type": "Keyword"
        },
        {
            "actual_name": "Halo",
            "dimension": "What",
            "disambiguated_name": "Halo",
            "doccount": -1,
            "frequency": 1,
            "index": "halo/keyword",
            "relevance": 0.461833,
            "sentiment": 0.168822,
            "totalfrequency": -1,
            "type": "Keyword"
        },
        {
            "actual_name": "Master Chief Incentives",
            "dimension": "What",
            "disambiguated_name": "Master Chief Incentives",
            "doccount": -1,
            "frequency": 1,
            "index": "master chief incentives/keyword",
            "relevance": 0.981457,
            "sentiment": 0.168876,
            "totalfrequency": -1,
            "type": "Keyword"
        },

Regex

TODO IN PROGRESS

Unlike many other extractors, the regex extractor does nothing by default - the configuration defines its functionality, and can vary from simple to more sophisticated. This section describes different cases:

Define regexes that create entities, running against the document's fullText

• The keys are either the type, or the type-then-"/"-then-the dimension (eg "Person" or "Person/Who"). If the dimension is not specified then the system tries to guess, defaulting to "Who"
• The value is the regex, in one of the following formats
  • "/<regex-pattern>/<optional-flags>" - will extract the entire pattern matched as an entity
  • "s/<regex-pattern>/<replacement-string/<flags>" - will extract the replacement string (with $1, $2, etc representing the capturing groups)"

Example 1:

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"Sha256Hash": "/[0-9a-fA-F]{64}/",
			"Who/ExternalIp": "s/(?:^|[^0-9a-z])([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)($|[^0-9a-z])/hash:$1/i"
		}
	}
}

Define the default fields over which to search

By default, only the doc.fullText field is searched. The special key "$" defines the fields to be configured. It can have the following formats:

• comma-separated list of document fields, eg "fullText,description.title"
  • (the field list can also include "field variables" - this is discussed below)
• a single regex in the format "/<regex-pattern>/<optional-fields>" - this regex is applied to each field in the document and only matching fields are scanned.

Example 2 and 3: 

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"$": "fullText,description,title",
			"Where/StreetAddress": "/[0-9]+ [a-z_-]+ (?:Road|Street|Avenue)/i"
		}
	}
}
//alternative - will search fullText, description, and any metadata field with "address" in the dot-notation path
{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"$": "/(?:fullText|description|metadata\\..*\\.address.*)/",
			"Where/StreetAddress": "/[0-9]+ *,? *[a-z_-]+ *(?:Road|Street|Avenue)/i"
		}
	}
}

Specify different regexes for different fields

It is possible to restrict individual regexes to run on a subset of fields. This is performed with the following "key":

• "<field-list>/<entity-type>/<dimension>" - comma-separated list of fields to search (can also include "field variables" - see below)
• "<field-list>/<entity-type>" (same as above but with automatically inferred dimension)
  • EXAMPLE: "url,sourceUrl/FileType/What" - searches only the documents' "url" and "sourceUrl" fields
• "/<regex-pattern/<flags>/<entity-type>/<dimension>" - specifies a regex to apply to all fields, only matching ones are scanned
• "/<regex-pattern/<flags>/<entity-type>" (same as above but with automatically inferred dimension)
  
  • EXAMPLE: "/[^.]*url$|metadata\..*filename.*/i/FileName/What" - will only scan top level document fields ending with url (url and sourceUrl), or metadata fields containing the string "fieldname" in the dot notation path

The value field is the same (the regex to apply to the specified stream).

Example 4:

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"url,sourceUrl/What/FileType": "s/\\.([a-z]{3})$/$1/i",
			"/[^.]*url$|metadata\\..*filename.*/i/What/FileName": "s/[^\\/]+\\.[a-z]{3}$/i"
		}
	}
}

Build up complex sets of fields using "field variables"

Finally, it is possible to build up more complex sets of field lists in increments using "field variables". In these cases the keys are in the format "$<saved-field-name>", eg "$regexList", "$fieldList" and the value is either the field list (which can itself include earlier "field variables") or a regex that is matched across fields, ie in the same way as the "$" default.

Example 5:

In this example, the default is a small number of fields. The "Sha256Hash" entity type only looks over those. The "FileType" regex will run across a much larger set of fields.

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"$docFields": "fullText,description,title",
			"$extendedDocFields": "$docFields,url,sourceUrl",
			"$metaFields": "/metadata\\..*content.*/i",
			"$moreFields":"$metaFields,$extendedDocFields",
 
			"$": "$docFields",
 
			"Sha256Hash": "/[0-9a-fA-F]{64}/"
			"$moreFields/What/FileType": "s/\.([a-z]{3})$/$1/i",
		}
	}
}