...

IKANOWS supports the following feature extraction engines:

• Textrank* - TODO description from legacy page
• OpenCalais* - TODO description from legacy page
• AlchemyAPI** - TODO description from legacy page
• AlchemyAPI-metadata** - TODO description from legacy page
• salience* - TODO description from legacy page
• regex* - a mechanism for converting regexes into entities from text or metadata

*requires a text extractor beforehand.

...

{
	"featureEngine": {
		"engineName": "salience",
		"engineConfig": {
			"salience.shortFormContent": "true",
			"salience.kw_score_threshold": "0.5"
		}
	}
}

...

Standard feature extractors

This section describes the configuration details for the supported extractors, and provides examples where applicable.

Regex

TODO IN PROGRESS

Unlike many other extractors, the regex extractor does nothing by default - the configuration defines its functionality, and can vary from simple to more sophisticated. This section describes different cases:

Step 1: Define regexes that create entities, running against the document's fullText

• The keys are either the type, or the type-then-"/"-then-the dimension (eg "Person" or "Person/Who"). If the dimension is not specified then the system tries to guess, defaulting to "Who"
• The value is the regex, in one of the following formats
  • "/<regex-pattern>/<optional-flags>" - will extract the entire pattern matched as an entity
  • "s/<regex-pattern>/<replacement-string/<flags>" - will extract the replacement string (with $1, $2, etc representing the capturing groups)"

Example 1:

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"Sha256Hash": "/[0-9a-fA-F]{64}/",
			"ExternalIp/Who": "s/(?:^|[^0-9a-z])([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)($|[^0-9a-z])/hash:$1/i"
		}
	}
}

Step 2: Define the default fields over which to search

By default, only the doc.fullText field is searched. The special key "$" defines the fields to be configured. It can have the following formats:

• comma-separated list of document fields, eg "fullText,description.title"
  • (the field list can also include "field variables" - this is discussed below under Step 4)
• a single regex in the format "/<regex-pattern>/<optional-fields>" - this regex is applied to each field in the document and only matching fields are scanned.

Example 2 and 3: 

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"$": "fullText,description,title",
			"StreetAddress/Where": "/[0-9]+ [a-z_-]+ (?:Road|Street|Avenue)/i"
		}
	}
}
//alternative - will search fullText, description, and any metadata field with "address" in the dot-notation path
{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"$": "/(?:fullText|description|metadata\..*\.address.*)/",
			"StreetAddress/Where": "/[0-9]+ *,? *[a-z_-]+ *(?:Road|Street|Avenue)/i"
		}
	}
}

Step 3: Specify different regexes for different fields

It is possible to restrict individual regexes to run on a subset of fields. This is performed with the following "key":

• "<field-list>/<entity-type>/<dimension>" - comma-separated list of fields to search (can also include "field variables" - see below)
• "<field-list>/<entity-type>" (same as above but with automatically inferred dimension)
  • EXAMPLE: "url,sourceUrl/FileType/What" - searches only the documents' "url" and "sourceUrl" fields
• "/<regex-pattern/<flags>/<entity-type>/<dimension>" - specifies a regex to apply to all fields, only matching ones are scanned
• "/<regex-pattern/<flags>/<entity-type>" (same as above but with automatically inferred dimension)
  
  • EXAMPLE: "/[^.]*url$|metadata\..*filename.*/i/FileName/What" - will only scan top level document fields ending with url (url and sourceUrl), or metadata fields containing the string "fieldname" in the dot notation path

The value field is the same (the regex to apply to the specified stream).

Example 4:

{
	"featureEngine": {
		"engineName": "regex",
		"engineConfig": {
			"url,sourceUrl/FileType/What": "s/\.([a-z]{3})$/$1/i",
			"/[^.]*url$|metadata\..*filename.*/i/FileName/What": "s/[^/]+\.[a-z]{3}$/i"
		}
	}
}

Step 4: Build up complex sets of fields using "field variables"

XXX

OpenCalais

The following custom configuration parameters are possible for Open Calais and can be set using the engineConfig parameter.

...

Examples

The following example source uses Alchemy API as the text engine, and OpenCalais as the feature engine.  In both cases, the default configuration of these engines is used to output entities and associations for the ingested RSS data.

...

Examples

...