Overview

For Web and Feed extractors, will scan the raw content of extracted documents and pull out new documents (eg from links in web pages) that are then extracted using the following pipeline elements. The original documents can either be discarded or processing with the same pipeline.

For other extractors (eg File or Database), can be called in one of 2 ways:

• Follow web links ("links") - just uses the userAgent/proxyOverride/httpFields to control any HTTP requests that are made either from the subsequent "split document" elements
• Split documents ("split") - takes the current document and uses its metadata to create new documents (by outputting an object in the format immediately below). The current document can then be retained or discarded. Newly spawned documents appear at the current point in the pipeline, ie skip by earlier elements.

This page has been broken into the following sections for ease of localization.

TODO

Format

{
	"display": string,
	"links": {
	    "userAgent": string, // (Optional) Overrides the "parent" (rss) setting for "search" operations (see usage guide)
	    "proxyOverride": string, // (Optional) "direct" to bypass proxy (the default), or a proxy specification "(http|socks)://host:port" 
	    "script": string, // (Mandatory) Script, must "return" (last statement evaluated) an array of the following format:
		                // [ { "url": string, "title": string /* optional-ish */, 
		                //     "description": string /* optional */, publishedDate: string /* optional */,
		                //     "spiderOut": string /*optional */ }
	    "scriptlang": string, // (Mandatory) Only "javascript" is supported, use extraMeta for different script types
	    "scriptflags": string, // (Optional) The flags to apply to the above script, see "unstructuredAnalysis.meta" for more details
	    "extraMeta": [ {...} ], // (Optional) A pipeline of metadata extraction operations that are applied prior to "script", see "Using The Feed Harvester" overview
	    "pageChangeRegex": string, // (Optional) If non-null, this regex should be used to match the pagination URL parameter (which will be replaced by pageChangeReplace)
				                    // Also, group 1 should be the start, to allow any offsets specified in the URL to be respected
	    "pageChangeReplace": string, // (Optional) Mandatory if pageChangeRegex is non-null, must be a replace string where $1 is the page*numResultsPerPage
	    "numPages": integer, // (Optional) Mandatory if pageChangeRegex is non-null - controls the number of pages deep the search will go
	    "stopPaginatingOnDuplicate": boolean, // (Ignored unless pageChangeRegex is non-null) If true (default: false) then will stop harvesting as soon as an already harvested link is encountered
	                                            // (for APIs that return docs in time order, ensures that no time is wasted harvesting and then discarding duplicate links)
	    "numResultsPerPage": integer, // (Optional) Mandatory if pageChangeRegex is non-null - controls the number of results per page
	    "waitTimeBetweenPages_ms": integer, // (Optional) Only used if pageChangeRegex is non-null - controls a wait between successive pages if set
 
	    "maxDepth": integer // (Optional, defaults to 2) If spidering out (returning "spiderOut": "true" from the script) the maximum depth to go
	}
}

Description

Follow Web Links can be used for API parsing (JSON, XML) or for more advanced HTML parsing.

It takes as its input documents that have been generated by an extractor, and then creates new documents based on the url links.  The original documents can then be retained or discarded.

API-Style Parsing

Scriptlang is set to javascript, to enable "Follow Web Links" to parse the additional urls.  The script field is passed a variable called "text" which returns the results of the specified url (original document).

For each document an array of the following objects is populated.

 "url": string, // Mandatory - this URL is copied into the "URL" field of the generated document,                     // and is used to fetch the content unless "fullText" is set.
    "title": string, // Mandatory (unless "spiderOut" set, see below) - this is used to generate the document's title.
    "description": string, // Optional, if set then used to generate the document's description.
    "publishedDate": string, // Optional, if not set then the current date is used instead.
 
    "fullText": string, // Optional, if set then this is the content for the generated document, ie "url" is not followed.
 
    "spiderOut": integer //Optional, if set to true then the searchConfig.script is applied to the resulting document,
                            // for a depth of up to "searchConfig.maxDepth" times
                            // Note spiderOut only works if rss.extraUrls is non-empty (ie use that instead of url)

spiderOut can be used to apply the Follow Web Links script to the resulting document.  This means that if the newly generated document also contains additional urls, the script will run again on these urls and return the array to make additional documents.

When spiderOut is enabled it will continue to follow additional urls (if present) until the parameter setting for maxDepth has been reached.

Example

The links parameter is configured using script to spiderout to the additional urls in the input document.  In the example, the script parameter is used to determine the behavior of follow web links when an additional URL is found in the input data.

 "links": {                "script": "var
 retVals = [];\nvar n = -1;\nvar url = 
_doc.url.replace(/[?].*/,\"\");\nvar start = 0;\nwhile (start < 
text.length) {\n    var end = text.indexOf('\\n', start);\n    if (end 
== -1) end = text.length;\n    var line = text.substr(start,end-1);\n   
 start = end + 1;    \n    \n    n++;\n    if (0 == n) continue;\n    
\n    var title = 'line #' + n.toString();\n    var url2 = url + '#' + 
n.toString();\n    var fullText = line;\n    var retVal = { 
'title':title, 'url':url2, 'fullText':line };\n    retVals.push(retVal);
 \n}\nretVals;\n      "

HTML

When Follow Web Links is used to ingest HTML pages, some additional considerations are required for HTML.

Parsing HTML

The extraMeta field is used by "Follow Web Links" to enable other script types to be used.

This enables the use of the xpath scripting language to extract the required urls from the HTML pages, before script can put them into the required array format.

After the extraMeta array has been used by script, it is discarded.

The extraMeta field can also be used for 2 debugging/error handling cases:

• If a field called "_ONERROR_" is generated then if no links are returned from the first page (ie likely due to a formatting error) then the contents of _ONERROR_ (assumed to be a string) are dumped to the harvest message.
• Only when running from the "Config - Source - Test" API call (including from the Source Editor GUI), then for every page, all of the _ONDEBUG_ field values (can be string or object) are dumped to the harvest message.

Example:

-insert example for parsing html

Working with Multiple Pages

Follow Web Links can be used to follow links when multiple pages must be traversed (results pages for Intranet search pages).  Follow Web Links can then generate new documents based on these urls.

The following parameters are part of the configuration

• numPages: the total number of pages that will be checked each search cycle.
• pageChangeRegex: a regex that must have at least one capturing group and must match the entire part of the URL that controls the page number. See example below.
• pageChangeReplace: the above string that controls the page number, with $1 used to represent the page number.
• (slightly misnamed) numResultsPerPage: If the "page number" in the URL is actually a result offset and not a page offset, then this field should be the number of results per page (which is then multiplied by the page number to generate the "$1" string mentioned above). See example.

Examples:

For example, consider a URL of the form:

• http://www.blahblah.com/search?q=search_terms&page=1

Then the following parameters would be used: "pageChangeRegex": "(page=\d+)", "pageChangeReplace": "page=$1", "numResultsPerPage": 1

And for a URL of the form:

• http://www.blahblahblah.com/search?q=search_terms&pagesize=20&start_result=0

The the following parameters would be used: "pageChangeRegex": "(start_result=\d+)", "pageChangeReplace": "start_result=$1", "numResultsPerPage": 20

Standard Web Crawling Methods for HTML Pages

it is likely that standard web-crawling measures are needed such as custom user-agents, and per-page wait times. Because these might well be different from the search engine to the pages themselves, "Follow Web Links" has its own waitTimeBetweenPages_ms, and userAgent fields (if not specified these are inherited from the parent object).

IN PROGRESS

Legacy documentation:

• See under basic/advanced "use of searchConfig"

TODO

Examples

TODO