...
Field | Description |
---|---|
feedType | Currently not used - will allow for RSS vs Atom in future releases (currently only RSS is supported) |
waitTimeOverride_ms | Optional - if specified, controls the amount of time between successive reads to a site (default: 10000ms): // ie if a site is timing out it may limit the number of accesses from a given IP - set the number higher // for large sites you can increase the performance of the harvester by setting this number lower. |
updateCycle_secs | Optional - if present harvested URLs may be replaced if they are older than this time and are encountered from the RSS or in the "extraUrls" |
regexInclude | Optional - if specified, only URLs matching the regex will be harvested |
regexExclude | Optional - if specified, any URLs matching the regex will not be harvested |
extraUrls | This array allows for manually specified URLs to be harvested once { "url": string // The URLComplex Type "url": string, // The URL "title": string, // The title that the document will be given (ie the equivalent to the RSS title). See below. "description": string, // (Optional) The description that the document will be given (ie the equivalent to the RSS description) "publishedData": string, // (Optional) The date that will be assigned to the document (default: now) - this can be overridden from "structuredAnalysis" "fullText": string // (Optional) If present and "useTextExtractor" is "none", then uses the specified string instead of the URL contents (mainly for debugging) |
userAgent | (Optional) If present overrides the system default user agent string |
proxyOverride | (Optional) "direct" to bypass proxy (the default), or a proxy specification "(http|socks)://host:port" |
httpFields | (Optional) Additional HTTP fields to be applied to the request headers |
Anchor | ||||
---|---|---|---|---|
|
Usage of the title
string impacts how the web extractor will generate documents. There is a dependency with the links
or splitter
elements which can be specified downstream in the source pipeline. For more information about links
and splitter
see Follow Web links.
Links or Splitter is Not Included
When neither a links
or splitter
element is included downstream, specifying a "title" for extraUrls
will cause Web Extractor to process the included url as a web page. When no title is specified, the url is treated as an RSS feed. This functionality enables you to mix both RSS and web pages within the same source configuration.
Links or Splitter is Included
If a links
element is included downstream, specifying a "title" will cause Web Extractor to treat the url as a web page. The original page will be preserved as a document, and links can still be followed based on how the links
element is setup.
When no "title" is specified, Web Extractor will simply see the web page as an API endpoint and will discard it as a document. Documents are only generated based on the responses from the API.
Examples
extraUrls
In the following example, the Web Extractor is used to run extraUrls
parameter against the web content. extraUrls
is used to indicate URLs that should only be harvested once.
...
In this example, the updateCycle_secs
parameter is also used to specify the refresh rate of the harvested urls.
"Title" is also specified, which can impact document generation depending on the presence of the links
or splitter
elements downstream in the source pipeline.
Code Block |
---|
{ "display": "Extract each document (re-extracting every 'updateCycle_secs') with the specified title and summary text", "web": { "extraUrls": [{ "description": "Optional", "title": "Page Title", "url": "http://youraddress.com/title.html" }], "updateCycle_secs": 86400 } } |
...