...
In the following example, the web extractor is used to run extraUrls
parameter against the web content. extraUrls
is used to indicate URLs that should only be harvested once.
In addition, when using the Web Extractor (as opposed to the feed extractor) it is also possible to specify title, description, publisheddate,
and fullText
attributes of extraUrls.
In this way, you can manually specify the values that these attributes should take when the extraUrls
are harvested.
TODO-an example is required in the source gallery that includes the attributes of extraUrls
Code Block |
---|
{ "description": "For cyber demo", "isPublic": false, "mediaType": "Log", "searchCycle_secs": 3600, "tags": [ "cyber", "structured" ], "title": "Cyber Logs Test", "processingPipeline": [ { "feed": { "extraUrls": [ { "url": "http://INFINITE_ENDPOINT/api/share/get/51ad28a440b4a4f0f757824c?infinite_api_key=API_KEY" } ] } }, |
...