...
In the following example, the web extractor Web Extractor is used to run extraUrls parameter against the web content. extraUrls is used to indicate URLs that should only be harvested once.
In addition, when using the Web Extractor (as opposed to the feed extractorFeed Extractor) it is also possible to specify title, description, publisheddate, and fullText attributes of extraUrls.
...
TODO-an example is required in the source gallery that includes the attributes of extraUrls.
| Code Block |
|---|
{
"description": "For cyber demo",
"isPublic": false,
"mediaType": "Log",
"searchCycle_secs": 3600,
"tags": [
"cyber",
"structured"
],
"title": "Cyber Logs Test",
"processingPipeline": [
{
"feed": {
"extraUrls": [
{
"url": "http://INFINITE_ENDPOINT/api/share/get/51ad28a440b4a4f0f757824c?infinite_api_key=API_KEY"
}
]
}
},
|
...