Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejs
{
	"display": string,
	"logstash": {
	    "config": string, // contains the complex - there are some custom substitution variables available, see below
 		"streaming": boolean, // defaults to true - data is stored only for a rolling 30 days, if false data is stored forever (useful for demos, should be used with care)
		"distributed": boolean, // defaults to false - if true, then the logstash source is run on all available nodes (if in file/s3 mode, normally used in conjunction with substitution variables - see below)
 
		"testDebugOutput": boolean, // defaults to false - if true then when running "test" eg from source editor, will generate additional debug output from logstash
		"testInactivityTimeout_secs": integer // (default 10s) when running "test" eg from source editor, controls how long after starting logstash will wait between successive input records before exiting
	} 
}

...

Info

System administrators can configure the set of allowed inputs via the 2 configuration parameters: "harvest.logstash.allowed_inputs" and "harvest.logstash.allowed_filters". This can be used either to ban elements considered unsafe, or to add new or custom elements.

Info

SUBSTITUTION VARIABLES

The following strings will subsitute within the config string:

  • "#LOGSTASH{host}" - resolves to the fully qualified hostname of the logstash on which the source is running (ie the master if non-distributed, each slave if distributed)
  • The standard Ikanow substitution (#IKANOW{xxx})
    • (not yet implemented - link to documentation once complete)
Warning

When using the "file" input (which only adminstrators can do), note that deleting the documents for the source does not appear to reset logstash's state for that file and therefore it does not re-ingest the data. To work around this, it is currently necessary to suspend the source, wait 5 minutes, and then re-enable it (which causes the logstash process to restart).

...