Logstash

A common Manager use case is to import logs, in order to enrich them with additional structure and enhanced meaning.

IKANOW leverages Logstash to apply structure to imported records, and harnesses Kibana for visualization.  The advantages of using IKANOW is the ability to combine logstash-based structured analysis of records, with the platform's strengths in unstructured data analysis.

For more background information, reference the following IKANOW blog post Log Analysis and Big Data Cyber Analytics In One Platform.

Getting Logs Into The Platform

Logstash is an open source log management platform that can parse logs from a variety of sources.  A commonly used logstash input plugin is the Amazon S3 connection to AWS S3 storage.  You can centralize log storage to an Amazon S3 account, and then point Community Edition/logstash to this location, in order to process the log data.

Defining the input to S3 is done as part of Inifinit.e Source configuration.

Creating the New Source

Before you can define the connection to Amazon S3, you must create a new Logstash source.

To create the new source

1. Navigate to Source Editor>New Source.
2. Select the Logstash template
3. Click on Select.
4. Fill in the remaining information, and ensure you select the correct Community
5. Click on Save Source.

Editing the Source

As part of editing the source, you can define the connection to Amazon S3 and specify any logstash filters using the LS Editor.

For more detailed information concerning the logstash configuration, see Logstash extractor.

To define the connection to S3

1. From your logstash source, click on LS.  The logstash editor is displayed.
2. Provide your Amazon AWS Access ID and secret key, as indicated in the screenshot below.
3. Provide the bucket name where the log files reside.  You can use "prefix" to specify the log file name.  "Type" is used to indicate the log file type.

Testing the Source

Once you have provided the correct URL and saved the source you can test it to verify if documents are returned.  

To test the source

1. Click on Test Source.  The platform will perform data processing and should then return the documents.  

2. A Source Test Output window will open displaying either a success or error message.  Provided there are no problems a record should be returned, in addition to the source test output.

Publishing the Source

Once you are satisfied with the results, you can publish the source.

To publish the source