Overview

You can use the Record Analyzer (encapsulation of the Kibana GUI) to gain insights into structured records such as logs.  This enables analysis of "record" objects.

About the Kibana Implementation

Ingest of records into the platform is currently only possible via the Logstash extractor.

Once the records have been ingested, they can be visualized using the record Analyzer widget.

Live and Saved Modes

When records are ingested, there are two modes: "streaming: true", and "streaming: false"

In "streaming" mode, records are only retained for 30 days.  In "stashed" mode, records are retained until manually deleted

 The "View Live" and "View Saved" toggle buttons on the widget toolbar toggle between viewing of these two modes.

Using the Widget

Querying the Record Analyzer

When you query the main GUI, you can pass the query to the record Analayzer by clicking on "Add Doc Query."

Provided that you have "records" within the selected communities/sources they will be displayed by the Record Analyzer.

To query the Record Analyzer

1. Provide a query to the GUI and ensure that the Record Analyzer widget is added to the workspace.
2. In the Record Analyzer widget toolbar, click on Add Doc Query.
3. Ensure that the "Show:" settings are tuned appropriately.  You will need to at least show some Logs or Docs.  Provided the query is successful, the Record Analyzer is populated with the Data Types, Sources, and Event data that correspond to the Live template or Saved template.

For more information, see section Record Analyzer Interface.

Viewing and Filtering Logs in the Event Table

Once you have selected your Communities and performed a query against the record objects, you can use the Record Analyzer widget to view the events in the Event Table.

To view the logs

1. From the widget interface, scroll down to All Events.
2. Select the fields to be displayed from the left navigation.
3. Click on one of the fields to filter accordingly.  For example, you can filter by type (eg. apache, DNS, Snort etc.)

Investigating an Event

A common cyber threat analysis procedure is to investigate a specific log file as part of malware analysis.

To perform an investigation

1. From the Record Analyzer widget, scroll own to the All Events table
2. Use log analysis and filtering techniques to hone in on the records for further analysis
3. Click on the log file entry to expand it.  You can view the data in a table, as JSON, or in raw format.  In some cases, the message will contain links to more information concerning a specific cyber threat.

Refreshing a Community

In some cases you will want to enable/disable scanning of Communities.  When this control is toggled on/of you must also perform a refresh from within the Record Analyzer widget.

To refresh a Community

1. Change the state of the community filter on/off.
2. From the Kibana GUI click on Refresh.  The Community filter setting is changed and the widgets update to reflect the new setting.

Refreshing the Data Types

You can view three data types from the record Analyzer widget: Logs, Custom, Docs.

To change the state

1. Toggle on/off the three different record types.
2. Click the Kibana refresh button.  The data type setting is changed and the widgets update to reflect the new setting.

Creating Custom Dashboards

Using the Record Analyzer widget it is possible to create custom Dashboards.

To create a custom Dashboard

1. Using the Record Analyzer widget add/delete Rows and Panels as required.  For more information, see the Kibana Documentation.
2. When the Dashboard is to your liking, click Save.
3. Provide a name for the new dashboard and click on the Save icon again.

About Creating Custom Dashboards

If creating a custom dashboard:

For records:

In live mode only "daily" timestamps are supported, together with the following index names:

• "[logstash-]YYYY.MM.DD" or "[ls-]YYYY.MM.DD" - will shows records from all selected communities. (Just a short cut that maps to:)
• "[recs_t_<community id>_]YYYY.MM.DD" - for any community id to which the user belongs.

Note that this does not override the community selection in the main GUI.

In stashed mode, only "none" timestamps are supported, together with the following index names:

• "_all" - will show records from all selected communities. (Just a short cut that maps to:)
• "recs_<community id>" - for any community id to which the user belongs. 

(Within each community index, the "sourceKey" field can be used to pull out individual sources)

Note that this does not override the community selection in the main GUI.

For documents:

• Documents are all stored in "docs_<community id>" with type "document_index". Note that:
  • Entities and associations are nested (so eg it is not possible to aggregate over them from Kibana, though it is by hand)
  • Only a small subset of the fields are in the "_source" field. The others can be aggregated over (including in Kibana) or retrieved from the store (Kibana does not support this)
  • Within each community index, the "sourceKey" field can be used to pull out individual sources

For custom objects:

• If the custom source sets the data to be indexed, then the data is stored in "custom_<job id>_1" or "custom_<job id>_2", depending on which of the 2 output collections is active.
  • Each job is aliased to the community index, "customs_<community id>"
  • Within these indexes, the "sourceKey" field can be used to pull out individual sources

For V2 objects:

• The set of indexes is deterministically generated from the bucket full name (see BucketTemplateUtils.getUniqueSignature for an example of how to generate the base index). 
• The actual indexes have both temporal suffixes, suffixes for the different active buffer (similar to the _1 or _2 in the custom jobs), and suffix for segments, but:
  • The safe form for reading all records from a bucket is "r__<base index>*"
  • Given knowledge of the temporal signature, then "r__<base index>_<time signature>*" can be used to cut down on the amount of data
• The bucket indexes are not guaranteed to have a field (other than "_index", which has some restrictions on its used) that can be used to pull out desired buckets from a multi-index query (though nothing stops the source developer from adding such a thing!), but...
  • ...(Roadmap: the record server API will provide an override param "&v2_path=<list of virtual file path globs, supporting * and ** wildcards>" to pull out only the desired buckets

Importing Dashboards

It is possible to save Kibana dashboards and import them into the platform as JSON shares.  Once imported as shares, the Dashboards can be loaded into the Record Analyzer.

To import a Dashboard

1. From the Record Analyzer widget, click on the "Load" icon.  The uploaded JSON shares from the selected communities are available from the menu.  
2. Select the Dashboard of choice.  The new Dashboard is loaded into the interface.