...

Overview

You can use the Record Analyzer (encapsulation of the Kibana GUI that enables flexible analysis and dashboarding ) to gain insights into structured records such as logs.  This enables analysis of Infinit.e 's new "record" object.

From Sep 2014, the Kibana GUI also lets you view appropriately indexed custom jobs and documents.

Image Removed

objects.

About the Kibana Implementation

Ingest of records into Infinit.e is currently only possible via the Logstash extractor (Later on it will be possible to create records using the custom engine or from the standard harvester).

Most of the documentation on the main Kibana site holds for the Infinit.e implementation.

The following are Infinit.e-specific details:

...

.

Once the records have been ingested into Infinit.e they can be visualized using the record Analyzer widget.

Live and Saved Modes

When records are ingested, there are two modes: "streaming: true", and "streaming: false"

In "streaming" mode, records are only retained for 30 days.

...

 In "stashed" mode, records are retained until manually deleted

...

 The "View Live" and "View Saved" toggle buttons on the widget toolbar toggle between viewing of these two modes.

...

• (Note that the "refresh button" in the top right of the Kibana view must currently be pressed after changing the toggle.)

...

• (After changing the toggles, the Kibana refresh button must be used to update the display)
• Logs: Show records harvested using the Logstash extractor
• Custom: Shows the results of custom jobs that have been configure with "$output.indexMode": "custom"
  • The custom fields get given "_type": "custom", and "sourceKey": <"custom:" then the custom job title>
• Docs: Shows a subset of "normal" (Infinit.e) documents. 
  • Note that communities created before 
  • Only the following fields are (currently) visible in the "Table" view (eg "All Events"): "message" (from the title), "@timestamp" (from publishedDate), "url", "displayUrl", "tags", "type" (from mediaType)
    • Other fields (except entities and assocations) can be used in the other dashboards and queries. 
      • Various dropdowns list the available fields (Eg Fields>All) in the "Table" view (eg "All Events")
      • (Entities and assocations use "nested" fields, which Kibana does not currently support)
    • Note that fields configured to be non-indexed by the harvester (eg Search index settings pipeline element) cannot be viewed

...

Using the Widget

Viewing and Filtering Logs in the Event Table

Once you have selected your communities and performed a query against the record objects, you can use the record Analyzer widget to view the events in the Event Table.

To view the logs

1. From the widget interface, scroll down to All Events
2. Select the fields to be displayed from the left navigation.
3. Click on one of the fields to filter accordingly.  For example, you can filter by type (eg. apache, DNS, Snort etc.)

Investigating an Event

One of the common cyber threat analysis procedures is to investigate a specific log file as part of a malware analysis.

To perform the investigation

1. From the Record Analyzer widget, scroll own to the All Events table
2. Use log analysis and filtering techniques to hone in on the records for further analysis
3. Click on the log file entry to expand it.  You can view the data in a table, as JSON, or in raw format.  In some cases the message will contain links to more information concerning specific cyber threat.

Refreshing a Community

In some cases you will want to enable/disable scanning of Infinit.e communities.  When this control is toggled on/of you must also perform a refresh from within the Record Analyzer widget.

To refresh a community

1. Change the state of the community filter on/off.
2. From the Kibana GUI click on Refresh.  The Community filter setting is changed and the widgets update to reflect the new setting.

Refreshing the Data Types

You can view three data types from the record Analyzer widget: Logs, Custom, Docs.

To change the state

1. Toggle on/off the three different record types.
2. Click the Kibana refresh button.  The data type setting is changed and the widgets update to reflect the new setting.

Creating Custom Dashboards

If creating custom dashboard:

In live mode only "daily" timestamps are supported, together with the following index names:

• "[logstash-]YYYY.MM.DD" or "[ls-]YYYY.MM.DD" - will shows records from all selected communities. (Just a short cut that maps to:)
• "[recs_t_<community id>_]YYYY.MM.DD" - for any community id to which the user belongs. Note that this does not override the community selection in the main GUI.

In stashed mode, only "none" timestamps are supported, together with the following index names:

• "_all" - will shows records from all selected communities. (Just a short cut that maps to:)
• "recs_<community id>" - for any community id to which the user belongs. Note that this does not override the community selection in the main GUI.

Note that the Kibana web page can also be accessed in a normal browser window/tab, via: "<ROOT_URL>/infinit.e.records/static/kibana/". By default this will show "stashed" mode and all communities - this can be adjusted by the following URL parameters:

• "cids=<comma separated list of community ids>"
• "mode=<live|stashed>"
• Which data types to view:
  • "records=<true|false>" (default true)
  • "custom=<true|false>" (default false)
  • "docs=<true|false>" (default false)

This view does not provide a login option - you must login via one of the standard routes (manager or main GUI).