Overview
You can use the Record Analyzer (encapsulation of the Kibana GUI) to gain insights into structured records such as logs. This enables analysis of Infinit.e "record" objects.
About the Kibana Implementation
Ingest of records into Infinit.e is currently only possible via the Logstash extractor.
Once the records have been ingested into Infinit.e they can be visualized using the record Analyzer widget.
Live and Saved Modes
When records are ingested, there are two modes: "streaming: true", and "streaming: false"
In "streaming" mode, records are only retained for 30 days. In "stashed" mode, records are retained until manually deleted
The "View Live" and "View Saved" toggle buttons on the widget toolbar toggle between viewing of these two modes.
Querying the Record Analyzer
When you query the main Infinit.e GUI, you can pass the query to the record Analayzer by clicking on "Add Doc Query."
Provided that you have Infinit.e "records" within the selected communities/sources they will be displayed by the Record Analyzer.
To query the Record Analyzer
- Provide a query to the Infinit.e GUI and ensure that the Record Analyzer widget is added to the workspace.
- In the Record Analyzer widget toolbar, click on Add Doc Query.
- Ensure that the "Show:" settings are tuned appropriately. You will need to at least show some Logs or Docs. Provided the query is successful, the Record Analyzer is populated with the Data Types, Sources, and Event data that correspond to the Live template or Saved template.
For more information, see section Record Analyzer Interface.
Viewing and Filtering Logs in the Event Table
Once you have selected your Communities and performed a query against the record objects, you can use the Record Analyzer widget to view the events in the Event Table.
To view the logs
- From the widget interface, scroll down to All Events.
- Select the fields to be displayed from the left navigation.
- Click on one of the fields to filter accordingly. For example, you can filter by type (eg. apache, DNS, Snort etc.)
Investigating an Event
A common cyber threat analysis procedure is to investigate a specific log file as part of malware analysis.
To perform an investigation
- From the Record Analyzer widget, scroll own to the All Events table
- Use log analysis and filtering techniques to hone in on the records for further analysis
- Click on the log file entry to expand it. You can view the data in a table, as JSON, or in raw format. In some cases, the message will contain links to more information concerning a specific cyber threat.
In some cases you will want to enable/disable scanning of Infinit.e communities. When this control is toggled on/of you must also perform a refresh from within the Record Analyzer widget.
To refresh a Community
- Change the state of the community filter on/off.
- From the Kibana GUI click on Refresh. The Community filter setting is changed and the widgets update to reflect the new setting.
Refreshing the Data Types
You can view three data types from the record Analyzer widget: Logs, Custom, Docs.
To change the state
- Toggle on/off the three different record types.
- Click the Kibana refresh button. The data type setting is changed and the widgets update to reflect the new setting.
Creating Custom Dashboards
If creating a custom dashboard:
In live mode only "daily" timestamps are supported, together with the following index names:
- "[logstash-]YYYY.MM.DD" or "[ls-]YYYY.MM.DD" - will shows records from all selected communities. (Just a short cut that maps to:)
- "[recs_t_<community id>_]YYYY.MM.DD" - for any community id to which the user belongs.
Note that this does not override the community selection in the main GUI.
In stashed mode, only "none" timestamps are supported, together with the following index names:
- "_all" - will show records from all selected communities. (Just a short cut that maps to:)
- "recs_<community id>" - for any community id to which the user belongs.
Note that this does not override the community selection in the main GUI.