Record Analyzer

An encapsulation of the Kibana GUI that enables flexible analysis and dashboarding of Infinit.e's new "record" object.

From Sep 2014, the Kibana GUI also lets you view appropriately indexed custom jobs and documents.

Ingest of records into Infinit.e is currently only possible via the Logstash extractor (Later on it will be possible to create records using the custom engine or from the standard harvester).

Most of the documentation on the main Kibana site holds for the Infinit.e implementation.

The following are Infinit.e-specific details:

• The widget only renders records for the selected communities from the main GUI source manager (not to be confused with the source editor in the Infinit.e manager!)
• When records are ingested, there are two modes: "streaming: true", and "streaming: false"
  • In "streaming" mode, records are only retained for 30 days. 
  • In "stashed" mode, records are retained until manually deleted
• The "View Live" and "View Saved" toggle buttons on the widget toolbar toggle between viewing of these two modes.
• The "Community Filter" toggle is by default "Off" meaning that the list of communities in the main GUI is ignored. If "On", only currently selected communities are scanned.
  • (Note that the "refresh button" in the top right of the Kibana view must currently be pressed after changing the toggle.)
• The data types viewed can be selected using the three "Show:" toggles:
  • (After changing the toggles, the Kibana refresh button must be used to update the display)
  • Logs: Show records harvested using the Logstash extractor
  • Custom: Shows the results of custom jobs that have been configure with "$output.indexMode": "custom"
    • The custom fields get given "_type": "custom", and "sourceKey": <"custom:" then the custom job title>
  • Docs: Shows a subset of "normal" (Infinit.e) documents. 
    • Note that communities created before 
    • Only the following fields are (currently) visible in the "Table" view (eg "All Events"): "message" (from the title), "@timestamp" (from publishedDate), "url", "displayUrl", "tags", "type" (from mediaType)
      • Other fields (except entities and assocations) can be used in the other dashboards and queries. 
        • Various dropdowns list the available fields (Eg Fields>All) in the "Table" view (eg "All Events")
        • (Entities and assocations use "nested" fields, which Kibana does not currently support)
      • Note that fields configured to be non-indexed by the harvester (eg Search index settings pipeline element) cannot be viewed
• In the Kibana widget all users can see all dashboards. The Infinit.e version is more restrictive:
  • Dashboards are stored as Infinit.e shares that can be edited/shared/deleted from the File Uploader
  • "Live" dashboards are only visible in "live" mode, and similarly for "Saved"
  • When a dashboard is first saved, it is shared across all currently selected communities and for the given live/stashed mode.
• If creating custom dashboard:
  • In live mode only "daily" timestamps are supported, together with the following index names:
    • "[logstash-]YYYY.MM.DD" or "[ls-]YYYY.MM.DD" - will shows records from all selected communities. (Just a short cut that maps to:)
    • "[recs_t_<community id>_]YYYY.MM.DD" - for any community id to which the user belongs. Note that this does not override the community selection in the main GUI.
  • In stashed mode, only "none" timestamps are supported, together with the following index names:
    • "_all" - will shows records from all selected communities. (Just a short cut that maps to:)
    • "recs_<community id>" - for any community id to which the user belongs. Note that this does not override the community selection in the main GUI.

Note that the Kibana web page can also be accessed in a normal browser window/tab, via: "<ROOT_URL>/infinit.e.records/static/kibana/". By default this will show "stashed" mode and all communities - this can be adjusted by the following URL parameters:

• "cids=<comma separated list of community ids>"
• "mode=<live|stashed>"
• Which data types to view:
  • "records=<true|false>" (default true)
  • "custom=<true|false>" (default false)
  • "docs=<true|false>" (default false)

This view does not provide a login option - you must login via one of the standard routes (manager or main GUI).

Note finally that, as for all Javascript-based widgets, the "add to clipboard" function is not available.