Overview
Once you have ingested data into the system using Source Builder, you can run queries against the data to return IKANOW Threat Analytics documents. The documents include entities, associations, as well as geo/temporal data which can then be visualized.
Data sources are associated to Data Groups and Users and User Groups at source creation time.
Searches are run within the context of a Project, which brings together Data Sources, Data Groups, as well as Users and User Groups.
About Documents and Records
You can query the Threat Analytics platform using the query bar, as displayed below. When querying the platform you can choose to display either documents or records. This will depend on the type of data comprised by your data sources.
Documents: The Threat Analytics document is a representation of the source data after it has passed through the various stages in the source pipeline: Ingest, Transform, Enrich, and Output. Documents contain their own fields as well as sub-objects such as entities, associations, metadata, and query enrichment. This format is good for viewing structured and unstructured data such as "office-style" documents, web pages, RSS feeds etc.
Records: Records are a lighter weight Threat Analytics object format for storing logs, term/record volumes, or statistics. A very common use case for logs/records is in Dev. Ops environments where log files need to be filtered and appropriately analyzed. When you query the platform with records selected, the output is visualized using Kibana.
Querying the Sources
Once you have created a Project and associated the Data Groups, Users and User Groups, and Data Sources, you can run searches within your Project Workspace.
To query the Project Data
- Ensure that you are within the desired Project, by referencing the selected project at top left of the interface. To change to a new project, see the procedure below.
- Select either Documents or Records.
- Enter your search term into the Search bar and click Enter.
The display of the Search results varies based on object format selection: Documents or records.
Documents:
Records:
Records are displayed in Kibana.
Selecting a Project
To select a project
- From the Threat Analytics interface, click on the name of the current project at top left.
- Find the project of choice in the dropdown menu.
- Select the project. You are taken into the new Project workspace, and your search results, as well as any open visualizations are updated accordingly to reflect the data in the Project Data Groups.
Advanced Search and Filtering
When you run queries against your data sources, it is possible to use Advanced Search, for advanced filtering capabilities. Filtering enables you to narrow your results by applying constraints around Entities, Tags, Verb categories, and Weightings.
To use the advanced query builder
- From the Search results, click on Filter Results. It is also possible to access the settings by clicking Advanced Search next to the Search bar.
Configure the filters, as described in the table below. Based on the various settings, the search results are filtered accordingly.
| Filter | Description |
|---|---|
| Entities | Search results can be easily filtered by entity type so that only documents including those entity types are returned to Dashboards, score cards etc. Possible values:
|
| Tags | When sources are added to the platform tags can be applied. These tags can then be used to limit a query to a subset of documents within a Project based on document tags. |
| Verb Categories | You can filter returned associations by using verb types. For example, you can only return associations with the verb category "travel," to encompass associations with verbs such as "flew" and "drove". Possible verb category filters are populated by the platform. |
| Weightings | In scoring, weightings enable you to further alter query output results. For example, you can set a central point in time, around which results are promoted. For more information see below. |
2. Click on Search.
The advanced query is executed against the sources and the results are returned.
About Weight by Time Decay:
You can set a central date (date and time) around which results will be promoted. Results after that central point in time are demoted as per the half-life setting.
Half-Life Setting Example:
1m (one month) time decay - results within 1 month of the entered date are promoted to top of results; results between 1 to 2 months from decay time are halved; results 2 to 3 months from decay time are quartered, etc.
In this section:
Related Documentation: