...
- The user query is turned into an ElasticSearch query and applied across the cluster.
- The number of documents returned from ElasticSearch is capped at a "large" number (default 1000, eg 10x the documents to return). The documents are ordered by their Lucene score (or optionally just by descending date).
- Each returned document is then assigned a Significance score as described below.
- The significance and relevance scores are then normalized against each other based on a relative importance specified by the user (default 2:1 in favor of significance) and combined, with the mean score set to 100 (like the "+" stats in baseball, eg 120 is 20% higher than average).
- (All three scores are attached to the documents, as "queryRelevance", "aggregateSignif", and "score" respectively)
- The top scoring documents or entities are returned to the client.
...
It is beyond the scope of this documentation to go into much detail about significance, but this section provides a brief description (a 1-line summary is also provided below!):
- Each document has a set of entities
- For each entity,document pair, a TF-IDF score is generated, the entity's "significance". This score is adjusted in a number of ways:
- Entities with low document counts have their significance suppressed by 33% (well below a dynamically calculated "noise floor") or 66% (just below/at the "noise floor")
- When only a subset of the matching documents are returned (eg > 1000 documents), the significance is adjusted to estimate the TF-IDF across the entire matching dataset, not just the returned subset.
- The document significance ("aggSignificance") is the sum of the entity,document significances. This score can be adjusted in a number of ways:
- Temporally, using the document's "publishedDate" field and a standard "decay algorithm"
- Geo-spatially, similarly to the above but based on distance using the closest entity to the decay origin (lat,long).
- Entities are also assigned a "datasetSignificance", which is just the average of the significances across all documents in which it appears.
- Note that neither entity scores are currently adjusted for time or geo-spatial decay, though this will be added as an option in a future release.
...