Overview of the Infinit.e Data Harvesting Process
The Infinit.e platform features a robust set of data harvesters that give Infinit.e a powerful data extraction and transformation (enrichment) capabilitie. Infinit.e's harvesters are designed to consume data from a variety of sources and media types including:
- Web based content accessible via URL including:
- Static HTML content;
- RSS and ATOM based news feeds;
- Restful web services interfaces.
- Traditional relational database management systems (RDBMS) via Java Database Connectivity (JDBC) drivers;
- Files located on local and network attached storage devices.
Need pretty graphic for these steps:
- Extract data from source
- Create feed document from source data
- Enrich source data by extracting entities, events, geographic/location data, etc.
- Update entity counts/aggregates
- Store finished within Infinit.e's MongoDB data store
Creating a Source
The following WIKI pages describe detail the steps involved with creating sources:
- Specifying a data source
How to specify the mechanics required to extract data from a source system: - Structured Analysis - Overview
An introduction to the Structured Analysis Harvester and how to specify the methods for enriching structured data sources with geographic information, entities, and events. - Unstructured Analysis - Overview
An introduction to the Unstructured Analysis Harvester.
Source Reference Documents
Source Document Specification
The following links provide detailed information regarding the objects that make up a Source document and the individual fields within each object to support the introductory materials above.
Sample Source Documents
The following sample source documents are provided as an aid to learning how to create your own sources: