Advanced Source Builder
andrew johnston (Unlicensed)
Overview
You can use the Advanced source builder to edit raw JSON directly in the browser. There is also a list of JSON templates that you can choose from to get started.
About Source Configuration
Access to the Advanced Source Builder is for power users and Developers. Editing JSON provides direct access to the source pipeline elements, a series of JSON objects that are configured to ingest, transform, enrich, and output data. Data ingested into the ISA platform is stored in a platform-specific JSON format to enable flexible transformation and enrichment.
For more information, see section Source Pipeline Documentation.
Using the Advanced Source Builder
To use the Advanced Source Builder
- From the Treat Analytics Dashboard, click on Data Sources (top right).
- Click on Add New Source.
- Under "What kind of source would you like to create?" specify "Advanced Source Builder."
- Click on Next.
Specify the the fields as defined in the table below.
Field Description Source Title Provide a title for the source. Tags Comma-separated list of tags Description Description of the source. Data Group Data Group where the source data will be saved. Media Type Pick from the list of Media Types. Data Origin Internal or External. For more information, see Data Sources. - Click on Next.
Configuring and Testing
As part of configuring and testing, you will edit the raw JSON in the browser, either starting from an empty template or specifying one of the templates from the dropdown.
To configure and test
- Choose an optional source template or specify empty.
- Edit the JSON as required
- Specify the Test Options. See below.
- Click on Test Source.
About Testing
If the source has been configured properly testing with return test results, and you will be able to move forward with Publishing the new source. Otherwise, a failure message is generated which can be used for troubleshooting (currently it only says FAIL). You can always Save your source and come back to fix any testing errors later.
Testing Options:
| Field | Description |
|---|---|
| Test: | Select the amount of documents under test. 1, 10 or 20. For testing purposes, it is often convenient to limit the mount of documents under test to an easily manageable number. |
| Full text | By default, the full text of a document is not returned (it can be quite long). For testing text extractors (eg "boilerpipe" vs "none" vs "AlchemyAPI"), or for testing "unstructured analysis" transformations, the Full text maybe useful or essential though. In these instances, it is recommended to enable the checkbox. |
| Update test mode | Setting which enables/disables test mode to check the database to see if documents already exist. This enables additional logic to be setup elsewhere in the source pipeline for testing eg. compare old and new sources, ignore sources older than N days etc. |
Saving or Publishing
Saving
To save the source after testing
- Click on Save.
The source is saved and you are re-directed to the Source Manager.
Publishing
To publish the source after testing
- Click on Publish.
The source is published and you are re-directed to the list Source Manager.
In this section: