Versions Compared

Old Version 9

changes.mady.by.user andrew johnston

Saved on

compared with

New Version 10

changes.mady.by.user andrew johnston

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following parameters are used in the configuration of manual text transformation

ParameterDescriptionNoteData Type
fieldName

Specifies the data source that the script will execute against

"fullText," "description," or "title"

 

 
scriptSpecify your script  
flags

Standard Java regex field

Can have different values, based on scriptlang

See below.

  

javascript:

There are  a few flags that provide additional variables in the javascript:

  • "m" to get "_doc.metadata", written into the variable "_metadata"
    • (for example this flag can be used to copy a subset of the fields from one fieldname to another, before using the "metadataFields" field in the "structuredAnalysis" object to delete the larger field)
  • "d" to get "_doc", written into the variable "_doc",
  • "t" to return the full text of the document into "text". 
    • If the "flags" field is not specified, this is returned by default. If the "flags" field is specified, then "t" must be included or the "text" variable is not populated.
 

 

 

 

xpath (and regex, except for "O"):

  • 'H': will HTML-decode resulting fields. (Eg "&" -> "&")
  • 'o': if  the XPath expression points to an HTML (/XML) object, then this object is converted to JSON and stored as an object in the corresponding metadata field array. (Can also be done via the deprecated "groupNum":-1)
  • 'x': if the XPath expression points to an HTML (/XML) object, then the XML of the object is displayed with no decoding (eg stripping of fields)
  • 'D': described above 
  • 'c': if set then fields with the same name are chained together (otherwise they will all append their results to the field within metadata)

 

  
replacement

If scriptlang is regex or xpath, replacement can be used to replace the value indicated in the regex/xpath.

eg. You could find the instance C/M or C/F in a document and extract that it is important to note that the Race is Caucasian. The same can be done to extract M or F as a Sex meaning Male or Female.

  

scriptlang

Specifies the language of the script that will be provided

One of "javascript," "regex," or "xpath"

 

Examples

...

Supported Script Languages

You can program manual text extraction using the following supported languages

  • Javascript
  • Regex
  • Xpath

Javascript

For power users, metadata can be generated from the content using javascript. This gives a huge amount of flexibility to apply site/source-specific knowledge to pull out metadata that can be turned into entities or associations.

Log File From File Share:

In the following example, manual text transformation is used to parse a log file over the web, with a script of type javascript.

...

Javascript can also return more complex objects, arrays of objects, or array of primitives. 

Regex

XML:

The following example shows how a regex script can be used to manually parse the text of the ingested data:

...

Code Block
   }],        "multipledays": ["No"],
        "organization": ["No group"],
        "perpetrator": [{
            "characteristic": "Islamic Extremist (Sunni)",
            "nationality": "Unknown"
        }],

 

...

Xpath

Neither regex nor javascript are well suited for extracting fields from HTML and XML.

...