Versions Compared

Old Version 3

changes.mady.by.user andrew johnston

Saved on

compared with

New Version 4

changes.mady.by.user andrew johnston

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
{
	"display": string,
	"contentMetadata": [
		{} // see MetadataSpecPojo below
	]
}
//////////////////////////////////
 
	public static class MetadataSpecPojo {
		public String fieldName; // Any string, the key for generated array in "doc.metadata"
		public String scriptlang; // One of "javascript", "regex", "xpath"
		public String script; // The script that will generate the array in "doc.metadata" (under fieldName)
		public String flags; // Standard Java regex field (regex/xpath only), plus "H" to decode HTML, "D": will allow duplicate strings (by default they are de-duplicated), plus the following custom flags:
								// For javascript (defaults to "t" if none specified), "t" the script receives the doc fullText ("text"), "d" the script receives the entire doc (_doc), "m" the script receives the doc.metadata (_metadata)
								// For xpath: "o": if the XPath expression points to an HTML (/XML) object, then this object is converted to JSON and stored as an object in the corresponding metadata field array. (Can also be done via the deprecated "groupNum":-1)
		public String replace; // Replacement string for regex/xpath+regex matches, can include capturing groups as $1 etc
		public Boolean store; // Whether this field should be stored in the DB or discarded after the harvet processing
		public Boolean index; // Whether this field should be full-text indexed or just stored in the DB
	}

Legacy documentation:

TODO 

Description

ParameterDescriptionNoteData Types
fieldName
Any string, the key for generated array in "doc.metadata"
  
scriptlang   
script   
flags
For javascript (defaults to "t" if none specified), "t" the script receives the doc fullText ("text"), "d" the script receives the entire doc (_doc), "m" the script receives the doc.metadata
 

There are also a few flags that provide additional variables in the javascript:

  • "m" to get "_doc.metadata", written into the variable "_metadata"
    • (for example this flag can be used to copy a subset of the fields from one fieldname to another, before using the "metadataFields" field in the "structuredAnalysis" object to delete the larger field)
  • "d" to get "_doc", written into the variable "_doc",
  • "t" to return the full text of the document into "text". 
    • If the "flags" field is not specified, this is returned by default. If the "flags" field is specified, then "t" must be included or the "text" variable is not populated.
  
For xpath: "o": if the XPath expression points to an HTML (/XML) object, then this object is converted to JSON and stored as an object in the corresponding metadata field array.

For reference, here is the complete set of flags for xpath (and regex, except for "O"):

  • 'H': will HTML-decode resulting fields. (Eg "&" -> "&")
  • 'o': if  the XPath expression points to an HTML (/XML) object, then this object is converted to JSON and stored as an object in the corresponding metadata field array. (Can also be done via the deprecated "groupNum":-1)
  • 'x': if the XPath expression points to an HTML (/XML) object, then the XML of the object is displayed with no decoding (eg stripping of fields)
  • 'D': described above 
  • 'c': if set then fields with the same name are chained together (otherwise they will all append their results to the field within metadata)
  
replace   
store   
index   

...