The Infinit.e platform has been designed to be easy to install and configure, run with decent performance on commodity hardware without custom configuration, and expand by "scaling horizonally" (ie adding more compute nodes).
This section describes steps that can be taken to squeeze the most performance out of a cluster (at the expense of a more complex configuration).
In this page it is assumed that the user is running more powerful machines - for example 12 cores, 64GB, with 1 or 2 fast RAID volumes (see below). It is also assumed that the (Elasticsearch) real-time index and the (MongoDB) data store are on different nodes.
The configuration suggested below assumes at least this - where more CPU/memory would affect the suggested configuration this is noted.
(This section focuses on magnetic disks, SSD is briefly mentioned at the bottom)
Typically each node has 2 IO channels that require performance:
Infinit.e has two default ways in which it uses directory names to decide where to put data directories:
Elasticsearch and HDFS and MongoDB have different recommended settings:
We have not tested Infinit.e using SSD, though both MongoDB and Elasticsearch have been used. The general approach to utilizing SSD is:
|
Currently we are tested against Oracle's JDK6 and JDK7. Oracle JDK8 testing is ongoing. Once JDK8 is tested it is expected to be significantly faster, for at least two reasons:
For now the recommended version is the latest Oracle JDK7.
It is recommended that there be SWAP space equal to at least 10GB - probably 20GB for 60GB of RAM.
(Relative to the central configuration file described here):
Other configuration notes:
One "trick" MongoDB have suggested in the past to speed up write performance is to run multiple instances of MongoDB shards on a single node (this comes at the price of a slight degradation in read performance for reads that aren't targeted based on their shard key).
We are currently increasing the ways in which we will advantage of this performance-wise - and shards can be added in this way dynamically in any case.
For the moment it is recommended to add 2 shards to each node. This can be done very easily from the "db.replica.sets" parameter .. eg if you have 4 hosts with hostnames A, B, C and D, then the configuration would look like:
(ie (A,B) and (C,D) pairwise contain the same data, and (A,B) hosts data from shards 1 and 2 and (C,D) hosts data from shards 3 and 4)
Assuming Nx API nodes and Mx DB nodes, the "standard" mapping is:
To maximize the ingest performance, you can also install the infinit.e-processing-engine RPM on the DB nodes. This doubles the number of harvesters. Note that it is necessary to copy any additional JARS into the DB nodes' plugins/extractors/unbundled directories (see here), just like for the API nodes.
In the Hadoop configuration guide, the following settings are recommended:
For an 8 core system this should be set to 4 and 2. (For a 16 core system, 8 and 4, etc).
This section describes changes that are made to the processes' configuration files after RPMs have been installed.
The files described in here are designated as RPM configuration files, meaning that they will only be overwritten if the RPM-side version of the file is updated (in which case the "old" version is saved to "<filename>.RPMSAVE". Care must therefore be taken while updating RPMs to note if this happens (and then the user must merge the files by hand if so) |
JAVA_OPTS="$JAVA_OPTS -Xms1024m -Xmx1024m -Xmn256m" && [[ `cat /proc/meminfo | grep MemTotal | gawk '{ print $2 }
' | grep -P "[0-9]{8,}"` ]] && JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m -Xmn512m"
Change the second JAVA_OPTS clause to (changes shown in bold)
export JAVA_OPTS="-Xms2048m -Xmx2048m -Xmn512m" && [[ `cat /proc/meminfo | grep MemTotal | gawk '{ print $2 }' |
grep -P "[0-9]{8,}"` ]] && \
JAVA_OPTS="-Xms7656m -Xmx7656m -Xmn2048m"
Currently we do not take advantage of HDFS for file extraction - this is coming soon.
In the meantime to provide a shared filesystem, there are a few options:
It is not known which is best from a performance standpoint - the second (NFS) is recommended for now (Samba would be preferred but see below).
There is currently an issue with multi-threading in the NetBIOS interface - as a result only one thread can perform all the File operations (including slow bulk operations like de-duplication), and this makes the Samba method very low performance if multi-threaded. For the moment, the Samba method is not recommended. UPDATE (11 Nov): there is a fix in the trunk that will be pushed out to the Nov 2014 release. With the fix implemented, the Samba share method is preferred again) |
The file extractor is the most optimized one, so wherever possible that should be used.
The fastest configuration for reading in files is as follows:
This will delete the files in the input directory as they are processed. If you want to preserve the files, they should therefore be copied into the input directory. (One alternative is to have an "archive" sub-directory of each input directory and then set "file.renameAfterParse" to "$path/archive/$name" and then set "file.pathExclude" to ".*/archive/.*") |
For a single bulk ingest, the "harvest.distributionFactor" should be set to 80 .. this corresponds to:
If you are expecting to be ingesting multiple sources at the same time then scale down the distributionFactor accordingly.
Note that the limiting factor on performance will often be the NLP processing that needs to occur. For example Salience will run at ~2 docs per second per thread on "average" sized documents (so at unrealistic 100% duty cycle on an 8 harvest node cluster with 5 files threads that would give you 80 docs/second, or about 250K/hour). Some NLTK-based extractors are even slower.