Overview
This page describes command line utilities that system administrators can use to modify the data within Infinit.e.
...
See below, under "Backups".
infinite-db: (database compaction) "/opt/db-home/master_compact-script.sh",
...
on Saturday at midnight (replicas) or 3am (masters)
| Code Block |
|---|
05 00 * * 6 root /opt/db-home/master_compact-script.sh 1 > /tmp/compact_script.log 05 03 * * 6 root /opt/db-home/master_compact-script.sh 2 >> /tmp/compact_script.log |
...
Note finally that compaction is not automatically run for databases that don't use replicas (eg single node databases). It must be run manually during planned outages.
More details are provided about the compaction scripts in the "DB backups" section below.
infinite-db: (extra MongoDB file management), every day at 2am
...
Sources that appear to be generating mostly transient errors (eg RSS sources returning lots of 5xx HTTP errors) are suspended for a day (setting "harvestBadSource":"true" in the source JSON). This call unsuspends them.
...
| Info |
|---|
Note the approved true/false is not currently very useful - it used to be that commonly erroring sources had their approved flag set to false to prevent them from running - but now this is achieved by setting the "searchCycle_secs" field to be a negative number. |
infinite-px-engine: (Checks for Hadoop jobs to run) "/opt/infinite-home/bin/custommr.sh", every minute
| Code Block |
|---|
*/1 * * * * tomcat if [ ! -f /opt/infinite-home/bin/STOP_CUSTOM ]; then /opt/infinite-home/bin/custommr.sh; fi |
...
This calls the core server JAR, which will check if there are any Infinit,e-scheduled Hadoop jobs (eg from the GUI or from the API) pending or completed.
Backup scripts
...
Lucene Backups (elasticsearch)
...
- Every day - creates a backup of the local index, in /opt/elasticsearch-infinite/backups/index_backup_<<cluster_name>>_<<hostname>>_most_recentlatest.tgz
- (note translog flushing is disabled during the backup)
- (note the previous backup will be overwritten by the new one every night)
- Only if the parameter "s3.url" exists in "/opt/infinite-home/config/infinite.service.properties" (and "s3cmd" has been configured, run "s3cmd --configure" if not):
- Every day:
- Uploads the backed-up index to the S3 bucket called "elasticsearch.<<s3.url>>", both as indexas index_backup_<<cluster_name>>_<<hostname>>_most_recentlatest.tgz tgz (ie overwriting the previous "most recent") and index_backup_<<cluster_name>>_<<hostname>>_<<DayOfMonth>><<DAY_OF_WEEK>>.tgz
- (as a result, a rolling previous ~30 ~7 days of indexes are stored for each index)
- Only on Sundays:
- Uploads the backed-up index to the S3 bucket pointed "backup.elasticsearch.<<s3.url>>" with the name index_backup_<<cluster_name>>_<<hostname>>_<<YYYYMMDD>><<WEEK_OF_YEAR>.tgz
- (It is recommended that this bucket be placed in a different region for redundancy)
- Uploads the backed-up index to the S3 bucket pointed "backup.elasticsearch.<<s3.url>>" with the name index_backup_<<cluster_name>>_<<hostname>>_<<YYYYMMDD>><<WEEK_OF_YEAR>.tgz
- Every day:
...
- Every day - creates a backup of the any shards for which the host is the master, in /opt/db-home/backups/db_backup_<<cluster_<<hostname>>_most_recent.tgzname>>_<<hostname>>_latest.tgz (for config DBs: "db_backup_<<hostname>>_latest_27016.tgz")
- (note shard re-balancing is disabled during the backup - note this doesn't work properly because each shard sets/unsets the global balancer)
- (note the previous backup will be overwritten by the new one every night)
- Only if the parameter "s3.url" exists in "/opt/infinite-home/config/infinite.service.properties" (and "s3cmd" has been configured, run "s3cmd --configure" if not):
- Every day:
- Uploads the backed-up database fileto the S3 bucket called "mongo.<<s3.url>>" both as db_backup_<<cluster_name>>_<<hostname>>_most_recentlatest.tgz (ie overwriting the previous "most recent") and db_backup_<<cluster_name>>_<<hostname>>_<<DayOfMonth>>.tgz
- (as a result, a rolling previous ~30 days of databases are stored for each index)
- Only on Sundays:
- Uploads the backed-up database file to the S3 bucket called "backup.mongo.<<s3.url>>" with the name db_backup_<<cluster_name>>_<<hostname>>_<<YYYYMMDD>><<WeekOfYear>>.tgz
- (It is recommended that this bucket be placed in a different region for redundancy)
- Uploads the backed-up database file to the S3 bucket called "backup.mongo.<<s3.url>>" with the name db_backup_<<cluster_name>>_<<hostname>>_<<YYYYMMDD>><<WeekOfYear>>.tgz
- Every day:
| Warning |
|---|
TODO Note that this is not quite the 10gen MongoDB approved backup process - is on the roadmap, describe differences from JIRA issue |
/opt/db-home/backup_script.sh <<port>>
TODO
/opt/db-home/restore_script.sh
TODO
/opt/db-home/sync_from_master.sh
...
which is described here. The most significant difference is that the backup of the config database is not directly synchronized with the data backup, only by the data of the cron job. In addition, the replica set is not locked (by shutting down a secondary on each one of the replicas). Also "point in time" backups are not supported. The net effect is the risk that the shards will be slightly out of sync with one other - but there is not really anyway consistency requirements across the different tables/shards, so the current strategy should be adequate. |
/opt/db-home/backup_script.sh <<port>>
This performs the low level backup operations, called from "master_backup_script.sh" with the port (eg 27106 for the config DB server, (27017 + shard) number for the data replicas). It does nothing on replicas other than the master.
/opt/db-home/restore_script.sh
This is currently not a useful script and should not be used. There is no automated way of restoring backups, administrators should decide what to do and then use "mongorestore" manually. There are a couple of options for the manual restore process:
- Smaller clusters, to an empty DB:
- For each shard do a mongorestore on the entire backup (just untar it) directly into a mongos on one of the nodes ... the config DB backup can be ignored, it will be recreated during the restore.
- (NOTE: if a directory called "config" exists you must delete it)
- For each shard do a mongorestore on the entire backup (just untar it) directly into a mongos on one of the nodes ... the config DB backup can be ignored, it will be recreated during the restore.
- Larger clusters, to an empty DB:
- Follow the steps described here
- Sub-sections of the entire database (eg one of the databases/collections has been corrupted)
- If an unsharded database/collection:
- mongorestore using "-d" and optionally "-c" directly into mongos on any one of the nodes.
- If a sharded collection:
- If "small enough": (what this means in practice is unclear, try this way first)
- For each shard, mongorestore using "-d" and optionally "-c" directly into mongos on any one of the nodes.
- If "larger":
- Follow the steps described here but only for the one database/collection.
- If "small enough": (what this means in practice is unclear, try this way first)
- If an unsharded database/collection:
/opt/db-home/sync_from_master.sh
This is an old script and should not be used. A restore guide is provided above.
/opt/db-home/master_compact_script.sh
/opt/db-home/compact_script.js
This is not strictly speaking a backup process, but it is included here anyway.
These scripts are called from a cron job and are described in outline under the "Database related" section of "Regularly scheduled jobs".
"master_compact_script" is called on Saturday night with one parameter: 1 for replicas (midnight), and 2 for masters (3am). The actual compaction is perfomed by a js file (compant_script.js) invoked using the mongo shell.
"master_company_script.sh 2" will result in the master abdicating and one of the replicas being elected in its place. Because of this, the master role will rotate through the replicas every week.
Batch aggregation (IN PROGRESS)
...