MongoDB upgrade notes

Overview

This page provides some details on MongoDB upgrades

To MongoDB 2.6

From MongoDB 2.4

The official MongoDB guide here can essentially be followed.

Note that the upgrade procedure requires no DB downtime in theory (the Infinit.e harvester will be interrupted for the duration of the upgrade, which will take 10 minutes unless something goes wrong).

The only difference from the steps detailed under the 2.2->2.4 instructions are:

• step 3:  upgrading to 2.6 is simpler: "yes | yum install mongodb-org-2.6"
  
  

From MongoDB 2.2

Follow the instructions below to get from MongoDB 2.2 to 2.4 (except bypassing the "start all Infinit.e services" steps at the end). Then follow the instructions above to get from MongoDB 2.4 to 2.6 (except bypassing all "stop all infinit.e services steps" at the start).

To MongoDB 2.4

From MongoDB 2.2

The official MongoDB guide here can essentially be followed.

Note that the upgrade procedure requires no DB downtime in theory (the Infinit.e harvester will be interrupted for the duration of the upgrade, which will take 10 minutes unless something goes wrong).

The steps we have taken, together with any Infinit.e peculiarities are:

1. Stop all harvesters and custom schedulers on the API nodes
   1. (eg "service infinite-px-engine stop; touch /opt/infinite-home/bin/STOP_CUSTOM")
2. Log onto the mongos of any node and stop the balancer
   1. ("sh.stopBalancer()" from the mongos shell)
3. Upgrade the MongoDB RPMs on all of the Config DB servers
   1. ("yes | yum install mongo-10gen-2.4.10 mongo-10gen-server-2.4.10 -y --exclude=mongodb-org*")
4. Log onto one of the Config DB servers and upgrade the metadata
   1. ("mongos --upgrade --configdb <the 3 db config nodes>")
   2. (NOTE: on a number of occasions, this step has failed, reporting transport errors to one of the nodes. On all such occasions, the following steps have worked around this:
      1. Remove the config upgrade lock: 
         • mongo config --eval 'db.locks.remove({_id: "configUpgrade"})'
      2. Log onto the Config DB node for which the transport error was reported and rerun the upgrade exactly as above)
5. Restart all 3 Config DB servers
   1. ("service mongo_infinite restart")
6. Upgrade MongoDB on all API and DB nodes
   1. (See above for yum command)
7. Restart all the servers and routers on all API and DB nodes
   1. ("service mongo_infinite restart")
8. Restart tomcat (unless you are moving onto MongoDB 2.6 next)
   1. ("service tomcat6-interface-engine restart")
9. Start the harvester and custom scheduler (unless you are moving onto MongoDB 2.6 next)
   1. (eg "service infinite-px-engine start; rm -f /opt/infinite-home/bin/STOP_CUSTOM")
10. Restart the balancer
    1. (log onto any mongos shell and run "sh.startBalancer()")