Hadoop configuration is taken from a number of places. This page is intended to provide a quick guide.
- The default configuration is taken from the hadoop-core JAR, ie cannot be changed
- When a job is created from the custom API/GUI, the following parameters are overridden:
- pre November 2014:
- Where to look for the jobtracker and FS ("mapred.job.tracker", "mapred.job.tracker", "fs.default.name")
- These are taken from the "*-site.xml" files found in "hadoop.configpath" (in the "/hadoop" subdir)
- all the per-job parameters (Infinit.e configuration, MongoDB configuration, mapper classes etc)
- Where to look for the jobtracker and FS ("mapred.job.tracker", "mapred.job.tracker", "fs.default.name")
- November 2014 onwards:
- All settings from the "*-site.xml" files found in "hadoop.configpath" (in the "/hadoop" subdir) override the defaults
- pre November 2014:
- Many of these configuration parameters are overridden by the settings maintained in the Cloudera Manager
- It isn't currently clear which, it should probably be assumed that if a configuration parameter controls the environment in which the job runs, rather than the job itself, then it will be overridden by Cloudera Manager
- (Note that the Cloudera Manager configuration for a given service on each node lives in a subdirectory of "/var/run/cloudera-scm-agent/process/" - which sub-directory can be found by getting the process id ("ps -ef"), then the working directory of that process ("pwdx $HADOOP_PID") ... the current configurations can also be viewed from the Cloudera Manager GUI)
- Note as a result of the above, it is not necessary to redistribute the client configuration when an "environmental" setting is changed.