Versions Compared

Old Version 16

changes.mady.by.user AlexI

Saved on

compared with

New Version Current

changes.mady.by.user AlexI

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This guide is meant to guide and give suggestions as to how to setup and configure the Hadoop installation once you have completed the RPM installation instructions found here: Infinit.e-hadoop-installer.

Info

This section covers the install of CDH3, which is supported for v0.4-. For v0.5, install CDH5 - see Installing CDH5 (including upgrading from CDH3)

Log in

This guide will start off from the point which you navigate to http://INSTALL_SERVER:7180 in your browser (ie INSTALL_SERVER is the node on which "sh install.sh full" was run):

...

Warning

The following items must be changed for the full functionality of Infinit.e to be available:

  • Set "MapReduce Service Environment Safety Valve" to 'HADOOP_TASKTRACKER_OPTS="-Djava.security.policy=/opt/infinite-home/config/security.policy $HADOOP_TASKTRACKER_OPTS"'
  • Set "" to "-Djava.security.policy=/opt/infinite-home/config/security.policy"
    • (these 2 shouldn't both be necessary, however empirically across different OS configurations sometimes one would work, sometimes the other)
  • Set "Map Tasks Speculative Execution" and " " to be false

We recommend changing these items:

  1. Number of tasks to run per JVM (mapred.job.reuse.jvm.num.tasks) to -1
  2. Use Compression on Map Outputs (mapred.compress.map.output) to false (uncheck the box)
  3. Maximum Number of Simultaneous Map Tasks to 2
  4. Maximum Number of Simultaneous Reduce Tasks to 1
  5. (On systems with large amount of RAM available: increase the size of "MapReduce Child Java Maximum Heap Size")

After making these changes, navigate to Instances (from the toolbar at the top of the page), the task trackers will show as having "outdated instances". Select them all and restart them.

...