HADOOP(1)

NAME
hadoop - software platform to process vast amounts of data

SYNOPSIS

Usage: hadoop [--config confdir] COMMAND

DESCRIPTION

Here’s what makes Hadoop especially useful:

Scalable
    Hadoop can reliably store and process petabytes.

Economical
    It distributes the data and processing across clusters of commonly available computers. These clusters can number into the thousands of nodes.

Efficient
    By distributing the data, Hadoop can process it in parallel on the nodes where the data is located. This makes it extremely rapid.

Reliable
    Hadoop automatically maintains multiple copies of data and automat‐ ically redeploys computing tasks based on failures.

    Hadoop implements MapReduce, using the Hadoop Distributed File System (HDFS) (see figure below.) MapReduce divides applications into many small blocks of work. HDFS creates multiple replicas of data blocks for reliability, placing them on compute nodes around the cluster. MapRe‐ duce can then process the data where it is located.

    For more details about hadoop, see the Hadoop Wiki at http://wiki.apache.org/hadoop/.

OPTIONS

--config configdir
    Overrides the "HADOOP_CONF_DIR" environment variable. See "ENVI‐ RONMENT" section below.

COMMANDS

namenode -format
    format the DFS filesystem

secondarynamenode
    run the DFS secondary namenode

namenode
    run the DFS namenode

datanode
    run a DFS datanode

dfsadmin
    run a DFS admin client

fsck
    run a DFS filesystem checking utility

fs  run a generic filesystem user client

balancer
    run a cluster balancing utility

jobtracker
    run the MapReduce job Tracker node

pipes
    run a Pipes job

tasktracker
    run a MapReduce task Tracker node

job manipulate MapReduce jobs

version
    print the version

jar <jar>
    run a jar file

distcp <srcurl> <desturl>
    copy file or directories recursively

archive -archiveName NAME <src>* <dest>
    create a hadoop archive

daemonlog
    get/set the log level for each daemon

CLASSNAME
    run the class named CLASSNAME

    Most commands print help when invoked w/o parameters.

FILESYSTEM COMMANDS

The following commands can be used with the fs command like hadoop fs [filesystem command]

•   -ls <path>

•   -lsr <path>

•   -du <path>

•   -dus <path>

•   -count[-q] <path>

•   -mv <src> <dst>

•   -cp <src> <dst>

•   -rm [-skipTrash] <path>

•   -rmr [-skipTrash] <path>

•   -expunge

•   -put <localsrc> ... <dst>

•   -copyFromLocal <localsrc> ... <dst>

•   -moveFromLocal <localsrc> ... <dst>

•   -get [-ignoreCrc] [-crc] <src> <localdst>

•   -getmerge <src> <localdst> [addnl]

•   -cat <src>

•   -text <src>

•   -copyToLocal [-ignoreCrc] [-crc] <src> <localdst>

•   -moveToLocal [-crc] <src> <localdst>

•   -mkdir <path>

•   -setrep [-R] [-w] <rep> <path/file>

•   -touchz <path>

•   -test -[ezd] <path>

•   -text <src>

•   -copyToLocal [-ignoreCrc] [-crc] <src> <localdst>

•   -moveToLocal [-crc] <src> <localdst>

•   -mkdir <path>

•   -setrep [-R] [-w] <rep> <path/file>

•   -touchz <path>

•   -test -[ezd] <path>

•   -stat [format] <path>

•   -tail [-f] <file>

•   -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...

•   -chown [-R] [OWNER][:[GROUP]] PATH...

•   -chgrp [-R] GROUP PATH...

•   -help [cmd]

    Generic options supported are

-conf <configuration file>
    specify an application configuration file

-D <property=value>
    use value for given property

-fs <local|namenode:port>
    specify a namenode

-jt <local|jobtracker:port>
    specify a job tracker

-files <comma separated list of files>
    specify comma separated files to be copied to the map reduce cluster

-libjars <comma separated list of jars>
    specify comma separated jar files to include in the classpath.

-archives <comma separated list of archives>
    specify comma separated archives to be unarchived on the compute machines.

FILES

/etc/hadoop/conf
    This symbolic link points to the currently active Hadoop configura‐ tion directory.

    Note to Hadoop System Admins

    The "/etc/hadoop/conf" link is managed by the alterna‐ tives(8) command so you should not change this symlink directly.

    To see what current alternative(8) Hadoop configurations you have, run the following command:

    # alternatives --display hadoop
    hadoop - status is auto.
     link currently points to /etc/hadoop/conf.pseudo
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.pseudo - priority 30
    Current 'best' version is /etc/hadoop/conf.pseudo.

    This shows that the link point to "/etc/hadoop/conf.pseudo" (for the Hadoop Pseudo-Distributed configuration).

    To add a new custom configuration, run the following com‐ mands as root:

    # cp -r /etc/hadoop/conf.empty /etc/hadoop/conf.my

    This will create a new configuration directory, "/etc/hadoop/conf.my", that serves as a starting point for a new configuration. Edit the configuration files in "/etc/hadoop/conf.my" until you have the configuration you want.

    To activate your new configuration and see the new configu‐ ration list:

    # alternatives --install /etc/hadoop/conf hadoop /etc/hadoop/conf.my 90

    You can verify your new configuration is active by running the following:

    # alternatives --display hadoop
    hadoop - status is auto.
     link currently points to /etc/hadoop/conf.my
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.pseudo - priority 30
    /etc/hadoop/conf.my - priority 90
    Current 'best' version is /etc/hadoop/conf.my.

    At this point, it might be a good idea to restart your ser‐ vices with the new configuration, e.g.,

    # /etc/init.d/hadoop-namenode restart

/etc/hadoop/conf/hadoop-site.xml
    This is the path to the currently deployed Hadoop site configura‐ tion. See "/etc/hadoop/conf" above.

/usr/bin/hadoop-config.sh
    This script searches for a useable "JAVA_HOME" location if "JAVA_HOME" is not already set. It also sets up environment vari‐ ables that Hadoop components need at startup (see "ENVIRONMENT" section).

/etc/init.d/hadoop-namenode
    Service script for starting and stopping the Hadoop NameNode

/etc/init.d/hadoop-datanode
    Service script for starting and stopping the Hadoop DataNode

/etc/init.d/hadoop-secondarynamenode
    Service script for starting and stopping the Hadoop Secondary NameNode

/etc/init.d/hadoop-jobtracker
    Service script for starting and stopping the Hadoop JobTracker

/etc/init.d/hadoop-tasktracker
    Service script for starting and stopping the Hadoop TaskTracker

ENVIRONMENT

HADOOP_CONF_DIR
    The location of the Hadoop configuration files. Defaults to "/etc/hadoop/conf". For more details, see the "FILES" section.

HADOOP_LOG_DIR
    All Hadoop services log to "/var/log/hadoop" by default. You can change the location with this environment variable.

HADOOP_ROOT_LOGGER
    Setting for log4j. Defaults to ERROR,console. You can try INFO,console for more verbose output.

EXAMPLES

$ mkdir input
$ cp <txt files> input
$ hadoop jar /usr/lib/hadoop/*example*.jar input output 'grep string'
$ cat output/*

BUGS

The Debian package of Hadoop is still in beta state. Use it at your own risk!

SEE ALSO

alternatives(8)

AUTHOR
Cloudera, Thomas Koch <thomas.koch@ymc.ch>

COPYRIGHT
2008 The Apache Software Foundation. All rights reserved.