Please note, that the packaging of hadoop for Debian is still work in
progress.  It is not yet tested enough to work reliable in your
production critical cluster. However it is included in Debian so that
people can easily setup a hadoop cluster to learn about hadoop, test
the packaging and help improving it.

See the TODO file!

Getting started
===============

Starting all daemons
--------------------

For a first test, you can install all hadoop daemons on the same
(developer) machine. The standard installations comes with a pseudo
distributed configuration that runs all parts of the cluster on the
same machine.

Format the hdfs filesystem:

# /etc/init.d/hadoop-namenoded stop
# su -c"hadoop namenode -format" hadoop
Re-format filesystem in /var/lib/hadoop/cache/hadoop/dfs/name ? (Y or N) Y

Open the namenode webinterface at http://localhost:50070

Start the secondary namenode
# /etc/init.d/hadoop-secondarynamenoded stop
And verify it's logfile under /var/log/hadoop. After some time you
should see a line like
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done. New Image Size: 

Start one datanode
# /etc/init.d/hadoop-datanoded start
After some moments, the webinterface of the namenode should list one
living datanone.

Start the jobtracker
# /etc/init.d/hadoop-jobtrackerd start
And verify the webinterface at http://localhost:50030

Start one tasktracker
# /etc/init.d/hadoop-tasktrackerd start
And verify the webinterface at http://localhost:50060
Also verify, that the webinterface of the jobtracker now lists one
live tasktracker.

Run an example job
------------------

The following commands must be run as user hadoop, therefor:
# su hadoop

Prepare input data:

$ hadoop fs -mkdir input
$ hadoop fs -put /etc/hadoop/conf/*.xml input
$ hadoop fs -ls input
Found 3 items
-rw-r--r--   1 hadoop supergroup        333 2010-02-02 17:01 /user/hadoop/input/core-site.xml
-rw-r--r--   1 hadoop supergroup        491 2010-02-02 17:01 /user/hadoop/input/hdfs-site.xml
-rw-r--r--   1 hadoop supergroup        213 2010-02-02 17:01 /user/hadoop/input/mapred-site.xml

$ hadoop jar /usr/share/java/hadoop-examples.jar grep input output 'dfs[a-z.]+'

Let's see what the job has produced:

$ hadoop fs -ls
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2010-02-02 17:01 /user/hadoop/input
drwxr-xr-x   - hadoop supergroup          0 2010-02-02 17:04 /user/hadoop/output
$ hadoop fs -ls output
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2010-02-02 17:03 /user/hadoop/output/_logs
-rw-r--r--   1 hadoop supergroup         51 2010-02-02 17:04 /user/hadoop/output/part-00000

Print the result:

$ hadoop fs -cat output/part-00000
1       dfs.name.dir
1       dfs.permissions
1       dfs.replication

Add your own configuration
==========================

The hadoop configuration is managed by the update-alternatives
mechanism. This means, you can build a debian package, that adds an
alternive for hadoop-conf with a higher priority then the default one
(10) and /etc/hadoop/conf will point to your new configuration. When
done by hand, this looks like:

# cp -r /etc/hadoop/conf.empty /etc/hadoop/conf.mine
# update-alternatives --install /etc/hadoop/conf hadoop-conf \
                                /etc/hadoop/conf.mine 15

