首页   快速返回

cdh5 mapreduce 例子     所属分类 bigdata
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars

ls -a|grep example

hadoop-examples-2.6.0-mr1-cdh5.14.2.jar
hadoop-examples.jar
hadoop-mapreduce-examples-2.6.0-cdh5.14.2.jar
hbase-examples-1.2.0-cdh5.14.2.jar
mahout-examples-0.9-cdh5.14.2.jar
mahout-examples-0.9-cdh5.14.2-job.jar
oozie-examples.jar
spark-examples-1.6.0-cdh5.14.2-hadoop2.6.0-cdh5.14.2.jar

PI计算
hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.14.2.jar pi 5 5 




单词统计 wordcount

test.txt

hello hadoop
apache hadoop is a framework for running applications on large cluster built of commodity hardware
the hadoop framework transparently provides applications both reliability and data motion
hadoop implements a computational paradigm named Map/Reduce
where the application is divided into many small fragments of work
each of which may be executed or re-executed on any node in the cluster


hadoop fs -ls /
hadoop fs -mkdir /tmp/dugang/
hadoop fs -ls /tmp/dugang

hadoop fs -put test.txt /tmp/dugang

hadoop fs -cat /tmp/dugang/test.txt


hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.14.2.jar wordcount /tmp/dugang/test.txt /tmp/dugang/out

输出到  /tmp/dugang/out 目录中 


 hadoop fs -ls /tmp/dugang/out
Found 33 items
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/_SUCCESS
-rw-r--r--   3 root supergroup         16 2019-10-09 12:55 /tmp/dugang/out/part-r-00000
-rw-r--r--   3 root supergroup          7 2019-10-09 12:55 /tmp/dugang/out/part-r-00001
-rw-r--r--   3 root supergroup         26 2019-10-09 12:55 /tmp/dugang/out/part-r-00002
-rw-r--r--   3 root supergroup          7 2019-10-09 12:55 /tmp/dugang/out/part-r-00003
-rw-r--r--   3 root supergroup         31 2019-10-09 12:55 /tmp/dugang/out/part-r-00004
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00005
-rw-r--r--   3 root supergroup         30 2019-10-09 12:55 /tmp/dugang/out/part-r-00006
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00007
-rw-r--r--   3 root supergroup         20 2019-10-09 12:55 /tmp/dugang/out/part-r-00008
-rw-r--r--   3 root supergroup         11 2019-10-09 12:55 /tmp/dugang/out/part-r-00009
-rw-r--r--   3 root supergroup         10 2019-10-09 12:55 /tmp/dugang/out/part-r-00010
-rw-r--r--   3 root supergroup         18 2019-10-09 12:55 /tmp/dugang/out/part-r-00011
-rw-r--r--   3 root supergroup         23 2019-10-09 12:55 /tmp/dugang/out/part-r-00012
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00013
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00014
-rw-r--r--   3 root supergroup         23 2019-10-09 12:55 /tmp/dugang/out/part-r-00015
-rw-r--r--   3 root supergroup         17 2019-10-09 12:55 /tmp/dugang/out/part-r-00016
-rw-r--r--   3 root supergroup         37 2019-10-09 12:55 /tmp/dugang/out/part-r-00017
-rw-r--r--   3 root supergroup          7 2019-10-09 12:55 /tmp/dugang/out/part-r-00018
-rw-r--r--   3 root supergroup         11 2019-10-09 12:55 /tmp/dugang/out/part-r-00019
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00020
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00021
-rw-r--r--   3 root supergroup          6 2019-10-09 12:55 /tmp/dugang/out/part-r-00022
-rw-r--r--   3 root supergroup          9 2019-10-09 12:55 /tmp/dugang/out/part-r-00023
-rw-r--r--   3 root supergroup         13 2019-10-09 12:55 /tmp/dugang/out/part-r-00024
-rw-r--r--   3 root supergroup         24 2019-10-09 12:55 /tmp/dugang/out/part-r-00025
-rw-r--r--   3 root supergroup         21 2019-10-09 12:55 /tmp/dugang/out/part-r-00026
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00027
-rw-r--r--   3 root supergroup          8 2019-10-09 12:55 /tmp/dugang/out/part-r-00028
-rw-r--r--   3 root supergroup         20 2019-10-09 12:55 /tmp/dugang/out/part-r-00029
-rw-r--r--   3 root supergroup         26 2019-10-09 12:55 /tmp/dugang/out/part-r-00030
-rw-r--r--   3 root supergroup          0 2019-10-09 12:55 /tmp/dugang/out/part-r-00031


hadoop fs -cat /tmp/dugang/out/part-r-00030
running	1
transparently	1

hadoop fs -cat /tmp/dugang/out/part-r-00000
a	2
many	1
on	2

上一篇     下一篇
hdfs namenode HA 高可用

HDFS Federation 机制

CDH 5.14.2 各种角色

HDFS特点及优缺点

hdfs安全模式

文件内容对比技巧