大数据技术与应用000220

大数据技术与应用苏州大学苏州大学唐煜安装jdk安装jdk拷贝文件至b系统•拷贝文件至Debian系统ty@MacBook:~$ls‐l‐rw‐r‐‐r‐‐1  ty  ty  97643815  Jan 23 10:21  jdk‐7u51‐linux‐i586.tar.gz改变状态•改变状态ty@MacBook:~$ chmod755 jdk‐7u51‐linux‐i586.tar.gz‐rwxr‐xr‐x1  ty  ty  97643815  Jan 23 10:21  jdk‐7u51‐linux‐i586.tar.gz解压缩•解压缩ty@MacBook:~$tar ‐xzvfjdk‐7u51‐linux‐i586.tar.gz设置路径和类路径设置路径和类路径生成录•生成目录ty@MacBook:~$ls‐ly@$drwxr‐xr‐x  8 tyty4096  Dec 18 19:25 jdk1.7.0_51建立软链接•建立软链接ty@MacBook:~$ln‐s jdk1.7.0_51 jdk•编辑\etc下配置文件.bashrc或者profile文件•或者在\etc\profiled下建立Shell文件来设置•或者在\etc\profile.d下建立Shell文件来设置环境变量localsh文件local.sh文件进录•进入/etc/profile.d目录ty@MacBook:~$cd/etc/profile.dy@$//pty@MacBook: /etc/profile.d$ ls‐lrwrr1rootroot660Jun172012‐rw‐r‐‐r‐‐1 root root660 Jun 17 2012 bash_completion.sh编辑文件利用用户•编辑local.sh文件(利用root用户)ty@MacBook: /etc/profile.d$ surooty@//proot@MacBook:/etc/profile.d# nanolocal.sh环境变量环境变量•export JAVA_HOME=/home/ty/jdk•exportexport CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/lib/toolsjar:$CLASSPATHb/tools.jar:$CLASSPATH•export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$PATH改变文件用户改变文件用户改变文件用户属性•改变文件用户属性root@MacBook:/etc/profile.d# chownty:tylocal.sh‐rw‐r‐‐r‐‐1 root root 152 Feb 4 03:08 local.sh‐rw‐r‐‐r‐‐1 ty ty 152 Feb 4 03:08 local.shyyroot@MacBook:/etc/profile.d# chmod755 local.sh•退出root用户并改变路径•退出root用户并改变路径root@MacBook:/etc/profile.d# exitt@MBk/t/fild$llhty@MacBook:/etc/profile.d$ source local.shty@MacBook: /etc/profile.d$cdHadoop安装Hadoop安装拷文件至系统•拷贝文件至Debian系统•解压缩解压缩ty@MacBook:~$tar ‐xzvfhadoop‐2.2.0.tar.gz建立软链接•建立软链接ty@MacBook:~$ln‐s hadoop‐2.2.0 hadoopypp•设置环境变量t@MBk~$d/t/fildty@MacBook:~$cd/etc/profile.dty@MacBook:/etc/profile.d$ nanolocal.sh环境变量环境变量HADOOPPREFIX/h//hd•exportHADOOP_PREFIX=/home/ty/hadoop•exportHADOOP_HOME=$HADOOP_PREFIX•eportHADOOPCOMMONHOME$HADOOPPREFIX•exportHADOOP_COMMON_HOME=$HADOOP_PREFIX•exportHADOOP_HDFS_HOME=$HADOOP_PREFIX•exportHADOOPMAPREDHOME=$HADOOPPREFIX•exportHADOOP_MAPRED_HOME=$HADOOP_PREFIX•exportHADOOP_YARN_HOME=$HADOOP_PREFIX•exportexportHADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop•exportPATH=$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$PATH修改配置文件修改配置文件进入hd配置文件路径•进入hadoop配置文件路径ty@MacBook:/etc/profile.d$ cdty@MacBook~$cdhadoop/etc/hadoopty@MacBook:~$cdhadoop/etc/hadoopty@MacBook:~/hadoop/etc/hadoop$ls‐l•修改配置文件•修改配置文件–‐systems/hadoop‐yarn‐installation‐definitive‐guidesystems/hadoopyarninstallationdefinitiveguide–core‐site.xml–hdfs‐site.xml–yarn‐site.xml (修改资源)–hadoop‐env.sh coresitexmlcore‐site.xmlfiiconfiguration property ppynamefs.defaultFS/name valuehdfs://localhost:9000/valuevaluehdfs://localhost:9000/value descriptionNameNode/URI/description /property /ppy/configurationhdfssitexmlhdfs‐site.xmlconfigurationconfiguration property namedfs.datanode.data.dir/namevaluefile:///home/ty/hadoop‐220/hdfs/datanode/valuevaluefile:///home/ty/hadoop‐2.2.0/hdfs/datanode/value descriptionComma separated list of paths on the local filesystemof a DataNodewhere it should store its blocks./description /p/property property namedfs.namenode.name.dir/name valuefile:///home/ty/hadoop‐2.2.0/hdfs/namenode/value descriptionPath on the local filesystemwhere the NameNodestores the namespace and transaction logs persistently./diti/description /property /configurationhadoopenvshhadoop‐env.sh文件中句文件中语句exportJAVAHOME=${JAVAHOME}export JAVA_HOME${JAVA_HOME}修改成# export JAVA_HOME=${JAVA_HOME}exportJAVAHOME=/home/ty/jdkexport JAVA_HOME=/home/ty/jdk免密码ssh登陆免密码ssh登陆•RSA公钥密码生成•RSA公钥密码生成ty@MacBook:~$ssh‐keygen@MBk$dhty@MacBook:~$cd.sshty@MacBook:~/.ssh$ls‐lty@MacBook:~/.ssh$catidrsa.pub  y@/$_pauthorized_keysty@MacBook:~/.ssh$sshlocalhosty@/$启动Hadoop启动Hadoop•格式化Hadoop•格式化Hadoopty@MacBook:~$hadoopnamenode‐format启动•启动Hadoopty@MacBook:~$start‐dfs.shty@MacBook:~$start‐yarn.shty@MacBook:~$jps网页浏览网页浏览页面端•NameNode页面端口：50070–localhost:50070•Secondary NameNode页面端口：50090页面端•DataNode页面端口：50075•yarn.resourcemanage页面端口：8088yarn.resourcemanage页面端口：8088•……NameNode页面NameNode页面测试Hadoop测试Hadoop•建立目录•建立目录ty@MacBook:~$hadoopfs‐mkdir/in上传文件•上传文件ty@MacBook:~$echo “Hello World“  text1.txtty@MacBook:~$echo “Hello Hadoop“  text2.txtty@MacBook:~$hadoopfs‐put text*.txt /inty@MacBook:~$hadoopfs‐ls/inWordCount测试WordCount测试•统计文件中的字数•统计文件中的字数–利用Hadoop自带程序ty@MacBook:~$ hadoopjar /home/ty/hadoop/share/hadoop/mapreduce/hadoop‐mapreduceexamples220jarwordcount/in/outmapreduce‐examples‐2.2.0.jar wordcount/in /outty@MacBook:~$hadoopfs‐ls/outty@MacBook:~$hadoopfscat/out/partr00000ty@MacBook:~$hadoopfs‐cat /out/part‐r‐00000关闭Hadoop关闭Hadoop•关闭Hadoop•关闭Hadoopty@MacBook:~$stop‐yarn.sht@MBk~$tdfhty@MacBook:~$stop‐dfs.shty@MacBook:~$jps安装模式安装模式单机模式()•单机模式(×)•伪分布模式–在PC机上熟练安装和配置–进一步的配置信息参阅9‐project‐dist/hadoop‐hdfs/hdfs‐default.xml9http://hadoopapacheorg/docs/current/hadoop9‐yarn/hadoop‐yarn‐common/yarn‐default.xml•完全分布模式完全分布模式–‐systems/hadoop‐yarn‐installation‐definitive‐guide