HDFS详解大数据

jdk:jdk-8u131-linux-x64.tar.gz
hadoop:hadoop-2.8.0.tar.gz
centos:CentOS-7-x86_64-Minimal-1611.iso

一.关闭防火墙和强制安全策略

systemctl stop firewalld.service 
systemctl disable firewalld.service

修改配置vi /etc/selinux/config

 SELINUX=disabled

二.修改主机名

修改配置 vi /etc/sysconfig/network

HOSTNAME=centos201

三.配置三台主机名映射

修改配置 vi /etc/hosts

192.168.193.201 centos201 
192.168.193.202 centos202 
192.168.193.203 centos203

重启系统 生效

四.添加用户

groupadd -g 601 wuyang 
useradd -g wuyang -u 601 wuyang 
(echo '123456';sleep 1;echo '123456')| passwd wuyang 

五.免密钥登录

su wuyang 
ssh-keygen -t rsa 
ssh-copy-id centos201 
ssh-copy-id centos202 
ssh-copy-id centos203 

六.安装JDK

A:解压/usr/java
B:vi /etc/profile
C:添加

    export JAVA_HOME=/usr/java/jdk1.8.0_131 
    export JRE_HOME=/usr/java/jdk1.8.0_131/jre 
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 
    export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

D:source /etc/profile
E:alternatives –install /usr/bin/java java /usr/java/jdk1.8.0_131/bin/java 300
F:alternatives –config java
G:alternatives –install /usr/bin/javac javac /usr/java/jdk1.8.0_131/bin/java 300
H:alternatives –config javac

七.安装HDFS

A:解压到/usr/hadoop
B:把Hadoop的安装路径添加到”/etc/profile”中

    vi /etc/profile

    export HADOOP_HOME=/usr/hadoop   
    export PATH=$PATH:$HADOOP_HOME/bin

C:source /etc/profile
D:配置hadoop-env.sh，并确认生效

    vi /usr/hadoop/etc/hadoop/hadoop-env.sh

    export JAVA_HOME=/usr/java/jdk1.8.0_131 
    export HADOOP_CONF_DIR=/usr/hadoop/etc/hadoop/

    source /usr/hadoop/etc/hadoop/hadoop-env.sh 

    hadoop version

E:在/usr/hadoop目录下创建子目录

    [root@centos201 hadoop]# mkdir tmp 
    [root@centos201 hadoop]# mkdir hdfs 
    [root@centos201 hadoop]# cd hdfs 
    [root@centos201 hdfs]# mkdir name 
    [root@centos201 hdfs]# mkdir tmp 
    [root@centos201 hdfs]# mkdir data

F:配置core-site.xml文件 在/usr/hadoop/etc/hadoop”目录下。修改Hadoop核心配置文件core-site.xml，这里配置的是HDFS master（即namenode）的地址和端口号。

vi /usr/hadoop/etc/hadoop/core-site.xml 

<configuration> 
<property> 
    <name>hadoop.tmp.dir</name> 
    <value>/usr/hadoop/tmp</value> 
    <final>true</final> 
<!--拢篓卤赂注拢潞??? /usr/hadoop 目录?陆篓b tmp ?录镁录校漏 --> 
    <description>A base for other temporary directories.</description> 
</property> 
<property> 
    <name>fs.default.name</name> 
    <value>hdfs://centos201:9000</value>    
<!-- hdfs://Master.Hadoop:22--> 
        <final>true</final> 
</property> 
<property> 
     <name>io.file.buffer.size</name> 
     <value>131072</value> 
</property>   
</configuration>

G:配置hdfs-site.xml文件

vi /usr/hadoop/etc/hadoop/hdfs-site.xml

<property>    
        <name>dfs.replication</name>    
        <value>3</value>    
    </property>    
    <property>    
        <name>dfs.name.dir</name>    
        <value>/usr/hadoop/hdfs/name</value>    
    </property>    
    <property>    
        <name>dfs.data.dir</name>    
        <value>/usr/hadoop/hdfs/data</value>    
    </property>    
    <property>     
         <name>dfs.namenode.secondary.http-address</name>     
         <value>centos201:9001</value>     
    </property>     
    <property>     
         <name>dfs.webhdfs.enabled</name>     
         <value>true</value>     
    </property>     
    <property>     
         <name>dfs.permissions</name>     
         <value>false</value>     
    </property>

H:配置mapred-site.xml文件

cp /usr/hadoop/etc/hadoop/mapred-site.xml.template /usr/hadoop/etc/hadoop/mapred-site.xml 
vi /usr/hadoop/etc/hadoop/mapred-site.xml

<configuration>   
    <property>     
          <name>mapreduce.framework.name</name>     
          <value>yarn</value>     
    </property>      
</configuration>

I:配置yarn-site.xml文件

vi /usr/hadoop/etc/hadoop/yarn-site.xml

<property>     
  <name>yarn.resourcemanager.address</name>     
  <value>centos201:18040</value>     
</property>     
<property>     
  <name>yarn.resourcemanager.scheduler.address</name>     
  <value>centos201:18030</value>     
</property>     
<property>     
  <name>yarn.resourcemanager.webapp.address</name>     
  <value>centos201:18088</value>     
</property>     
<property>     
  <name>yarn.resourcemanager.resource-tracker.address</name>     
  <value>centos201:18025</value>     
</property>     
<property>     
  <name>yarn.resourcemanager.admin.address</name>     
  <value>centos201:18141</value>     
</property>     
<property>     
  <name>yarn.nodemanager.aux-services</name>     
  <value>mapreduce_shuffle</value>     
</property>     
<property>     
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>     
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>     
</property>

J:配置slaves文件

vi /usr/hadoop/etc/hadoop/slaves

centos201 
centos202 
centos203

K:给权限

chown -R wuyang:wuyang /usr/hadoop/ 
chown -R wuyang:wuyang /usr/java/ 
sudo chmod -R a+w /usr/hadoop/

L:切换用户

su wuyang

M:初始化

/usr/hadoop/bin/hdfs namenode -format

N:启动

/usr/hadoop/sbin/start-all.sh

O:测试

http://192.168.193.201:18088

重点内容

1. 资源设定

2.参数配置