标签: 数据治理

  • hadoop中NameNode、DataNode和Client三者之间协作关系及通信方式介绍详解大数据

    <ignore_js_op> 

    1)NameNode、DataNode和Client
             NameNode可以看作是分布式文件系统中的管理者,主要负责管理文件系统的命名空间、集群配置信息和存储块的复制等。NameNode会将文件系统的Meta-data存储在内存中,这些信息主要包括了文件信息、每一个文件对应的文件块的信息和每一个文件块在DataNode的信息等。
    DataNode是文件存储的基本单元,它将Block存储在本地文件系统中,保存了Block的Meta-data,同时周期性地将所有存在的Block信息发送给NameNode。
    Client就是需要获取分布式文件系统文件的应用程序。

    2)文件写入
        Client向NameNode发起文件写入的请求。
        NameNode根据文件大小和文件块配置情况,返回给Client它所管理部分DataNode的信息。
        Client将文件划分为多个Block,根据DataNode的地址信息,按顺序写入到每一个DataNode块中。

    3)文件读取
        Client向NameNode发起文件读取的请求。
        NameNode返回文件存储的DataNode的信息。
        Client读取文件信息。

    —————————————————————————————————————————————————————-

    通信方式介绍:

    在hadoop系统中,master/slaves/client的对应关系是:
    master—namenode;
    slaves—datanode;
    client—dfsclient;
    那究竟是通过什么样的方式进行通信的呢,在这里从大体介绍一下:
    简单地讲:
    client和namenode之间是通过rpc通信;
    datanode和namenode之间是通过rpc通信;
    client和datanode之间是通过简单的socket通信。
    随便拔一下DFSClient的代码,可以看到它有一个成员变量public final ClientProtocolnamenode;
    而再拔一下DataNode的代码,可以看到它也有一个成员变量public DatanodeProtocolnamenode

     

    文章转自:http://www.aboutyun.com/thread-6794-1-1.html

  • 解决客户端通过zookeeper连接到hbase时连接过多的问题详解大数据

    原因:客户端程序通过zookeeper访问hbase的连接数超过设置的默认链接数(默认数是30),连接数不够用会导致后续的连接连接不上去。

    解决办法:设置hbase-site.xml配置文件,添加如下属性

    <property>
        <name>hbase.zookeeper.property.maxClientCnxns</name>
        <value>300</value>
        <description>Property from ZooKeeper’s config zoo.cfg.
        Limit on number of concurrent connections (at the socket level) that a
        single client, identified by IP address, may make to a single member of
        the ZooKeeper ensemble. Set high to avoid zk connection issues running
        standalone and pseudo-distributed.
        </description>
      </property>

    将最大连接数我这设置成了300,后来发现仍然提示同样的问题,最大连接数并没有起作用,根据属性提示,直接修改zoo.cfg配置文件

    添加:maxClientCnxns=300

    重启下zookeeper,hbase,重新测试,问题解决。

     

    文章转自:http://blog.csdn.net/chlaws/article/details/7101020

  • Hive安装过程(mysql/oracle存储元数据)详解大数据

    Hive安装过程(mysql/oracle存储元数据)

    前置条件:
    – mysql数据库已经安装成功
    – hadoop环境已经配置正确,且可以提供正常服务
    说明:
    -由于资源有限,下面配置的hadoop集群只有一个节点,所有服务都在一个节点上启动

    (一)mysql存储元数据

    (1)创建mysql用户

    [email protected]:~$ mysql -uroot -pmysql 
    mysql> CREATE USER 'hive' IDENTIFIED BY 'mysql'; 
    mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' WITH GRANT OPTION; 
    mysql> flush privileges;

    (2)重启mysql服务

    sudo service mysql restart

    (3)建立 Hive 专用的元数据库(使用刚才创建的hive用户登陆)

    mysql> exit; 
    hadoop@ubuntu:~$ mysql -uhive -pmysql 
    mysql> create database hive;

    (4)Hive环境变量及配置项

    • 设置环境变量
      在/home/hadoop/.bashrc文件中加入环境变量
    export HIVE_HOME=/home/hadoop/apache-hive-2.1.0-bin 
    export PATH=$HIVE_HOME/bin:$PATH
    • 修改配置文件

    将 $HIVE_HOME/conf/hive-default.xml.template修改为$HIVE_HOME/conf/hive-site.xml,修改hive-site.xml文件内容如下:

      <property> 
        <name>javax.jdo.option.ConnectionURL</name> 
        <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> 
      </property> 
       <property> 
        <name>javax.jdo.option.ConnectionUserName</name> 
        <value>hive</value> 
        <description>Username to use against metastore database</description> 
      </property> 
      <property> 
        <name>javax.jdo.option.ConnectionPassword</name> 
        <value>mysql</value> 
        <description>password to use against metastore database</description> 
       </property>

    (5)下载mysqlJDBC驱动包,复制到Hive的lib目录下

    cp  mysql-connector-java-5.1.39-bin.jar  $HIVE_HOME/lib/

    (6)初始化hive元数据表

     schematool  -initSchema   -dbType mysql

    (7)启动 Hive Shell

    启动 Hive Shell, 执行“show tables;”命令,如果不报错,表明基于独立元数据库的 Hive 已经安装成功了

    (8)测试

    • 创建表
    hive> create table mytable(str STRING);
    • 查看表信息
    hive> describe mytable; 
    OK 
    str                     string                                       
    Time taken: 0.075 seconds, Fetched: 1 row(s)
    • 添加数据
     echo "test hive table row" >/tmp/myfile 
     hive -e "LOAD DATA LOCAL INPATH '/tmp/myfile' INTO TABLE mytable" ;
    • 查询数据
    hive> select * from mytable; 
    OK 
    test hive table row 
    Time taken: 0.152 seconds, Fetched: 1 row(s)
    • 删除表
    hive> drop table  mytable; 
    OK 
    Time taken: 2.133 seconds

    (二)oracle 存储元数据

    (1)创建用户

    sqlplus "sys as sysdba" 
     
    SQL> create user hive identified by hive; 
     
    User created. 
     
    SQL> grant connect to hive; 
     
    Grant succeeded. 
     
    SQL> grant all privileges to hive; 
     
    Grant succeeded. 
    

    (2)配置文件

       <property> 
            <name>javax.jdo.option.ConnectionURL</name> 
            <value>jdbc:oracle:thin:@myhost:1521:xe</value> 
       </property> 
       <property> 
            <name>javax.jdo.option.ConnectionDriverName</name> 
             <value>oracle.jdbc.OracleDriver</value> 
       </property> 
     
      <property> 
            <name>javax.jdo.option.ConnectionUserName</name> 
            <value>hive</value> 
      </property> 
     
     <property> 
            <name>javax.jdo.option.ConnectionPassword</name> 
             <value>hive</value> 
     </property>

    (3)下载oracle驱动包

    驱动包地址

    (4)初始化表

    schematool   -dbType  oracle  -initSchema 

    (5)示例(使用Hive自带的数据)

    • 创建表
    hive> CREATE TABLE testtable (foo INT, bar STRING) PARTITIONED BY (ds STRING);
    • 插入数据
    hive> LOAD DATA LOCAL INPATH '$HIVE_HOME/examples/files/kv1.txt' OVERWRITE INTO TABLE testtable PARTITION (ds='2016-07-28'); 
     
    hive> LOAD DATA LOCAL INPATH '$HIVE_HOME/examples/files/kv2.txt' OVERWRITE INTO TABLE testtable PARTITION (ds='2016-07-29');

    请将$HIVE_HOME替换成具体路径执行
    - 查询数据

    hive> select t.foo from testtable t where t.ds='2016-07-29'; 

    (6)参考

    http://www.cloudera.com/documentation/archive/cdh/4-x/4-2-0/CDH4-Installation-Guide/cdh4ig_topic_18_4.html
    https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_hive_metastore_configure.html

  • Hive Example详解大数据

    Hive Example

    此实例主要学习Hive的基本操作

    准备数据

    • 下载数据

      wget http://files.grouplens.org/datasets/movielens/ml-100k.zip

    • 解压数据

      unzip ml-100k.zip

    • 数据说明(具体请查看解压目录中README)
    u.data     -- The full u data set, 100000 ratings by 943 users on 1682 items. 
                  Each user has rated at least 20 movies.  Users and items are 
                  numbered consecutively from 1.  The data is randomly 
                  ordered. This is a tab separated list of 
                     user id | item id | rating | timestamp. 
                  The time stamps are unix seconds since 1/1/1970 UTC 
     
    u.info     -- The number of users, items, and ratings in the u data set. 
     
    u.item     -- Information about the items (movies); this is a tab separated 
                  list of 
                  movie id | movie title | release date | video release date | 
                  IMDb URL | unknown | Action | Adventure | Animation | 
                  Children's | Comedy | Crime | Documentary | Drama | Fantasy | 
                  Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi | 
                  Thriller | War | Western | 
                  The last 19 fields are the genres, a 1 indicates the movie 
                  is of that genre, a 0 indicates it is not; movies can be in 
                  several genres at once. 
                  The movie ids are the ones used in the u.data data set. 
     
    u.genre    -- A list of the genres. 
     
    u.user     -- Demographic information about the users; this is a tab 
                  separated list of 
                  user id | age | gender | occupation | zip code 
                  The user ids are the ones used in the u.data data set. 
     
    u.occupation -- A list of the occupations. 
    

    创建表

    接下来使用上述数据中的其中三个数据做接下来的练习,创建文件 hiveExample.sql,hiveExample.sql内容如下:

    CREATE TABLE IF NOT EXISTS u_data ( 
            userid INT, 
            movieid INT, 
            rating INT, 
            unixtime STRING) 
    ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY '/t' 
    STORED AS TEXTFILE; 
     
    CREATE TABLE IF NOT EXISTS u_user( 
            userid INT, 
            age  INT, 
            gender STRING, 
            occupation STRING , 
            zipcode INT) 
    ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY '|' 
    STORED AS TEXTFILE; 
     
    CREATE TABLE IF NOT EXISTS u_item( 
            movieid INT, 
            movietitle STRING, 
            releasedate  STRING, 
            videoreleasedate STRING, 
            IMDbURL STRING, 
            unknown  INT, 
            Action INT, 
            Adventure  INT, 
            Animation INT, 
            Childrens INT, 
            Comedy INT, 
            Crime INT, 
            Documentary INT, 
            Drama INT, 
            Fantasy INT, 
            FilmNoir INT, 
            Horror INT, 
            Musical INT, 
            Mystery INT, 
            Romance INT, 
            SciFi INT, 
            Thriller INT, 
            War INT, 
            Western INT) 
    ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY '|' 
    STORED AS TEXTFILE;

    执行如下命令,创建表:
    hive -f hiveExample.sql

    导入数据:

    创建文件 importData.sql,importData.sql内容如下:

    LOAD DATA LOCAL INPATH '/home/dev/storeFile/ml-100k/u.data' OVERWRITE INTO TABLE u_data; 
     
    LOAD DATA LOCAL INPATH '/home/dev/storeFile/ml-100k/u.user' OVERWRITE INTO TABLE u_user; 
     
    LOAD DATA LOCAL INPATH '/home/dev/storeFile/ml-100k/u.item' OVERWRITE INTO TABLE u_item;

    其中/home/dev/storeFile/ml-100k/u.data为本地文件路径。
    执行如下命令,导入数据:
    hive -f importData.sql
    导入成功后打印如下信息:

    Loading data to table default.u_data 
    OK 
    Time taken: 3.852 seconds 
    Loading data to table default.u_user 
    OK 
    Time taken: 0.563 seconds 
    Loading data to table default.u_item 
    OK 
    Time taken: 0.706 seconds

    HiveQL查询实例

    select / order by /limit

    将 u_user表中用户按照age降序排列,并查询出前5位用户信息:

    hive> select * from u_user u order by u.age desc limit 5 ;

    结果:

    481 73  M   retired 37771 
    860 70  F   retired 48322 
    767 70  M   engineer    0 
    803 70  M   administrator   78212 
    585 69  M   librarian   98501

    group by

    • group by 按照一个或者多个列对结果进行分组。
    • count / distinct 一同使用 DISTINCT 和 COUNT 关键词,来计算非重复结果的数目
      统计u_user表中不同性别的人数。
    select u.gender, count(distinct u.userid) from u_user u group by u.gender;

    结果:

    OK 
    F   273 
    M   670 
    Time taken: 19.786 seconds, Fetched: 2 row(s)

    参考资料

    https://cwiki.apache.org/confluence/display/Hive/GettingStarted

  • Hive order by/sort by/distribute by/cluster by作用详解大数据

    Hive order by/sort by/distribute by/cluster by作用

    order by

    Hive中的order by跟传统的sql语言中的order by作用是一样的,会对查询的结果做一次全局排序,所以说,只有hive的sql中制定了order by所有的数据都会到同一个reducer进行处理(不管有多少map,也不管文件有多少的block只会启动一个reducer)。但是对于大量数据这将会消耗很长的时间去执行。

    sort by

    sort by 只会对每一个reducer 中的数据进行排序,也就是执行一个局部的排序,这个可以保证每一个reducer的输出数据都是有序的(但并非全局有序)。这样可以提高后面进行全局排序的效率。

    distribute by

    distribute by 控制map的输出在reducer中是如何划分的。distribute by语句必须写在sort by语句之前。

    cluster by

    cluster by的功能就是distribute by和sort by相结合。

  • Hbase完全分布式集群安装配置(Hbase1.0.0,Hadoop2.6.0)详解大数据

    1.安装软件

        OS:centos6.5

        Hadoop:hadoop2.6.0

        Hbase:hbase.1.0.0

       JDK: jdk1.7.0_51

    集群机器:

        192.168.153.130(hadoop130 namenode)

        192.168.153.131 (hadoop131datanode)

        192.168.153.132 (hadoop132datanode)

    2.安装步骤

        如下安装步骤假设的JDK已经安装,如果未安装请自行查找JDK安装教程。

    2.1 安装hadoop

    (1)下载hadoop版本(hadoop2.6.0),下载地址:http://www.apache.org/dyn/closer.cgi/hadoop/common/   在此路径下选择相应的版本下载。

    (2)解压hadoop-2.6.0.tar.gz

        tar zxvf hadoop-2.6.0.tar.gz  -C /home/hadoop001/thirdparty/

        将hadoop解压到/home/hadoop001/thirdparty/ 目录下。

    (3)将hadoop添加到环境变量中

        vim ~/.bashrc 打开.bashrc文件添加HADOOP_HOME

        exportJAVA_HOME=/home/hadoop001/thirdparty/jdk1.7.0_51

        export PATH=$HADOOP_HOME/bin:$PATH

    (4)修改配置文件($HADOOP_HOME/etc/hadoop目录下)

     配置core-site.xml

     

    配置hdfs-site.xml:

    配置mapred-site.xml文件

     

    配置yarn-site.xml文件


    修改slaves文件,添加datanode节点hostname到slaves文件中

       hadoop131

       hadoop130

    (5)格式化集群,在hadoop130节点上执行如下命令:

        hadoopnamenode -format

    (6)启动集群,在hadoop130节点上执行如下命令:

        start-all.sh

    (7) 通过浏览器查看hadoop是否安装成功:

        输入: http://hadoop130:50070

       

        输入:http://hadoop130:8088/

       

    2.2 安装Hbase

    (1)下载hbase版本

        下载地址:http://www.apache.org/dyn/closer.cgi/hbase/,在此路径下选择相应的版本下载,本次安装下载hadoop1.0.0版本

    (2)解压hbase-1.0.0-bin.tar.gz

        tar zxvf hbase-1.0.0-bin.tar.gz –C/home/hadoop001/thirdparty/

    (3)将hbase添加到环境变量中

        exportHBASE_HOME=/home/hadoop001/thirdparty/hbase-1.0.0

        export PATH=$HBASE_HOME/bin:$PATH

    (4)修改配置文件

        修改hbase-env.sh

        exportJAVA_HOME=/home/hadoop001/thirdparty/jdk1.7.0_51

        修改hbase-site.xml

       

    备注:

        在上面的配置文件中,第一个属性指定本机的hbase的存储目录;第二个属性指定hbase的运行模式,true代表全分布模式;第三和第四个属性是关于Zookeeper集群的配置。我的Zookeeper安装在hadoop130,hadoop131和hadoop132上。

    修改regionservers,在regionservers文件中添加如下内容:

        hadoop131

        hadoop130

    (5)启动hbase

        启动hbase时要确保hdfs已经启动。在主节点上执行:

        start-hbase.sh

        启动成功后集群会多出如下进程:

        NameNode节点:

       

        Datanode节点:

             

        通过浏览器查看:

        输入:http://hadoop130:16030

        

    2.3 Hbase shell测试

    (1)执行hbase shell 命令:

    (2)创建testtable表

         create ‘testtable’, ‘colfaml’

    (3)put数据

         hbase shell是基于Ruby实现的,因此使用过程中可以将hbase shell与Ruby代码混合使用,此示例参考《HBase权威指南》p-66页shell介绍。

        for i in ‘a’..’z’ do for j in ‘a’..’z’ do /

        put ‘testtable’ , “row-#{i}#{j}”,”colfaml:#{j}” ,”#{j}” end end(4)查看插入数据

        scan ‘testtable’

        部分结果截图:

       

    3.常用参数(待续)

    4.备注

        目前安装所有配置都是最简配置,并没有考虑参数优化,此教程的目的是让Hbase先跑起来,优化后期继续做。

  • HBase Default Configuration(Hbase1.0.0)详解大数据

    The documentation below is generated using the default hbase configuration file,hbase-default.xml, as source.

    hbase.tmp.dir
    Description

    Temporary directory on the local filesystem. Change this setting to point to a location more permanent than ‘/tmp’, the usual resolve for java.io.tmpdir, as the ‘/tmp’ directory is cleared on machine restart.

    Default

    ${java.io.tmpdir}/hbase-${user.name}

    hbase.rootdir
    Description

    The directory shared by region servers and into which HBase persists. The URL should be ‘fully-qualified’ to include the filesystem scheme. For example, to specify the HDFS directory ‘/hbase’ where the HDFS instance’s namenode is running at namenode.example.org on port 9000, set this value to: hdfs://namenode.example.org:9000/hbase. By default, we write to whatever ${hbase.tmp.dir} is set too — usually /tmp — so change this configuration or else all data will be lost on machine restart.

    Default

    ${hbase.tmp.dir}/hbase

    hbase.cluster.distributed
    Description

    The mode the cluster will be in. Possible values are false for standalone mode and true for distributed mode. If false, startup will run all HBase and ZooKeeper daemons together in the one JVM.

    Default

    false

    hbase.zookeeper.quorum
    Description

    Comma separated list of servers in the ZooKeeper ensemble (This config. should have been named hbase.zookeeper.ensemble). For example, “host1.mydomain.com,host2.mydomain.com,host3.mydomain.com”. By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper ensemble servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which hbase will start/stop ZooKeeper on as part of cluster start/stop. Client-side, we will take this list of ensemble members and put it together with the hbase.zookeeper.clientPort config. and pass it into zookeeper constructor as the connectString parameter.

    Default

    localhost

    hbase.local.dir
    Description

    Directory on the local filesystem to be used as a local storage.

    Default

    ${hbase.tmp.dir}/local/

    hbase.master.info.port
    Description

    The port for the HBase Master web UI. Set to -1 if you do not want a UI instance run.

    Default

    16010

    hbase.master.info.bindAddress
    Description

    The bind address for the HBase Master web UI

    Default

    0.0.0.0

    hbase.master.logcleaner.plugins
    Description

    A comma-separated list of BaseLogCleanerDelegate invoked by the LogsCleaner service. These WAL cleaners are called in order, so put the cleaner that prunes the most files in front. To implement your own BaseLogCleanerDelegate, just put it in HBase’s classpath and add the fully qualified class name here. Always add the above default log cleaners in the list.

    Default

    org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner

    hbase.master.logcleaner.ttl
    Description

    Maximum time a WAL can stay in the .oldlogdir directory, after which it will be cleaned by a Master thread.

    Default

    600000

    hbase.master.hfilecleaner.plugins
    Description

    A comma-separated list of BaseHFileCleanerDelegate invoked by the HFileCleaner service. These HFiles cleaners are called in order, so put the cleaner that prunes the most files in front. To implement your own BaseHFileCleanerDelegate, just put it in HBase’s classpath and add the fully qualified class name here. Always add the above default log cleaners in the list as they will be overwritten in hbase-site.xml.

    Default

    org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner

    hbase.master.catalog.timeout
    Description

    Timeout value for the Catalog Janitor from the master to META.

    Default

    600000

    hbase.master.infoserver.redirect
    Description

    Whether or not the Master listens to the Master web UI port (hbase.master.info.port) and redirects requests to the web UI server shared by the Master and RegionServer.

    Default

    true

    hbase.regionserver.port
    Description

    The port the HBase RegionServer binds to.

    Default

    16020

    hbase.regionserver.info.port
    Description

    The port for the HBase RegionServer web UI Set to -1 if you do not want the RegionServer UI to run.

    Default

    16030

    hbase.regionserver.info.bindAddress
    Description

    The address for the HBase RegionServer web UI

    Default

    0.0.0.0

    hbase.regionserver.info.port.auto
    Description

    Whether or not the Master or RegionServer UI should search for a port to bind to. Enables automatic port search if hbase.regionserver.info.port is already in use. Useful for testing, turned off by default.

    Default

    false

    hbase.regionserver.handler.count
    Description

    Count of RPC Listener instances spun up on RegionServers. Same property is used by the Master for count of master handlers.

    Default

    30

    hbase.ipc.server.callqueue.handler.factor
    Description

    Factor to determine the number of call queues. A value of 0 means a single queue shared between all the handlers. A value of 1 means that each handler has its own queue.

    Default

    0.1

    hbase.ipc.server.callqueue.read.ratio
    Description

    Split the call queues into read and write queues. The specified interval (which should be between 0.0 and 1.0) will be multiplied by the number of call queues. A value of 0 indicate to not split the call queues, meaning that both read and write requests will be pushed to the same set of queues. A value lower than 0.5 means that there will be less read queues than write queues. A value of 0.5 means there will be the same number of read and write queues. A value greater than 0.5 means that there will be more read queues than write queues. A value of 1.0 means that all the queues except one are used to dispatch read requests. Example: Given the total number of call queues being 10 a read.ratio of 0 means that: the 10 queues will contain both read/write requests. a read.ratio of 0.3 means that: 3 queues will contain only read requests and 7 queues will contain only write requests. a read.ratio of 0.5 means that: 5 queues will contain only read requests and 5 queues will contain only write requests. a read.ratio of 0.8 means that: 8 queues will contain only read requests and 2 queues will contain only write requests. a read.ratio of 1 means that: 9 queues will contain only read requests and 1 queues will contain only write requests.

    Default

    0

    hbase.ipc.server.callqueue.scan.ratio
    Description

    Given the number of read call queues, calculated from the total number of call queues multiplied by the callqueue.read.ratio, the scan.ratio property will split the read call queues into small-read and long-read queues. A value lower than 0.5 means that there will be less long-read queues than short-read queues. A value of 0.5 means that there will be the same number of short-read and long-read queues. A value greater than 0.5 means that there will be more long-read queues than short-read queues A value of 0 or 1 indicate to use the same set of queues for gets and scans. Example: Given the total number of read call queues being 8 a scan.ratio of 0 or 1 means that: 8 queues will contain both long and short read requests. a scan.ratio of 0.3 means that: 2 queues will contain only long-read requests and 6 queues will contain only short-read requests. a scan.ratio of 0.5 means that: 4 queues will contain only long-read requests and 4 queues will contain only short-read requests. a scan.ratio of 0.8 means that: 6 queues will contain only long-read requests and 2 queues will contain only short-read requests.

    Default

    0

    hbase.regionserver.msginterval
    Description

    Interval between messages from the RegionServer to Master in milliseconds.

    Default

    3000

    hbase.regionserver.regionSplitLimit
    Description

    Limit for the number of regions after which no more region splitting should take place. This is not a hard limit for the number of regions but acts as a guideline for the regionserver to stop splitting after a certain limit. Default is MAX_INT; i.e. do not block splitting.

    Default

    2147483647

    hbase.regionserver.logroll.period
    Description

    Period at which we will roll the commit log regardless of how many edits it has.

    Default

    3600000

    hbase.regionserver.logroll.errors.tolerated
    Description

    The number of consecutive WAL close errors we will allow before triggering a server abort. A setting of 0 will cause the region server to abort if closing the current WAL writer fails during log rolling. Even a small value (2 or 3) will allow a region server to ride over transient HDFS errors.

    Default

    2

    hbase.regionserver.hlog.reader.impl
    Description

    The WAL file reader implementation.

    Default

    org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader

    hbase.regionserver.hlog.writer.impl
    Description

    The WAL file writer implementation.

    Default

    org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter

    hbase.master.distributed.log.replay
    Description

    Enable ‘distributed log replay’ as default engine splitting WAL files on server crash. This default is new in hbase 1.0. To fall back to the old mode ‘distributed log splitter’, set the value to ‘false’. ‘Disributed log replay’ improves MTTR because it does not write intermediate files. ‘DLR’ required that ‘hfile.format.version’ be set to version 3 or higher.

    Default

    true

    hbase.regionserver.global.memstore.size
    Description

    Maximum size of all memstores in a region server before new updates are blocked and flushes are forced. Defaults to 40% of heap. Updates are blocked and flushes are forced until size of all memstores in a region server hits hbase.regionserver.global.memstore.size.lower.limit.

    Default

    0.4

    hbase.regionserver.global.memstore.size.lower.limit
    Description

    Maximum size of all memstores in a region server before flushes are forced. Defaults to 95% of hbase.regionserver.global.memstore.size. A 100% value for this value causes the minimum possible flushing to occur when updates are blocked due to memstore limiting.

    Default

    0.95

    hbase.regionserver.optionalcacheflushinterval
    Description

    Maximum amount of time an edit lives in memory before being automatically flushed. Default 1 hour. Set it to 0 to disable automatic flushing.

    Default

    3600000

    hbase.regionserver.catalog.timeout
    Description

    Timeout value for the Catalog Janitor from the regionserver to META.

    Default

    600000

    hbase.regionserver.dns.interface
    Description

    The name of the Network Interface from which a region server should report its IP address.

    Default

    default

    hbase.regionserver.dns.nameserver
    Description

    The host name or IP address of the name server (DNS) which a region server should use to determine the host name used by the master for communication and display purposes.

    Default

    default

    hbase.regionserver.region.split.policy
    Description

    A split policy determines when a region should be split. The various other split policies that are available currently are ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy, DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.

    Default

    org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy

    hbase.regionserver.regionSplitLimit
    Description

    Limit for the number of regions after which no more region splitting should take place. This is not hard limit for the number of regions but acts as a guideline for the regionserver to stop splitting after a certain limit. Default is set to 1000.

    Default

    1000

    zookeeper.session.timeout
    Description

    ZooKeeper session timeout in milliseconds. It is used in two different ways. First, this value is used in the ZK client that HBase uses to connect to the ensemble. It is also used by HBase when it starts a ZK server and it is passed as the ‘maxSessionTimeout’. See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions. For example, if a HBase region server connects to a ZK ensemble that’s also managed by HBase, then the session timeout will be the one specified by this configuration. But, a region server that connects to an ensemble managed with a different configuration will be subjected that ensemble’s maxSessionTimeout. So, even though HBase might propose using 90 seconds, the ensemble can have a max timeout lower than this and it will take precedence. The current default that ZK ships with is 40 seconds, which is lower than HBase’s.

    Default

    90000

    zookeeper.znode.parent
    Description

    Root ZNode for HBase in ZooKeeper. All of HBase’s ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase’s ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed.

    Default

    /hbase

    zookeeper.znode.rootserver
    Description

    Path to ZNode holding root region location. This is written by the master and read by clients and region servers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, this means the root location is stored at /hbase/root-region-server.

    Default

    root-region-server

    zookeeper.znode.acl.parent
    Description

    Root ZNode for access control lists.

    Default

    acl

    hbase.zookeeper.dns.interface
    Description

    The name of the Network Interface from which a ZooKeeper server should report its IP address.

    Default

    default

    hbase.zookeeper.dns.nameserver
    Description

    The host name or IP address of the name server (DNS) which a ZooKeeper server should use to determine the host name used by the master for communication and display purposes.

    Default

    default

    hbase.zookeeper.peerport
    Description

    Port used by ZooKeeper peers to talk to each other. Seehttp://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper for more information.

    Default

    2888

    hbase.zookeeper.leaderport
    Description

    Port used by ZooKeeper for leader election. See http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper for more information.

    Default

    3888

    hbase.zookeeper.useMulti
    Description

    Instructs HBase to make use of ZooKeeper’s multi-update functionality. This allows certain ZooKeeper operations to complete more quickly and prevents some issues with rare Replication failure scenarios (see the release note of HBASE-2611 for an example). IMPORTANT: only set this to true if all ZooKeeper servers in the cluster are on version 3.4+ and will not be downgraded. ZooKeeper versions before 3.4 do not support multi-update and will not fail gracefully if multi-update is invoked (see ZOOKEEPER-1495).

    Default

    true

    hbase.config.read.zookeeper.config
    Description

    Set to true to allow HBaseConfiguration to read the zoo.cfg file for ZooKeeper properties. Switching this to true is not recommended, since the functionality of reading ZK properties from a zoo.cfg file has been deprecated.

    Default

    false

    hbase.zookeeper.property.initLimit
    Description

    Property from ZooKeeper’s config zoo.cfg. The number of ticks that the initial synchronization phase can take.

    Default

    10

    hbase.zookeeper.property.syncLimit
    Description

    Property from ZooKeeper’s config zoo.cfg. The number of ticks that can pass between sending a request and getting an acknowledgment.

    Default

    5

    hbase.zookeeper.property.dataDir
    Description

    Property from ZooKeeper’s config zoo.cfg. The directory where the snapshot is stored.

    Default

    ${hbase.tmp.dir}/zookeeper

    hbase.zookeeper.property.clientPort
    Description

    Property from ZooKeeper’s config zoo.cfg. The port at which the clients will connect.

    Default

    2181

    hbase.zookeeper.property.maxClientCnxns
    Description

    Property from ZooKeeper’s config zoo.cfg. Limit on number of concurrent connections (at the socket level) that a single client, identified by IP address, may make to a single member of the ZooKeeper ensemble. Set high to avoid zk connection issues running standalone and pseudo-distributed.

    Default

    300

    hbase.client.write.buffer
    Description

    Default size of the HTable client write buffer in bytes. A bigger buffer takes more memory — on both the client and server side since server instantiates the passed write buffer to process it — but a larger buffer size reduces the number of RPCs made. For an estimate of server-side memory-used, evaluate hbase.client.write.buffer * hbase.regionserver.handler.count

    Default

    2097152

    hbase.client.pause
    Description

    General client pause value. Used mostly as value to wait before running a retry of a failed get, region lookup, etc. See hbase.client.retries.number for description of how we backoff from this initial pause amount and how this pause works w/ retries.

    Default

    100

    hbase.client.retries.number
    Description

    Maximum retries. Used as maximum for all retryable operations such as the getting of a cell’s value, starting a row update, etc. Retry interval is a rough function based on hbase.client.pause. At first we retry at this interval but then with backoff, we pretty quickly reach retrying every ten seconds. See HConstants#RETRY_BACKOFF for how the backup ramps up. Change this setting and hbase.client.pause to suit your workload.

    Default

    35

    hbase.client.max.total.tasks
    Description

    The maximum number of concurrent tasks a single HTable instance will send to the cluster.

    Default

    100

    hbase.client.max.perserver.tasks
    Description

    The maximum number of concurrent tasks a single HTable instance will send to a single region server.

    Default

    5

    hbase.client.max.perregion.tasks
    Description

    The maximum number of concurrent connections the client will maintain to a single Region. That is, if there is already hbase.client.max.perregion.tasks writes in progress for this region, new puts won’t be sent to this region until some writes finishes.

    Default

    1

    hbase.client.scanner.caching
    Description

    Number of rows that we try to fetch when calling next on a scanner if it is not served from (local, client) memory. This configuration works together with hbase.client.scanner.max.result.size to try and use the network efficiently. The default value is Integer.MAX_VALUE by default so that the network will fill the chunk size defined by hbase.client.scanner.max.result.size rather than be limited by a particular number of rows since the size of rows varies table to table. If you know ahead of time that you will not require more than a certain number of rows from a scan, this configuration should be set to that row limit via Scan#setCaching. Higher caching values will enable faster scanners but will eat up more memory and some calls of next may take longer and longer times when the cache is empty. Do not set this value such that the time between invocations is greater than the scanner timeout; i.e. hbase.client.scanner.timeout.period

    Default

    2147483647

    hbase.client.keyvalue.maxsize
    Description

    Specifies the combined maximum allowed size of a KeyValue instance. This is to set an upper boundary for a single entry saved in a storage file. Since they cannot be split it helps avoiding that a region cannot be split any further because the data is too large. It seems wise to set this to a fraction of the maximum region size. Setting it to zero or less disables the check.

    Default

    10485760

    hbase.client.scanner.timeout.period
    Description

    Client scanner lease period in milliseconds.

    Default

    60000

    hbase.client.localityCheck.threadPoolSize
    Default

    2

    hbase.bulkload.retries.number
    Description

    Maximum retries. This is maximum number of iterations to atomic bulk loads are attempted in the face of splitting operations 0 means never give up.

    Default

    10

    hbase.balancer.period
    Description

    Period at which the region balancer runs in the Master.

    Default

    300000

    hbase.regions.slop
    Description

    Rebalance if any regionserver has average + (average * slop) regions.

    Default

    0.2

    hbase.server.thread.wakefrequency
    Description

    Time to sleep in between searches for work (in milliseconds). Used as sleep interval by service threads such as log roller.

    Default

    10000

    hbase.server.versionfile.writeattempts
    Description

    How many time to retry attempting to write a version file before just aborting. Each attempt is seperated by the hbase.server.thread.wakefrequency milliseconds.

    Default

    3

    hbase.hregion.memstore.flush.size
    Description

    Memstore will be flushed to disk if size of the memstore exceeds this number of bytes. Value is checked by a thread that runs every hbase.server.thread.wakefrequency.

    Default

    134217728

    hbase.hregion.percolumnfamilyflush.size.lower.bound
    Description

    If FlushLargeStoresPolicy is used, then every time that we hit the total memstore limit, we find out all the column families whose memstores exceed this value, and only flush them, while retaining the others whose memstores are lower than this limit. If none of the families have their memstore size more than this, all the memstores will be flushed (just as usual). This value should be less than half of the total memstore threshold (hbase.hregion.memstore.flush.size).

    Default

    16777216

    hbase.hregion.preclose.flush.size
    Description

    If the memstores in a region are this size or larger when we go to close, run a “pre-flush” to clear out memstores before we put up the region closed flag and take the region offline. On close, a flush is run under the close flag to empty memory. During this time the region is offline and we are not taking on any writes. If the memstore content is large, this flush could take a long time to complete. The preflush is meant to clean out the bulk of the memstore before putting up the close flag and taking the region offline so the flush that runs under the close flag has little to do.

    Default

    5242880

    hbase.hregion.memstore.block.multiplier
    Description

    Block updates if memstore has hbase.hregion.memstore.block.multiplier times hbase.hregion.memstore.flush.size bytes. Useful preventing runaway memstore during spikes in update traffic. Without an upper-bound, memstore fills such that when it flushes the resultant flush files take a long time to compact or split, or worse, we OOME.

    Default

    4

    hbase.hregion.memstore.mslab.enabled
    Description

    Enables the MemStore-Local Allocation Buffer, a feature which works to prevent heap fragmentation under heavy write loads. This can reduce the frequency of stop-the-world GC pauses on large heaps.

    Default

    true

    hbase.hregion.max.filesize
    Description

    Maximum HFile size. If the sum of the sizes of a region’s HFiles has grown to exceed this value, the region is split in two.

    Default

    10737418240

    hbase.hregion.majorcompaction
    Description

    Time between major compactions, expressed in milliseconds. Set to 0 to disable time-based automatic major compactions. User-requested and size-based major compactions will still run. This value is multiplied by hbase.hregion.majorcompaction.jitter to cause compaction to start at a somewhat-random time during a given window of time. The default value is 7 days, expressed in milliseconds. If major compactions are causing disruption in your environment, you can configure them to run at off-peak times for your deployment, or disable time-based major compactions by setting this parameter to 0, and run major compactions in a cron job or by another external mechanism.

    Default

    604800000

    hbase.hregion.majorcompaction.jitter
    Description

    A multiplier applied to hbase.hregion.majorcompaction to cause compaction to occur a given amount of time either side of hbase.hregion.majorcompaction. The smaller the number, the closer the compactions will happen to the hbase.hregion.majorcompaction interval.

    Default

    0.50

    hbase.hstore.compactionThreshold
    Description

    If more than this number of StoreFiles exist in any one Store (one StoreFile is written per flush of MemStore), a compaction is run to rewrite all StoreFiles into a single StoreFile. Larger values delay compaction, but when compaction does occur, it takes longer to complete.

    Default

    3

    hbase.hstore.flusher.count
    Description

    The number of flush threads. With fewer threads, the MemStore flushes will be queued. With more threads, the flushes will be executed in parallel, increasing the load on HDFS, and potentially causing more compactions.

    Default

    2

    hbase.hstore.blockingStoreFiles
    Description

    If more than this number of StoreFiles exist in any one Store (one StoreFile is written per flush of MemStore), updates are blocked for this region until a compaction is completed, or until hbase.hstore.blockingWaitTime has been exceeded.

    Default

    10

    hbase.hstore.blockingWaitTime
    Description

    The time for which a region will block updates after reaching the StoreFile limit defined by hbase.hstore.blockingStoreFiles. After this time has elapsed, the region will stop blocking updates even if a compaction has not been completed.

    Default

    90000

    hbase.hstore.compaction.min
    Description

    The minimum number of StoreFiles which must be eligible for compaction before compaction can run. The goal of tuning hbase.hstore.compaction.min is to avoid ending up with too many tiny StoreFiles to compact. Setting this value to 2 would cause a minor compaction each time you have two StoreFiles in a Store, and this is probably not appropriate. If you set this value too high, all the other values will need to be adjusted accordingly. For most cases, the default value is appropriate. In previous versions of HBase, the parameter hbase.hstore.compaction.min was named hbase.hstore.compactionThreshold.

    Default

    3

    hbase.hstore.compaction.max
    Description

    The maximum number of StoreFiles which will be selected for a single minor compaction, regardless of the number of eligible StoreFiles. Effectively, the value of hbase.hstore.compaction.max controls the length of time it takes a single compaction to complete. Setting it larger means that more StoreFiles are included in a compaction. For most cases, the default value is appropriate.

    Default

    10

    hbase.hstore.compaction.min.size
    Description

    A StoreFile smaller than this size will always be eligible for minor compaction. HFiles this size or larger are evaluated by hbase.hstore.compaction.ratio to determine if they are eligible. Because this limit represents the “automatic include”limit for all StoreFiles smaller than this value, this value may need to be reduced in write-heavy environments where many StoreFiles in the 1-2 MB range are being flushed, because every StoreFile will be targeted for compaction and the resulting StoreFiles may still be under the minimum size and require further compaction. If this parameter is lowered, the ratio check is triggered more quickly. This addressed some issues seen in earlier versions of HBase but changing this parameter is no longer necessary in most situations. Default: 128 MB expressed in bytes.

    Default

    134217728

    hbase.hstore.compaction.max.size
    Description

    A StoreFile larger than this size will be excluded from compaction. The effect of raising hbase.hstore.compaction.max.size is fewer, larger StoreFiles that do not get compacted often. If you feel that compaction is happening too often without much benefit, you can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes.

    Default

    9223372036854775807

    hbase.hstore.compaction.ratio
    Description

    For minor compaction, this ratio is used to determine whether a given StoreFile which is larger than hbase.hstore.compaction.min.size is eligible for compaction. Its effect is to limit compaction of large StoreFiles. The value of hbase.hstore.compaction.ratio is expressed as a floating-point decimal. A large ratio, such as 10, will produce a single giant StoreFile. Conversely, a low value, such as .25, will produce behavior similar to the BigTable compaction algorithm, producing four StoreFiles. A moderate value of between 1.0 and 1.4 is recommended. When tuning this value, you are balancing write costs with read costs. Raising the value (to something like 1.4) will have more write costs, because you will compact larger StoreFiles. However, during reads, HBase will need to seek through fewer StoreFiles to accomplish the read. Consider this approach if you cannot take advantage of Bloom filters. Otherwise, you can lower this value to something like 1.0 to reduce the background cost of writes, and use Bloom filters to control the number of StoreFiles touched during reads. For most cases, the default value is appropriate.

    Default

    1.2F

    hbase.hstore.compaction.ratio.offpeak
    Description

    Allows you to set a different (by default, more aggressive) ratio for determining whether larger StoreFiles are included in compactions during off-peak hours. Works in the same way as hbase.hstore.compaction.ratio. Only applies if hbase.offpeak.start.hour and hbase.offpeak.end.hour are also enabled.

    Default

    5.0F

    hbase.hstore.time.to.purge.deletes
    Description

    The amount of time to delay purging of delete markers with future timestamps. If unset, or set to 0, all delete markers, including those with future timestamps, are purged during the next major compaction. Otherwise, a delete marker is kept until the major compaction which occurs after the marker’s timestamp plus the value of this setting, in milliseconds.

    Default

    0

    hbase.offpeak.start.hour
    Description

    The start of off-peak hours, expressed as an integer between 0 and 23, inclusive. Set to -1 to disable off-peak.

    Default

    -1

    hbase.offpeak.end.hour
    Description

    The end of off-peak hours, expressed as an integer between 0 and 23, inclusive. Set to -1 to disable off-peak.

    Default

    -1

    hbase.regionserver.thread.compaction.throttle
    Description

    There are two different thread pools for compactions, one for large compactions and the other for small compactions. This helps to keep compaction of lean tables (such ashbase:meta) fast. If a compaction is larger than this threshold, it goes into the large compaction pool. In most cases, the default value is appropriate. Default: 2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size (which defaults to 128MB). The value field assumes that the value of hbase.hregion.memstore.flush.size is unchanged from the default.

    Default

    2684354560

    hbase.hstore.compaction.kv.max
    Description

    The maximum number of KeyValues to read and then write in a batch when flushing or compacting. Set this lower if you have big KeyValues and problems with Out Of Memory Exceptions Set this higher if you have wide, small rows.

    Default

    10

    hbase.storescanner.parallel.seek.enable
    Description

    Enables StoreFileScanner parallel-seeking in StoreScanner, a feature which can reduce response latency under special conditions.

    Default

    false

    hbase.storescanner.parallel.seek.threads
    Description

    The default thread pool size if parallel-seeking feature enabled.

    Default

    10

    hfile.block.cache.size
    Description

    Percentage of maximum heap (-Xmx setting) to allocate to block cache used by a StoreFile. Default of 0.4 means allocate 40%. Set to 0 to disable but it’s not recommended; you need at least enough cache to hold the storefile indices.

    Default

    0.4

    hfile.block.index.cacheonwrite
    Description

    This allows to put non-root multi-level index blocks into the block cache at the time the index is being written.

    Default

    false

    hfile.index.block.max.size
    Description

    When the size of a leaf-level, intermediate-level, or root-level index block in a multi-level block index grows to this size, the block is written out and a new block is started.

    Default

    131072

    hbase.bucketcache.ioengine
    Description

    Where to store the contents of the bucketcache. One of: onheap, offheap, or file. If a file, set it to file:PATH_TO_FILE. Seehttps://hbase.apache.org/apidocs/org/apache/hadoop/hbase/io/hfile/CacheConfig.html for more information.

    Default

    none

    hbase.bucketcache.combinedcache.enabled
    Description

    Whether or not the bucketcache is used in league with the LRU on-heap block cache. In this mode, indices and blooms are kept in the LRU blockcache and the data blocks are kept in the bucketcache.

    Default

    true

    hbase.bucketcache.size
    Description

    The size of the buckets for the bucketcache if you only use a single size. Defaults to the default blocksize, which is 64 * 1024.

    Default

    65536

    hbase.bucketcache.sizes
    Description

    A comma-separated list of sizes for buckets for the bucketcache if you use multiple sizes. Should be a list of block sizes in order from smallest to largest. The sizes you use will depend on your data access patterns.

    Default

    none

    hfile.format.version
    Description

    The HFile format version to use for new files. Version 3 adds support for tags in hfiles (Seehttp://hbase.apache.org/book.html#hbase.tags). Distributed Log Replay requires that tags are enabled. Also see the configuration ‘hbase.replication.rpc.codec’.

    Default

    3

    hfile.block.bloom.cacheonwrite
    Description

    Enables cache-on-write for inline blocks of a compound Bloom filter.

    Default

    false

    io.storefile.bloom.block.size
    Description

    The size in bytes of a single block (“chunk”) of a compound Bloom filter. This size is approximate, because Bloom blocks can only be inserted at data block boundaries, and the number of keys per data block varies.

    Default

    131072

    hbase.rs.cacheblocksonwrite
    Description

    Whether an HFile block should be added to the block cache when the block is finished.

    Default

    false

    hbase.rpc.timeout
    Description

    This is for the RPC layer to define how long HBase client applications take for a remote call to time out. It uses pings to check connections but will eventually throw a TimeoutException.

    Default

    60000

    hbase.rpc.shortoperation.timeout
    Description

    This is another version of “hbase.rpc.timeout”. For those RPC operation within cluster, we rely on this configuration to set a short timeout limitation for short operation. For example, short rpc timeout for region server’s trying to report to active master can benefit quicker master failover process.

    Default

    10000

    hbase.ipc.client.tcpnodelay
    Description

    Set no delay on rpc socket connections. See http://docs.oracle.com/javase/1.5.0/docs/api/java/net/Socket.html#getTcpNoDelay()

    Default

    true

    hbase.master.keytab.file
    Description

    Full path to the kerberos keytab file to use for logging in the configured HMaster server principal.

    Default

    none

    hbase.master.kerberos.principal
    Description

    Ex. “[email protected]". The kerberos principal name that should be used to run the HMaster process. The principal name should be in the form: [email protected] If "_HOST" is used as the hostname portion, it will be replaced with the actual hostname of the running instance.

    Default

    none

    hbase.regionserver.keytab.file
    Description

    Full path to the kerberos keytab file to use for logging in the configured HRegionServer server principal.

    Default

    none

    hbase.regionserver.kerberos.principal
    Description

    Ex. "[email protected]". The kerberos principal name that should be used to run the HRegionServer process. The principal name should be in the form: [email protected] If "_HOST" is used as the hostname portion, it will be replaced with the actual hostname of the running instance. An entry for this principal must exist in the file specified in hbase.regionserver.keytab.file

    Default

    none

    hadoop.policy.file
    Description

    The policy configuration file used by RPC servers to make authorization decisions on client requests. Only used when HBase security is enabled.

    Default

    hbase-policy.xml

    hbase.superuser
    Description

    List of users or groups (comma-separated), who are allowed full privileges, regardless of stored ACLs, across the cluster. Only used when HBase security is enabled.

    Default

    none

    hbase.auth.key.update.interval
    Description

    The update interval for master key for authentication tokens in servers in milliseconds. Only used when HBase security is enabled.

    Default

    86400000

    hbase.auth.token.max.lifetime
    Description

    The maximum lifetime in milliseconds after which an authentication token expires. Only used when HBase security is enabled.

    Default

    604800000

    hbase.ipc.client.fallback-to-simple-auth-allowed
    Description

    When a client is configured to attempt a secure connection, but attempts to connect to an insecure server, that server may instruct the client to switch to SASL SIMPLE (unsecure) authentication. This setting controls whether or not the client will accept this instruction from the server. When false (the default), the client will not allow the fallback to SIMPLE authentication, and will abort the connection.

    Default

    false

    hbase.display.keys
    Description

    When this is set to true the webUI and such will display all start/end keys as part of the table details, region names, etc. When this is set to false, the keys are hidden.

    Default

    true

    hbase.coprocessor.enabled
    Description

    Enables or disables coprocessor loading. If 'false' (disabled), any other coprocessor related configuration will be ignored.

    Default

    true

    hbase.coprocessor.user.enabled
    Description

    Enables or disables user (aka. table) coprocessor loading. If 'false' (disabled), any table coprocessor attributes in table descriptors will be ignored. If "hbase.coprocessor.enabled" is 'false' this setting has no effect.

    Default

    true

    hbase.coprocessor.region.classes
    Description

    A comma-separated list of Coprocessors that are loaded by default on all tables. For any override coprocessor method, these classes will be called in order. After implementing your own Coprocessor, just put it in HBase’s classpath and add the fully qualified class name here. A coprocessor can also be loaded on demand by setting HTableDescriptor.

    Default

    none

    hbase.rest.port
    Description

    The port for the HBase REST server.

    Default

    8080

    hbase.rest.readonly
    Description

    Defines the mode the REST server will be started in. Possible values are: false: All HTTP methods are permitted - GET/PUT/POST/DELETE. true: Only the GET method is permitted.

    Default

    false

    hbase.rest.threads.max
    Description

    The maximum number of threads of the REST server thread pool. Threads in the pool are reused to process REST requests. This controls the maximum number of requests processed concurrently. It may help to control the memory used by the REST server to avoid OOM issues. If the thread pool is full, incoming requests will be queued up and wait for some free threads.

    Default

    100

    hbase.rest.threads.min
    Description

    The minimum number of threads of the REST server thread pool. The thread pool always has at least these number of threads so the REST server is ready to serve incoming requests.

    Default

    2

    hbase.rest.support.proxyuser
    Description

    Enables running the REST server to support proxy-user mode.

    Default

    false

    hbase.defaults.for.version.skip
    Description

    Set to true to skip the 'hbase.defaults.for.version' check. Setting this to true can be useful in contexts other than the other side of a maven generation; i.e. running in an ide. You’ll want to set this boolean to true to avoid seeing the RuntimException complaint: "hbase-default.xml file seems to be for and old version of HBase (/${hbase.version}), this version is X.X.X-SNAPSHOT"

    Default

    false

    hbase.coprocessor.master.classes
    Description

    A comma-separated list of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase’s classpath and add the fully qualified class name here.

    Default

    none

    hbase.coprocessor.abortonerror
    Description

    Set to true to cause the hosting server (master or regionserver) to abort if a coprocessor fails to load, fails to initialize, or throws an unexpected Throwable object. Setting this to false will allow the server to continue execution but the system wide state of the coprocessor in question will become inconsistent as it will be properly executing in only a subset of servers, so this is most useful for debugging only.

    Default

    true

    hbase.online.schema.update.enable
    Description

    Set true to enable online schema changes.

    Default

    true

    hbase.table.lock.enable
    Description

    Set to true to enable locking the table in zookeeper for schema change operations. Table locking from master prevents concurrent schema modifications to corrupt table state.

    Default

    true

    hbase.table.max.rowsize
    Description

    Maximum size of single row in bytes (default is 1 Gb) for Get’ting or Scan’ning without in-row scan flag set. If row size exceeds this limit RowTooBigException is thrown to client.

    Default

    1073741824

    hbase.thrift.minWorkerThreads
    Description

    The "core size" of the thread pool. New threads are created on every connection until this many threads are created.

    Default

    16

    hbase.thrift.maxWorkerThreads
    Description

    The maximum size of the thread pool. When the pending request queue overflows, new threads are created until their number reaches this number. After that, the server starts dropping connections.

    Default

    1000

    hbase.thrift.maxQueuedRequests
    Description

    The maximum number of pending Thrift connections waiting in the queue. If there are no idle threads in the pool, the server queues requests. Only when the queue overflows, new threads are added, up to hbase.thrift.maxQueuedRequests threads.

    Default

    1000

    hbase.thrift.htablepool.size.max
    Description

    The upper bound for the table pool used in the Thrift gateways server. Since this is per table name, we assume a single table and so with 1000 default worker threads max this is set to a matching number. For other workloads this number can be adjusted as needed.

    Default

    1000

    hbase.regionserver.thrift.framed
    Description

    Use Thrift TFramedTransport on the server side. This is the recommended transport for thrift servers and requires a similar setting on the client side. Changing this to false will select the default transport, vulnerable to DoS when malformed requests are issued due to THRIFT-601.

    Default

    false

    hbase.regionserver.thrift.framed.max_frame_size_in_mb
    Description

    Default frame size when using framed transport, in MB

    Default

    2

    hbase.regionserver.thrift.compact
    Description

    Use Thrift TCompactProtocol binary serialization protocol.

    Default

    false

    hbase.data.umask.enable
    Description

    Enable, if true, that file permissions should be assigned to the files written by the regionserver

    Default

    false

    hbase.data.umask
    Description

    File permissions that should be used to write data files when hbase.data.umask.enable is true

    Default

    000

    hbase.metrics.showTableName
    Description

    Whether to include the prefix "tbl.tablename" in per-column family metrics. If true, for each metric M, per-cf metrics will be reported for tbl.T.cf.CF.M, if false, per-cf metrics will be aggregated by column-family across tables, and reported for cf.CF.M. In both cases, the aggregated metric M across tables and cfs will be reported.

    Default

    true

    hbase.metrics.exposeOperationTimes
    Description

    Whether to report metrics about time taken performing an operation on the region server. Get, Put, Delete, Increment, and Append can all have their times exposed through Hadoop metrics per CF and per region.

    Default

    true

    hbase.snapshot.enabled
    Description

    Set to true to allow snapshots to be taken / restored / cloned.

    Default

    true

    hbase.snapshot.restore.take.failsafe.snapshot
    Description

    Set to true to take a snapshot before the restore operation. The snapshot taken will be used in case of failure, to restore the previous state. At the end of the restore operation this snapshot will be deleted

    Default

    true

    hbase.snapshot.restore.failsafe.name
    Description

    Name of the failsafe snapshot taken by the restore operation. You can use the {snapshot.name}, {table.name} and {restore.timestamp} variables to create a name based on what you are restoring.

    Default

    hbase-failsafe-{snapshot.name}-{restore.timestamp}

    hbase.server.compactchecker.interval.multiplier
    Description

    The number that determines how often we scan to see if compaction is necessary. Normally, compactions are done after some events (such as memstore flush), but if region didn’t receive a lot of writes for some time, or due to different compaction policies, it may be necessary to check it periodically. The interval between checks is hbase.server.compactchecker.interval.multiplier multiplied by hbase.server.thread.wakefrequency.

    Default

    1000

    hbase.lease.recovery.timeout
    Description

    How long we wait on dfs lease recovery in total before giving up.

    Default

    900000

    hbase.lease.recovery.dfs.timeout
    Description

    How long between dfs recover lease invocations. Should be larger than the sum of the time it takes for the namenode to issue a block recovery command as part of datanode; dfs.heartbeat.interval and the time it takes for the primary datanode, performing block recovery to timeout on a dead datanode; usually dfs.client.socket-timeout. See the end of HBASE-8389 for more.

    Default

    64000

    hbase.column.max.version
    Description

    New column family descriptors will use this value as the default number of versions to keep.

    Default

    1

    hbase.dfs.client.read.shortcircuit.buffer.size
    Description

    If the DFSClient configuration dfs.client.read.shortcircuit.buffer.size is unset, we will use what is configured here as the short circuit read default direct byte buffer size. DFSClient native default is 1MB; HBase keeps its HDFS files open so number of file blocks * 1MB soon starts to add up and threaten OOME because of a shortage of direct memory. So, we set it down from the default. Make it > the default hbase block size set in the HColumnDescriptor which is usually 64k.

    Default

    131072

    hbase.regionserver.checksum.verify
    Description

    If set to true (the default), HBase verifies the checksums for hfile blocks. HBase writes checksums inline with the data when it writes out hfiles. HDFS (as of this writing) writes checksums to a separate file than the data file necessitating extra seeks. Setting this flag saves some on i/o. Checksum verification by HDFS will be internally disabled on hfile streams when this flag is set. If the hbase-checksum verification fails, we will switch back to using HDFS checksums (so do not disable HDFS checksums! And besides this feature applies to hfiles only, not to WALs). If this parameter is set to false, then hbase will not verify any checksums, instead it will depend on checksum verification being done in the HDFS client.

    Default

    true

    hbase.hstore.bytes.per.checksum
    Description

    Number of bytes in a newly created checksum chunk for HBase-level checksums in hfile blocks.

    Default

    16384

    hbase.hstore.checksum.algorithm
    Description

    Name of an algorithm that is used to compute checksums. Possible values are NULL, CRC32, CRC32C.

    Default

    CRC32

    hbase.client.scanner.max.result.size
    Description

    Maximum number of bytes returned when calling a scanner’s next method. Note that when a single row is larger than this limit the row is still returned completely. The default value is 2MB, which is good for 1ge networks. With faster and/or high latency networks this value should be increased.

    Default

    2097152

    hbase.status.published
    Description

    This setting activates the publication by the master of the status of the region server. When a region server dies and its recovery starts, the master will push this information to the client application, to let them cut the connection immediately instead of waiting for a timeout.

    Default

    false

    hbase.status.publisher.class
    Description

    Implementation of the status publication with a multicast message.

    Default

    org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher

    hbase.status.listener.class
    Description

    Implementation of the status listener with a multicast message.

    Default

    org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener

    hbase.status.multicast.address.ip
    Description

    Multicast address to use for the status publication by multicast.

    Default

    226.1.1.3

    hbase.status.multicast.address.port
    Description

    Multicast port to use for the status publication by multicast.

    Default

    16100

    hbase.dynamic.jars.dir
    Description

    The directory from which the custom filter/co-processor jars can be loaded dynamically by the region server without the need to restart. However, an already loaded filter/co-processor class would not be un-loaded. See HBASE-1936 for more details.

    Default

    ${hbase.rootdir}/lib

    hbase.security.authentication
    Description

    Controls whether or not secure authentication is enabled for HBase. Possible values are 'simple' (no authentication), and 'kerberos'.

    Default

    simple

    hbase.rest.filter.classes
    Description

    Servlet filters for REST service.

    Default

    org.apache.hadoop.hbase.rest.filter.GzipFilter

    hbase.master.loadbalancer.class
    Description

    Class used to execute the regions balancing when the period occurs. See the class comment for more on how it workshttp://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.html It replaces the DefaultLoadBalancer as the default (since renamed as the SimpleLoadBalancer).

    Default

    org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

    hbase.security.exec.permission.checks
    Description

    If this setting is enabled and ACL based access control is active (the AccessController coprocessor is installed either as a system coprocessor or on a table as a table coprocessor) then you must grant all relevant users EXEC privilege if they require the ability to execute coprocessor endpoint calls. EXEC privilege, like any other permission, can be granted globally to a user, or to a user on a per table or per namespace basis. For more information on coprocessor endpoints, see the coprocessor section of the HBase online manual. For more information on granting or revoking permissions using the AccessController, see the security section of the HBase online manual.

    Default

    false

    hbase.procedure.regionserver.classes
    Description

    A comma-separated list of org.apache.hadoop.hbase.procedure.RegionServerProcedureManager procedure managers that are loaded by default on the active HRegionServer process. The lifecycle methods (init/start/stop) will be called by the active HRegionServer process to perform the specific globally barriered procedure. After implementing your own RegionServerProcedureManager, just put it in HBase’s classpath and add the fully qualified class name here.

    Default

    none

    hbase.procedure.master.classes
    Description

    A comma-separated list of org.apache.hadoop.hbase.procedure.MasterProcedureManager procedure managers that are loaded by default on the active HMaster process. A procedure is identified by its signature and users can use the signature and an instant name to trigger an execution of a globally barriered procedure. After implementing your own MasterProcedureManager, just put it in HBase’s classpath and add the fully qualified class name here.

    Default

    none

    hbase.coordinated.state.manager.class
    Description

    Fully qualified name of class implementing coordinated state manager.

    Default

    org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager

    hbase.regionserver.storefile.refresh.period
    Description

    The period (in milliseconds) for refreshing the store files for the secondary regions. 0 means this feature is disabled. Secondary regions sees new files (from flushes and compactions) from primary once the secondary region refreshes the list of files in the region (there is no notification mechanism). But too frequent refreshes might cause extra Namenode pressure. If the files cannot be refreshed for longer than HFile TTL (hbase.master.hfilecleaner.ttl) the requests are rejected. Configuring HFile TTL to a larger value is also recommended with this setting.

    Default

    0

    hbase.region.replica.replication.enabled
    Description

    Whether asynchronous WAL replication to the secondary region replicas is enabled or not. If this is enabled, a replication peer named "region_replica_replication" will be created which will tail the logs and replicate the mutatations to region replicas for tables that have region replication > 1. If this is enabled once, disabling this replication also requires disabling the replication peer using shell or ReplicationAdmin java class. Replication to secondary region replicas works over standard inter-cluster replication. So replication, if disabled explicitly, also has to be enabled by setting "hbase.replication" to true for this feature to work.

    Default

    false

    hbase.http.filter.initializers
    Description

    A comma separated list of class names. Each class in the list must extend org.apache.hadoop.hbase.http.FilterInitializer. The corresponding Filter will be initialized. Then, the Filter will be applied to all user facing jsp and servlet web pages. The ordering of the list defines the ordering of the filters. The default StaticUserWebFilter add a user principal as defined by the hbase.http.staticuser.user property.

    Default

    org.apache.hadoop.hbase.http.lib.StaticUserWebFilter

    hbase.security.visibility.mutations.checkauths
    Description

    This property if enabled, will check whether the labels in the visibility expression are associated with the user issuing the mutation

    Default

    false

    hbase.http.max.threads
    Description

    The maximum number of threads that the HTTP Server will create in its ThreadPool.

    Default

    10

    hbase.replication.rpc.codec
    Description

    The codec that is to be used when replication is enabled so that the tags are also replicated. This is used along with HFileV3 which supports tags in them. If tags are not used or if the hfile version used is HFileV2 then KeyValueCodec can be used as the replication codec. Note that using KeyValueCodecWithTags for replication when there are no tags causes no harm.

    Default

    org.apache.hadoop.hbase.codec.KeyValueCodecWithTags

    hbase.http.staticuser.user
    Description

    The user name to filter as, on static web filters while rendering content. An example use is the HDFS web UI (user to be used for browsing files).

    Default

    dr.stack

    hbase.regionserver.handler.abort.on.error.percent
    Description

    The percent of region server RPC threads failed to abort RS. -1 Disable aborting; 0 Abort if even a single handler has died; 0.x Abort only when this percent of handlers have died; 1 Abort only all of the handers have died.

    Default

    0.5

  • HBase RegionServer详解大数据

    HBase RegionServer详解

    RegionServer组件介绍

    RegionServer是HBase集群运行在每个工作节点上的服务。它是整个HBase系统的关键所在,一方面它维护了Region的状态,提供了对于Region的管理和服务;另一方面,它与Master交互,参与Master的分布式协调管理。

    MemStoreFlusher

    MemStoreFlusher主要功能是将MemStore刷新到文件中,当满足一下条件时会出发MemStore执行flush操作,最小的flush单元是region:

    • 当一个MemStore的大小等于hbase.hregion.memstore.flush.size指定大小时,所有属于当前region的memstores都将写入到文件;
    • 当MemStore使用内存总量达到hbase.regionserver.global.memstore.upperLimit指定值时,将会有多个MemStores flush到文件中,MemStore flush 顺序是按照大小降序执行的,直到刷新到MemStore使用内存略小于hbase.regionserver.global.memstore.lowerLimit。
    • 当每一个region server WAL数量达到hbase.regionserver.max.logs指定的值时,不同的region将按照时间先后顺序flush memstores ,较早的将先被刷新,直到WAL数量低于hbase.regionserver.max.logs为止。

      MemStoreFlusher的主要成员变量:

      class MemStoreFlusher implements FlushRequester { 
      static final Log LOG = LogFactory.getLog(MemStoreFlusher.class); 
       
      // These two data members go together.  Any entry in the one must have 
      // a corresponding entry in the other. 
      private final BlockingQueue<FlushQueueEntry> flushQueue = 
      new DelayQueue<FlushQueueEntry>(); 
      private final Map<HRegion, FlushRegionEntry> regionsInQueue = 
      new HashMap<HRegion, FlushRegionEntry>(); 
      private AtomicBoolean wakeupPending = new AtomicBoolean(); 
       
      private final long threadWakeFrequency; 
      private final HRegionServer server; 
      private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(); 
      private final Object blockSignal = new Object(); 
       
      protected long globalMemStoreLimit; 
      protected float globalMemStoreLimitLowMarkPercent; 
      protected long globalMemStoreLimitLowMark; 
       
      private long blockingWaitTime; 
      private final Counter updatesBlockedMsHighWater = new Counter(); 
       
      private final FlushHandler[] flushHandlers; 
      private List<FlushRequestListener> flushRequestListeners = new ArrayList<FlushRequestListener>(1);
    • flushQueue :代表某一个region 的Flush请求,Flusher线程不断地从该队列中获取 请求信息,完成Region的Flush操作;
    • regionsInQueue :维护HRegion实例与请求FlushRegionEntry之间的对应关系;某一个FlushQueueEntry实例存在regionsInQueue 中也必然存在于flushQueue中 。
    • threadWakeFrequency:用于flushQueue执行poll操作时,最大等待时间,配置项为hbase.server.thread.wakefrequency,默认值10000ms。

      MemStoreFlusher相关配置项:

    HeapMemoryManager

    CompactSplitThread

    合并文件清理不需要的数据,控制Region的规模。在Store内的文件个数超过阈值时,触发Compact合并文件操作,一是清理被删除的数据,二是多余版本的清理。在Region内的Store文件大小超过阈值,会触发Region的Split操作,一个Region被切分成两个Region。这两个操作都是在CompactSplitThread的各自的线程池中被触发。

    ZooKeeperWatcher

    MasterAddressTracker

    捕获Master服务节点的变化。HBase使用多Master来解决Master单点故障的问题,主Master服务故障时,它与ZooKeeper的心跳延迟超过阈值,ZooKeeeper路径下的数据被清理,备Master上的ActiveMaserManager服务会竞争该Master路径,成为主Master。MasterAddresTracker是RS内部监听Master节点变化的追踪器。

    ClusterStatusTracker

    HBase集群状态追踪器。该选项可以标识当前集群的状态,及它的启动时间。该设置选项有利于集群中的各个工作节点(RS)统一执行启动和退出操作。

    SplitLogWorker

    基于Region的HLog文件切分器。在RS宕机之后,RS上的保存的HLog文件,需要按照Region进行切分。HMaster会把这些文件作为任务放置到Zookeeper的splitlog路径下,RS上SplitLogWorker会尝试获取任务,对获取到的HLog文件按照Region进行分组,处理的结果保存到相应Region的recovered.edits目录下。

    RegionServer启动过程分析

    1.HRegionServer main方法
    RegionServer是一个独立的服务,有一个mian方法在启动时被调用。mian方法内部调用HRegionServerCommandLine实现RegionServer的启动。
    具体代码如下:

      public static void main(String[] args) throws Exception { 
        VersionInfo.logVersion(); 
        Configuration conf = HBaseConfiguration.create(); 
        @SuppressWarnings("unchecked") 
        Class<? extends HRegionServer> regionServerClass = (Class<? extends HRegionServer>) conf 
            .getClass(HConstants.REGION_SERVER_IMPL, HRegionServer.class); 
     
        new HRegionServerCommandLine(regionServerClass).doMain(args); 
      }
    1. HRegionServer run方法

    2. HRegionServer preRegistrationInitialization方法
      此方法内部主要包括初始化zookeeper相关的服务以及RegionServer服务组件的初始化。

     private void preRegistrationInitialization(){ 
        try { 
          setupClusterConnection(); 
     
          // Health checker thread. 
          if (isHealthCheckerConfigured()) { 
            int sleepTime = this.conf.getInt(HConstants.HEALTH_CHORE_WAKE_FREQ, 
              HConstants.DEFAULT_THREAD_WAKE_FREQUENCY); 
            healthCheckChore = new HealthCheckChore(sleepTime, this, getConfiguration()); 
          } 
          this.pauseMonitor = new JvmPauseMonitor(conf); 
          pauseMonitor.start(); 
     
          initializeZooKeeper(); 
          if (!isStopped() && !isAborted()) { 
            initializeThreads(); 
          } 
        } catch (Throwable t) { 
          // Call stop if error or process will stick around for ever since server 
          // puts up non-daemon threads. 
          this.rpcServices.stop(); 
          abort("Initialization of RS failed.  Hence aborting RS.", t); 
        } 
      }

    initializeThreads主要初始化RegionServer服务组件,主要包括初始化compactSplitThread,cacheFlusher,compactionChecker,以及Leases等。具体代码如下:

      private void initializeThreads() throws IOException { 
        // Cache flushing thread. 
        this.cacheFlusher = new MemStoreFlusher(conf, this); 
     
        // Compaction thread 
        this.compactSplitThread = new CompactSplitThread(this); 
     
        // Background thread to check for compactions; needed if region has not gotten updates 
        // in a while. It will take care of not checking too frequently on store-by-store basis. 
        this.compactionChecker = new CompactionChecker(this, this.threadWakeFrequency, this); 
        this.periodicFlusher = new PeriodicMemstoreFlusher(this.threadWakeFrequency, this); 
        this.leases = new Leases(this.threadWakeFrequency); 
     
        // Create the thread to clean the moved regions list 
        movedRegionsCleaner = MovedRegionsCleaner.createAndStart(this); 
     
        if (this.nonceManager != null) { 
          // Create the chore that cleans up nonces. 
          nonceManagerChore = this.nonceManager.createCleanupChore(this); 
        } 
     
        // Setup RPC client for master communication 
        rpcClient = RpcClientFactory.createClient(conf, clusterId, new InetSocketAddress( 
            rpcServices.isa.getAddress(), 0)); 
     
        int storefileRefreshPeriod = conf.getInt( 
            StorefileRefresherChore.REGIONSERVER_STOREFILE_REFRESH_PERIOD 
          , StorefileRefresherChore.DEFAULT_REGIONSERVER_STOREFILE_REFRESH_PERIOD); 
        if (storefileRefreshPeriod > 0) { 
          this.storefileRefresher = new StorefileRefresherChore(storefileRefreshPeriod, this, this); 
        } 
        registerConfigurationObservers(); 
      }
  • ganglia安装教程(centos7)详解大数据

    ganglia 安装教程

    依赖软件

    1. http://nchc.dl.sourceforge.net/project/pcre/pcre/8.32/pcre-8.32.tar.gz
    • tar xvzf pcre-8.32.tar.gz
    • cd pcre-8.32
      -./configure –prefix=/usr/local
    • make && make install

    2.http://savannah.nongnu.org/download/confuse/confuse-2.7.tar.gz

    • tar xvzf confuse-2.7.tar.gz
    • cd confuse-2.7
    • CFLAGS=-fPIC ./configure –prefix=/usr/local –disable-nls
    • make CFLAGS=-fPIC
    • make CFLAGS=-fPIC install

    3.http://nchc.dl.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz

    • tar -xvzf expat-2.1.0.tar.gz
    • cd expat-2.1.0
    • ./configure –prefix=/usr/local
    • make && make install

    4.http://zlib.net/zlib-1.2.8.tar.gz

    • tar xvzf zlib-1.2.8.tar.gz
    • cd zlib-1.2.8
    • CFLAGS=-fPIC ./configure –prefix=/usr/local
    • make CFLAGS=-fPIC
    • make CFLAGS=-fPIC install

    5.ftp://xmlsoft.org/libxml2/libxml2-2.7.8.tar.gz

    • tar xvzf libxml2-2.7.8.tar.gz
    • cd libxml2-2.7.8
    • ./configure –prefix=/usr/local –with-zlib=/usr/local
    • make && make install

    6.http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.8.tar.gz

    • tar xvzf rrdtool-1.4.8.tar.gz
    • cd rrdtool-1.4.8
    • export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
    • ./configure –prefix=/usr/local
    • make && make install

    7.http://cznic.dl.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.6.0/ganglia-3.6.0.tar.gz

    • ./configure –enable-gexec
    • make && make install

    ganglia配置

    gmetad配置

    在ganglia安装目录执行如下操作

    • cp -a gmetad/gmetad.init /etc/init.d/gmetad
    • chkconfig –add gmetad
    • chkconfig –level 345 gmetad on
    • gmetad -t | tee /usr/local/etc/gmetad.conf
    • mkdir -p /var/lib/ganglia/rrds
    • chown nobody:nobody /var/lib/ganglia/rrds

    配置gmetad.conf

    1.修改gmetad.conf 中 data_source

    data_source “hadoop201” hadoop201

    gmond配置

    • cp -a gmond/gmond.init /etc/init.d/gmond
    • chkconfig –add gmond
    • chkconfig –level 345 gmond on
    • gmond -t | tee /usr/local/etc/gmond.conf

    配置gmond.conf

    修改 cluster中 name 为在gmetad.conf 中的data_source

     cluster {       
      name = "hadoop201" 
      owner = "nobody" 
      latlong = "unspecified" 
      url = "unspecified" 
    } 
    

    Ganglia-web环境部署

    • tar xvzf ganglia-web-3.6.2.tar.gz -C /var/www/html/
    • cd /var/www/html/
    • mv ganglia-web-3.5.12 ganglia
    • chmod -R 777 /var/www/html/ganglia
    • cd /var/www/html/ganglia
    • cp conf_default.php conf.php
    • vim conf.php

    conf.php中有些默认配置和以上设置不一样的需要进行修改:

    # Gmetad-webfrontend version. Used to check for updates. 
    # 
    $conf['gweb_root'] = "/var/www/html/ganglia"; 
    $conf['gweb_confdir'] = "/var/www/html/ganglia"; 
     
    include_once $conf['gweb_root'] . "/version.php"; 
     
    # 
    # 'readonly': No authentication is required.  All users may view all resources.  No edits are allowed. 
    #  'enabled': Guest users may view public clusters.  Login is required to make changes.   
    #             An administrator must configure an authentication scheme and ACL rules. 
    # 'disabled': Guest users may perform any actions, including edits.  No authentication is required. 
    $conf['auth_system'] = 'readonly'; 
     
    # 
    # The name of the directory in "./templates" which contains the 
    # templates that you want to use. Templates are like a skin for the 
    # site that can alter its look and feel. 
    # 
    $conf['template_name'] = "default"; 
     
    # 
    # If you installed gmetad in a directory other than the default 
    # make sure you change it here. 
    # 
     
    # Where gmetad stores the rrd archives. 
    $conf['gmetad_root'] = "/var/lib/ganglia"; 
    $conf['rrds'] = "${conf['gmetad_root']}/rrds"; 
     
    # Where Dwoo (PHP templating engine) store compiled templates 
    $conf['dwoo_compiled_dir'] = "${conf['gweb_confdir']}/dwoo/compiled"; 
    $conf['dwoo_cache_dir'] = "${conf['gweb_confdir']}/dwoo/cache"; 
     
    # Where to store web-based configuration 
    $conf['views_dir'] = $conf['gweb_confdir'] . '/conf'; 
    $conf['conf_dir'] = $conf['gweb_confdir'] . '/conf'; 
    

    问题及解决方式

    1. 编译rrdtool 错误
      /.libs/librrd.so: undefined reference to [email protected]_2.4.30'
      ./.libs/librrd.so: undefined reference [email protected]_2.4.30’
      ./.libs/librrd.so: undefined reference to [email protected]_2.4.30'
      ./.libs/librrd.so: undefined reference [email protected]_2.4.30’
      ./.libs/librrd.so: undefined reference to [email protected]_2.4.30'
      ./.libs/librrd.so: undefined reference [email protected]_2.4.30’
      ./.libs/librrd.so: undefined reference to [email protected]_2.4.30'
      ./.libs/librrd.so: undefined reference [email protected]_2.4.30’
      ./.libs/librrd.so: undefined reference to [email protected]_2.4.30'
      ./.libs/librrd.so: undefined reference [email protected]_2.6.17’
      ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ganglia编译必须依赖rrdtool,而rrdtool安装不同机器可能出现莫名的安装错误,此时如果机器可以连接网络可以使用 yum -y install rrdtool 方式安装,但此时安装后,编译ganglia时仍然无法找到rrdtool相应的库,此时可以采用如下方法,使ganglia编译安装成功:

    下载rrdtool-1.4.8.tar.gz ,解压后,将rrdtool-1.4.8/src/.libs 中相应库拷贝到如下目录:

    home/hadoop/为我本机rrdtool-1.4.8所在路径:

    cp /home/hadoop/rrdtool-1.4.8/src/.libs/.so /usr/local/lib/

    cp /home/hadoop/rrdtool-1.4.8/src/.libs/*.a /usr/local/lib/

    cp /home/hadoop/rrdtool-1.4.8/src/.libs/*.la /usr/local/lib/

    cp /home/hadoop/rrdtool-1.4.8/src/rrd*.h /usr/local/include

    效果展示

    这里写图片描述

  • ganglia安装教程详解大数据

    ganglia 安装教程

    依赖软件

    1. http://nchc.dl.sourceforge.net/project/pcre/pcre/8.32/pcre-8.32.tar.gz
    • tar xvzf pcre-8.32.tar.gz

    • cd pcre-8.32

    • ./configure –prefix=/usr/local

    • make && make install

    2.http://savannah.nongnu.org/download/confuse/confuse-2.7.tar.gz

    • tar xvzf confuse-2.7.tar.gz

    • cd confuse-2.7

    • CFLAGS=-fPIC ./configure –prefix=/usr/local –disable-nls

    • make CFLAGS=-fPIC

    • make CFLAGS=-fPIC install

    3.http://nchc.dl.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz

    • tar -xvzf expat-2.1.0.tar.gz

    • cd expat-2.1.0

    • ./configure –prefix=/usr/local

    • make && make install

    4.http://zlib.net/zlib-1.2.8.tar.gz

    • tar xvzf zlib-1.2.8.tar.gz

    • cd zlib-1.2.8

    • CFLAGS=-fPIC ./configure –prefix=/usr/local

    • make CFLAGS=-fPIC

    • make CFLAGS=-fPIC install

    5.ftp://xmlsoft.org/libxml2/libxml2-2.7.8.tar.gz

    • tar xvzf libxml2-2.7.8.tar.gz

    • cd libxml2-2.7.8

    • ./configure –prefix=/usr/local –with-zlib=/usr/local

    • make && make install

    6.http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.8.tar.gz

    • tar xvzf rrdtool-1.4.8.tar.gz

    • cd rrdtool-1.4.8

    • export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

    • ./configure –prefix=/usr/local

    • make && make install

    7.http://cznic.dl.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.6.0/ganglia-3.6.0.tar.gz

    • ./configure –enable-gexec –with-gmetad (gmetad节点)
    • ./configure –enable-gexec (gmond节点)

    • make && make install

    1. 如果安装过程中还有其他包缺失。可以执行如下命令,从本地源安装
    • yum -y install apr-devel apr-util check-devel cairo-devel pango-devel libxml2-devel rpmbuild glib2-devel dbus-devel freetype-devel fontconfig-devel gcc-c++ expat-devel python-devel libXrender-devel

    ganglia配置

    gmetad配置

    在ganglia安装目录执行如下操作

    • cp -a gmetad/gmetad.init /etc/init.d/gmetad

    • chkconfig –add gmetad

    • chkconfig –level 345 gmetad on

    • gmetad -t | tee /usr/local/etc/gmetad.conf

    • mkdir -p /var/lib/ganglia/rrds

    • chown nobody:nobody /var/lib/ganglia/rrds

    配置gmetad.conf

    1.修改gmetad.conf 中 data_source

    data_source “hadoop201” hadoop201

    gmond配置

    • cp -a gmond/gmond.init /etc/init.d/gmond

    • chkconfig –add gmond

    • chkconfig –level 345 gmond on

    • gmond -t | tee /usr/local/etc/gmond.conf

    配置gmond.conf

    修改 cluster中 name 为在gmetad.conf 中的data_source

     
     cluster {       
     
      name = "hadoop201" 
     
      owner = "nobody" 
     
      latlong = "unspecified" 
     
      url = "unspecified" 
     
    } 
     
    

    Ganglia-web环境部署

    • tar xvzf ganglia-web-3.6.2.tar.gz -C /var/www/html/

    • cd /var/www/html/

    • mv ganglia-web-3.5.12 ganglia

    • chmod -R 777 /var/www/html/ganglia

    • cd /var/www/html/ganglia

    • cp conf_default.php conf.php

    • vim conf.php

    conf.php中有些默认配置和以上设置不一样的需要进行修改:

     
    # Gmetad-webfrontend version. Used to check for updates. 
     
    # 
     
    $conf['gweb_root'] = "/var/www/html/ganglia"; 
     
    $conf['gweb_confdir'] = "/var/www/html/ganglia"; 
     
     
     
    include_once $conf['gweb_root'] . "/version.php"; 
     
     
     
    # 
     
    # 'readonly': No authentication is required.  All users may view all resources.  No edits are allowed. 
     
    #  'enabled': Guest users may view public clusters.  Login is required to make changes.   
     
    #             An administrator must configure an authentication scheme and ACL rules. 
     
    # 'disabled': Guest users may perform any actions, including edits.  No authentication is required. 
     
    $conf['auth_system'] = 'readonly'; 
     
     
     
    # 
     
    # The name of the directory in "./templates" which contains the 
     
    # templates that you want to use. Templates are like a skin for the 
     
    # site that can alter its look and feel. 
     
    # 
     
    $conf['template_name'] = "default"; 
     
     
     
    # 
     
    # If you installed gmetad in a directory other than the default 
     
    # make sure you change it here. 
     
    # 
     
     
     
    # Where gmetad stores the rrd archives. 
     
    $conf['gmetad_root'] = "/var/lib/ganglia"; 
     
    $conf['rrds'] = "${conf['gmetad_root']}/rrds"; 
     
     
     
    # Where Dwoo (PHP templating engine) store compiled templates 
     
    $conf['dwoo_compiled_dir'] = "${conf['gweb_confdir']}/dwoo/compiled"; 
     
    $conf['dwoo_cache_dir'] = "${conf['gweb_confdir']}/dwoo/cache"; 
     
     
     
    # Where to store web-based configuration 
     
    $conf['views_dir'] = $conf['gweb_confdir'] . '/conf'; 
     
    $conf['conf_dir'] = $conf['gweb_confdir'] . '/conf'; 
     
     
    

    问题及解决方式

    1. #### 编译rrdtool 错误

    /.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ./.libs/librrd.so: undefined reference to [email protected]_2.6.17’

    ./.libs/librrd.so: undefined reference to [email protected]_2.4.30’

    ganglia编译必须依赖rrdtool,而rrdtool安装不同机器可能出现莫名的安装错误,此时如果机器可以连接网络可以使用 yum -y install rrdtool 方式安装,但此时安装后,编译ganglia时仍然无法找到rrdtool相应的库,此时可以采用如下方法,使ganglia编译安装成功:

    下载rrdtool-1.4.8.tar.gz ,解压后,将rrdtool-1.4.8/src/.libs 中相应库拷贝到如下目录,拷贝后可以不用再编译将rrdtool:

    进入rrdtool-1.4.8所在目录执行如下拷贝操作:

    cp src/.libs/*.so* /usr/local/lib/

    cp src/.libs/*.a /usr/local/lib/

    cp src/.libs/*.la /usr/local/lib/

    cp src/rrd*.h /usr/local/include

    1. #### metad: error while loading shared libraries

    metad: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory

    解决方法

    执行如下命令:

    whereis libpcre.so.1
    显示如下:
    libpcre.so: /lib64/libpcre.so.0 /usr/local/lib/libpcre.so.1 /usr/local/lib/libpcre.so

    然后建立连接:
    ln -s /usr/local/lib/libpcre.so /lib64/libpcre.so.1

    效果展示