Hadoop Pseudo-Distributed Environment Setup Guide (Linux)

1. Preparation

1.1 Configure Hostname and Hosts File

# 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
# ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.241.128 nodeQLy

1.2 SSH Key Configuration

[root@nodeQLy opt]# ssh-keygen -t rsa
[root@nodeQLy opt]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@nodeQLy
[root@nodeQLy opt]# ssh root@nodeQLy

1.3 Install Java Development Kit (JDK)

vim /etc/profile
export JAVA_HOME=/usr/local/java/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH

2. Hadoop Configuration

2.1 Hadoop Environment Setup

export HADOOP_HOME=/opt/hadoop-2.7.7
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

2.2 Core Configuration

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://nodeQLy:9000</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/hadoop-2.7.7/tmp</value>
</property>

2.3 HDFS Configuration

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>

2.4 MapReduce Configuration

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

2.5 YARN Configuration

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>nodeQLy</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

2.6 Format Namenode

[root@nodeQLy ~]# hdfs namenode -format

2.7 Start Hadoop Services

[root@nodeQLy ~]# start-dfs.sh
[root@nodeQLy ~]# start-yarn.sh
[root@nodeQLy ~]# jps
4081 DataNode
4853 Jps
4728 NodeManager
4409 ResourceManager
3947 NameNode
4252 SecondaryNameNode

2.8 Verify Hadoop Services

Access HDFS Web UI at: http://nodeQLy:50070/

Access YARN Web UI at: http://nodeQLy:8088/

3. HBase Configuration

3.1 HBase Environment Setup

export JAVA_HOME=/usr/local/java/jdk1.8.0_221
export HBASE_MANAGES_ZK=false

3.2 HBase Configuration File

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://nodeQLy:9000/hbase</value>
</property>

<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>

<property>
    <name>hbase.zookeeper.quorum</name>
    <value>nodeQLy:2181</value>
</property>

<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/opt/zookeeper-3.4.5/tmp</value>
</property>

3.3 Start Zookeeper

[root@nodeQLy bin]# ./zkServer.sh start
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

3.4 Start HBase

[root@nodeQLy bin]# ./start-hbase.sh
[root@nodeQLy bin]# jps
... HBase Processes ...

4. Hive Configuration

4.1 MySQL Setup for Hive

create database hive;
create user 'hiveowner'@'%' identified by 'Welcome_1';
grant all on hive.* TO 'hiveowner'@'%';
grant all on hive.* TO 'hiveowner'@'localhost' identified by 'Welcome_1';

4.2 Hive Configuration File

<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hiveowner</value>
</property>

<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>Welcome_1</value>
</property>

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://nodeQLy:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value>
</property>

<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
</property>

<property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
</property>

<property>
    <name>datanucleus.schema.autoCreateAll</name>
    <value>true</value>
</property>

<property>
    <name>hive.server2.thrift.bind.host</name>
    <value>nodeQLy</value>
</property>

4.3 Start Hive Server 2 and Connect with Beeline

[root@nodeQLy hive-2.3]# ./bin/hiveserver2 &
[root@nodeQLy hive-2.3]# ./bin/beeline
!connect jdbc:hive2://nodeQLy:10000/

Thẻ: hadoop hbase hive linux

Đăng vào ngày 10 tháng 7 lúc 14:42

Thẻ Phổ Biến

©2026 Thành phố Cuồng loạn