# centos7离线安装Ambari2.6+HDP2.6.4+ Hadoop集群 [TOC] # 第1章 概述 安装总共分为四个部分 1.准备工作,服务器的基本配置 2.安装Ambari 3.使用Ambari安装HDP 4.排错和调优 # 第2章 准备工作,服务器基本配置 ## 2.1 机器准备 hosts和主机名,这里选用三台主机,1个控制节点,2个数据节点,控制节点的内存给大点 ``` 192.168.56.210 master.vim26.com 192.168.56.220 data1.vim26.com 192.168.56.230 data2.vim26.com ``` ## 2.2 软件准备 因为默认的源都是在国外,而且很大,一共加起来差不多6个G左右,所以国内安装的话很容易超时中断导致安装失败,所以这里采用创建本地仓库的方式来安装Ambari以及HDP的CentOS7版本 ### 2.2.1 Ambari 2.6.1: ``` http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari-2.6.1.0-centos7.tar.gz ``` ### 2.2.2 HDP-2.6.4.0: ``` http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz ``` ### 2.2.3 HDP-UTILS-1.1.0.22: ``` http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz ``` ### 2.2.4 HDP-GPL-2.6.4.0: ``` http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos7-rpm.tar.gz ``` ### 2.2.5 其他操作系统版本的官方地址: ``` https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-installation/content/ambari_repositories.html ``` ### 2.2.6 mysql的repo源,非必须 这里如果不添加的话当安装HDP的时候会提示找不到mysql,当然你也可以手动安装mysql,然后配置帐号密码, ``` rpm -ivh http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql-community-release-el7-7.noarch.rpm ``` ### 2.2.7 jdbc驱动,非必须 后面安装服务的时候会有提示下载的地址: ``` https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.45.tar.gz ``` ### 2.2.8 java环境 可以自己安装,也可以使用默认安装脚本自动安装,不过脚本安装的时候是从国外的源下载,可能会很慢 ### 2.2.9 apache httpd 提供离线仓库的下载服务 yum install httpd -y ### 2.2.10 ntp时间服务 :所有节点都必须安装ntpd服务,不然安装的时候会提示警告 ``` yum install ntp -y ``` ## 2.3 环境设置 每个机器的hosts,hostname,关闭防火墙,selinux,ssh免密钥登录. ``` #hosts地址 192.168.56.210 master210.vim.com 192.168.56.220 data220.vim.com 192.168.56.230 data230.vim.com #关闭selinux sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config setenforce 0 #关闭防火墙 systemctl stop firewalld.service systemctl disable firewalld.service #生成密钥并免密钥认证 ssh-keygen ssh-copy-id -i .ssh/id_rsa.pub 192.168.56.240 ssh-copy-id -i .ssh/id_rsa.pub 192.168.56.230 ssh-copy-id -i .ssh/id_rsa.pub 192.168.56.220 ``` ## 2.4 所有节点安装ntp ``` yum install ntp -y ``` ## 2.5 主节点安装httpd ``` yum install httpd -y ``` ## 2.6 上传仓库软件 ``` #把下载下来的文件都解压到/var/www/html/ambari下,最终目录结构如下: [root@master210 ~]#mkdir /var/www/html/ambari/ [root@master210 ~]# tree -Ld 3 /var/www/html/ambari/ /var/www/html/ambari/ ├── ambari │ └── centos7 │ └── 2.6.1.0-143 ├── HDP │ └── centos7 │ └── 2.6.4.0-91 ├── HDP-GPL │ └── centos7 │ └── 2.6.4.0-91 └── HDP-UTILS-1.1.0.22 ├── openblas ├── repodata ├── RPM-GPG-KEY └── snappy ``` ## 2.7 修改repo源文件 这里可以从每个目录下的repo文件拷贝过来,或者直接新建文件然后复制以下内容: ambari.repo ``` [root@master210 ~]# cat /etc/yum.repos.d/ambari.repo #VERSION_NUMBER=2.6.1.0-143 [ambari-2.6.1.0] name=ambari Version - ambari-2.6.1.0 baseurl=http://192.168.56.210/ambari/ambari/centos7/2.6.1.0-143/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/ambari/centos7/2.6.1.0-143/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 ``` hdp.repo ``` [root@master210 ~]# cat /etc/yum.repos.d/hdp.repo #VERSION_NUMBER=2.6.4.0-91 [HDP-2.6.4.0] name=HDP Version - HDP-2.6.4.0 baseurl=http://192.168.56.210/ambari/HDP/centos7/2.6.4.0-91/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.22] name=HDP-UTILS Version - HDP-UTILS-1.1.0.22 baseurl=http://192.168.56.210/ambari/HDP-UTILS-1.1.0.22/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/HDP-UTILS-1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 ``` hdp.gpl.repo ``` [root@master210 ~]# cat /etc/yum.repos.d/hdp.gpl.repo #VERSION_NUMBER=2.6.4.0-91 [HDP-GPL-2.6.4.0] name=HDP-GPL Version - HDP-GPL-2.6.4.0 baseurl=http://192.168.56.210/ambari/HDP-GPL/centos7/2.6.4.0-91/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/HDP-GPL/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 ``` 如果使用mysql的话也需要mysql的repo,不然安装的时候会提示找不到,这里很奇怪,因为下载的离线源里并没有mysql,并且安装的时候会去mysql的官方源里去找,所以是个坑.下面有解决 ``` rpm -ivh http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql-community-release-el7-7.noarch.rpm ``` 清理缓存并生成源文件 ``` yum clean all yum makecache ``` # 第3章 使用自定义mysql数据库安装Ambari 可以使用默认配置安装的hadoop集群,也可以使用自己定义的环境,比如mysql和java ## 3.1 yum安装mysql和java环境 ``` [root@hadoop-node1 html]# yum install -y java-1.8.0 mariadb mariadb-server [root@hadoop-node1 html]# java -version openjdk version "1.8.0_161" OpenJDK Runtime Environment (build 1.8.0_161-b14) OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode) [root@hadoop-node1 html]# systemctl start mariadb [root@hadoop-node1 html]# systemctl enable mariadb Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service. [root@hadoop-node1 html]# systemctl status mariadb ● mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-01-29 17:44:54 CST; 23s ago Main PID: 1800 (mysqld_safe) CGroup: /system.slice/mariadb.service ├─1800 /bin/sh /usr/bin/mysqld_safe --basedir=/usr └─1962 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin- ................................. ``` ## 3.2 初始化mysql数据库 ``` [root@hadoop-node1 html]# mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] ... Success! By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB! ``` ## 3.3 创建Ambari帐号和hive服务帐号以及数据库 ``` [root@hadoop-node1 html]# mysql -u root -p123123 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 16 Server version: 5.5.56-MariaDB MariaDB Server Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database ambari default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on ambari.* to ambari@localhost identified by 'bigdata'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> grant all on ambari.* to ambari@'%' identified by 'bigdata'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database hive default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on hive.* to hive@localhost identified by 'hive'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> grant all on hive.* to hive@'%' identified by 'hive'; Query OK, 0 rows affected (0.00 sec) ``` ## 3.4 安装Ambari服务 ``` [root@hadoop-node1 ~]# yum install ambari-server.x86_64 ``` ## 3.5 配置Ambari服务 这里配置的过程中需要修改java环境的变量以及导入mysql文件,先把完整过程放出,修改的部分在后面 ``` [root@hadoop-node1 ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):ambari Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/ ###这里填写本机已经安装好的java软件目录 Validating JDK on Ambari Server...done. Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Configuring ambari database... WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties. ###这里提示需要配置java的环境变量到ambari.properties这个文件里,参考下面java配置修改内容 Press <enter> to continue. Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? ###这里提示需要导入SQL的DDL文件,操作参考下面mysql的部分 Extracting system views... ambari-admin-2.6.1.0.143.jar ........... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. ``` ### 3.5.1 java路径 默认yum安装到/usr/lib/jvm ,后面的版本号可能不同,注意修改成自己本机的安装位置 ``` /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/ ``` ### 3.5.2 配置mysql的jdbc ``` [root@hadoop-node1 hadoop_soft]# ls ambari.repo hdp.repo mysql-connector-java-5.1.45.tar.gz [root@hadoop-node1 hadoop_soft]# tar zxf mysql-connector-java-5.1.45.tar.gz [root@hadoop-node1 hadoop_soft]# cd mysql-connector-java-5.1.45/ [root@hadoop-node1 mysql-connector-java-5.1.45]# ls build.xml CHANGES COPYING mysql-connector-java-5.1.45-bin.jar README README.txt src [root@hadoop-node1 mysql-connector-java-5.1.45]# mv mysql-connector-java-5.1.45-bin.jar /usr/share/java/ [root@hadoop-node1 mysql-connector-java-5.1.45]# vim /etc/ambari-server/conf/ambari.properties ###在jdbc区下面找个位置添加如下如下语句 server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.45-bin.jar ``` ### 3.5.3 导入mysql的DDL文件 ``` [root@hadoop-node1 ~]# mysql -uroot -p123123 ambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql [root@hadoop-node1 ~]# ``` ## 3.6 启动Ambari服务 ``` [root@hadoop-node1 ~]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................................... Server started listening on 8080 DB configs consistency check: no errors and warnings were found. Ambari Server 'start' completed successfully. ###查看端口 [root@hadoop-node1 ~]# lsof -i:8080 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 2932 ambari 1454u IPv6 31635 0t0 TCP *:webcache (LISTEN) ``` # 第4章 使用mysql数据库配置Ambari Web安装 下面的配置和前面的默认配置大部分一样,但是选择数据库的时候有区别 ## 4.1 登录 帐号密码均为admin  ## 4.2 安装向导创建集群  ## 4.3 集群命名  ## 4.4 选择安装版本并使用离线仓库  ## 4.5 配置安装节点信息  ## 4.6 安装ambari客户端 这里报了个错,提示2个节点没有安装java,看来需要所有节点都安装java环境  安装完成后重新检测一下警告消除了,显示全部节点安装成功  ## 4.7 选择安装服务 只选择spark2之后点击next会自动帮我们把需要依赖的服务都安装上 ## 4.8 分配服务器角色 这里保持默认即可,或者将控制服务都安装在一个节点上  ## 4.9 分配slaves服务 这里不太明白,不过可以都勾上    ## 4.10 配置服务 ### 4.10.1 hive修改 这里会有三处需要修改的地方: 1,填写hive的密码 2,选择mysql 3,上传mysql-jdbc包并运行设置命令  ``` [root@hadoop-node1 java]# cd /usr/share/java [root@hadoop-node1 java]# ambari-server setup --jdbc-db=mysql --jdbc-driver=mysql-connector-java-5.1.45-bin.jar Using python /usr/bin/python Setup ambari-server Copying mysql-connector-java-5.1.45-bin.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.45-bin.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. ``` ### 4.10.2 Ambari Metrics配置 密码都设置为admin即可  ### 4.10.3 SmartSense修改 设置为admin即可  ## 4.11 查看和确认预配置  ## 4.12 安装组件   点击每个选项可以查看具体执行的过程内容输出  ## 4.13 安装完成  如果全部都是sucess那无疑是很开心的 ## 4.14 配置修改以及服务启动 这里是展示页面,还可以调整配置  配置服务参数的页面  设置服务启动页面  如果用不到的服务也可以删掉