9、高可用部署 · Kubernetes部署和实战

一、部署master组件 1、基本环境搭建 ``` scp -r /opt/kubernetes root@192.168.254.202:/opt scp -r /opt/etcd/ssl root@192.168.254.202:/opt/etcd scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.254.202:/usr/lib/systemd/system scp /usr/bin/kubectl root@192.168.254.202:/usr/bin ``` 2、修改配置文件 ``` # cat /opt/kubernetes/cfg/kube-apiserver.conf KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://192.168.254.201:2379,https://192.168.254.202:2379,https://192.168.254.203:2379 \ --bind-address=192.168.254.202 \ --secure-port=6443 \ --advertise-address=192.168.254.202 \ ``` 3、启动 ``` systemctl start kube-apiserver systemctl start kube-controller-manager systemctl start kube-scheduler systemctl enable kube-apiserver systemctl enable kube-controller-manager systemctl enable kube-scheduler systemctl daemon-reload ``` ``` for i in $(ls /opt/kubernetes/bin/);do systemctl start $i;systemctl enable $i;done ``` 4、验证 4.1、查看 ``` ps -ef | grep kube ``` ![](https://img.kancloud.cn/5c/74/5c7402835a5396b9537b944c39ec5193_946x174.png) 4.2、执行命令 ``` kubectl get node ``` 二、部署nginx负载均衡 1、下载 http://nginx.org/packages/rhel/7/x86\_64/RPMS/ http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm 2、建立nginx的yum仓库 rpm -ivh nginx-release-centos-7-0.el7.ngx.noarch.rpm 3、下载并安装nginx yum install nginx 4、修改配置文件 ``` # vim /etc/nginx/nginx.conf …… stream { log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; access_log /var/log/nginx/k8s-access.log main; upstream k8s-apiserver { server 192.168.254.201:6443; server 192.168.254.202:6443; } server { listen 6443; proxy_pass k8s-apiserver; } } ``` 5、启动nginx服务 ``` systemctl start nginx ``` 或 ``` service nginx start ``` 6、开机启动 ``` systemctl enable nginx ``` 7、验证 ![](https://img.kancloud.cn/d5/f7/d5f7540f9cc04c679e939d79c03f120c_1572x579.png) 查看进程 ``` ps -ef | grep nginx ``` 三、 keepalived高可用 1、主节点 1.1、安装 yum install -y keepalived 1.2、修改配置文件 ``` # vi /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 100 # 优先级，备服务器设置 90 advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.254.200/24 } track_script { check_nginx } } # cat /etc/keepalived/check\_nginx.sh #!/bin/bash count=$(ps -ef |grep nginx |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then exit 1 else exit 0 fi ``` 1.3、执行 ``` chmod +x check\_nginx.sh systemctl start keepalived systemctl enable keepalived ``` 2、备用节点 2.1、安装 yum install -y keepalived 2.2、修改配置文件 ``` # cat /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_BACKUP } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 90 # 优先级，备服务器设置 90 advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.254.200/24 } track_script { check_nginx } } # cat /etc/keepalived/check_nginx.sh #!/bin/bash count=$(ps -ef |grep nginx |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then exit 1 else exit 0 fi ``` 3.2、执行 ``` chmod +x check\_nginx.sh systemctl start keepalived systemctl enable keepalived ``` 3、验证 3.1、查看ip地址 ``` ip addr ``` 3.2、查看进程 ``` ps -ef | grep keep* ``` 3.3、 tail -f /var/log/messages 3.4、停止nginx ``` systemctl stop nginx ``` 四、修改node链接vip 1、将Node连接VIP： ``` # cd /opt/kubernetes/cfg # grep 192 * bootstrap.kubeconfig: server: https://192.168.254.201:6443 kubelet.kubeconfig: server: https://192.168.254.2016443 kube-proxy.kubeconfig: server: https://192.168.254.201:6443 ``` 批量修改： ``` sed -i 's#192.168.254.201#192.168.254.200#g' * ``` 2.、执行 ``` systemctl restart kubelet systemctl restart kube-proxy ``` 3、验证 ![](https://img.kancloud.cn/1a/88/1a882a97ebe6949cad465d51caaa4aad_1026x164.png) 查看日志 ``` tail -f /var/log/nginx/k8s-access.log ``` 五、注意 [https://blog.csdn.net/u010801994/article/details/86691777](https://blog.csdn.net/u010801994/article/details/86691777) ``` kubectl get node ``` ![](https://img.kancloud.cn/02/be/02bef2655acaa1003f40b617ccd86648_837x105.png) ``` kubectl describe node k8s-node1 ``` ![](https://img.kancloud.cn/2b/1e/2b1e2ca22c8bae0a4c4d7998321cb6bf_1470x206.png) 由错误日志可知是节点中的Cgroup不支持子系统pids所致。于是用uname -r查看内核版本： [root@localhost ~]# uname -r 3.10.0-327.el7.x86_64 然后查看该内核所支持的CGROUP，发现的确不支持PIDS [root@localhost ~]# cat /boot/config-3.10.0-327.el7.x86_64 | grep CGROUP CONFIG_CGROUPS=y #CONFIG_CGROUP\_DEBUG is not set CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_HUGETLB=y CONFIG_CGROUP_PERF=y CONFIG_CGROUP_SCHED=y CONFIG_BLK_CGROUP=y #CONFIG_DEBUG_BLK_CGROUP is not set CONFIG_NETFILTER_XT_MATCH_CGROUP=m CONFIG_NET_CLS_CGROUP=y CONFIG_NETPRIO_CGROUP=m 然后在运行yum update -y后，使用yum list kernel命令查看当前安装的内核. [root@lbw-master ~]# yum list kernel Installed Packages kernel.x86_64 3.10.0-327.el7 @anaconda kernel.x86_64 3.10.0-862.3.2.el7 @updates kernel.x86_64 3.10.0-957.21.3.el7 @updates kernel.x86_64 3.10.0-957.27.2.el7 @updates 查看新版内核所支持的CGOURP [root@lbw-master ~]# cat /boot/config-3.10.0-957.27.2.el7.x86_64 | grep CGROUP CONFIG_CGROUPS=y #CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_PIDS=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_HUGETLB=y CONFIG_CGROUP_PERF=y CONFIG_CGROUP_SCHED=y CONFIG_BLK_CGROUP=y #CONFIG_DEBUG_BLK_CGROUP is not set CONFIG_NETFILTER_XT_MATCH_CGROUP=m CONFIG_NET_CLS_CGROUP=y CONFIG_NETPRIO_CGROUP=y 发现的确有PIDS支持。于是接下来就是想办法将内核进行升级了。用以下命令查看所有可用的内核 [root@lbw-master ~]# awk -F\\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-957.27.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-957.21.3.el7.x86_64) 7 (Core) 2 : CentOS Linux (3.10.0-862.3.2.el7.x86_64) 7 (Core) 3 : CentOS Linux (3.10.0-327.el7.x86_64) 7 (Core) 4 : CentOS Linux (0-rescue-c4da2e677e384e85b9fd9f27eb3a9f8a) 7 (Core) 用grub2-set-default命令设置默认启动内核。利用设为0表示使用上一个命令输出的第一个内核。 grub2-set-default 0 然后用grub2-mkconfig命令生成配置文件并应用在grub.config文件中。 grub2-mkconfig -o /boot/grub2/grub.cfg 执行完毕后，用reboot命令重启机器即可。