💎一站式轻松地调用各大LLM模型接口,支持GPT4、智谱、星火、月之暗面及文生图 广告
## 部署Node节点 **kubernetes Node节点包含如下组件:** * flanneld * docker * kubelet * kube-proxy #### 环境变量 ```shell $ source /usr/k8s/bin/env.sh $ export KUBE_APISERVER="https://${MASTER_URL}" #如果你没有安装`haproxy`的话,还是需要使用6443端口的哦 $ export NODE_IP=192.168.1.170 # 当前部署的节点 IP ``` #### 配置docker ```shell $ cat /usr/lib/systemd/system/docker.service # 用systemctl status docker 命令可查看unit 文件路径 [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker EnvironmentFile=-/run/flannel/docker ExecStart=/usr/bin/dockerd --log-level=info $DOCKER_NETWORK_OPTIONS ExecReload=/bin/kill -s HUP $MAINPID # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Uncomment TasksMax if your systemd version supports it. # Only systemd 226 and above support this version. #TasksMax=infinity TimeoutStartSec=0 # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process # restart the docker process if it exits prematurely Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target ``` **kube-node-61的docker.service文件如下** ```shell [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target rhel-push-plugin.socket registries.service Wants=docker-storage-setup.service Requires=docker-cleanup.timer [Service] Type=notify NotifyAccess=all EnvironmentFile=-/run/containers/registries.conf EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network Environment=GOTRACEBACK=crash Environment=DOCKER_HTTP_HOST_COMPAT=1 Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin ExecStart=/usr/bin/dockerd-current \ --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \ --default-runtime=docker-runc \ --exec-opt native.cgroupdriver=cgroupfs \ --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $ADD_REGISTRY \ $BLOCK_REGISTRY \ $INSECURE_REGISTRY\ $REGISTRIES ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity TimeoutStartSec=0 Restart=on-abnormal MountFlags=slave KillMode=process [Install] WantedBy=multi-user.target ``` #### 启动docker ```shell $ sudo systemctl daemon-reload $ sudo systemctl stop firewalld $ sudo systemctl disable firewalld $ sudo iptables -F && sudo iptables -X && sudo iptables -F -t nat && sudo iptables -X -t nat $ sudo systemctl enable docker $ sudo systemctl start docker ``` * 需要关闭 firewalld(centos7)/ufw(ubuntu16.04),否则可能会重复创建的 iptables 规则 * 最好清理旧的 iptables rules 和 chains 规则 * 执行命令:docker version,检查docker服务是否正常 #### 安装和配置kubelet kubelet 启动时向kube-apiserver 发送TLS bootstrapping 请求,需要先将bootstrap token 文件中的kubelet-bootstrap 用户赋予system:node-bootstrapper 角色,然后kubelet 才有权限创建认证请求(certificatesigningrequests): > kubelet就是运行在Node节点上的,所以这一步安装是在所有的Node节点上,如果你想把你的Master也当做Node节点的话,当然也可以在Master节点上安装的。 ```shell $ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap ``` * --user=kubelet-bootstrap 是文件 /etc/kubernetes/token.csv 中指定的用户名,同时也写入了文件 /etc/kubernetes/bootstrap.kubeconfig 另外1.8 版本中还需要为Node 请求创建一个RBAC 授权规则: ```shell $ kubectl create clusterrolebinding kubelet-nodes --clusterrole=system:node --group=system:nodes ``` 然后下载最新的kubelet 和kube-proxy 二进制文件(前面下载kubernetes 目录下面其实也有): ```shell $ wget https://dl.k8s.io/v1.8.2/kubernetes-server-linux-amd64.tar.gz $ tar -xzvf kubernetes-server-linux-amd64.tar.gz $ cd kubernetes $ tar -xzvf kubernetes-src.tar.gz $ sudo cp -r ./server/bin/{kube-proxy,kubelet} /usr/k8s/bin/ ``` **创建kubelet bootstapping kubeconfig文件** ```shell # 设置集群参数 $ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig # 设置客户端认证参数 $ kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=bootstrap.kubeconfig # 设置上下文参数 $ kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig # 设置默认上下文 $ kubectl config use-context default --kubeconfig=bootstrap.kubeconfig $ mv bootstrap.kubeconfig /etc/kubernetes/ ``` * --embed-certs 为 true 时表示将 certificate-authority 证书写入到生成的 bootstrap.kubeconfig 文件中; * 设置 kubelet 客户端认证参数时没有指定秘钥和证书,后续由 kube-apiserver 自动生成; **创建kubelet的systemd unit文件** ```shell $ sudo mkdir /var/lib/kubelet # 必须先创建工作目录 $ cat > kubelet.service <<EOF [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/usr/k8s/bin/kubelet \ --fail-swap-on=false \ --cgroup-driver=cgroupfs \ --address=192.168.10.61 \ --hostname-override=192.168.10.61 \ --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \ --require-kubeconfig \ --cert-dir=/etc/kubernetes/ssl \ --cluster-dns=10.254.0.2 \ --cluster-domain=cluster.local. \ --hairpin-mode promiscuous-bridge \ --allow-privileged=true \ --serialize-image-pulls=false \ --logtostderr=true \ --v=2 ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT ExecStartPost=/sbin/iptables -A INPUT -s 172.17.0.0/12 -p tcp --dport 4194 -j ACCEPT ExecStartPost=/sbin/iptables -A INPUT -s 192.168.1.0/16 -p tcp --dport 4194 -j ACCEPT ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` > 请仔细阅读下面的注意事项,不然可能会启动失败。 * --fail-swap-on参数,这个一定要注意,Kubernetes 1.8开始要求关闭系统的Swap,如果不关闭,默认配置下kubelet将无法启动,也可以通过kubelet的启动参数–fail-swap-on=false来避免该问题 * --cgroup-driver参数,kubelet 用来维护主机的的 cgroups 的,默认是cgroupfs,但是这个地方的值需要你根据docker 的配置来确定(docker info |grep cgroup) * -address 不能设置为 127.0.0.1,否则后续 Pods 访问 kubelet 的 API 接口时会失败,因为 Pods 访问的 127.0.0.1指向自己而不是 kubelet * 如果设置了 --hostname-override 选项,则 kube-proxy 也需要设置该选项,否则会出现找不到 Node 的情况 * --experimental-bootstrap-kubeconfig 指向 bootstrap kubeconfig 文件,kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求 * 管理员通过了 CSR 请求后,kubelet 自动在 --cert-dir 目录创建证书和私钥文件(kubelet-client.crt 和 kubelet-client.key),然后写入 --kubeconfig 文件(自动创建 --kubeconfig 指定的文件) * 建议在 --kubeconfig 配置文件中指定 kube-apiserver 地址,如果未指定 --api-servers 选项,则必须指定 --require-kubeconfig 选项后才从配置文件中读取 kue-apiserver 的地址,否则 kubelet 启动后将找不到 kube-apiserver (日志中提示未找到 API Server),kubectl get nodes 不会返回对应的 Node 信息 * --cluster-dns 指定 kubedns 的 Service IP(可以先分配,后续创建 kubedns 服务时指定该 IP),--cluster-domain 指定域名后缀,这两个参数同时指定后才会生效 #### 启动kubelet ```shell $ sudo cp kubelet.service /etc/systemd/system/kubelet.service $ sudo systemctl daemon-reload $ sudo systemctl enable kubelet $ sudo systemctl start kubelet $ systemctl status kubelet ``` **通过kubelet的TLS证书请求** kubelet 首次启动时向kube-apiserver 发送证书签名请求,必须通过后kubernetes 系统才会将该 Node 加入到集群。查看未授权的CSR 请求: ```shell $ kubectl get csr NAME AGE REQUESTOR CONDITION node-csr--k3G2G1EoM4h9w1FuJRjJjfbIPNxa551A8TZfW9dG-g 2m kubelet-bootstrap Pending $ kubectl get nodes No resources found. ``` 通过CSR请求 ```shell $ kubectl certificate approve node-csr--k3G2G1EoM4h9w1FuJRjJjfbIPNxa551A8TZfW9dG-g certificatesigningrequest "node-csr--k3G2G1EoM4h9w1FuJRjJjfbIPNxa551A8TZfW9dG-g" approved $ kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.1.170 Ready <none> 48s v1.8.1 ``` 自动生成了kubelet kubeconfig文件和公私钥 ```shell $ ls -l /etc/kubernetes/kubelet.kubeconfig -rw------- 1 root root 2280 Nov 7 10:26 /etc/kubernetes/kubelet.kubeconfig $ ls -l /etc/kubernetes/ssl/kubelet* -rw-r--r-- 1 root root 1046 Nov 7 10:26 /etc/kubernetes/ssl/kubelet-client.crt -rw------- 1 root root 227 Nov 7 10:22 /etc/kubernetes/ssl/kubelet-client.key -rw-r--r-- 1 root root 1115 Nov 7 10:16 /etc/kubernetes/ssl/kubelet.crt -rw------- 1 root root 1675 Nov 7 10:16 /etc/kubernetes/ssl/kubelet.key ``` #### 配置kube-proxy **创建kube-proxy证书签名请求:** ```shell $ cat > kube-proxy-csr.json <<EOF { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF ``` **生成kube-proxy客户端证书和私钥** ```shell $ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \ -ca-key=/etc/kubernetes/ssl/ca-key.pem \ -config=/etc/kubernetes/ssl/ca-config.json \ -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy $ ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem $ sudo mv kube-proxy*.pem /etc/kubernetes/ssl/ ``` #### 创建kube-proxy kubeconfig文件 ```shell # 设置集群参数 $ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig # 设置客户端认证参数 $ kubectl config set-credentials kube-proxy \ --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \ --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig # 设置上下文参数 $ kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig # 设置默认上下文 $ kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig $ mv kube-proxy.kubeconfig /etc/kubernetes/ ``` #### 创建kube-proxy的systemd unit文件 ```shell $ sudo mkdir -p /var/lib/kube-proxy # 必须先创建工作目录 $ cat > kube-proxy.service <<EOF [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/usr/k8s/bin/kube-proxy \\ --bind-address=${NODE_IP} \\ --hostname-override=${NODE_IP} \\ --cluster-cidr=${SERVICE_CIDR} \\ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\ --logtostderr=true \\ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ``` **kube-node-61机器的kube-proxy.service文件如下** ```shell [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/usr/k8s/bin/kube-proxy \ --bind-address=192.168.10.61 \ --hostname-override=192.168.10.61 \ --cluster-cidr=10.254.0.0/16 \ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \ --logtostderr=true \ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ``` #### 启动kube-proxy ```shell $ sudo cp kube-proxy.service /etc/systemd/system/ $ sudo systemctl daemon-reload $ sudo systemctl enable kube-proxy $ sudo systemctl start kube-proxy $ systemctl status kube-proxy ```