[TOC]
# 环境变量
**规划端口**
| - | 端口范围 |
| ------------------------------------ | ------------------------------------------------------------ |
| etcd数据库 | 2379、2380、2381； |
| k8s组件端口 | 6443、10257、10257、10250、10249、10256 |
| k8s插件端口 | Calico: 179、9099; |
| k8s NodePort端口 | 30000 - 32767 |
| ip_local_port_range | 32768 - 65535 |
下面对上面的各端口类型进行解释：
a) `etcd端口`：所需端口
b) `k8s组件端口`：基础组件
c) `k8s插件端口`：calico端口、nginx-ingress-controller端口
d) `k8s NodePort端口`：跑在容器里面的应用，可以通过这个范围内的端口向外暴露服务，所以应用的对外端口要在这个范围内
e) `ip_local_port_range`：主机上一个进程访问外部应用时，需要与外部应用建立TCP连接，TCP连接需要本机的一个端口，主机会从这个范围内选择一个没有使用的端口建立TCP连接；
**添加hosts文件记录**
```shell
cat >> /etc/hosts <<-EOF
{IP} {HOSTNAMW}
EOF
```
> 替换成实际的地址和域名
**关闭防火墙**
```shell
sudo systemctl disable firewalld --now
```
**关闭selinux**
```shell
#临时生效
sudo setenforce 0
sed -ri 's/(SELINUX=).*/\1disabled/g' /etc/selinux/config
```
**关闭交换分区**
```shell
#临时生效
swapoff -a
#永久生效，需要重启
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
```
**加载ipvs模块**
```shell
cat > /etc/sysconfig/modules/ipvs.modules <<-EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
modprobe -- br_netfilter
EOF
# 生效ipvs模块
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules
# 验证
lsmod | grep -e ip_vs -e nf_conntrack -e br_netfilter
```
注意：在 `/etc/sysconfig/modules/` 目录下的modules文件，重启会自动加载。
**安装ipset依赖包**
```shell
yum install ipvsadm conntrack-tools vim -y # 确保安装ipset包
```
**优化内核参数**
```shell
cat > /etc/sysctl.d/kubernetes.conf << EOF
# 二层的网桥在转发包时也会被iptables的FORWARD规则所过滤
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
# kernel转发功能
net.ipv4.ip_forward = 1
# 允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭
net.ipv4.tcp_tw_reuse = 1
# TCP连接中TIME-WAIT sockets的快速回收, 默认是0，表示关闭。对于位于NAT设备(容器转发)后面的Client来说，就是一场灾难
net.ipv4.tcp_tw_recycle = 0
# 允许系统打开的端口范围，即用于向外连接的端口范围
net.ipv4.ip_local_port_range = 32768 65535
# kernel中最多存在的TIME_WAIT数量, 默认是4096
net.ipv4.tcp_max_tw_buckets = 65535
# 控制系统是否开启对数据包源地址的校验(0 不校验)
net.ipv4.conf.all.rp_filter = 0
# 开启ipv6路由转发
net.ipv6.conf.all.forwarding = 1
# 开启ipv4路由转发
net.ipv4.conf.all.forwarding = 1
# FIN-WAIT-2状态保持时间
net.ipv4.tcp_fin_timeout = 15
EOF
# 生效 kubernetes.conf 文件
sysctl -p /etc/sysctl.d/kubernetes.conf
```
**设置时间同步**
```shell
# 安装chrony包
yum install -y chrony
# 注释原有的同步信息
sed -ri 's/(server .* iburst)/# \1/g' /etc/chrony.conf
# 添加ntp同步源
echo "server ntp.aliyun.com iburst" >> /etc/chrony.conf
# 重启chronyd服务
systemctl restart chronyd
# 验证服务
chronyc sources
```
# 安装docker
**创建docker安装目录及环境变量**
```shell
mkdir -p /etc/docker/ /data/docker
```
**下载docker二进制包**
```shell
curl -SL -o /usr/local/src/docker-20.10.24.tgz https://download.docker.com/linux/static/stable/x86_64/docker-20.10.24.tgz
```
**解压二进制包**
```shell
tar xf /usr/local/src/docker-20.10.24.tgz -C /opt
cp /opt/docker/* /usr/local/bin/
rm -rf /opt/docker
```
**创建docker 的systemd 模板**
```shell
cat > /usr/lib/systemd/system/docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/local/bin/dockerd --config-file /etc/docker/daemon.json
ExecReload=/bin/kill -s HUP
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
```
**创建 daemon.json 文件**
```shell
cat > /etc/docker/daemon.json << EOF
{
"data-root": "/data/docker/",
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://1nj0zren.mirror.aliyuncs.com",
"https://docker.mirrors.ustc.edu.cn",
"http://f1361db2.m.daocloud.io",
"https://registry.docker-cn.com"
],
"log-driver": "json-file",
"log-level": "info"
}
}
EOF
```
**启动docker**
```shell
systemctl daemon-reload
systemctl enable docker.service --now
```
**安装docker-compose**
```shell
curl -L https://github.com/docker/compose/releases/download/v2.23.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose version
```
# 安装etcd
**ca证书**
```shell
mkdir -p /etc/kubernetes/pki/etcd
cd /etc/kubernetes/pki/etcd && openssl genrsa -out ca.key 2048
cat <<-EOF | sudo tee ca-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = etcd-ca
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = etcd-ca
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment, Certificate Sign
basicConstraints=CA:TRUE
subjectKeyIdentifier=hash
subjectAltName=@alt_names
EOF
openssl req -x509 -new -nodes -key ca.key -days 36500 -out ca.crt -config ca-csr.conf -extensions v3_ext
rm -rf /etc/kubernetes/pki/etcd/{ca.srl,ca-csr.conf,ca.csr}
```
**服务证书**
etcd服务之间的证书
```shell
ETCD_IP01=192.168.31.95
ETCD_IP02=192.168.31.78
ETCD_IP03=192.168.31.253
cd /etc/kubernetes/pki/etcd && openssl genrsa -out server.key 2048
cat <<-EOF | sudo tee server-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = localhost
IP.1 = 127.0.0.1
IP.2 = $ETCD_IP01
IP.3 = $ETCD_IP02
IP.4 = $ETCD_IP03
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=serverAuth,clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
subjectAltName=@alt_names
EOF
openssl req -new -key server.key -out server.csr -config server-csr.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out server.crt -days 36500 \
-extensions v3_ext -extfile server-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/server.crt
rm -rf /etc/kubernetes/pki/etcd/{server-csr.conf,server.csr}
cd /etc/kubernetes/pki/etcd && openssl genrsa -out peer.key 2048
cat <<-EOF | sudo tee peer-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = localhost
IP.1 = 127.0.0.1
IP.2 = $ETCD_IP01
IP.3 = $ETCD_IP02
IP.4 = $ETCD_IP03
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=serverAuth,clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
subjectAltName=@alt_names
EOF
openssl req -new -key peer.key -out peer.csr -config peer-csr.conf
openssl x509 -req -in peer.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out peer.crt -days 36500 \
-extensions v3_ext -extfile peer-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/peer.crt
rm -rf /etc/kubernetes/pki/etcd/{peer-csr.conf,peer.csr}
```
**分发etcd相关证书到各个节点**
```shell
scp -r /etc/kubernetes/pki/etcd root@k8s-node01:/etc/kubernetes/pki/
scp -r /etc/kubernetes/pki/etcd root@k8s-node02:/etc/kubernetes/pki/
scp -r /etc/kubernetes/pki/etcd root@k8s-node03:/etc/kubernetes/pki/
```
**创建etcd数据目录**
```shell
mkdir -p /var/lib/etcd
chmod 700 /var/lib/etcd
```
**下载etcd包**
```shell
curl -SL -o /usr/local/src/etcd-v3.5.6-linux-amd64.tar.gz https://mirrors.huaweicloud.com/etcd/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz
tar xf /usr/local/src/etcd-v3.5.6-linux-amd64.tar.gz -C /opt
cp -r /opt/etcd-v3.5.6-linux-amd64/etcd* /usr/local/bin
rm -rf /opt/etcd-v3.5.6-linux-amd64
```
**分发etcd程序到各个etcd节点**
```shell
scp /usr/local/bin/etcd* root@k8s-node01:/usr/local/bin
scp /usr/local/bin/etcd* root@k8s-node02:/usr/local/bin
scp /usr/local/bin/etcd* root@k8s-node03:/usr/local/bin
```
**创建etcd 的systemd 模板**
```shell
netcar=`ip r | awk '/default via/ {print $5}'`
[ ! -z $netcar ] && LOCAL_IP=`ip r | awk -v netcar=$netcar '{if($3==netcar) print $9}'` || echo '$netcar is null'
ETCD_IP01=192.168.31.95
ETCD_IP02=192.168.31.78
ETCD_IP03=192.168.31.253
cat <<EOF | sudo tee /usr/lib/systemd/system/etcd.service > /dev/null
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://etcd.io/docs/v3.5/
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
--name=etcd01 \\
--data-dir=/var/lib/etcd \\
--snapshot-count=1000000 \\
--experimental-initial-corrupt-check=true \\
--experimental-watch-progress-notify-interval=5s \\
--advertise-client-urls=https://${LOCAL_IP}:2379 \\
--initial-advertise-peer-urls=https://${LOCAL_IP}:2380 \\
--initial-cluster=etcd01=https://${ETCD_IP01}:2380,etcd02=https://${ETCD_IP02}:2380,etcd03=https://${ETCD_IP03}:2380 \\
--initial-cluster-token=etcd-cluster \\
--initial-cluster-state=new \\
--client-cert-auth=true \\
--listen-client-urls=https://${LOCAL_IP}:2379 \\
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \\
--cert-file=/etc/kubernetes/pki/etcd/server.crt \\
--key-file=/etc/kubernetes/pki/etcd/server.key \\
--peer-client-cert-auth=true \\
--listen-peer-urls=https://${LOCAL_IP}:2380 \\
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \\
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \\
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
LimitNOFILE=65536
Restart=always
RestartSec=30
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
```
**分发etcd 的systemd 模板**
```shell
scp /usr/lib/systemd/system/etcd.service k8s-node01:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service k8s-node02:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service k8s-node03:/usr/lib/systemd/system/
```
**启动etcd**
```shell
systemctl daemon-reload
systemctl enable etcd.service --now
```
**验证etcd**
```shell
etcd_cert_dir=/etc/kubernetes/pki/etcd
ETCDCTL_API=3 etcdctl --cacert=${etcd_cert_dir}/ca.crt --cert=${etcd_cert_dir}/server.crt --key=${etcd_cert_dir}/server.key --endpoints="https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379" endpoint health -w table
```
说明：需要修改上面的 `IP地址`。
# 部署master节点
**下载kubernetes二进制包**
[kubernetes官方地址](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md)，需要上外网。
```shell
curl -SL -o /usr/local/src/kubernetes-server-linux-amd64.tar.gz https://dl.k8s.io/v1.18.18/kubernetes-server-linux-amd64.tar.gz
```
说明：亲测没有外网可以下载。但是可以会出现超时，或者连接错误。可以重试几次即可。
**解压kubernetes的安装包**
```shell
tar xf /usr/local/src/kubernetes-server-linux-amd64.tar.gz -C /opt/
```
## ca证书
kubernetes整个集群ca证书
```shell
cd /etc/kubernetes/pki && openssl genrsa -out ca.key 2048
cat <<-EOF | sudo tee ca-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = kubernetes-ca
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = kubernetes
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment, Certificate Sign
basicConstraints=CA:TRUE
subjectKeyIdentifier=hash
subjectAltName=@alt_names
EOF
openssl req -x509 -new -nodes -key ca.key -days 36500 -out ca.crt -config ca-csr.conf -extensions v3_ext
rm -rf /etc/kubernetes/pki/{ca.srl,ca-csr.conf}
```
kubernetes集群前端代理客户ca证书
```shell
cd /etc/kubernetes/pki && openssl genrsa -out front-proxy-ca.key 2048
cat <<-EOF | sudo tee front-proxy-ca-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = front-proxy-ca
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = front-proxy-ca
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment, Certificate Sign
basicConstraints=CA:TRUE
subjectKeyIdentifier=hash
subjectAltName=@alt_names
EOF
openssl req -x509 -new -nodes -key front-proxy-ca.key -days 36500 -out front-proxy-ca.crt -config front-proxy-ca-csr.conf -extensions v3_ext
rm -rf /etc/kubernetes/pki/{front-proxy-ca-csr.conf,front-proxy-ca.srl}
```
## 安装kube-apiserver
**创建目录**
```shell
mkdir -p /etc/kubernetes/{conf,manifests,pki}
mkdir -p /var/log/kubernetes/kube-apiserver
```
**拷贝命令**
```shell
cp /opt/kubernetes/server/bin/{kube-apiserver,kubectl} /usr/local/bin/
```
**生成服务证书**
apiserver服务使用的证书
```shell
# apiserver证书
SERVICE_IP=10.96.0.1 # default命名空间下的kubernetesservice IP地址
MASTER_VIP=192.168.31.100 # master节点的VIP地址
MASTER_IP01=192.168.31.103
MASTER_IP02=192.168.31.79
cd /etc/kubernetes/pki && openssl genrsa -out apiserver.key 2048
cat <<-EOF | sudo tee apiserver-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
CN = kube-apiserver
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = $SERVICE_IP
IP.2 = $MASTER_IP01
IP.3 = $MASTER_IP02
IP.5 = $MASTER_VIP
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=serverAuth,clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
subjectAltName=@alt_names
EOF
openssl req -new -key apiserver.key -out apiserver.csr -config apiserver-csr.conf
openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out apiserver.crt -days 36500 \
-extensions v3_ext -extfile apiserver-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{apiserver.csr,apiserver-csr.conf}
```
kube-apiserver与etcd通讯的证书
```shell
cd /etc/kubernetes/pki && openssl genrsa -out apiserver-etcd-client.key 2048
cat <<-EOF | sudo tee apiserver-etcd-client-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
O = system:masters
CN = kube-apiserver-etcd-client
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key apiserver-etcd-client.key -out apiserver-etcd-client.csr -config apiserver-etcd-client-csr.conf
openssl x509 -req -in apiserver-etcd-client.csr -CA etcd/ca.crt -CAkey etcd/ca.key \
-CAcreateserial -out apiserver-etcd-client.crt -days 36500 \
-extensions v3_ext -extfile apiserver-etcd-client-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/apiserver-etcd-client.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{apiserver-etcd-client-csr.conf,apiserver-etcd-client.csr}
```
kube-apiserver与kubelet通讯的证书
```shell
cd /etc/kubernetes/pki && openssl genrsa -out apiserver-kubelet-client.key 2048
cat <<-EOF | sudo tee apiserver-kubelet-client-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
O = system:masters
CN = kube-apiserver-kubelet-client
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -config apiserver-kubelet-client-csr.conf
openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out apiserver-kubelet-client.crt -days 36500 \
-extensions v3_ext -extfile apiserver-kubelet-client-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{apiserver-kubelet-client.csr,apiserver-kubelet-client-csr.conf}
```
前端代理客户的证书
```shell
cd /etc/kubernetes/pki && openssl genrsa -out front-proxy-client.key 2048
cat <<-EOF | sudo tee front-proxy-client-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
CN = front-proxy-client
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key front-proxy-client.key -out front-proxy-client.csr -config front-proxy-client-csr.conf
openssl x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key \
-CAcreateserial -out front-proxy-client.crt -days 36500 \
-extensions v3_ext -extfile front-proxy-client-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-client.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{front-proxy-client.csr,front-proxy-client-csr.conf}
```
serviceaccount证书
```shell
cd /etc/kubernetes/pki && openssl genrsa -out sa.key 2048
openssl rsa -in sa.key -pubout -out sa.pub
```
**创建审计策略配置文件**
```shell
cat > /etc/kubernetes/conf/kube-apiserver-audit.yml <<-EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# 所有资源都记录请求的元数据（请求的用户、时间戳、资源、动词等等）， 但是不记录请求或者响应的消息体。
- level: Metadata
EOF
```
**创建kube-apiserver的systemd模板**
```shell
cat <<'EOF' | sudo tee /usr/lib/systemd/system/kube-apiserver.service >> /dev/null
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--bind-address=0.0.0.0 \
--insecure-port=0 \
--secure-port=6443 \
--allow-privileged=true \
--service-cluster-ip-range=10.96.0.0/12 \
--service-node-port-range=30000-32767 \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--client-ca-file=/etc/kubernetes/pki/ca.crt \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--etcd-servers=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 \
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
--requestheader-allowed-names=front-proxy-client \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--alsologtostderr=true --logtostderr=false --v=4 \
--log-dir=/var/log/kubernetes/kube-apiserver \
--audit-log-path=/var/log/kubernetes/kube-apiserver/apiserver.audit \
--audit-policy-file=/etc/kubernetes/conf/kube-apiserver-audit.yml \
--audit-log-maxsize=100 --audit-log-maxage=7
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
```
> 说明：需要修改 `etcd-servers(etcd 服务地址)` 和 `service-cluster-ip-range(service IP段)` 。
**启动kube-apiserver**
```shell
systemctl daemon-reload
systemctl enable kube-apiserver.service --now
```
**验证**
```shell
curl -sk --cert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver.crt --key /etc/kubernetes/pki/apiserver.key https://localhost:6443/healthz && echo
```
## 安装kube-controller-manager
**创建日志目录**
```shell
mkdir /var/log/kubernetes/kube-controller-manager
```
**拷贝命令**
```shell
cp /opt/kubernetes/server/bin/kube-controller-manager /usr/local/bin/
```
**生成证书**
```shell
cd /etc/kubernetes/pki && openssl genrsa -out controller-manager.key 2048
cat <<-EOF | sudo tee controller-manager-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
CN = system:kube-controller-manager
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key controller-manager.key -out controller-manager.csr -config controller-manager-csr.conf
openssl x509 -req -in controller-manager.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out controller-manager.crt -days 36500 \
-extensions v3_ext -extfile controller-manager-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/controller-manager.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{controller-manager.csr,controller-manager-csr.conf}
```
**生成连接集群的kubeconfig文件**
```shell
KUBE_APISERVER="https://192.168.31.103:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=/etc/kubernetes/controller-manager.conf
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.crt \
--client-key=/etc/kubernetes/pki/controller-manager.key \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.conf
kubectl config set-context default \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.conf
kubectl config use-context default \
--kubeconfig=/etc/kubernetes/controller-manager.conf
```
**kube-controller-manager的systemd模板**
```shell
cat <<'EOF' | sudo tee /usr/lib/systemd/system/kube-controller-manager.service >> /dev/null
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--bind-address=0.0.0.0 \
--port=0 \
--secure-port=10257 \
--leader-elect=true \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--node-cidr-mask-size=24 \
--service-cluster-ip-range=10.96.0.0/12 \
--authentication-kubeconfig=/etc/kubernetes/controller-manager.conf \
--authorization-kubeconfig=/etc/kubernetes/controller-manager.conf \
--client-ca-file=/etc/kubernetes/pki/ca.crt \
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt \
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key \
--root-ca-file=/etc/kubernetes/pki/ca.crt \
--service-account-private-key-file=/etc/kubernetes/pki/sa.key \
--kubeconfig=/etc/kubernetes/controller-manager.conf \
--controllers=*,bootstrapsigner,tokencleaner \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--use-service-account-credentials=true \
--experimental-cluster-signing-duration=87600h0m0s \
--alsologtostderr=true --logtostderr=false --v=4 \
--log-dir=/var/log/kubernetes/kube-controller-manager
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
```
> 说明：需要修改 `service-cluster-ip-range(service IP段)` 、 `cluster-cidr(pod IP段)` 的值。
**启动kube-controller-manager**
```shell
systemctl daemon-reload
systemctl enable kube-controller-manager.service --now
```
**验证**
```shell
curl -sk --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/controller-manager.crt --key /etc/kubernetes/pki/controller-manager.key https://localhost:10257/healthz && echo
```
## 安装kube-scheduler
**拷贝命令**
```shell
cp /opt/kubernetes/server/bin/kube-scheduler /usr/local/bin/
```
**创建日志目录**
```shell
mkdir /var/log/kubernetes/kube-scheduler
```
**生成kube-scheduler证书**
```shell
cd /etc/kubernetes/pki && openssl genrsa -out scheduler.key 2048
cat <<-EOF | sudo tee scheduler-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
CN = system:kube-scheduler
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key scheduler.key -out scheduler.csr -config scheduler-csr.conf
openssl x509 -req -in scheduler.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out scheduler.crt -days 36500 \
-extensions v3_ext -extfile scheduler-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/scheduler.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{scheduler.csr,scheduler-csr.conf}
```
**生成连接集群的kubeconfig文件**
```shell
KUBE_APISERVER="https://192.168.31.103:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=/etc/kubernetes/scheduler.conf
kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.crt \
--client-key=/etc/kubernetes/pki/scheduler.key \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.conf
kubectl config set-context default \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.conf
kubectl config use-context default \
--kubeconfig=/etc/kubernetes/scheduler.conf
```
**创建kube-scheduler的systemd模板**
```shell
cat <<'EOF' | sudo tee /usr/lib/systemd/system/kube-scheduler.service >> /dev/null
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-scheduler \
--bind-address=0.0.0.0 \
--port=0 \
--secure-port=10259 \
--leader-elect=true \
--kubeconfig=/etc/kubernetes/scheduler.conf \
--authentication-kubeconfig=/etc/kubernetes/scheduler.conf \
--authorization-kubeconfig=/etc/kubernetes/scheduler.conf \
--alsologtostderr=true --logtostderr=false --v=4 \
--log-dir=/var/log/kubernetes/kube-scheduler
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
```
**启动kube-scheduler**
```shell
systemctl daemon-reload
systemctl enable kube-scheduler.service --now
```
**验证**
```shell
curl -sk --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/scheduler.crt --key /etc/kubernetes/pki/scheduler.key https://localhost:10259/healthz && echo
```
## 客户端设置及验证
**客户端设置**
```shell
cd /etc/kubernetes/pki && openssl genrsa -out admin.key 2048
cat <<-EOF | sudo tee admin-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
O = system:masters
CN = kubernetes-admin
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key admin.key -out admin.csr -config admin-csr.conf
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out admin.crt -days 36500 \
-extensions v3_ext -extfile admin-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/admin.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/admin{.csr,-csr.conf}
KUBE_APISERVER="https://192.168.31.103:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=/etc/kubernetes/admin.conf
kubectl config set-credentials system:admin \
--client-certificate=/etc/kubernetes/pki/admin.crt \
--client-key=/etc/kubernetes/pki/admin.key \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/admin.conf
kubectl config set-context default \
--cluster=kubernetes \
--user=system:admin \
--kubeconfig=/etc/kubernetes/admin.conf
kubectl config use-context default \
--kubeconfig=/etc/kubernetes/admin.conf
cp /etc/kubernetes/admin.conf ~/.kube/config
```
# 部署工作节点
## Bootstrap Tokens认证
1. 生成bootstrap token
```shell
token_id=`cat /dev/urandom | head -c 10 | md5sum | head -c 6`
token_secret=`cat /dev/urandom | head -c 10 | md5sum | head -c 16`
echo $token_id | grep [^0-9] > /dev/null || echo -e '\n\033[31m【警告】\033[0m$token_id 是纯数字，请重新生成token_id。'
cat <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: bootstrap-token-${token_id} # 格式：bootstrap-token-[TOKENID]
namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
token-id: ${token_id} # 格式：[a-z0-9]{6}
token-secret: ${token_secret} # 格式：[a-z0-9]{16}
expiration: `date -d '1 day' +%F`T`date +%T`+08:00 # 过期时间，当该bootstrap token过期则删除，默认有效期为一天
usage-bootstrap-authentication: "true" # 表示令牌可以作为持有者令牌用于 API 服务器的身份认证
usage-bootstrap-signing: "true" # 该令牌可用于如下所述地对 cluster-info 的 ConfigMap 进行签名
auth-extra-groups: system:bootstrappers:worker,system:bootstrappers:ingress
EOF
```
> **注意**：需要修改过期时间。
2. 授予bootstrap token创建CSR证书签名请求的权限,即授予kubelet创建CSR证书签名请求的权限
```shell
cat <<-EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: create-csrs-for-bootstrapping
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:node-bootstrapper
apiGroup: rbac.authorization.k8s.io
EOF
```
3. 授予bootstrap token权限，让kube-controller-manager可以自动审批其发起的CSR
```shell
cat <<-EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.io
EOF
```
4. 授予kubelet权限，让kube-controller-manager自动批复kubelet的证书轮换请求
```shell
cat <<-EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
EOF
```
## 安装kubelet
**创建日志目录**
```shell
mkdir /var/log/kubernetes/kubelet
mkdir /var/lib/kubelet
```
**拷贝命令**
```shell
cp /opt/kubernetes/server/bin/kubelet /usr/local/bin/
```
**生成连接集群的kubeconfig文件**
```shell
KUBE_APISERVER="https://192.168.31.103:6443"
if [ `kubectl -n kube-system get secret --field-selector type=bootstrap.kubernetes.io/token -o name | wc -l` -ge 1 ];then
token_id=`kubectl -n kube-system get secret --field-selector type=bootstrap.kubernetes.io/token -ojsonpath='{.items[0].data.token-id}' | base64 -d`
token_secret=`kubectl -n kube-system get secret --field-selector type=bootstrap.kubernetes.io/token -ojsonpath='{.items[0].data.token-secret}' | base64 -d`
TOKEN="${token_id}.${token_secret}"
echo ${TOKEN}
else
echo -e "\n\033[31m【警告】\033[0m没有Bootstrap Tokens认证，请重新生成bootstrap token..."
fi
kubectl config set-cluster bootstrap \
--server=$KUBE_APISERVER \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
kubectl config set-credentials kubelet-bootstrap \
--token=$TOKEN \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
kubectl config set-context bootstrap --user=kubelet-bootstrap \
--cluster=bootstrap --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
kubectl config use-context bootstrap \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
```
**创建kubelet启动参数**
```shell
netcar=`ip r | awk '/default via/ {print $5}'`
[ ! -z $netcar ] && ipaddr=`ip r | awk -v netcar=$netcar '{if($3==netcar) print $9}'` || echo '$netcar is null'
cat > /etc/kubernetes/conf/kubelet.conf <<EOF
KUBELET_KUBECONFIG_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
KUBELET_CONFIG_ARGS="--config=/var/lib/kubelet/config.yaml"
KUBELET_NETWORK_ARGS="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
KUBELET_DATA_ARGS="--root-dir=/var/lib/kubelet --cert-dir=/var/lib/kubelet/pki --rotate-certificates"
KUBELET_LOG_ARGS="--alsologtostderr=true --logtostderr=false --v=4 --log-dir=/var/log/kubernetes/kubelet"
KUBELET_EXTRA_ARGS="--hostname-override=$ipaddr --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2"
EOF
```
> 说明：修改 `hostname-override` 为当前的 IP地址 。 `cni-conf-dir` 默认是 /etc/cni/net.d，`cni-bin-dir` 默认是/opt/cni/bin。`root-dir` 默认是/var/lib/kubelet目录
**创建kubelet配置参数文件**
```shell
cat > /var/lib/kubelet/config.yaml <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 0
cgroupDriver: systemd
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
anthorization:
mode: Webhook
Webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
clusterDomain: cluster.local
healthzBindAddress: 127.0.0.1
healthzPort: 10248
rotateCertificates: true
staticPodPath: /etc/kubernetes/manifests
maxOpenFiles: 1000000
maxPods: 100
clusterDNS:
- 10.96.0.10
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
EOF
```
> 说明：需要修改 `clusterDNS` 的IP地址为 `server IP段` 。
> 参考地址： https://github.com/kubernetes/kubelet
> https://kubernetes.io/zh/docs/reference/config-api/kubelet-config.v1beta1/
> https://pkg.go.dev/k8s.io/kubelet/config/v1beta1#KubeletConfiguration
**创建kubelet的systemd模板**
```shell
cat > /usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/etc/kubernetes/conf/kubelet.conf
ExecStart=/usr/local/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_NETWORK_ARGS \$KUBELET_DATA_ARGS \$KUBELET_LOG_ARGS \$KUBELET_EXTRA_ARGS
Restart=on-failure
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
EOF
```
**启动kubelet**
```shell
systemctl daemon-reload
systemctl enable kubelet.service --now
```
**验证**
```shell
curl http://localhost:10248/healthz && echo
kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.31.103 NotReady <none> 27s v1.18.18
# 将节点标记为master节点
kubectl label node 192.168.31.103 node-role.kubernetes.io/master=""
# 将role为master节点，设置不可调度
kubectl taint node -l node-role.kubernetes.io/master node-role.kubernetes.io/master="":NoSchedule --overwrite
```
**删除bootstrap证书**
```shell
rm -rf /etc/kubernetes/bootstrap-kubelet.conf
```
## 安装kube-proxy
**创建日志目录**
```shell
mkdir /var/log/kubernetes/kube-proxy
```
**拷贝命令**
```shell
cp /opt/kubernetes/server/bin/kube-proxy /usr/local/bin/
```
**生成kube-proxy证书**
```shell
cd /etc/kubernetes/pki && openssl genrsa -out proxy.key 2048
cat <<-EOF | sudo tee proxy-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
CN = system:kube-proxy
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key proxy.key -out proxy.csr -config proxy-csr.conf
openssl x509 -req -in proxy.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out proxy.crt -days 36500 \
-extensions v3_ext -extfile proxy-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/proxy.crt
[ $? -eq 0 ] && rm -rf /etc/kubernetes/pki/{proxy.csr,proxy-csr.conf}
```
**生成连接集群的kubeconfig文件**
```shell
KUBE_APISERVER="https://192.168.31.103:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=/etc/kubernetes/proxy.conf
kubectl config set-credentials system:kube-proxy \
--client-certificate=/etc/kubernetes/pki/proxy.crt \
--client-key=/etc/kubernetes/pki/proxy.key \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/proxy.conf
kubectl config set-context default \
--cluster=kubernetes \
--user=system:kube-proxy \
--kubeconfig=/etc/kubernetes/proxy.conf
kubectl config use-context default \
--kubeconfig=/etc/kubernetes/proxy.conf
```
**创建配置参数文件**
```shell
netcar=`ip r | awk '/default via/ {print $5}'`
[ ! -z $netcar ] && ipaddr=`ip r | awk -v netcar=$netcar '{if($3==netcar) print $9}'` || echo '$netcar is null'
cat > /etc/kubernetes/conf/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
kubeconfig: /etc/kubernetes/proxy.conf
hostnameOverride: $ipaddr
clusterCIDR: 10.244.0.0/16
mode: ipvs
ipvs:
minSyncPeriod: 5s
syncPeriod: 5s
scheduler: "rr"
EOF
```
> 说明：修改 `hostnameOverride` 的值为IP地址。`clusterCIDR` 的值为pod IP段。
> 参考地址： https://github.com/kubernetes/kube-proxy
> https://godoc.org/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration
> https://kubernetes.io/zh/docs/reference/config-api/kube-proxy-config.v1alpha1/
**创建kube-proxy的systemd模板**
```shell
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-proxy \\
--config=/etc/kubernetes/conf/kube-proxy-config.yml \\
--alsologtostderr=true --logtostderr=false --v=4 \\
--log-dir=/var/log/kubernetes/kube-proxy
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
```
**启动kube-proxy**
```shell
systemctl daemon-reload
systemctl enable kube-proxy.service --now
```
**验证**
```shell
curl http://localhost:10249/healthz && echo
```
# 部署插件
## 安装calico
详细的参数信息，请查看 [calico官网](https://docs.projectcalico.org/about/about-calico)
1. 下载manifest文件
```shell
mkdir /etc/kubernetes/addons
curl https://docs.projectcalico.org/archive/v3.18/manifests/calico-etcd.yaml -o /etc/kubernetes/addons/calico.yaml
```
2. 生成calico证书
```shell
pki_dir=/etc/kubernetes/pki/etcd
cd ${pki_dir} && openssl genrsa -out calico-etcd-client.key 2048
cat <<-EOF | sudo tee calico-etcd-client-csr.conf > /dev/null
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C = CN
ST = Guangdong
L = Guangzhou
CN = calico-etcd-client
[ v3_ext ]
keyUsage=Digital Signature, Key Encipherment
extendedKeyUsage=clientAuth
basicConstraints=CA:FALSE
authorityKeyIdentifier=keyid:always
EOF
openssl req -new -key calico-etcd-client.key -out calico-etcd-client.csr -config calico-etcd-client-csr.conf
openssl x509 -req -in calico-etcd-client.csr -CA /etc/kubernetes/pki/etcd/ca.crt -CAkey /etc/kubernetes/pki/etcd/ca.key \
-CAcreateserial -out calico-etcd-client.crt -days 36500 \
-extensions v3_ext -extfile calico-etcd-client-csr.conf -sha256
openssl verify -CAfile /etc/kubernetes/pki/etcd/ca.crt ${pki_dir}/calico-etcd-client.crt
[ $? -eq 0 ] && rm -rf ${pki_dir}/{calico-etcd-client-csr.conf,calico-etcd-client.csr}
```
3. 修改calico连接etcd的地址
>[info] 修改etcd地址
```bash
sed -ri 's@http://<ETCD_IP>:<ETCD_PORT>@https://192.168.32.127:2379,https://192.168.32.128:2379,https://192.168.32.129:2379@g' /etc/kubernetes/addons/calico.yaml
```
4. 修改calico连接etcd证书
```bash
ETCD_CA=$(cat /etc/kubernetes/pki/etcd/ca.crt | base64 -w 0)
ETCD_CERT=$(cat /etc/kubernetes/pki/etcd/calico-etcd-client.crt | base64 -w 0)
ETCD_KEY=$(cat /etc/kubernetes/pki/etcd/calico-etcd-client.key | base64 -w 0)
sed -ri "s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" /etc/kubernetes/addons/calico.yaml
sed -ri "s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g" /etc/kubernetes/addons/calico.yaml
sed -ri "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g" /etc/kubernetes/addons/calico.yaml
sed -ri 's@etcd_ca: ""@etcd_ca: "/calico-secrets/etcd-ca"@g' /etc/kubernetes/addons/calico.yaml
sed -ri 's@etcd_cert: ""@etcd_cert: "/calico-secrets/etcd-cert"@g' /etc/kubernetes/addons/calico.yaml
sed -ri 's@etcd_key: ""@etcd_key: "/calico-secrets/etcd-key"@g' /etc/kubernetes/addons/calico.yaml
```
5. 设置calico网段地址
将 DaemonSet 类型，calico-node 的 `spec.template.spec.containers.env` 下添加一段下面的内容
>[info] 默认是192.168.0.0/16地址
```yaml
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
```
6. 设置calico使用网卡
将 DaemonSet 类型，calico-node 的 `spec.template.spec.containers.env` 下添加一段下面的内容
```yaml
- name: IP_AUTODETECTION_METHOD
value: "interface=eth.*|em.*|enp.*"
```
7. calico开启metrics功能
将 DaemonSet 类型，calico-node 的 `spec.template.spec.containers.env` 下添加一段下面的内容
```yaml
- name: FELIX_PROMETHEUSMETRICSENABLED
value: "True"
- name: FELIX_PROMETHEUSMETRICSPORT
value: "9091"
```
将 DaemonSet 类型，calico-node 的 `spec.template.spec.containers` 下添加一段下面的内容
```yaml
ports:
- containerPort: 9091
name: http-metrics
protocol: TCP
```
**部署calico**
```shell
kubectl apply -f /etc/kubernetes/addons/calico.yaml
```
**验证calico**
```shell
kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-f4c6dbf-tkq77 1/1 Running 1 42h
calico-node-c4ccj 1/1 Running 1 42h
calico-node-crs9k 1/1 Running 1 42h
calico-node-fm697 1/1 Running 1 42h
kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.31.103 Ready master 5d23h v1.18.18
192.168.31.253 Ready <none> 5d23h v1.18.18
192.168.31.78 Ready <none> 5d23h v1.18.18
192.168.31.95 Ready <none> 5d23h v1.18.18
**注意**：status不是为ready的话，稍等一段时间再看看。一直都没有变成ready，请检查 kubelet 配置文件是否设置cni-bin-dir参数。默认是 `/opt/cni/bin`、`/etc/cni/net.d/`
kubectl run busybox --image=jiaxzeng/busybox:1.24.1 sleep 3600
kubectl run nginx --image=nginx
kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 6 42h 20.0.58.194 192.168.31.78 <none> <none>
nginx 1/1 Running 1 42h 20.0.85.194 192.168.31.95 <none> <none>
kubectl exec busybox -- ping 20.0.85.194 -c4
PING 20.0.85.194 (20.0.85.194): 56 data bytes
64 bytes from 20.0.85.194: seq=0 ttl=62 time=0.820 ms
64 bytes from 20.0.85.194: seq=1 ttl=62 time=0.825 ms
64 bytes from 20.0.85.194: seq=2 ttl=62 time=0.886 ms
64 bytes from 20.0.85.194: seq=3 ttl=62 time=0.840 ms
--- 20.0.85.194 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.820/0.842/0.886 ms
```
**除ping不通跨节点容器外，其他都没有问题的话**
可能是IP隧道的原因。可以手动测试一下两台主机IP隧道是否可以通信。
```sehll
modprobe ipip
ip tunnel add ipip-tunnel mode ipip remote 对端外网IP local 本机外网IP
ifconfig ipip-tunnel 虚IP netmask 255.255.255.0
```
如上述不通，请核查主机IP隧道通信问题。如果是openstack创建的虚机出现这种情况，可以禁用安全端口功能。
```shell
openstack server show 主机名称
openstack server remove security group 主机名称 安全组名称
openstack port set --disable-port-security `openstack port list | grep '主机IP地址' | awk '{print $2}'`
```
**安装calicoctl客户端**
```shell
curl -L https://github.com/projectcalico/calicoctl/releases/download/v3.18.6/calicoctl -o /usr/local/bin/calicoctl
chmod +x /usr/local/bin/calicoctl
```
**配置calicoctl**
```shell
mkdir -p /etc/calico
cat <<EOF | sudo tee /etc/calico/calicoctl.cfg > /dev/null
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
etcdEndpoints: https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379
etcdKeyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
etcdCertFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
etcdCACertFile: /etc/kubernetes/pki/etcd/ca.crt
EOF
```
**验证**
```shell
$ calicoctl get node -owide
```
## 部署coreDNS
**下载coredns部署yaml文件**
```shell
mkdir ~/coredns && cd ~/coredns
# https://github.com/kubernetes/kubernetes/blob/v1.18.18/cluster/addons/dns/coredns/coredns.yaml.sed
cat <<-EOF | sudo tee ~/coredns/coredns.yaml > /dev/null
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes \$DNS_DOMAIN in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. In order to make Addon Manager do not reconcile this replicas parameter.
# 2. Default is 1.
# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.6.5
imagePullPolicy: IfNotPresent
resources:
limits:
memory: \$DNS_MEMORY_LIMIT
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: \$DNS_SERVER_IP
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP
EOF
```
**修改参数**
```shell
vim coredns.yaml
...
kubernetes $DNS_DOMAIN in-addr.arpa ip6.arpa {
...
memory: $DNS_MEMORY_LIMIT
...
clusterIP: $DNS_SERVER_IP
...
image: k8s.gcr.io/coredns:1.6.5
# 添加 pod 反亲和，在 deploy.spec.template.spec 添加以下内容
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
k8s-app: kube-dns
```
- 将 `$DNS_DOMAIN` 替换成 `cluster.local.` 。默认 DNS_DOMAIN 就是 cluster.local. 。
- 将 `$DNS_MEMORY_LIMIT` 替换成合适的资源。
- 将 `$DNS_SERVER_IP` 替换成和 kubelet-config.yaml 的 `clusterDNS` 字段保持一致
- 如果不能上外网的话，将 image 的镜像设置为 `coredns/coredns:x.x.x`。
- 生产环境只有一个副本数不合适，所以在 `deployment`控制器的 `spec` 字段下，添加一行 `replicas: 3` 参数。
**部署coredns**
```shell
kubectl apply -f coredns.yaml
```
**验证**
```shell
kubectl get pod -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-75d9bd4f59-df94b 1/1 Running 0 7m55s
coredns-75d9bd4f59-kh4rp 1/1 Running 0 7m55s
coredns-75d9bd4f59-vjkpb 1/1 Running 0 7m55s
kubectl run dig --rm -it --image=jiaxzeng/dig:latest /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup kubernetes.default.svc.cluster.local.
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
/ # nslookup kube-dns.kube-system.svc.cluster.local.
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kube-dns.kube-system.svc.cluster.local
Address: 10.96.0.10
```
# k8s命令补全
```shell
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
```
# 附加iptables规则
```shell
# ssh 服务
iptables -t filter -A INPUT -p icmp --icmp-type 8 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 22 -m comment --comment "sshd service" -j ACCEPT
iptables -t filter -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t filter -A INPUT -i lo -j ACCEPT
iptables -t filter -P INPUT DROP
# etcd数据库
iptables -t filter -I INPUT -p tcp --dport 2379:2381 -m comment --comment "etcd Component ports" -j ACCEPT
# matster服务
iptables -t filter -I INPUT -p tcp -m multiport --dport 6443,10257,10259 -m comment --comment "k8s master Component ports" -j ACCEPT
# node服务
iptables -t filter -I INPUT -p tcp -m multiport --dport 10249,10250,10256 -m comment --comment "k8s node Component ports" -j ACCEPT
# k8s使用到的端口
iptables -t filter -I INPUT -p tcp --dport 32768:65535 -m comment --comment "ip_local_port_range ports" -j ACCEPT
iptables -t filter -I INPUT -p tcp --dport 30000:32767 -m comment --comment "k8s service nodeports" -j ACCEPT
# calico服务端口
iptables -t filter -I INPUT -p tcp -m multiport --dport 179,9099 -m comment --comment "k8s calico Component ports" -j ACCEPT
iptables -t filter -I INPUT -p tcp --dport 9091 -m comment --comment "k8s calico metrics ports" -j ACCEPT
# coredns服务端口
iptables -t filter -I INPUT -p udp -m udp --dport 53 -m comment --comment "k8s coredns ports" -j ACCEPT
# pod 到 service 网络。没有设置的话，启动coredns失败。
iptables -t filter -I INPUT -p tcp -s 20.0.0.0/16 -d 10.183.0.0/24 -m comment --comment "pod to service" -j ACCEPT
# 记录别drop的数据包，日志在 /var/log/messages，过滤关键字"iptables-drop: "
iptables -t filter -A INPUT -j LOG --log-prefix='iptables-drop: '
```
- 前言
- 架构
- 部署
- kubeadm部署
- kubeadm扩容节点
- 二进制安装基础组件
- 添加master节点
- 添加工作节点
- 选装插件安装
- Kubernetes使用
- k8s与dockerfile启动参数
- hostPort与hostNetwork异同
- 应用上下线最佳实践
- 进入容器命名空间
- 主机与pod之间拷贝
- events排序问题
- k8s会话保持
- 容器root特权
- CNI插件
- calico
- calicoctl安装
- calico网络通信
- calico更改pod地址范围
- 新增节点网卡名不一致
- 修改calico模式
- calico数据存储迁移
- 启用 kubectl 来管理 Calico
- calico卸载
- cilium
- cilium架构
- cilium/hubble安装
- cilium网络路由
- IP地址管理(IPAM)
- Cilium替换KubeProxy
- NodePort运行DSR模式
- IP地址伪装
- ingress使用
- nginx-ingress
- ingress安装
- ingress高可用
- helm方式安装
- 基本使用
- Rewrite配置
- tls安全路由
- ingress发布管理
- 代理k8s集群外的web应用
- ingress自定义日志
- ingress记录真实IP地址
- 自定义参数
- traefik-ingress
- traefik名词概念
- traefik安装
- traefik初次使用
- traefik路由(IngressRoute)
- traefik中间件(middlewares)
- traefik记录真实IP地址
- cert-manager
- 安装教程
- 颁布者CA
- 创建证书
- 外部存储
- 对接NFS
- 对接ceph-rbd
- 对接cephfs
- 监控平台
- Prometheus
- Prometheus安装
- grafana安装
- Prometheus配置文件
- node_exporter安装
- kube-state-metrics安装
- Prometheus黑盒监控
- Prometheus告警
- grafana仪表盘设置
- 常用监控配置文件
- thanos
- Prometheus
- Sidecar组件
- Store Gateway组件
- Querier组件
- Compactor组件
- Prometheus监控项
- grafana
- Querier对接grafana
- alertmanager
- Prometheus对接alertmanager
- 日志中心
- filebeat安装
- kafka安装
- logstash安装
- elasticsearch安装
- elasticsearch索引生命周期管理
- kibana安装
- event事件收集
- 资源预留
- 节点资源预留
- imagefs与nodefs验证
- 资源预留 vs 驱逐 vs OOM
- scheduler调度原理
- Helm
- Helm安装
- Helm基本使用
- 安全
- apiserver审计日志
- RBAC鉴权
- namespace资源限制
- 加密Secret数据
- 服务网格
- 备份恢复
- Velero安装
- 备份与恢复
- 常用维护操作
- container runtime
- 拉取私有仓库镜像配置
- 拉取公网镜像加速配置
- runtime网络代理
- overlay2目录占用过大
- 更改Docker的数据目录
- Harbor
- 重置Harbor密码
- 问题处理
- 关闭或开启Harbor的认证
- 固定harbor的IP地址范围
- ETCD
- ETCD扩缩容
- ETCD常用命令
- ETCD数据空间压缩清理
- ingress
- ingress-nginx header配置
- kubernetes
- 验证yaml合法性
- 切换KubeProxy模式
- 容器解析域名
- 删除节点
- 修改镜像仓库
- 修改node名称
- 升级k8s集群
- 切换容器运行时
- apiserver接口
- 其他
- 升级内核
- k8s组件性能分析
- ETCD
- calico
- calico健康检查失败
- Harbor
- harbor同步失败
- Kubernetes
- 资源Terminating状态
- 启动容器报错