calico更改pod地址范围 · Kubernetes

[TOC] ## 背景默认pod地址范围是26位的子网掩码，地址可能不够使用。或者PodIP地址范围不符合需求的。如果有设置防火墙的话，请提前放通pod到svc的网络。 ## 备份数据 ```shell $ calicoctl get ippool default-ipv4-ippool -oyaml > default-ipv4-ippool.yml ``` ## 修改k8s配置文件 - kube-apiserver: - --cluster-cidr - --node-cidr-mask-size - kube-proxy: - --cluster-cidr ```shell # 修改kube-controller-manager的字段 --cluster-cidr=20.188.0.0/16 --node-cidr-mask-size=26 sed -ri 's@(--cluster-cidr).*@\1=20.188.0.0/16 \\@g' /data/k8s/conf/kube-controller-manager.conf sed -ri 's@(--node-cidr-mask-size).*@\1=26 \\@g' /data/k8s/conf/kube-controller-manager.conf # 修改kube-proxy的字段 clusterCIDR: 20.188.0.0/16 sed -ri 's@(clusterCIDR:).*@\1 20.180.0.0/16@g' /data/k8s/conf/kube-proxy-config.yml ``` ## 删除节点 ```shell $ kubectl delete nodes --all ``` > 注意： > - 删除节点后，节点相关的标签都会丢失。请备份好节点标签 > - 重启kubelet服务是自动添加节点 ## 停服务 ```shell # master节点执行 $ systemctl stop kube-controller-manager.service # 所有节点执行 $ systemctl stop kubelet.service kube-proxy.service ``` ## 修改PodIP 1. 下载calicoctl ```shell $ curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.18.6/calicoctl $ mv calicoctl /usr/local/bin/ $ chmod +x /usr/local/bin/calicoctl ``` 2. 配置calicoctl连接信息 ```shell $ cat >> /etc/calico/calicoctl.cfg <<-EOF apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: etcdEndpoints: "https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379" etcdKeyFile: "/data/etcd/certs/etcd-key.pem" etcdCertFile: "/data/etcd/certs/etcd.pem" etcdCACertFile: "/data/etcd/certs/ca.pem" EOF ``` 3. 将默认的ippool修改成disable 将备份的 `default-ipv4-ippool.yml` 最下面添加一行 ```shell $ cat default-ipv4-ippool.yml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: blockSize: 26 cidr: 20.0.0.0/16 ipipMode: Always natOutgoing: true nodeSelector: all() vxlanMode: Never # 添加字段 disabled: true $ calicoctl apply -f default-ipv4-ippool.yml Successfully applied 1 'IPPool' resource(s) $ calicoctl get ippool -owide NAME CIDR NAT IPIPMODE VXLANMODE DISABLED SELECTOR default-ipv4-ippool 20.0.0.0/16 true Always Never true all() ``` 4. 添加新的ippool ```shell $ cat new-ipv4-ippool.yml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: new-ipv4-ippool spec: # 子网掩码与kube-controller-manager保持一致 blockSize: 26 # podIP地址范围 cidr: 20.188.0.0/16 ipipMode: Always natOutgoing: true nodeSelector: all() vxlanMode: Never $ calicoctl apply -f new-ipv4-ippool.yml Successfully applied 1 'IPPool' resource(s) $ calicoctl get ippool -owide NAME CIDR NAT IPIPMODE VXLANMODE DISABLED SELECTOR default-ipv4-ippool 20.0.0.0/16 true Always Never true all() new-ipv4-ippool 20.188.0.0/16 true Always Never false all() ``` ## 启动服务 ```shell # master节点执行 $ systemctl start kube-controller-manager.service # 所有节点执行 $ systemctl start kubelet.service kube-proxy.service ``` ## 重启所有的容器 > 前提：把原来有节点标签的，需要重新打回标签。 ```shell kubectl label nodes k8s-master02 node-role.kubernetes.io/master=true kubectl label nodes k8s-master01 node-role.kubernetes.io/master=true kubectl label nodes k8s-node01 kubernetes.io/node=monitor kubectl label nodes k8s-node02 kubernetes.io/ingress=nginx kubectl label nodes k8s-node03 kubernetes.io/ingress=nginx $ kubectl delete pod --all --all-namespaces ``` ## 验证 ```shell $ calicoctl get ippool -owide NAME CIDR NAT IPIPMODE VXLANMODE DISABLED SELECTOR default-ipv4-ippool 20.0.0.0/16 true Always Never true all() new-ipv4-ippool 20.188.0.0/16 true CrossSubnet Never false all() # 该条结果网段不准，但是掩码是正确的。 $ kubectl get nodes -o custom-columns=Name:.metadata.name,podCIDR:.spec.podCIDR Name podCIDR k8s-master01 20.188.1.0/24 k8s-master02 20.188.4.0/24 k8s-node01 20.188.2.0/24 k8s-node02 20.188.0.0/24 k8s-node03 20.188.3.0/24 # 以下列的CIDR的值为准 $ calicoctl ipam show --show-blocks +----------+-----------------+-----------+------------+--------------+ | GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE | +----------+-----------------+-----------+------------+--------------+ | IP Pool | 20.0.0.0/16 | 65536 | 0 (0%) | 65536 (100%) | | IP Pool | 20.188.0.0/16 | 65536 | 20 (0%) | 65516 (100%) | | Block | 20.188.130.0/24 | 256 | 5 (2%) | 251 (98%) | | Block | 20.188.131.0/24 | 256 | 4 (2%) | 252 (98%) | | Block | 20.188.30.0/24 | 256 | 5 (2%) | 251 (98%) | | Block | 20.188.93.0/24 | 256 | 3 (1%) | 253 (99%) | | Block | 20.188.96.0/24 | 256 | 3 (1%) | 253 (99%) | +----------+-----------------+-----------+------------+--------------+ $ kubectl get pod -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES app-v1-68db595855-dm9lb 1/1 Running 0 2m19s 20.188.30.4 k8s-node03 <none> <none> app-v1-68db595855-mvzwf 1/1 Running 0 2m19s 20.188.93.1 k8s-master02 <none> <none> app-v1-68db595855-rxnn8 1/1 Running 0 2m19s 20.188.131.1 k8s-node01 <none> <none> app-v2-595cf6b7f-mchhr 1/1 Running 0 2m19s 20.188.93.2 k8s-master02 <none> <none> app-v2-595cf6b7f-rxf8x 1/1 Running 0 2m18s 20.188.30.0 k8s-node03 <none> <none> app-v2-595cf6b7f-sjm45 1/1 Running 0 2m19s 20.188.96.1 k8s-node02 <none> <none> busybox-79b94f5dd8-2hzbr 1/1 Running 0 2m19s 20.188.96.2 k8s-node02 <none> <none> ```