### Kubespray Arch  ### cluster HA-mode  [参考链接](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/ha-mode.md) ### 阿里云 容器服务 vs kubespray - 集群托管 vs 自行维护 - 容器镜像 vs harbor - 应用配置管理 vs git+zookeeper+configmap - 阿里 SLB 蓝绿发布 vs nginx-ingress - Nas oss vs ClusterFS [slb金丝雀发布最佳实践](https://www.alibabacloud.com/help/zh/doc-detail/73980.htm?spm=a2c63.p38356.b99.246.20d65044xK7ALo) [alicloud-controller-manager](https://github.com/AliyunContainerService/alicloud-controller-manager) ## 应用部署 ### Jenkins 动态伸缩slave jenkins地址：http://jenkins-k8s.jiedai361.com * 服务高可用，当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master 容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用。 * 动态伸缩，合理使用资源，每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave 自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配 Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。 * 扩展性好，当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个 Kubernetes Node 到集群中，从而实现扩展。  1. 创建应用jenkins job 2. 创建应用配置：启动参数 setenv.sh 3. 创建应用环境变量：configmap 4. 创建服务路由ingress 启动参数配置： setenv.sh cat FTC_DEMO_FTC-PAAS-CONTRACT-SERVICE/bin/setenv.sh JAVA_OPTS=" -DZK_URL=${DR_CFG_ZOOKEEPER_ENV_URL} -DAppLogs=/volume_logs/ -Xms2G -Xmx2G -Dspring.application.name=ftc-paas-contract-service ${JAVA_OPTS} " 应用环境变量：configmap cat ftc-paas-contract-service-configmap.yaml kind: ConfigMap apiVersion: v1 metadata: name: ftc-paas-contract-service-env-config namespace: ftc-demo data: CFG_ADDR: '10.34.11.186:4181' DR_CFG_ZOOKEEPER_ENV_URL: '10.34.11.186:4181' CFG_FILES: 'bin/setenv.sh' 创建服务路由ingress： cat app-ingress/ftc-paas-contract-service-ingres.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ftc-paas-contract-service-ingress namespace: ftc-demo annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: ftc-paas-contract-k8s-demo.dalianyun.com http: paths: - backend: serviceName: ftc-paas-contract-service servicePort: 8073 service 服务请求地址 k8s-demo set /config/contract/ftc-paas-contract-service/cfca.seal_endpoint http://ftc-paas-cfca-seal.ftc-demo.svc.cluster.local:8080/Seal/ set /config/contract/ftc-paas-contract-service/cfca.ra_endpoint http://ftc-paas-cfca-ra.ftc-demo.svc.cluster.local:8080/raWeb/CSHttpServlet set /config/contract/ftc-paas-contract-service/cfca.kt_ip ftc-paas-cfca-kt.ftc-demo.svc.cluster.local set /config/contract/ftc-paas-contract-service/cfca.kt_port 9040 [kubernetes 命名规范](https://wiki.dianrong.com/pages/viewpage.action?pageId=36098232) jenkins 应用部署 kind: Deployment metadata: name: ftc-paas-contract-service namespace: ftc-demo labels: app: ftc-paas-contract-service version: CC-321 AppEnv: demo spec: replicas: 2 selector: matchLabels: app: ftc-paas-contract-service template: metadata: labels: app: ftc-paas-contract-service spec: containers: - name: ftc-paas-contract-service image: dl-harbor.dianrong.com/ftc/ftc-paas-contract-service:6676f059485f2e38703b8aeaadf6614184fb01a3 ports: - containerPort: 8073 livenessProbe: httpGet: path: /ccc/api/health port: 8073 initialDelaySeconds: 90 timeoutSeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: /ccc/api/health port: 8073 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 5 # configmap env env: - name: CFG_LABEL value: /instances/FTC_DEMO_FTC-PAAS-CONTRACT-SERVICE - name: CFG_ADDR valueFrom: configMapKeyRef: name: ftc-paas-contract-service-env-config key: CFG_ADDR - name: DR_CFG_ZOOKEEPER_ENV_URL valueFrom: configMapKeyRef: name: ftc-paas-contract-service-env-config key: DR_CFG_ZOOKEEPER_ENV_URL - name: CFG_FILES valueFrom: configMapKeyRef: name: ftc-paas-contract-service-env-config key: CFG_FILES # configMap volume volumeMounts: - name: applogs mountPath: /volume_logs/ volumes: - name: applogs hostPath: path: /opt/app_logs/ftc-paas-contract-service imagePullSecrets: - name: regcred # 设置jenkins全局变量 --- apiVersion: v1 kind: Service metadata: name: ftc-paas-contract-service namespace: ftc-demo labels: app: ftc-paas-contract-service spec: ports: - port: 8073 targetPort: 8073 selector: app: ftc-paas-contract-service ### 集群资源管理 - Node Taints: node.kubernetes.io=unschedulable:NoSchedule - LimitRange ftx-demo=limits.yaml apiVersion: v1 kind: LimitRange metadata: name: ftc-demo spec: limits: - max: cpu: "2" memory: 4Gi min: cpu: "1" memory: "2Gi" # maxLimitRequestRatio: # cpu: 3 # memory: 2 type: Pod - default: cpu: 2 memory: 4Gi defaultRequest: cpu: 1 memory: 2Gi max: cpu: "2" memory: 4Gi min: cpu: 500m memory: 1024Mi # maxLimitRequestRatio: # cpu: 4 # memory: 4 type: Container ### 近期任务计划 - 应用日志搜集 - 集群监控 - helm 应用软件仓库 - 容器自动伸缩 - ### k8s 监控prometheus-operator [金融云grafana展示](http://grafana-demo.dalianyun.com/) [金融云prometheus 控制台](http://prometheus-demo.dalianyun.com)