ingress-nginx安装 · Kubernetes

### 一、Ingress 简介 ~~~ 在Kubernetes中，服务和Pod的IP地址仅可以在集群网络内部使用，对于集群外的应用是不可见的。为了使外部的应用能够访问集群内的服务，在Kubernetes 目前提供了以下几种方案： NodePort LoadBalancer Ingress ~~~ #### Ingress 组成 ~~~ ingress controller 　　将新加入的Ingress转化成Nginx的配置文件并使之生效 ingress服务　　将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可 ~~~ #### Ingress 工作原理 ~~~ 1.ingress controller通过和kubernetes api交互，动态的去感知集群中ingress规则变化， 2.然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段nginx配置， 3.再写到nginx-ingress-control的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中， 4.然后reload一下使配置生效。以此达到域名分配置和动态更新的问题。 ~~~ #### Ingress 可以解决什么问题 ~~~ 1.动态配置服务　　如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作. 2.减少不必要的端口暴露　　配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式 ~~~ ### 二、部署配置Ingress #### 1 .部署文件介绍、准备 **获取配置文件位置: [https://github.com/kubernetes/ingress-nginx/tree/nginx-0.20.0/deploy](https://github.com/kubernetes/ingress-nginx/tree/nginx-0.20.0/deploy)** ![](https://img2018.cnblogs.com/blog/911490/201811/911490-20181106155720393-1537032858.png) #### 下载部署文件 ~~~ 提供了两种方式：　　默认下载最新的yaml：　　　　wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml 　　指定版本号下载对应的yaml 　　　　 ~~~ #### 部署文件介绍 ~~~ 1.namespace.yaml 创建一个独立的命名空间 ingress-nginx 2.configmap.yaml ConfigMap是存储通用的配置变量的，类似于配置文件，使用户可以将分布式系统中用于不同模块的环境变量统一到一个对象中管理；而它与配置文件的区别在于它是存在集群的“环境”中的，并且支持K8S集群中所有通用的操作调用方式。从数据角度来看，ConfigMap的类型只是键值组，用于存储被Pod或者其他资源对象（如RC）访问的信息。这与secret的设计理念有异曲同工之妙，主要区别在于ConfigMap通常不用于存储敏感信息，而只存储简单的文本信息。 ConfigMap可以保存环境变量的属性，也可以保存配置文件。创建pod时，对configmap进行绑定，pod内的应用可以直接引用ConfigMap的配置。相当于configmap为应用/运行环境封装配置。 pod使用ConfigMap，通常用于：设置环境变量的值、设置命令行参数、创建配置文件。 3.default-backend.yaml 如果外界访问的域名不存在的话，则默认转发到default-http-backend这个Service，其会直接返回404： 4.rbac.yaml 负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding 5.with-rbac.yaml 是Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置 ~~~ #### 2.部署ingress #### 准备镜像，从这里mandatory.yaml查看需要哪些镜像 | 镜像名称 | 版本 | 下载地址 | | --- | --- | --- | | k8s.gcr.io/defaultbackend-amd64 | 1.5 | registry.cn-qingdao.aliyuncs.com/kubernetes\_xingej/defaultbackend-amd64 | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | 0.20.0 | registry.cn-qingdao.aliyuncs.com/kubernetes\_xingej/nginx-ingress-controller | 在每一个节点(Node)上下载镜像: ~~~ [root@k8s-node1 ~]# docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5 #注意版本号 Trying to pull repository registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64 ... 1.5: Pulling from registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64 26de8f6c1f4b: Pull complete Digest: sha256:2cdff48ab9b20ca5f9b0ee48bf3c139c51d6fb1a15245966583bc371c121c238 Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5 [root@k8s-node1 ~]# docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0 #注意版本号 Trying to pull repository registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller ... 0.20.0: Pulling from registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller 8e41b996a802: Pull complete f8e7d603ef88: Pull complete 610da4f3123f: Pull complete 62702a85e6c9: Pull complete 7cedf5b2083f: Pull complete 755f7fa719b6: Pull complete 6adbdd0c8aaf: Pull complete 2389d8b0d2a2: Pull complete 1ea794448393: Pull complete bb0c388ee432: Pull complete 9626641c5a97: Pull complete bd0bebb5ba38: Pull complete Digest: sha256:3f06079f7727b2fb7ad5c97d8152eb622ae504674395dfa71fda7ce315aaaf30 Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0 [root@k8s-node1 ~]# docker images #检查镜像是否下载成功 REPOSITORY TAG IMAGE ID CREATED SIZE registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64 1.5 d8f37b8cdaf4 2 weeks ago 5.13 MB registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller 0.20.0 3cc332ecde4f 3 weeks ago 513 MB k8s.gcr.io/kube-proxy-amd64 v1.10.0 bfc21aadc7d3 7 months ago 97 MB k8s.gcr.io/kube-controller-manager-amd64 v1.10.0 ad86dbed1555 7 months ago 148 MB k8s.gcr.io/kube-scheduler-amd64 v1.10.0 704ba848e69a 7 months ago 50.4 MB k8s.gcr.io/kube-apiserver-amd64 v1.10.0 af20925d51a3 7 months ago 225 MB k8s.gcr.io/etcd-amd64 3.1.12 52920ad46f5b 8 months ago 193 MB k8s.gcr.io/kubernetes-dashboard-amd64 v1.8.3 0c60bcf89900 8 months ago 102 MB quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 9 months ago 44.6 MB k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64 1.14.8 c2ce1ffb51ed 10 months ago 41 MB k8s.gcr.io/k8s-dns-sidecar-amd64 1.14.8 6f7f2dc7fab5 10 months ago 42.2 MB k8s.gcr.io/k8s-dns-kube-dns-amd64 1.14.8 80cc5ea4b547 10 months ago 50.5 MB k8s.gcr.io/pause-amd64 3.1 da86e6ba6ca1 10 months ago 742 kB docker.io/kubeguide/tomcat-app v1 a29e200a18e9 2 years ago 358 MB ~~~ #### 下载yaml文件并更新mandatory.yaml中的镜像地址（master上） ~~~ [root@k8s-master ~]# mkdir ingress-nginx [root@k8s-master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.20.0/deploy/mandatory.yaml [root@k8s-master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml #对外提供服务，如果不需要可以不下载 [root@k8s-master ingress-nginx]# sed -i 's#k8s.gcr.io/defaultbackend-amd64#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64#g' mandatory.yaml #替换defaultbackend-amd64镜像地址 sed -i 's#quay.io/kubernetes-ingress-controller/nginx-ingress-controller#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller#g' mandatory.yaml #替换nginx-ingress-controller镜像地址 [root@k8s-master ingress-nginx]# grep image mandatory.yaml #检查替换结果 # Any image is permissible as long as: image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5 image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0 ~~~ #### 修改service-nodeport.yaml文件，添加NodePort端口，默认为随机端口 ~~~ [root@k8s-master ingress-nginx]# cat service-nodeport.yaml apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP nodePort: 32080 #http - name: https port: 443 targetPort: 443 protocol: TCP nodePort: 32443 #https selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ~~~ #### 部署nginx-ingress-controller ~~~ [root@k8s-master ingress-nginx]# kubectl apply -f mandatory.yaml namespace "ingress-nginx" created deployment.extensions "default-http-backend" created service "default-http-backend" created configmap "nginx-configuration" created configmap "tcp-services" created configmap "udp-services" created serviceaccount "nginx-ingress-serviceaccount" created clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" created role.rbac.authorization.k8s.io "nginx-ingress-role" created rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" created clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" created deployment.extensions "nginx-ingress-controller" created [root@k8s-master ingress-nginx]# kubectl apply -f service-nodeport.yaml service "ingress-nginx" created ~~~ #### 3.查看ingress-nginx组件状态 ~~~ [root@k8s-master ingress-nginx]# kubectl get pods -n ingress-nginx #pod状态 NAME READY STATUS RESTARTS AGE default-http-backend-66c4fbf5b4-x2n8w 1/1 Running 0 58s nginx-ingress-controller-64bcff8657-5gdrd 1/1 Running 0 58s [root@k8s-master ingress-nginx]# kubectl get svc -n ingress-nginx #service状态及暴露端口 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.96.87.65 <none> 80/TCP 1m ingress-nginx NodePort 10.100.48.237 <none> 80:32080/TCP,443:32443/TCP 1m ~~~ **4.访问ingress-nginx服务，查看是否配置成功** **![](https://img2018.cnblogs.com/blog/911490/201811/911490-20181107144805697-2129132213.png)** 可以看到，提示404，这个因为当前ingress-nginx服务现在还没有后端服务，这是正常的 ### 三、创建ingress-nginx后端服务 #### 1.创建一个Service及后端Deployment(以nginx为例) ~~~ [root@k8s-master01 ingress]# cat deploy-demon.yaml apiVersion: v1 kind: Service metadata: name: myapp namespace: default spec: selector: app: myapp release: canary ports: - name: http port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: myapp-deploy spec: replicas: 5 selector: matchLabels: app: myapp release: canary template: metadata: labels: app: myapp release: canary spec: containers: - name: myapp image: ikubernetes/myapp:v2 ports: - name: httpd containerPort: 80 ~~~ **创建相关服务及检查状态是否就绪** ~~~ [root@k8s-master ingress-nginx]# kubectl apply -f deploy-demon.yaml service "myapp" created deployment.apps "myapp-deploy" created [root@k8s-master ingress-nginx]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-deploy-5cfd895984-ffzm5 1/1 Running 0 1m myapp-deploy-5cfd895984-ftg9t 1/1 Running 0 1m myapp-deploy-5cfd895984-jg887 1/1 Running 0 1m myapp-deploy-5cfd895984-mk4jq 1/1 Running 0 1m myapp-deploy-5cfd895984-nqz6s 1/1 Running 0 1m myweb-hrfqm 1/1 Running 0 8d myweb-pb5tb 1/1 Running 0 8d myweb-xrk22 1/1 Running 0 8d [root@k8s-master ingress-nginx]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d myapp ClusterIP 10.102.30.215 <none> 80/TCP 1m myweb NodePort 10.106.138.244 <none> 8080:31888/TCP 8d ~~~ **将myapp添加至ingress-nginx中** ~~~ [root@k8s-master ingress-nginx]# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-myapp namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: myapp.magedu.com #生产中该域名应当可以被公网解析 http: paths: - path: backend: serviceName: myapp servicePort: 80 ~~~ \[root@k8s-master ingress-nginx\]# kubectl apply -f ingress-myapp.yaml ingress.extensions "ingress-myapp" created **配置域名解析，当前测试环境我们使用hosts文件进行解析** ~~~ 172.33.16.241 myapp.magedu.com ~~~ 使用域名进行访问 ![](https://img2018.cnblogs.com/blog/911490/201811/911490-20181107151510875-1927214137.png) #### 2.再创建一个Service及后端Deployment(以tomcat为例) ~~~ [root@k8s-master ingress-nginx]# cat tomcat-deploy.yaml apiVersion: v1 kind: Service metadata: name: tomcat namespace: default spec: selector: app: tomcat release: canary ports: - name: http port: 8080 targetPort: 8080 - name: ajp port: 8009 targetPort: 8009 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-deploy spec: replicas: 3 selector: matchLabels: app: tomcat release: canary template: metadata: labels: app: tomcat release: canary spec: containers: - name: tomcat image: tomcat:7-alpine ports: - name: httpd containerPort: 8080 - name: ajp containerPort: 8009 [root@k8s-master ingress-nginx]# kubectl apply -f tomcat-deploy.yaml service "tomcat" created deployment.apps "tomcat-deploy" created [root@k8s-master ingress-nginx]# kubectl get pod #等待pod状态就绪 ~~~ **将tomcat添加至ingress-nginx中** ~~~ [root@k8s-master ingress-nginx]# cat ingress-tomcat.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat namespace: default annotations: kubernets.io/ingress.class: "nginx" spec: rules: - host: tomcat.magedu.com http: paths: - path: backend: serviceName: tomcat servicePort: 8080 [root@k8s-master ingress-nginx]# kubectl apply -f ingress-tomcat.yaml ingress.extensions "ingress-tomcat" created ~~~ **添加域名解析及访问服务** ![](https://img2018.cnblogs.com/blog/911490/201811/911490-20181107152122720-131540011.png) #### 3.下面我们对tomcat服务添加httpds服务 **创建私有证书及secret** ~~~ [root@k8s-master ingress-nginx]# openssl genrsa -out tls.key 2048 Generating RSA private key, 2048 bit long modulus .......+++ ..............................+++ e is 65537 (0x10001) [root@k8s-master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.magedu.com #注意域名要和服务的域名一致 [root@k8s-master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key #创建secret secret "tomcat-ingress-secret" created [root@k8s-master ingress-nginx]# kubectl get secret NAME TYPE DATA AGE default-token-bf52l kubernetes.io/service-account-token 3 9d tomcat-ingress-secret kubernetes.io/tls 2 7s [root@k8s-master ingress-nginx]# kubectl describe secret tomcat-ingress-secret Name: tomcat-ingress-secret Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 1294 bytes #base64加密 tls.key: 1679 bytes ~~~ **将证书应用至tomcat服务中** ~~~ [root@k8s-master01 ingress]# cat ingress-tomcat-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat-tls namespace: default annotations: kubernets.io/ingress.class: "nginx" spec: tls: - hosts: - tomcat.magedu.com #与secret证书的域名需要保持一致 secretName: tomcat-ingress-secret #secret证书的名称 rules: - host: tomcat.magedu.com http: paths: - path: backend: serviceName: tomcat servicePort: 8080 ~~~ ~~~ [root@k8s-master01 ingress]# kubectl apply -f ingress-tomcat-tls.yaml ~~~ **访问服务** ![](https://img2018.cnblogs.com/blog/911490/201811/911490-20181107154547795-742410055.png) **#参考文档：** 原文参考：**[https://www.cnblogs.com/panwenbin-logs/p/9915927.html](https://www.cnblogs.com/panwenbin-logs/p/9915927.html)** **[https://www.jianshu.com/p/e30b06906b77](https://www.jianshu.com/p/e30b06906b77)** **[https://kubernetes.io/docs/concepts/services-networking/ingress/](https://kubernetes.io/docs/concepts/services-networking/ingress/)** **[https://kubernetes.github.io/ingress-nginx/deploy/](https://kubernetes.github.io/ingress-nginx/deploy/)** **[https://github.com/kubernetes/ingress-nginx/blob/nginx-0.20.0/docs/deploy/index.md](https://github.com/kubernetes/ingress-nginx/blob/nginx-0.20.0/docs/deploy/index.md)**