[TOC] kube-state-metrics 是关于在不修改的情况下从 `Kubernetes API` 对象生成指标。这确保了 `kube-state-metrics` 提供的特性与 Kubernetes API 对象本身具有相同等级的稳定性。反过来，这意味着 `kube-state-metrics` 在某些情况下可能不会显示与 kubectl 完全相同的值，因为 kubectl 应用某些启发式方法来显示可理解的消息。`kube-state-metrics` 公开了未经 `Kubernetes API` 修改的原始数据，这样用户就拥有了他们需要的所有数据，并在他们认为合适的时候执行启发式算法。 ## 版本兼容性 | kube-state-metrics version | kubernetes version | | :-: | :-: | | v1.6.0 | 1.10, 1.11, 1.12, 1.13, 1.14 | | v1.7.2 | 1.11, 1.12, 1.13, 1.14, 1.15 | | v1.8.0 | 1.11, 1.12, 1.13, 1.14, 1.15 | | v1.9.8 | 1.12, 1.13, 1.14, 1.15, 1.16 | | v2.0.0 | 1.17, 1.18, 1.19, 1.20 | | v2.1.1 | 1.17, 1.18, 1.19, 1.20, 1.21 | | v2.2.4 | 1.18, 1.19, 1.20, 1.21, 1.22 | | v2.3.0 | 1.19, 1.20, 1.21, 1.22, 1.23 | | v2.4.2 | 1.21, 1.22, 1.23 | | v2.5.0 | 1.22, 1.23, 1.24 | | v2.6.0 | 1.22, 1.23, 1.24 | | v2.7.0 | 1.25 | | v2.8.2 | 1.26 | | v2.9.2 | 1.26 | | v2.10.1 | 1.27 | | v2.11.0 | 1.28 | | v2.12.0 | 1.29 | ## 部署安装 0.创建目录 ```shell mkdir /etc/kubernetes/addons/kube-state-metrics ``` 1.rbac清单文件 ```yaml cat <<'EOF' | sudo tee /etc/kubernetes/addons/kube-state-metrics/rbac.yaml > /dev/null apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.6.0 name: kube-state-metrics rules: - apiGroups: - "" resources: - configmaps - secrets - nodes - pods - services - serviceaccounts - resourcequotas - replicationcontrollers - limitranges - persistentvolumeclaims - persistentvolumes - namespaces - endpoints verbs: - list - watch - apiGroups: - apps resources: - statefulsets - daemonsets - deployments - replicasets verbs: - list - watch - apiGroups: - batch resources: - cronjobs - jobs verbs: - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - list - watch - apiGroups: - authentication.k8s.io resources: - tokenreviews verbs: - create - apiGroups: - authorization.k8s.io resources: - subjectaccessreviews verbs: - create - apiGroups: - policy resources: - poddisruptionbudgets verbs: - list - watch - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests verbs: - list - watch - apiGroups: - storage.k8s.io resources: - storageclasses - volumeattachments verbs: - list - watch - apiGroups: - admissionregistration.k8s.io resources: - mutatingwebhookconfigurations - validatingwebhookconfigurations verbs: - list - watch - apiGroups: - networking.k8s.io resources: - networkpolicies - ingresses verbs: - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - list - watch - apiGroups: - rbac.authorization.k8s.io resources: - clusterrolebindings - clusterroles - rolebindings - roles verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.6.0 name: kube-state-metrics roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metrics subjects: - kind: ServiceAccount name: kube-state-metrics namespace: kube-system --- apiVersion: v1 automountServiceAccountToken: false kind: ServiceAccount metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.6.0 name: kube-state-metrics namespace: kube-system EOF ``` 2.deployment清单文件 ```yaml cat <<'EOF' | sudo tee /etc/kubernetes/addons/kube-state-metrics/deployment.yaml > /dev/null apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.6.0 name: kube-state-metrics namespace: kube-system spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: kube-state-metrics template: metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.6.0 spec: automountServiceAccountToken: true containers: - image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 timeoutSeconds: 5 name: kube-state-metrics ports: - containerPort: 8080 name: http-metrics - containerPort: 8081 name: telemetry readinessProbe: httpGet: path: / port: 8081 initialDelaySeconds: 5 timeoutSeconds: 5 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsUser: 65534 nodeSelector: kubernetes.io/os: linux serviceAccountName: kube-state-metrics EOF ``` 3.service清单文件 ```yaml cat <<'EOF' | sudo tee /etc/kubernetes/addons/kube-state-metrics/service.yaml > /dev/null apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.6.0 name: kube-state-metrics namespace: kube-system spec: clusterIP: None ports: - name: http-metrics port: 8080 targetPort: http-metrics - name: telemetry port: 8081 targetPort: telemetry selector: app.kubernetes.io/name: kube-state-metrics EOF ``` 4.创建相关资源 ```shell $ kubectl apply -f /etc/kubernetes/addons/kube-state-metrics/ deployment.apps/kube-state-metrics created clusterrole.rbac.authorization.k8s.io/kube-state-metrics created clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created serviceaccount/kube-state-metrics created service/kube-state-metrics created ``` 5.验证 ```shell $ kubectl -n kube-system get pod -l app.kubernetes.io/name=kube-state-metrics NAME READY STATUS RESTARTS AGE kube-state-metrics-6544d5656d-sjnhh 1/1 Running 0 53m $ kubectl -n kube-system get endpoints kube-state-metrics NAME ENDPOINTS AGE kube-state-metrics 10.0.0.222:8081,10.0.0.222:8080 55m $ curl -s 10.0.0.222:8080/metrics | head -20 # HELP kube_certificatesigningrequest_annotations Kubernetes annotations converted to Prometheus labels. # TYPE kube_certificatesigningrequest_annotations gauge # HELP kube_certificatesigningrequest_labels Kubernetes labels converted to Prometheus labels. # TYPE kube_certificatesigningrequest_labels gauge # HELP kube_certificatesigningrequest_created Unix creation timestamp # TYPE kube_certificatesigningrequest_created gauge # HELP kube_certificatesigningrequest_condition The number of each certificatesigningrequest condition # TYPE kube_certificatesigningrequest_condition gauge # HELP kube_certificatesigningrequest_cert_length Length of the issued cert # TYPE kube_certificatesigningrequest_cert_length gauge # HELP kube_configmap_annotations Kubernetes annotations converted to Prometheus labels. # TYPE kube_configmap_annotations gauge kube_configmap_annotations{namespace="kube-system",configmap="coredns"} 1 kube_configmap_annotations{namespace="kube-node-lease",configmap="kube-root-ca.crt"} 1 kube_configmap_annotations{namespace="kube-system",configmap="ingress-nginx-controller"} 1 kube_configmap_annotations{namespace="kube-system",configmap="extension-apiserver-authentication"} 1 kube_configmap_annotations{namespace="kube-system",configmap="kube-proxy"} 1 kube_configmap_annotations{namespace="kube-system",configmap="prometheus-target"} 1 kube_configmap_annotations{namespace="default",configmap="kube-root-ca.crt"} 1 kube_configmap_annotations{namespace="kube-system",configmap="metrics-proxy"} 1 ``` 6.监控获取指标 ```yaml - job_name: "kube-state-metrics" kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoints_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-state-metrics;http-metrics ``` > 需要重新加载Prometheus配置文件 7.查看Prometheus的targets  ## 参考 https://github.com/kubernetes/kube-state-metrics/tree/v2.6.0/examples/standard