在KubeSphere集群中采用prometheus-operator为集群内应用添加自动发现监控

prometheus-operator
CRDs:

• Prometheus:定义了所需的 Prometheus 部署
• alertmanager定义了所需的 alertmanager 部署
• ThanosRuler定义了所需的 Thanos Ruler 部署
• ServiceMonitor以声明方式指定应如何监控 Kubernetes 服务组。Operator 根据 API 服务器中对象的当前状态自动生成 Prometheus 抓取配置
• Podmonitor以声明方式指定应如何监视 Pod 组。Operator 根据 API 服务器中对象的当前状态自动生成 Prometheus 抓取配置
• Probe以声明方式指定应如何监视入口组或静态目标组。Operator 根据定义自动生成 Prometheus 抓取配置
• PrometheusRule定义了一组所需的 Prometheus 警报和/或记录规则。Operator 生成一个规则文件，可供 Prometheus 实例使用
• alertmanagerConfig以声明方式指定 alertmanager 配置的子部分，允许将警报路由到自定义接收器，并设置禁止规则

https://github.com/prometheus-operator/prometheus-operator/blob/master/documentation/api.md#podmetricsendpoint

监控配置

Configuration | Prometheushttps://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

Kubernetes内应用监控配置

1. 应用添加相应metric

(涉及开发内容不做过多讲解,我司项目为java开发,采用micrometer git地址:https://github.com/micrometer-metrics/micrometer)

2. 主要采用podmonitor配置自动发现(https://github.com/prometheus-operator/prometheus-operator/blob/master/documentation/api.md#podmetricsendpoint)

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    mill-app: appmonitor
  name: appmonitor
spec:
  namespaceSelector:
    any: true
  podMetricsEndpoints:
  - interval: 10s
    relabelings:
    - action: replace
      regex: (.+)
      sourceLabels:
      - __meta_kubernetes_pod_annotation_prometheus_io_path
      targetLabel: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::d+)?;(d+)
      replacement: $1:$2
      sourceLabels:
      - __address__
      - __meta_kubernetes_pod_annotation_prometheus_io_port
      targetLabel: __address__
    scheme: http
  selector:
    matchLabels:
      mill-monitor: app
# 关键配置需要发现的标签

3. 配置应用deployment 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mill-base
  labels:
    app: ywwl-base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mill-base
  template:
    metadata:
      annotations:
# 配置参数控制路径与端口
        prometheus.io/path: /metrics/prometheus
        prometheus.io/port: "8080"
      labels:
        app: mill-base
        #配置对应标记
        mill-monitor: app
    spec:
      containers:
        - name: mill-base
          image: localhub.xxx.com/xxx/xxx:latest
          ports:
            - containerPort: 8080

4. 打开prometheus界面发现相应配置实现

5. 如需配置全namespace监听,需要配置相应RBAC权限