prometheus 提供podMinitor与serviceMonitor用于用户自定义规则，发现需要监控的服务
以下以prometheus安装在monitoring，自定义发现seldon-core服务为例
通过kubectl get serviceMonitor -n monitoring 可以查询到现有的prometheus的服务监控
也可以从prometheus的页面上看到prometheus的service discovery
serviceMonitor 自定义发现规则
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: seldon-core-service
namespace: monitoring
labels: # ServiceMonitor的标签需要包含prometheus的selectorMonitorSelector中的条件
release: kube-prometheus-stack # kubectl get prometheus -n monitoring |grep serviceMonitorSelector
spec:
endpoints: # 入口规则 包含数据请求间隔、服务端口、请求路径、标签转换
- interval: 10s
port: http
#targetPort: http
path: /prometheus
relabelings:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+) # 将pod标签转换成 prometheus的指标参数 更多参考https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
namespaceSelector: # 命名空间过滤器器
any: true
selector: # 服务过滤器 k8s.labelSelector
matchexpressions:
- key: seldon-app
operator: Exists
servicemonitor 几分钟后生效，生效后就可以在prometheus的页面发现新增service discovery，发布seldon-core的服务并请求服务后就可以通过下列几个指标监控seldon-core服务
seldon-core自带指标参考 Metrics — seldon-core documentation
