prometheus配置AlertManager及dolphinscheduler http告警使用测试

AlertManager
- 下载部署
- prometheus 配置告警规则
- - 创建告警规则文件
  - 加载告警规则文件
  - 验证查看
- 验证告警
- - 修改告警规则
  - 验证告警
dolphinscheduler配置AlertManager告警
- AlertManager API
- dolphinscheduler HTTP告警代码修改
- - HttpSender源码
- 测试验证
- - 告警配置
  - 执行测试
  - 测试结果
  - - 告警内容
    - 测试api2
    - -  API2测试结果
其它
- dolphinscheduler 2.0.5 告警组件-HTTP

AlertManager

官方资料

下载部署

下载地址
解压查看
可以规范下目录（批量创建常见的bin、conf、data结构）

[prometheus@host1 ~]$ mkdir -p alertmanager/{bin,conf,data,logs,templates}
[prometheus@host1 ~]$ ll alertmanager
总用量 0
drwxrwxr-x. 2 prometheus prometheus 57 5月  11 17:12 bin
drwxrwxr-x. 2 prometheus prometheus 47 5月  13 10:27 conf
drwxrwxr-x. 3 prometheus prometheus 81 5月  16 04:45 data
drwxrwxr-x. 2 prometheus prometheus  6 5月  11 15:23 logs
drwxrwxr-x. 2 prometheus prometheus  6 5月  11 15:23 templates
[prometheus@host1 ~]$

移动解压目录文件到相应目录
执行文件移到bin目录，配置文件移到conf目录下

[prometheus@host1 ~]$ tar xf alertmanager-0.24.0.linux-amd64.tar.gz 
[prometheus@host1 ~]$  mkdir -p alertmanager/{bin,conf,data,logs,templates}
[prometheus@host1 ~]$ cp alertmanager-0.24.0.linux-amd64/alertmanager alertmanager/bin/
[prometheus@host1 ~]$ cp alertmanager-0.24.0.linux-amd64/amtool alertmanager/bin/
[prometheus@host1 ~]$ cp alertmanager-0.24.0.linux-amd64/alertmanager.yml alertmanager/conf/

conf/alertmanager.yml文件内容

## Alertmanager 配置文件
global:
  resolve_timeout: 5s
  # smtp配置
  smtp_from: "102XXX638@qq.com"
  smtp_smarthost: 'smtp.qq.com:25'
  smtp_auth_username: "102XXX638@qq.com"
  smtp_auth_password: "ovosgonmfliabbbf"
  smtp_require_tls: true
 
# 路由分组
route:
  receiver: email
  group_wait: 3s # 在组内等待所配置的时间，如果同组内，30秒内出现相同报警，在一个组内出现。
  group_interval: 1m # 如果组内内容不变化，合并为一条警报信息，5m后发送。
  repeat_interval: 5m # 发送报警间隔，如果指定时间内没有修复，则重新发送报警。
  group_by: [alertname]  # 报警分组

 
# 接收器指定发送人以及发送渠道
receivers:
# ops分组的定义
- name: email
  email_configs:
  - to: 'liuwei@copote.com'
    send_resolved: true
    headers:
      subject: "测试告警"
      from: "alertmanager"
      to: "test"

 
# 抑制器配置
inhibit_rules: # 抑制规则
  - source_match: # 源标签警报触发时抑制含有目标标签的警报，在当前警报匹配 status: 'High'
      status: 'High'  
    target_match:
      status: 'Warning' # 
    equal: ['alertname','operations', 'instance'] # 确保这个配置下的标签内容相同才会抑制，也就是说警报中必须有这三个标签值才会被抑制。

启动alertmanager
后台启动（也可以配置成系统服务启动）

nohup /home/prometheus/alertmanager/bin/alertmanager --storage.path=/home/prometheus/alertmanager/data/ --config.file=/home/prometheus/alertmanager/conf/alertmanager.yml &

查看日志，默认9093监听端口

prometheus 配置告警规则 创建告警规则文件

alert.yml （自定义名称，类型为yml）

[prometheus@host1 data]$ pwd
/home/prometheus/alertmanager/data
[prometheus@host1 data]$ cat alert.yml 
groups:
    - name: Node_Metrics
      rules:
       - alert: Node_Filesystem_Below_20MB
         expr: sum(node_filesystem_avail_bytes)by(instance)/1024/1024 < 20
         for: 2m
         labels:
           team: node
         annotations:
          summary: "{{$labels.instance}}: Filesystem Pressure"
          description: "{{$labels.instance}}: Filesystem is below 20MB"
       - alert: FLUME_CHANNEL_ChannelCapacity_1W
         expr: FLUME_CHANNEL_ChannelCapacity > 10000
         for: 2m
         labels:
           team: flume
         annotations:
           summary: "{{$labels.instance}}: low Capacity"
           description: "{{$labels.instance}}: low Capacity is above 10000 (current value is: {{ $value }}"
[prometheus@host1 data]$

加载告警规则文件

prometheus.yml

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - 192.168.56.10:9093
rule_files:
    - "/home/prometheus/alertmanager/data/alert.yml"

重启prometheus
prometheus安装部署热加载详见prometheus、influxdb安装及flume_export下载编译使用

验证查看

prometheus查看地址：

http://192.168.56.10:9090/alerts
http://192.168.56.10:9090/rules

Alertmanager查看地址：

http://192.168.56.10:9093/#/alerts

inactive-未触发告警规则，pending-触发告警规则，待发送，firing-发送告警

验证告警 修改告警规则

改成大于20，测试告警
重启prometheus,重新加载规则

验证告警

查看页面
等待状态变为firing后，查看是否收到告警邮件

dolphinscheduler配置AlertManager告警

AlertManager API

AlertManager API

dolphinscheduler HTTP告警代码修改

HttpSender源码


package org.apache.dolphinscheduler.plugin.alert.http;

import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.dolphinscheduler.alert.api.AlertResult;
import org.apache.dolphinscheduler.spi.utils.JSONUtils;
import org.apache.dolphinscheduler.spi.utils.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;

import java.util.HashMap;
import java.util.Map;

public final class HttpSender {
    private static final Logger log = org.slf4j.LoggerFactory.getLogger(HttpSender.class);
    private static final String URL_SPLICE_CHAR = "?";
    
    private static final String REQUEST_TYPE_POST = "POST";
    
    private static final String REQUEST_TYPE_GET = "GET";
    private static final String DEFAULT_CHARSET = "utf-8";
    private final String headerParams;
    private final String bodyParams;
    private final String contentField;
    private final String requestType;
    private String url;
    private HttpRequestBase httpRequest;

    public HttpSender(Map paramsMap) {

        url = paramsMap.get(HttpAlertConstants.URL);
        headerParams = paramsMap.get(HttpAlertConstants.HEADER_PARAMS);
        bodyParams = paramsMap.get(HttpAlertConstants.BODY_PARAMS);
        contentField = paramsMap.get(HttpAlertConstants.CONTENT_FIELD);
        requestType = paramsMap.get(HttpAlertConstants.REQUEST_TYPE);
    }

    public AlertResult send(String msg) {

        AlertResult alertResult = new AlertResult();

        createHttpRequest(msg);

        if (httpRequest == null) {
            alertResult.setStatus("false");
            alertResult.setMessage("Request types are not supported");
            return alertResult;
        }

        try {
            CloseableHttpClient httpClient = HttpClientBuilder.create().build();
            CloseableHttpResponse response = httpClient.execute(httpRequest);
            HttpEntity entity = response.getEntity();
            String resp = EntityUtils.toString(entity, DEFAULT_CHARSET);
            alertResult.setStatus("true");
            alertResult.setMessage(resp);
        } catch (Exception e) {
            log.error("send http alert msg  exception : {}", e.getMessage());
            alertResult.setStatus("false");
            alertResult.setMessage("send http request  alert fail.");
        }

        return alertResult;
    }

    private void createHttpRequest(String msg) {
        if (REQUEST_TYPE_POST.equals(requestType)) {
            httpRequest = new HttpPost(url);
            setHeader();
            //POST request add param in request body
            setMsgInRequestBody(msg);
        } else if (REQUEST_TYPE_GET.equals(requestType)) {
            //GET request add param in url
            setMsgInUrl(msg);
            httpRequest = new HttpGet(url);
            setHeader();
        }
    }

    
    private void setMsgInUrl(String msg) {

        if (StringUtils.isNotBlank(contentField)) {
            String type = "&";
            //check splice char is & or ?
            if (!url.contains(URL_SPLICE_CHAR)) {
                type = URL_SPLICE_CHAR;
            }
            url = String.format("%s%s%s=%s", url, type, contentField, msg);
        }
    }

    
    private void setHeader() {

        if (httpRequest == null) {
            return;
        }

        HashMap map = JSONUtils.parseObject(headerParams, HashMap.class);
        for (Map.Entry entry : map.entrySet()) {
            httpRequest.setHeader(entry.getKey(), String.valueOf(entry.getValue()));
        }
    }

    
    private void setMsgInRequestBody(String msg) {
        ObjectNode objectNode = JSONUtils.parseObject(bodyParams);
        //set msg content field
        objectNode.put(contentField, msg);
        try {
        	//rxz 20220517 modify
        	msg = int2Str(msg);
            StringEntity entity = new StringEntity(msg, ContentType.APPLICATION_JSON);
            log.info("send http alert msg: {}",msg);
            ((HttpPost) httpRequest).setEntity(entity);
        } catch (Exception e) {
            log.error("send http alert msg  exception : {}", e.getMessage());
        }
    }
    
    private String int2Str(String msg) {
    	String sendMsg = "[{";
    	JSONArray array = JSONObject.parseArray(msg);
		for(Object arr:array) {
			JSONObject jsonObject = (JSONObject)arr;
			for (Map.Entry entry: jsonObject.entrySet()) {
	            Object o = entry.getValue();
	            if(o instanceof Integer ||o instanceof Long) {
	            	entry.setValue(entry.getValue().toString());
	            }
	        }
			sendMsg+=(""labels":"+jsonObject.toString()+",");
		}
    	return sendMsg.substring(0, sendMsg.length()-1)+"}]";
    }
}

测试验证 告警配置

告警实例配置

http://192.168.56.10:9093/api/v1/alerts
POST
{"Content-Type":"application/json"}
{}
content

告警组配置

执行测试

工作流启动

测试结果

查看日志
邮件查看

告警内容

[{
	"labels": {
		"owner": "admin",
		"taskHost": "192.168.56.10:1234",
		"processDefinitionCode": "5327475818624",
		"taskStartTime": "2022-05-16 09:30:11",
		"taskType": "PROCEDURE",
		"taskState": "FAILURE",
		"taskCode": "5327458833408",
		"processId": "62822",
		"processName": "存储过程pgsql-6-20220516093011807",
		"logPath": "/home/dolphinscheduler/app/dolphinscheduler/logs/5327475818624_6/62822/483135.log",
		"taskName": "存储过程pgsql",
		"projectName": "dolphin",
		"projectId": "30",
		"taskEndTime": "2022-05-16 09:30:23"
	}
}]

测试api2

 API2测试结果

也是可以的，格式没有区别，只是url从v1换成了v2

其它

View In Alertmanager
进入 Alertmanager页面，看不到刚刚发送的记录，看现象，第三方触发的告警不会逗留，只有触发告警规则的记录才会一直在这，直到不满足告警规则。怎么把alert信息存到数据库？待研究

dolphinscheduler 2.0.5 告警组件-HTTP

告警组件-HTTP试用及改造

prometheus配置AlertManager及dolphinscheduler http告警使用测试

Linux相关栏目本月热门文章