Prometheus + node_exporter +process_exporter + CM Datasource + Grafana 监控安装部署
一、node_exporter安装与配置node_exporter是Prometheus的一个插件，部署该插件，可以将本地机器的系统资源信息筹集并发送给Prometheus。
在需要监控的机器上面都要安装该插件。
这里我只演示监控我的两台机器
192.168.0.113,
192.168.0.114
进入到http://prometheus.io/download官网,或者使用我给的安装包
#下载安装包并上传服务器/software：node_exporter-1.3.0.linux-amd64.tar.gz
1.1.解压
cd /software tar -zxvf node_exporter-1.3.0.linux-amd64.tar.gz -C /opt/ cd /opt/ mv node_exporter-1.3.0.linux-amd64 node_exporter cd node_exporter/ #启动node_exporter # nohup ./node_exporter --web.listen-address=":9100" >/dev/null 2>&1 & #注意：--web.listen-address=":9100" 端口一定不能被占用，否则会报错，可以自己手动修改为任意端口
1.2.添加到系统服务中
vim /etc/systemd/system/node_exporter.service
-------------------------------配置如下内容--------------
[Unit] Description=Node Export After=network.target documentation=https://prometheus.io/docs/guides/node-exporter/ [Service] Type=simple WorkingDirectory=/tmp/ ExecStart=/opt/node_exporter/node_exporter --web.listen-address=:9100 Restart=on-failure [Install] WantedBy=multi-user.target
#注意：–web.listen-address=:9100 端口一定不能被占用，否则会报错，可以自己手动修改为任意端口
#授权可执行 chmod +x /etc/systemd/system/node_exporter.service #重载systemd系统 systemctl daemon-reload #启动服务 systemctl start node_exporter systemctl status node_exporter systemctl enable node_exporter #检验是否安装成功，登录ip:9100进行查看二、processes_exporter安装与配置
process_exporter是Prometheus的一个插件，部署该插件，可以将本地机器的进程信息筹集并发送给Prometheus。
需要监控的机器上面都要安装该插件。
2.1.下载
下载地址：
https://github.com/ncabatoff/process-exporter/releases/tag/v0.5.0
下载：process-exporter-0.5.0.linux-amd64.tar.gz
2.2.上传服务器并解压
这里我已经提前上传到/software
cd /software tar -zxvf process_exporter-0.5.0.linux-amd64.tar.gz -C /opt/ cd /opt/ mv process_exporter-0.5.0.linux-amd64.tar.gz process_exporter cd process_exporter/
2.3.创建process-name.yaml
cd /opt/process_exporter
vi process-name.yaml //将下面内容写入，也可自己再下面添加相关的进程
process_names:
- name: "{{.Matches}}"
cmdline:
- 'DataNode'
- name: "{{.Matches}}"
cmdline:
- 'kafka'
- name: "{{.Matches}}"
cmdline:
- 'SecondaryNameNode'
- name: "{{.Matches}}"
cmdline:
- 'NodeManager'
- name: "{{.Matches}}"
cmdline:
- 'greenplum'
- name: "{{.Matches}}"
cmdline:
- 'hadoop'
- name: "{{.Matches}}"
cmdline:
- 'hbase'
- name: "{{.Matches}}"
cmdline:
- 'hudi'
- name: "{{.Matches}}"
cmdline:
- 'hive'
- name: "{{.Matches}}"
cmdline:
- 'zookeeper'
- name: "{{.Matches}}"
cmdline:
- 'QueryServer'
- name: "{{.Matches}}"
cmdline:
- 'QuorumPeerMain'
- name: "{{.Matches}}"
cmdline:
- 'java'
2.4.添加到系统服务中
vim /etc/systemd/system/process_exporter.service
-------------------------------配置如下内容--------------
[Unit] Description=process-exporter [Service] Type=simple WorkingDirectory=/tmp/ ExecStart=/opt/process_exporter/process_exporter --config.path=/opt/process_exporter/process-name.yaml --web.listen-address=:9256 Restart=on-failure [Install] WantedBy=multi-user.target
#注意：–web.listen-address=:9256 端口一定不能被占用，否则会报错，可以自己手动修改为任意端口
#授权可执行 chmod +x /etc/systemd/system/process_exporter.service #重载systemd系统 systemctl daemon-reload #启动服务 systemctl start process_exporter systemctl status process_exporter systemctl enable process_exporter
#检验是否安装成功，登录ip:9256进行查看
**
三、Prometheus安装与配置**
进入到http://prometheus.io官网
下载最新版本 prometheus-2.31.1.linux-amd64.tar.gz
这里只需要在一台机器上安装就好
//下载安装包：prometheus-2.31.1.linux-amd64.tar.gz
3.1.解压文件到/opt/目录下
tar -zxvf prometheus-2.31.1.linux-amd64.tar.gz -C /opt/
3.2.重命名
cd /opt mv prometheus-2.31.1.linux-amd64 prometheus cd prometheus
3.3.配置prometheus服务端
vim prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# alertmanager configuration:告警配置，集成alertmanager插件
alerting:
alertmanagers:
- static_configs:
- targets:
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: "prometheus"
static_configs:
- targets: ["192.168.0.113:9090"] #为当前机器ip
#通过配置file 获取target
- job_name: 'node_exporter'
file_sd_configs:
- files:
- /opt/prometheus/targets/nodes/*.json
3.4.创建我们的动态配置文件
vim /opt/prometheus/targets/nodes/host.json
[
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "113node",
"group": "node_exporter",
"host_ip": "192.168.0.113",
"hostname": "node2"
},
"targets": [
"192.168.0.113:9101"
]
},
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "114node",
"group": "node_exporter",
"host_ip": "192.168.0.114",
"hostname": "node3"
},
"targets": [
"192.168.0.114:9110"
]
},
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "111node",
"group": "node_exporter",
"host_ip": "192.168.0.111",
"hostname": "master"
},
"targets": [
"192.168.0.111:9100"
]
}
]
vim /opt/prometheus/targets/nodes/process.json
[
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "113node",
"group": "process_exporter",
"host_ip": "192.168.0.113",
"hostname": "node2"
},
"targets": [
"192.168.0.113:9256"
]
},
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "114node",
"group": "process_exporter",
"host_ip": "192.168.0.114",
"hostname": "node3"
},
"targets": [
"192.168.0.114:9256"
]
}
]
3.5.添加peometheus到系统服务脚本中
vim /etc/systemd/system/prometheus.service
[Unit] Description=Promtheus documentation=https://prometheus.io/ After=network.target [Service] User=root ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --web.listen-address=:9090 Type=simple Restart=on-failure [Install] WantedBy=multi-user.target
#启动脚本授权可执行 chmod +x /etc/systemd/system/prometheus.service #系统服务配置重新加载 systemctl daemon-reload #服务添加可随机启动 systemctl enable prometheus.service #启动服务 systemctl start prometheus.service #查看服务状态 systemctl status prometheus.service
3.6.登录验证：192.168.0.113:9090 如下
4.1.安装grafana
#下载安装包：grafana-8.2.5-1.x86_64.rpm 并上传到服务器
#1.使用yum安装
yum localinstall grafana-8.2.5-1.x86_64.rpm
#启动Grafana systemctl start grafana-server #停止Grafana systemctl stop grafana-server #重启Grafana systemctl restart grafana-server #设置开机自启动 systemctl enable grafana-server
#登录：192.168.0.113:3000,首次登录账号密码为：admin/admin
4.2.安装CM Datasource
将cloudera manager的数据信心作为datasource接入grafana进行可视化监控。
4.2.1安装插件
方法一：github下载，解压到指定位置
https://grafana.com/grafana/plugins/foursquare-clouderamanager-datasource/installation
4.2.2
下载完成后，上传服务器，解压到/var/lib/grafana/plugins
unzip foursquare-clouderamanager-datasource-0.9.3.zip
方法二：直接命令行安装
grafana-cli plugins install foursquare-clouderamanager-datasource
4.2.3重启grafana
systemctl restart grafana-server
延伸：grafana插件
grafana-cli plugins install grafana-clock-panel #时钟 grafana-cli plugins install kentik-app #网络流量 grafana-cli plugins install grafana-piechart-panel #饼图 grafana-cli plugins install novalabs-annotations-panel #注释 grafana-cli plugins install farski-blendstat-panel #混合 grafana-cli plugins install yesoreyeram-boomtable-panel #多表 grafana-cli plugins install yesoreyeram-boomtheme-panel #多主题 grafana-cli plugins install jeanbaptistewatenberg-percent-panel #百分比 grafana-cli plugins install corpglory-progresslist-panel #进程列表 grafana-cli plugins install mxswat-separator-panel #分隔符 grafana-cli plugins install aidanmountford-html-panel #网页
4.3.配置grafana
4.3.1添加数据源
配置prometheus数据源
配置CM插件数据源，前提：我们的cdh已经部署完成
4.3.2制作监控看板
这里我们直接将我给的模板导入即可。
4.4.看板展示
最终的页面如图展示
