prometheus、influxdb安装及flume

磊prometheus
- 賂创建用户
- 賂下载解压部署
- - 雷下载
  - 雷解压
  - 雷部署启动
- 賂登录验证
- 賂数据存储
磊influxdb
- 賂创建用户
- 賂下载解压部署
- - 雷下载
  - 雷解压部署
- 賂登录验证
- - 雷初始化用户信息，全都是必填项
  - 雷查看数据库信息
- 賂prometheus指定数据库配置
- - 雷prometheus.yml
  - 雷重新加载
  - 雷查看
磊flume_exporter
- 賂创建目录
- 賂下载
- 賂编译
- - 雷安装go环境
  - 雷编译flume_exporter项目
  - - 设置变量，初始化flume目录
    - 安装依赖包
    - 指定编译目录，执行安装
- 賂启动
- - 雷简单部署flume
- 賂promethues配置flume_exporter
- - 雷加载验证
  - 雷influx查看flume指标数据
  - -  安装telegrafs
    -  prometheus修改db配置
    -  再次查看flume指标数据
磊其它
- 賂查看系统及处理器型号
- 賂lsb_release: 未找到命令
- 賂解压报错
- 賂 influxdb2.0 写入数据库必须要telegrafs嘛

磊prometheus

一款监控告警软件，详见官方资料

賂创建用户

创建部署用户，赋予sudo权限

[root@host1 ~]# useradd prometheus
[root@host1 ~]# passwd prometheus
更改用户 prometheus 的密码 。
新的 密码：
无效的密码： 密码少于 8 个字符
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。
[root@host1 ~]# echo 'prometheus ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
[root@host1 ~]# su - prometheus
[prometheus@host1 ~]$ sudo -v
[prometheus@host1 ~]$

賂下载解压部署雷下载

prometheus下载地址

选择最新版amd64（网速差，可以先下载到本地，再上传到服务器）

wget https://github.com/prometheus/prometheus/releases/tag/v2.35.0/prometheus-2.35.0.linux-amd64.tar.gz  --no-check-certificate

雷解压

解压修改配置文件，默认localhost替换为具体地址

[prometheus@host1 ~]$ tar xf prometheus-2.35.0-rc0.linux-amd64.tar.gz 
[prometheus@host1 ~]$ ll
总用量 78748
drwxr-xr-x. 4 prometheus prometheus      132 4月   8 21:41 prometheus-2.35.0-rc0.linux-amd64
-rw-r--r--. 1 prometheus prometheus 80636489 5月   9 10:50 prometheus-2.35.0-rc0.linux-amd64.tar.gz
[prometheus@host1 ~]$ mv prometheus-2.35.0-rc0.linux-amd64 prometheus
[prometheus@host1 ~]$ cd prometheus
[prometheus@host1 prometheus]$ ll
总用量 198240
drwxr-xr-x. 2 prometheus prometheus        38 4月   8 21:38 console_libraries
drwxr-xr-x. 2 prometheus prometheus       173 4月   8 21:38 consoles
-rw-r--r--. 1 prometheus prometheus     11357 4月   8 21:38 LICENSE
-rw-r--r--. 1 prometheus prometheus      3773 4月   8 21:38 NOTICE
-rwxr-xr-x. 1 prometheus prometheus 105568448 4月   8 21:29 prometheus
-rw-r--r--. 1 prometheus prometheus       934 4月   8 21:38 prometheus.yml
-rwxr-xr-x. 1 prometheus prometheus  97405202 4月   8 21:33 promtool
[prometheus@host1 prometheus]$ sed -i 's/localhost/192.168.56.10/g' prometheus.yml 
[prometheus@host1 prometheus]$ pwd
/home/prometheus/prometheus

雷部署启动

nohup /home/prometheus/prometheus/prometheus --config.file=/home/prometheus/prometheus/prometheus.yml --web.enable-lifecycle  > prometheus.log 2>&1 &

参数说明：

nohup “xxxx” & ：后端启动
prometheus ：执行命令文件
–config.file ：指定配置文件
–web.enable-lifecycle：支持通过http请求重载配置（热加载）
prometheus.log 2>&1：>-覆盖写入日志、prometheus.log-日志名称、2>&1-写入全部日志，不管对错
查看日志，对应监听端口为9090

賂登录验证

登录页面查看

賂数据存储

默认情况下prometheus会将采集的数据存放到本地磁盘data下面（二进制文件），为了方便查看和扩展，接下来使用influxdb来存储数据。

磊influxdb 賂创建用户

创建部署用户，赋予sudo权限

[root@host1 ~]# useradd influx
[root@host1 ~]# passwd influx
更改用户 influx 的密码 。
新的 密码：
无效的密码： 密码少于 8 个字符
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。
[root@host1 ~]# echo 'influx ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
[root@host1 ~]# su -  influx
[influx@host1 ~]$ sudo -v
[influx@host1 ~]$

賂下载解压部署雷下载

官方资料，目前最新版本2.2

wget https://dl.influxdata.com/influxdb/releases/influxdb2-2.2.0-linux-amd64.tar.gz

雷解压部署

[influx@host1 ~]$ tar xf influxdb2-2.2.0-linux-amd64.tar.gz 
[influx@host1 ~]$ mv influxdb2-2.2.0-linux-amd64 influxdb
[influx@host1 ~]$ cd influxdb
[influx@host1 influxdb]$ ll
总用量 169660
-rwxr-xr-x. 1 influx influx 173712472 4月   7 01:39 influxd
-rw-rw-r--. 1 influx influx      1067 4月   7 01:40 LICENSE
-rw-rw-r--. 1 influx influx      9834 4月   7 01:40 README.md
[influx@host1 influxdb]$ nohup ./influxd &

查看日志，默认启动端口8086

賂登录验证

登录界面

雷初始化用户信息，全都是必填项

雷查看数据库信息

賂prometheus指定数据库配置雷prometheus.yml

添加如下配置

# set db
remote_write:
  - url: "http://192.168.56.10:8086/api/v1/prom/write?db=prometheus&u=prometheus&p=prometheus"
remote_read:
  - url: "http://192.168.56.10:8086/api/v1/prom/read?db=prometheus&u=prometheus&p=prometheus"

雷重新加载

直接刷新页面没反应，需执行如下命令

 curl -X POST http://192.168.56.10:9090/-/reload

雷查看

磊flume_exporter 賂创建目录

直接在prometheus用户下面建立exporter目录

[prometheus@host1 ~]$ ll
总用量 78748
drwxr-xr-x. 5 prometheus prometheus      166 5月  15 16:36 prometheus
-rw-r--r--. 1 prometheus prometheus 80636489 5月   9 10:50 prometheus-2.35.0-rc0.linux-amd64.tar.gz
[prometheus@host1 ~]$ mkdir exporter
[prometheus@host1 ~]$ cd exporter/
[prometheus@host1 exporter]$ ll
总用量 0
[prometheus@host1 exporter]$ 
[prometheus@host1 exporter]$

賂下载

下载地址，下载下来的是源文件，还要进行编译（网上没找到编译好的压缩包）

賂编译雷安装go环境

下载地址

解压配置环境变量

[prometheus@host1 ~]$ tar xf go1.18.1.linux-amd64.tar.gz 
[prometheus@host1 ~]$ ll
总用量 217128
drwxrwxr-x.  3 prometheus prometheus       106 5月  15 17:45 exporter
drwxr-xr-x. 10 prometheus prometheus       257 4月  12 23:42 go
-rw-r--r--.  1 prometheus prometheus 141699677 5月   9 15:22 go1.18.1.linux-amd64.tar.gz
drwxr-xr-x.  5 prometheus prometheus       166 5月  15 16:36 prometheus
-rw-r--r--.  1 prometheus prometheus  80636489 5月   9 10:50 prometheus-2.35.0-rc0.linux-amd64.tar.gz
[prometheus@host1 ~]$ vi .bash_profile 
[prometheus@host1 ~]$ . .bash_profile 
[prometheus@host1 ~]$ go version
go version go1.18.1 linux/amd64
[prometheus@host1 ~]$

.bash_profile 文件配置如下

export GOROOT=$HOME/go
export PATH=$PATH:$GOROOT/bin

雷编译flume_exporter项目 设置变量，初始化flume目录

[prometheus@host1 exporter]$ ll
总用量 67908
-rw-r--r--. 1 prometheus prometheus 67938106 5月   9 16:45 apache-flume-1.9.0-bin.tar.gz
drwxrwxr-x. 6 prometheus prometheus      267 2月  20 2019 flume_exporter-0.0.2
-rw-r--r--. 1 prometheus prometheus  1594290 5月  15 17:05 flume_exporter-0.0.2.tar.gz
[prometheus@host1 exporter]$ go env -w GO111MODULE=on
[prometheus@host1 exporter]$ go mod init flume_exporter-0.0.2
go: creating new go.mod: module flume_exporter-0.0.2
go: to add module requirements and sums:
	go mod tidy
[prometheus@host1 exporter]$ go env -w GOPROXY=https://goproxy.cn
[prometheus@host1 exporter]$

进入flume项目，执行安装

安装依赖包

下载logrus

[prometheus@host1 flume_exporter-0.0.2]$ go get github.com/Sirupsen/logrus
go: github.com/Sirupsen/logrus@v1.8.1: parsing go.mod:
	module declares its path as: github.com/sirupsen/logrus
	        but was required as: github.com/Sirupsen/logrus

下载余下依赖

[prometheus@host1 flume_exporter-0.0.2]$ go get github.com/prometheus/client_golang/prometheus &&go get github.com/prometheus/client_golang/prometheus/promhttp && go get github.com/prometheus/common/version && go get github.com/woozhijun/flume_exporter/exporter && go get gopkg.in/alecthomas/kingpin.v2

指定编译目录，执行安装

[prometheus@host1 flume_exporter-0.0.2]$  export GOPATH=/home/prometheus/exporter/flume_exporter-0.0.2
[prometheus@host1 flume_exporter-0.0.2]$ go install 
go: downloading github.com/Sirupsen/logrus v1.0.6
go: downloading github.com/prometheus/client_golang v1.12.2
go: downloading github.com/prometheus/common v0.34.0
go: downloading github.com/woozhijun/flume_exporter v0.0.2
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading golang.org/x/sys v0.0.0-20220114195835-da31bd327af9
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/prometheus/procfs v0.7.3
go: downloading google.golang.org/protobuf v1.26.0
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/bitly/go-simplejson v0.5.0
go: downloading gopkg.in/yaml.v2 v2.4.0

查看生成的文件，统一管理

賂启动

nohup /home/prometheus/exporter/flume_exporter/flume_exporter-0.0.2 --config-file=/home/prometheus/exporter/flume_exporter/config.yml --metric-file=/home/prometheus/exporter/flume_exporter/metrics.yml >flumexport.log 2>&1 &

config.yml 配置flume机器IP信息，为了查看效果，只能再搭建一个flume单机版了

雷简单部署flume

下载地址
下载解压

配置java变量

[prometheus@host1 flume]$ cd conf/
[prometheus@host1 conf]$ ls
flume-conf.properties.template  flume-env.ps1.template  flume-env.sh.template  log4j.properties
[prometheus@host1 conf]$ cp flume-env.sh.template flume-env.sh
[prometheus@host1 conf]$ cp flume-conf.properties.template flume-conf.properties
[prometheus@host1 conf]$ vi flume-env.sh

定义相关参数

[prometheus@host1 conf]$ vi flume-conf.properties
[prometheus@host1 conf]$ cat flume-conf.properties
#Agent: 名称定义为 a1
#Source:可以理解为输入端，定义名称为 r1
#channel：传输频道，定义为 c1，设置为内存模式
#sinks：可以理解为输出端，定义为 s1

＃命名此代理上的组件别名
a1.sources = r1
a1.channels = c1
a1.sinks = s1

＃描述配置源
a1.sources.r1.type = netcat 
a1.sources.r1.bind = 192.168.56.10
a1.sources.r1.port = 36001

＃描述接收器
#设置 Sink 为 logger 模式 
a1.sinks.s1.type = logger 

#设置 channel 信息 
a1.channels.c1.type = memory 
#内存模式 
a1.channels.c1.capacity = 1000 
#通道中最大的缓冲事件,要大于或等于batchSize的数
a1.channels.c1.transactionCapacity = 100   

＃将源和接收器绑定到通道
a1.sources.r1.channels  =  c1 
a1.sinks.s1.channel  =  c1
[prometheus@host1 conf]$

启动flume

nohup /home/prometheus/flume/bin/flume-ng agent -n a1 -c conf -f /home/prometheus/flume/conf/flume-conf.properties -Dflume.monitoring.type=http -Dflume.monitoring.port=36001 -Dflume.root.logger=INFO,console &

賂promethues配置flume_exporter

prometheus.yml 添加配置信息

# set flume export
  - job_name: flume
    file_sd_configs:
      - files:
        - '/home/prometheus/exporter/flume_exporter/flume-nodes.json'
    metric_relabel_configs:
      - source_labels: [192.168.56.10]
        regex: '192.168.56.10:36001'
        replacement: flume-92.168.56.10-36001
        target_label: logs
        action: replace

创建json配置文件

[prometheus@host1 prometheus]$ cd /home/prometheus/exporter/flume_exporter
[prometheus@host1 flume_exporter]$ vi flume-nodes.json
[prometheus@host1 flume_exporter]$ cat flume-nodes.json 
[{
    "targets": ["192.168.56.10:9360"],
    "labels": {
        "alias": "flume1",
        "job": "flume"
    }
}]
[prometheus@host1 flume_exporter]$

雷加载验证

[prometheus@host1 prometheus]$ curl -X POST http://192.168.56.10:9090/-/reload
[prometheus@host1 prometheus]$

监控目标

雷influx查看flume指标数据

数据并未写入influxdb,看页面，还需要创建telegraf,而且上面配置的api应该也不对，里面有v1,实际上安装的是v2

 安装telegrafs

下载

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.22.3_linux_amd64.tar.gz

登录influxdb,拷贝前面定义的telegrafs的内容，添加到telegraf.conf配置文件，同时配置环境变量

export INFLUX_TOKEN=-jExfGZAinmMYdY98xQLuKK3xtuJdjvFasLBq_X8wb_gtnkgdHozcb0ph-gW-8C8xFtx3JFIM7irEwKe2kjkxw==

telegraf.conf 内容（增加了inputs.prometheus和inputs.http_listener_v2）

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false
[[outputs.influxdb_v2]]
  urls = ["http://192.168.56.10:8086"]
  token = "$INFLUX_TOKEN"
  organization = "prometheus"
  bucket = "prometheus"
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.prometheus]]
  urls = ["http://192.168.56.10:9090/metrics","http://192.168.56.10:9360/metrics"]
[[inputs.http_listener_v2]]
  service_address = ":8087"
  paths = ["/receive"]
  data_format = "influx"
[influx@host1 telegraf]$

启动telegraf

nohup /home/influx/telegraf-1.22.3/usr/bin/telegraf --config /home/influx/telegraf-1.22.3/etc/telegraf/telegraf.conf  &

 prometheus修改db配置

修改为remote_write

remote_write:
   - url: "http://192.168.56.10:8087/receive"

重新加载

[prometheus@host1 prometheus]$ curl -X POST http://192.168.56.10:9090/-/reload

 再次查看flume指标数据

注意：[[inputs.prometheus]]对应下面targets,配置几个采集几个！！！

[[inputs.prometheus]]
  urls = ["http://192.168.56.10:9090/metrics","http://192.168.56.10:9360/metrics"]

磊其它賂查看系统及处理器型号

[prometheus@host1 ~]$ uname -a
Linux host1 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[prometheus@host1 ~]$ 
[prometheus@host1 ~]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.4.1708 (Core) 
Release:	7.4.1708
Codename:	Core
[prometheus@host1 ~]$

賂lsb_release: 未找到命令

[prometheus@host1 ~]$ lsb_release -a
-bash: lsb_release: 未找到命令
[prometheus@host1 ~]$ yum provides lsb_release
已加载插件：fastestmirror
base                                                                                                           | 3.6 kB  00:00:00     
extras                                                                                                         | 2.9 kB  00:00:00     
mysql-connectors-community                                                                                     | 2.6 kB  00:00:00     
mysql-tools-community                                                                                          | 2.6 kB  00:00:00     
mysql80-community                                                                                              | 2.6 kB  00:00:00     
updates                                                                                                        | 2.9 kB  00:00:00     
(1/4): mysql-tools-community/x86_64/primary_db                                                                 |  86 kB  00:00:01     
(2/4): mysql-connectors-community/x86_64/primary_db                                                            |  87 kB  00:00:01     
(3/4): updates/7/x86_64/primary_db                                                                             |  15 MB  00:00:01     
(4/4): mysql80-community/x86_64/primary_db                                                                     | 211 kB  00:00:02     
Determining fastest mirrors
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
base/7/x86_64/filelists_db                                                                                     | 7.2 MB  00:00:01     
extras/7/x86_64/filelists_db                                                                                   | 277 kB  00:00:00     
mysql-connectors-community/x86_64/filelists_db                                                                 |  35 kB  00:00:00     
mysql-tools-community/x86_64/filelists_db                                                                      | 325 kB  00:00:01     
mysql80-community/x86_64/filelists_db                                                                          | 1.9 MB  00:00:01     
updates/7/x86_64/filelists_db                                                                                  | 8.4 MB  00:00:01     
redhat-lsb-core-4.1-27.el7.centos.1.i686 : LSB Core module support
源    ：base
匹配来源：
文件名    ：/usr/bin/lsb_release



redhat-lsb-core-4.1-27.el7.centos.1.x86_64 : LSB Core module support
源    ：base
匹配来源：
文件名    ：/usr/bin/lsb_release



[prometheus@host1 ~]$ sudo yum install -y redhat-lsb

賂解压报错

包未下载完整，网速太差，可以浏览器下载之后，丢到服务器

賂 influxdb2.0 写入数据库必须要telegrafs嘛

看2.0API官网说明不用好像也行，问题是不知道prometheus该怎么配。Java开发、curl模拟都ok

prometheus、influxdb安装及flume

Linux相关栏目本月热门文章