Beats入门案例

轻量型数据采集器Beats是一个免费且开放的平台，集合了多种单一用途数据采集器。它们从成百上千或成千上万台机器和系统向 Logstash 或 Elasticsearch 发送数据。

如果采集数据不需要任何处理，那么可以直接发送到Elasticsearch中。
如果采集的数据需要处理，那么可以发送到Logstash中，处理完成后再发送到Elasticsearch。最后通过Kibana对数据进行一系列的可视化展示。

Filebeat是一款轻量型日志采集器，用于监控、收集服务器日志文件

首先Filebeat指定一些日志文件为数据输入源，之后使用Harvester（收割机）源源不断的读取日志，最后通过Spooler（卷轴）将日志数据传送到对应的目的地。

1. 使用rz工具将Filebeat压缩文件上传到Linux虚拟机
2. 解压：
   tar -zxvf filebeat-7.12.1-linux-x86_64.tar.gz -C /opt/

接下来我们使用filebeat读取一个普通的日志文件

1. 创建一个文本文件
   vim /usr/local/mylog.log # 为该文件随便添加一句话
   
2. 在filebeat中创建配置文件，配置文本文件的读取参数

# 进入filebeat文件夹下 
cd /opt/filebeat-7.12.1-linux-x86_64/
# 创建配置文件 
vim mylogconfig.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/log/mylog.log
output.console:
  pretty: true
  enable: true

3. 基于配置文件启动filebeat

./filebeat -e -c mylogconfig.yml

参数说明：
-e：标准输出，输出到控制台
-c：指定配置文件
4. 向文本文件追加数据，测试filebeat是否能为增量数据生成日志数据

# 打开另一个会话窗口，进入文本文件的目录下 
cd /opp/
# 向文本文件中追加内容，再次查看filebeat的控制台 
echo '科比是我的偶像' >> mylog.log

Filebeat读取日志文件后会生成json格式的日志，我们还可以为生成的日志添加一些自定义字段：

# 修改配置文件：
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/log/mylog.log
  tags: ["mylog","test"]#添加自定义标签，便于后续处理
  fields:
    from: mylog
  fields_under_root: true #true为添加到根节点，false为添加到子节点中
output.console:
  pretty: true
  enable: true
# 重启
filebeat ./filebeat -e -c mylogconfig.yml
# 向文本文件追加数据 
echo '科比是我的偶像' >> mylog.log

我们可以看到生成的日志数据多了两个字段

1. 安装Nginx

tar -zxvf nginx-1.21.1.tar.gz -C /usr/local/
# 安装依赖包
yum -y install gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel
# 安装Nginx
./configure
make & make install
# 启动Nginx
/usr/local/nginx/sbin/nginx
./nginx

Nginx的日志文件在/usr/local/nginx/logs中，正常日志存在access.log中，异常日志存在error.log
中。

2. 读取Nginx日志的配置文件

# 在filebeat中创建配置文件
cd /usr/local/filebeat-7.12.1-linux-x86_64/
vim nginxlogconfig.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /usr/local/nginx/logs/*.log 
  tags: ["nginx"]
output.console:
  pretty: true
  enable: true


# 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat 
./filebeat -e -c nginxlogconfig.yml

在收集Ngnix日志时，日志内容并没有处理，难以阅读其中的具体数据。Filebeat针对常见的服务提供了处理日志的模板。接下来我们讲解Filebeat中Module的使用。

1. 配置Nginx读取模板：

# 查看Filebeat的日志处理模板
 ./filebeat modules list
 
[root@node0 filebeat]# ./filebeat modules list
Enabled:
nginx

Disabled:
activemq
apache
auditd
aws
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
crowdstrike
cyberark
cylance
elasticsearch
envoyproxy
f5
fortinet
gcp
google_workspace
googlecloud
gsuite
haproxy
ibmmq
icinga
iis
imperva
infoblox
iptables
juniper
kafka
kibana
logstash
microsoft
misp
mongodb
mssql
mysql
mysqlenterprise
nats
netflow
netscout
o365
okta
oracle
osquery
panw
pensando
postgresql
proofpoint
rabbitmq
radware
redis
santa
snort
snyk
sonicwall
sophos
squid
suricata
system
threatintel
tomcat
traefik
zeek
zoom
zscaler

# 启用模板 
./filebeat modules enable nginx
cd modules.d/ 
vim nginx.yml
# 配置日志处理模板
- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/usr/local/nginx/logs/access.log"]
  # Error logs
  error:
    enabled: true
    var.paths: ["/usr/local/nginx/logs/error.log"]

2. 修改配置文件：

vim nginxlogconfig.yml

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml 
  reload.enabled: true
output.console:
  pretty: true
  enable: true

3. 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat

./filebeat -e -c nginxlogconfig.yml

{
  "@timestamp": "2021-12-03T04:57:55.131Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.1",
    "pipeline": "filebeat-7.12.1-nginx-access-pipeline"
  },
  "event": {
    "dataset": "nginx.access",
    "module": "nginx",
    "timezone": "+08:00"
  },
  "fileset": {
    "name": "access"
  },
  "ecs": {
    "version": "1.8.0"
  },
  "host": {
    "name": "node0"
  },
  "log": {
    "offset": 1746,
    "file": {
      "path": "/usr/local/nginx/logs/access.log"
    }
  },
  "message": "192.168.134.1 - - [03/Dec/2021:12:57:51 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"",
  "input": {
    "type": "log"
  },
  "service": {
    "type": "nginx"
  },
  "agent": {
    "id": "1043e956-ee86-47e0-8edd-084cab906fd9",
    "name": "node0",
    "type": "filebeat",
    "version": "7.12.1",
    "hostname": "node0",
    "ephemeral_id": "dda18f29-2c37-49e9-b1b0-755f61cf47ea"
  }
}

1. 启动Elasticsearch
2. 启动Kibana，连接Elasticsearch
3. 修改Filebeat配置文件：

vim nginxlogconfig.yml

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml 
  reload.enabled: true
output.elasticsearch:
  hosts: ["node0:9200"]

4. 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat

./filebeat -e -c nginxlogconfig.yml

5. 进入Kibana查看数据

GET /filebeat-7.12.1/_search
{
  "query": {
    "match_all": {}
  }
}

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.12.1",
        "_type" : "_doc",
        "_id" : "K4XUfn0BLb2Jb20VAvp9",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "hostname" : "node0",
            "name" : "node0",
            "id" : "1043e956-ee86-47e0-8edd-084cab906fd9",
            "ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af",
            "type" : "filebeat",
            "version" : "7.12.1"
          },
          "nginx" : {
            "access" : {
              "remote_ip_list" : [
                "192.168.134.1"
              ]
            }
          },
          "log" : {
            "file" : {
              "path" : "/usr/local/nginx/logs/access.log"
            },
            "offset" : 2319
          },
          "source" : {
            "address" : "192.168.134.1",
            "ip" : "192.168.134.1"
          },
          "fileset" : {
            "name" : "access"
          },
          "url" : {
            "original" : "/"
          },
          "input" : {
            "type" : "log"
          },
          "@timestamp" : "2021-12-03T05:46:00.000Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "related" : {
            "ip" : [
              "192.168.134.1"
            ]
          },
          "service" : {
            "type" : "nginx"
          },
          "host" : {
            "name" : "node0"
          },
          "http" : {
            "request" : {
              "method" : "GET"
            },
            "response" : {
              "status_code" : 304,
              "body" : {
                "bytes" : 0
              }
            },
            "version" : "1.1"
          },
          "event" : {
            "ingested" : "2021-12-03T05:46:02.766382193Z",
            "timezone" : "+08:00",
            "created" : "2021-12-03T05:46:01.739Z",
            "kind" : "event",
            "module" : "nginx",
            "category" : [
              "web"
            ],
            "type" : [
              "access"
            ],
            "dataset" : "nginx.access",
            "outcome" : "success"
          },
          "user_agent" : {
            "original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
            "os" : {
              "name" : "Windows",
              "version" : "10",
              "full" : "Windows 10"
            },
            "name" : "Chrome",
            "device" : {
              "name" : "Other"
            },
            "version" : "93.0.4577.82"
          }
        }
      },
      {
        "_index" : "filebeat-7.12.1",
        "_type" : "_doc",
        "_id" : "LIXUfn0BLb2Jb20VBPp8",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "hostname" : "node0",
            "name" : "node0",
            "id" : "1043e956-ee86-47e0-8edd-084cab906fd9",
            "type" : "filebeat",
            "ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af",
            "version" : "7.12.1"
          },
          "nginx" : {
            "access" : {
              "remote_ip_list" : [
                "192.168.134.1"
              ]
            }
          },
          "log" : {
            "file" : {
              "path" : "/usr/local/nginx/logs/access.log"
            },
            "offset" : 2510
          },
          "source" : {
            "address" : "192.168.134.1",
            "ip" : "192.168.134.1"
          },
          "fileset" : {
            "name" : "access"
          },
          "url" : {
            "original" : "/"
          },
          "input" : {
            "type" : "log"
          },
          "@timestamp" : "2021-12-03T05:46:02.000Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "related" : {
            "ip" : [
              "192.168.134.1"
            ]
          },
          "service" : {
            "type" : "nginx"
          },
          "host" : {
            "name" : "node0"
          },
          "http" : {
            "request" : {
              "method" : "GET"
            },
            "response" : {
              "status_code" : 304,
              "body" : {
                "bytes" : 0
              }
            },
            "version" : "1.1"
          },
          "event" : {
            "ingested" : "2021-12-03T05:46:03.760599179Z",
            "timezone" : "+08:00",
            "created" : "2021-12-03T05:46:02.757Z",
            "kind" : "event",
            "module" : "nginx",
            "category" : [
              "web"
            ],
            "type" : [
              "access"
            ],
            "dataset" : "nginx.access",
            "outcome" : "success"
          },
          "user_agent" : {
            "original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
            "os" : {
              "name" : "Windows",
              "version" : "10",
              "full" : "Windows 10"
            },
            "name" : "Chrome",
            "device" : {
              "name" : "Other"
            },
            "version" : "93.0.4577.82"
          }
        }
      }
    ]
  }
}