目录

去掉部分属性

JSON日志扁平化

去掉部分属性

filebeat采集的日志格式默认如下：

        {
          "@timestamp": "2022-05-10T06:04:40.804Z",
          "beat": {
            "hostname": "honeypot-filebeat-f2mj9",
            "name": "honeypot-filebeat-f2mj9",
            "version": "5.6.16"
          },
          "input_type": "log",
          "message": """{"code":200,"msg":"今天是个好天气"}""",
          "offset": 4930,
          "source": "/var/log/honeyrouter.log",
          "type": "log"
        }

beat、input_type、offset等字段都是filebeat加上去的，如果想要去掉这些不需要的key可以通过drop_fields配置删除这些属性，具体配置如下：

filebeat.prospectors:
    - input_type: log
      paths:
        - /var/log/*

    processors:
    - drop_fields:
        fields: ["input_type", "offset", "beat", "type"]

    output.elasticsearch:
      hosts: ['host:port']
      username: user
      password: pwd

JSON日志扁平化

通过采集到的日志文件查看可以发现所有的日志都放到了message属性中，如果想将message中JSON格式的日志拆分到json root下可以通过json.keys_under_root进行配置，具体配置如下：

filebeat.prospectors:
    - input_type: log
      paths:
        - /var/log/*
      json:
        keys_under_root: true

    processors:

    output.elasticsearch:
      hosts: ['host:port']
      username: user
      password: pwd

修改配置后，查看采集到的日志：

{
          "@timestamp": "2022-05-10T06:11:33.428Z",
          "beat": {
            "hostname": "honeypot-filebeat-rjdx5",
            "name": "honeypot-filebeat-rjdx5",
            "version": "5.6.16"
          },
          "code": 200,
          "input_type": "log",
          "msg": "今天是个好天气",
          "offset": 4973,
          "source": "/var/log/honeyrouter.log",
          "type": "log"
        }