logstash7.15.1读取ngnix的access日志文件并写入elasticsearch7.5.1

通过access日志记录，可以分析出并发访问量的大小，也可以分析访问的基本信息，比如请求ip地址，请求客户端信息以及访问的具体地址等。

input {
  file {
    start_position => beginning 
    path => "E:/logstash-test/access.log"
	type => "type1" ### 用去输出到es时判断存入哪个索引	
  }
 
}


filter{
	grok{
	  match => { "message" => "%{HTTPD_COMBINEDLOG}" }
	} ### 通过grok匹配内容并将
	date {        
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
		target => "@timestamp"
    }	
}


output {
  stdout{}
  elasticsearch {
   #es地址，可多个
   hosts => ["localhost:9200"]
    action => "index"
	#获取输出参数"indexname"值当做索引，如果没有则会自动创建对应索引(需要es开启自动创建索引)
    index => "qd_ngnix_access-%{+YYYY-MM}"
   }
}

其中，提起日志中的时间，作为日志时间。注意日期的格式应与日志文件中的日期格式一致。

date {        
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
		target => "@timestamp"
    }

日志文件参考如下：

223.104.190.204 - - [01/Mar/2022:03:50:01 +0800] "GET /mnt_qingdao/1055.jpg?random=fJeZCdWBhE HTTP/1.1" 200 140228 "https://www.demo.com/qingdao/dcep/" "Mozilla/5.0 (Linux; U; Android 11; zh-CN; NAM-AL00 Build/HUAWEINAM-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 UWS/3.22.2.18 Mobile Safari/537.36 UCBS/3.22.2.18_210803145558 ChannelId(9) NebulaSDK/1.8.100112 Nebula Bankabc/Portal BankabcAndroid/7.1.0 SDKVersion/30  mPaaSClient" "-"

logstash -f logstash2-ngnix.conf

COMMonAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{data:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

在kibana的Grok Debugger中进行验证：

参考数据：

223.104.190.204 - - [01/Mar/2022:03:50:01 +0800] "GET /mnt_qingdao/1055.jpg?random=fJeZCdWBhE HTTP/1.1" 200 140228 "https://www.demo.com/qingdao/dcep/" "Mozilla/5.0 (Linux; U; Android 11; zh-CN; NAM-AL00 Build/HUAWEINAM-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 UWS/3.22.2.18 Mobile Safari/537.36 UCBS/3.22.2.18_210803145558 ChannelId(9) NebulaSDK/1.8.100112 Nebula Bankabc/Portal BankabcAndroid/7.1.0 SDKVersion/30  mPaaSClient" "-"

Grok Pattern

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{data:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

结构化的数据：

{
  "request": "/mnt_qingdao/1055.jpg?random=fJeZCdWBhE",
  "agent": ""Mozilla/5.0 (Linux; U; Android 11; zh-CN; NAM-AL00 Build/HUAWEINAM-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 UWS/3.22.2.18 Mobile Safari/537.36 UCBS/3.22.2.18_210803145558 ChannelId(9) NebulaSDK/1.8.100112 Nebula Bankabc/Portal BankabcAndroid/7.1.0 SDKVersion/30  mPaaSClient"",
  "auth": "-",
  "ident": "-",
  "verb": "GET",
  "referrer": ""https://www.demo.com/qingdao/dcep/"",
  "response": "200",
  "bytes": "140228",
  "clientip": "223.104.190.204",
  "httpversion": "1.1",
  "timestamp": "01/Mar/2022:03:50:01 +0800"
}

验证界面类似：