SpringBoot利用ELK实现日志收集

ELK是Elasticsearch、Logstash、Kibana他们三个组合起来可以搭建日志系统，本文主要记录使

用ELK收集SoringBoot应用产生的日志

Elasticsearch、Logstash、Kibana作用

Elasticsearch:存储日志信息

Logstash: 日志收集，springboot利用Logstash把日志发送个Logstash，然后Logstash将日志传递

给Elasticsearch。

Kibana:通过web端对日志进行可视化操作对Elasticsearch安装

下载Elasticsearch镜像

docker pull Elasticsearch:7.6.2

修改虚拟内存地址，否则可能出现内存过小无法启动

sysctl -w vm.max_map_count=262144

启动Elasticsearch服务:

docker run -p 9200:9200 -p 9300:9300 --name elasticsearch  
-e "discovery.type=single-node"  
-e "cluster.name=elasticsearch"  
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins  
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data  
-d elasticsearch:7.6.2

启动时/usr/share/elasticsearch会出现没有访问权限，需要修改/mydata/elasticsearch/data/权

限,然后重新启动elasticsearch

chmod 777 /mydata/elasticsearch/data/

安装IKAnalyzer中文分词器,并重新启动:

docker exec -it elasticsearch /bin/bash 
#此命令需要在容器中运行 
elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis- ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip 
docker restart elasticsearch

注:离线安装elasticsearch中插件

1.下载elasticsearch-analysis-ik-7.6.2.zip

https://github.com/medcl/elasticsearch-analysis- ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip

2.上传到linux

3.上传的linux复制到elasticsearch容器中

docker cp elasticsearch-analysis-ik-7.6.2.zip elasticsearch:/

4.安装插件

docker exec -it elasticsearch /bin/bash elasticsearch-plugin install file:elasticsearch-analysis-ik-7.6.2.zip docker restart elasticsearch

如果防火墙没有关闭

firewall-cmd --zone=public --add-port=9200/tcp --permanent 
firewall-cmd --reload

安装Logstash的Docker镜像

1.下载Logstash镜像

docker pull logstash:7.6.2

2.添加Logstash配置文件logstash.conf

input {
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json_lines
    type => "manage"
  }
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4561
    codec => json_lines
    type => "star"
  }
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4562
    codec => json_lines
    type => "love"
  }
}
filter{
  if [type] == "record" {
    mutate {
      remove_field => "port"
      remove_field => "host"
      remove_field => "@version"
    }
    json {
      source => "message"
      remove_field => ["message"]
    }
  }
}
output {
  elasticsearch {
    hosts => "es:9200"
    index => "leinfty-%{type}-%{+YYYY.MM.dd}"
  }
}

3.创建/mydata/logstash，将logstash.conf拷贝到该目录

mkdir /mydata/logstash

4.启动logstash

docker run --name logstash -p 4560:4560 -p 4561:4561 -p 4562:4562 
--link elasticsearch:es 
-v /mydata/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf 
-d logstash:7.6.2

Kibana安装

1.下载Kibana镜像

docker pull kibana:7.6.2

2.启动Kibana

docker run --name kibana -p 5601:5601 
--link elasticsearch:es 
-e "elasticsearch.hosts=http://es:9200" 
-d kibana:7.6.2

3.如果防火墙没有关闭

firewall-cmd --zone=public --add-port=5601/tcp --permanent 
firewall-cmd --reload

4.将kibana变为中文

docker exec -it kibana bash 
cd config 
vi kibana.yml

5.在kibana.yml中添加

il8n.locale:"zh-CN"

6.访问http://xxxx:5601进行测试

SpringBoot集成Logstash 添加Logstash依赖

 
 
    net.logstash.logback 
    logstash-logback-encoder
    5.3

添加配置文件logback-spring.xml，使得logbach日志输入到logstash




    
    
    
    
    
    
    ${APP_NAME}
    
    
        
            ${LOG_FILE_PATH}/${APP_NAME}-%d{yyyy-MM-dd}.log
            30
        
        
            ${FILE_LOG_PATTERN}
        
    
    
    
        
        ip:4562

在application.yml中添加配置进行测试

logging: 
    file: 
        path: /var/logs 
    level: 
        root: info 
    config: classpath:logback-spring.xml

查看收集的日志

1.创建索引

权限控制

进入es容器

docker exec -it elasticsearch bash

修改配置

vi config/elasticsearch.yml

启用安全配置

xpack.security.enabled: true

xpack.license.self_generated.type: basic

xpack.security.transport.ssl.enabled: true

重启es容器

exit
docker restart elasticsearch

设置密码

docker exec -it elasticsearch bash

bin/elasticsearch-setup-passwords interactive

按提示填入各类应用的密码

进入kibana容器

docker exec -it kibana bash

配置kibana连接elastic的设置

vi config/kibana.yml

elasticsearch.username: "elastic"
elasticsearch.password: "xxx"

重启kibana容器

docker restart kibana

配置logstash连接elastic的设置

vi /mydata/logstash/logstash.conf

output {
  elasticsearch {
    hosts => "es:9200"
    user => "elastic"
    password => "xxx"
    index => "leinfty-%{type}-%{+YYYY.MM.dd}"
  }
}

重启logstash

docker restart logstash

验证账号登录

SpringBoot利用ELK实现日志收集

大数据系统相关栏目本月热门文章