[2022-07-14 10:03:30.310] [http-nio-8080-exec-2] [ERROR] (com.zz.framework.exception.handler.GlobalExceptionHandler:83) - 数据校验异常,异常字段下载
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.4-x86_64.rpm wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.4-x86_64.rpm wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.4-x86_64.rpm rpm包wget https://artifacts.elastic.co/downloads/enterprise-search/enterprise-search-7.17.4.rpm 或者wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.4-linux-x86_64.tar.gz安装配置elasticsearch 安装jdk环境
● 下载安装包
○ 登陆官网下载(需要oracle账号)
■ https://www.oracle.com/java/technologies/downloads/#java8
解压并配置环境变量
tar xf jdk-8u311-linux-x64.tar.gz mv jdk1.8.0_311 /usr/local/jdk cat >> /etc/profile <配置系统参数 ● 设置内核参数 echo "vm.max_map_count=655360" >> /etc/sysctl.conf ● 执行以下命令确保配置生效。 sysctl -p ● 设置资源参数 ○ # vi /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 * soft nproc 65536 * hard nproc 131072 设置es启动用户资源参数 echo "elastic soft nproc 65536" >> /etc/security/limits.d/20-nproc.conf解压elasticsearch到指定目录tar xf elasticsearch-7.17.4-linux-x86_64.tar.gz -C /usr/local/ mkdir -p /data/es/{data,logs} chown -R elastic.elastic /usr/local/elasticsearch-7.17.4 chown -R elastic.elastic /data/es su - elastic cat >> /usr/local/elasticsearch-7.17.4/config/elasticsearch.yml <设置密码 /usr/local/elasticsearch-7.17.4/bin/elasticsearch-setup-passwords interactive后台启动/usr/local/elasticsearch-7.17.4/bin/elasticsearch -d查看日志验证 安装配置logstash，并启动cat > /etc/logstash/conf.d/app-java.conf << EOF input { beats { port => 5044 } } filter { grok { match => [ "message", "[%{DATA:logtime}] " ] break_on_match => false } date { match=> ["logtime","yyyy-MM-dd HH:mm:ss.SSS"] target=>"@timestamp" } } output { if "jAPPname" in [tags] { elasticsearch { hosts => ["elasticsearch集群ip:9200"] index => "java-APPname-%{+YYYY-MM-dd}" user => "elastic" password => "123456" } }else if "APPname" in [tags] { elasticsearch { hosts => ["elasticsearch集群ip:9200"] index => "java-APPname-%{+YYYY-MM-dd}" user => "elastic" password => "123456" } }else { elasticsearch { hosts => ["elasticsearch集群ip:9200"] index => "java-%{+YYYY-MM-dd}" user => "elastic" password => "123456" } } } EOF systemctl start logstash安装配置filebeat,并启动rpm -ivh filebeat-7.17.4-x86_64.rpm cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak cat > /etc/filebeat/filebeat.yml << EOF filebeat.inputs: - type: log enabled: true paths: - /tmp/APPname.log tags: ["APPname"] fields: env: test app: APPname multiline.pattern: '^[' multiline.negate: true multiline.match: "after" - type: log enabled: true paths: - /tmp/APPname.log tags: ["APPname"] fields: env: stg app: APPname multiline.pattern: '^[' multiline.negate: true multiline.match: "after" - type: log enabled: true paths: - /tmp/APPname.log tags: ["APPname"] fields: env: prod app: APPname multiline.pattern: '^[' multiline.negate: true multiline.match: "after" - type: log enabled: true paths: - /tmp/APPname.log tags: ["APPname"] fields: env: test app: APPname multiline.pattern: '^[' multiline.negate: true multiline.match: "after" filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true output.logstash: hosts: ["logstashIP:5044"] EOF systemctl start filebeatfilebeat排错如果你发现没有启动成功，可以执行 cd /usr/bin，在这个目录下执行./filebeat -c /etc/filebeat/filebeat.yml -e，这样会提示具体的错误信息filebeat删除元数据，重新读取数据创建索引（解决）背景：filebeat搜集同一文件，修改filebeat中自定义tag和logstash中定义的索引名称，重启后不生效，es中未创还能对应的索引 解决办法 mv /var/lib/filebeat/registry/filebeat/log.json /tmp systemctl restart filebeat 查看对应索引是否创建 curl -u elastic:123456 -XGET 'http://localhost:9200/_cat/indices?v&pretty'安装配置kibana，并启动rpm -ivh kibana-7.17.4-x86_64.rpm cp /etc/kibana/kibana.yml /etc/kibana/kibana.yml.bak sed -i 's/#elasticsearch.username: "kibana_system"/elasticsearch.username: "kibana"/g' /etc/kibana/kibana.yml sed -i 's/#elasticsearch.password: "pass"/elasticsearch.password: "123456"/g' /etc/kibana/kibana.yml systemctl start kibana 防火墙开放5601端口暑期编程PK赛 得CSDN机械键盘等精美礼品！
