端口
elasticsearch:9200、9300 kibana:5601 logstash:5044 filebeat:
依赖jdk
elasticsearch: jdk11或以上 logstash: jdk11或以上
功能
ELKF 是 Elasticsearch 、 Logstash 、 Kibana 、 Filebeat 的简称。 elasticsearch: 存储、搜索和分析引擎，特点是高可伸缩、高可靠和易管理等。 kibana: 数据分析和可视化平台，通常依赖 elasticsearch 。 logstash: 数据收集引擎，可以对数据进行过滤、分析、丰富、统一格式等操作，存储到用户指定的位置，包含但不限于文件、 elasticsearch 。 filebeat: 轻量级的开源日志文件数据搜集器，负责对服务的日志进行收集。 简单归纳为：FileBeat收集日志、Logstash解析格式化、Elasticsearch存储、Kibana分析。0.2 文件上传
将相关文件相继上传到目录/home/elkf 下
0.3 文件解压cd /home/elkf tar -xzvf elasticsearch-7.17.0-linux-x86_64.tar.gz tar -xzvf kibana-7.17.0-linux-x86_64.tar.gz tar -xzvf logstash-7.17.0-linux-x86_64.tar.gz tar -xzvf filebeat-7.17.0-linux-x86_64.tar.gz一 elasticsearch 1.1 配置elasticsearch.yml
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #cluster.name: my-application cluster.name: elasticsearch_prod cluster.routing.allocation.disk.threshold_enabled: false # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: node-1 node.name: node-001-data # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data path.data: /home/elkf/elasticsearch-7.17.0/datas # # Path to log files: # #path.logs: /path/to/logs path.logs: /home/elkf/elasticsearch-7.17.0/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # By default Elasticsearch is only accessible on localhost. Set a different # address here to expose this node on the network: # #network.host: 192.168.0.1 # # By default Elasticsearch listens for HTTP traffic on the first free port it # finds starting at 9200. Set a specific HTTP port here: # #http.port: 9200 # # For more information, consult the network module documentation. # http.cors.enabled: true http.cors.allow-origin: "*" http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User" # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true # # ---------------------------------- Security ---------------------------------- # # *** WARNING *** # # Elasticsearch security features are not enabled by default. # These features are free, but require configuration changes to enable them. # This means that users don’t have to provide credentials and can get full access # to the cluster. Network connections are also not encrypted. # # To protect your data, we strongly encourage you to enable the Elasticsearch security features. # Refer to the following documentation for instructions. # # https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html xpack.security.enabled: true1.2 配置jvm.options（不配置）
## -Xms4g ## -Xmx4g #-Xms1g #-Xmx1g
错误：
encountered [2] errors parsing [/home/elkf/elasticsearch-7.17.0/config/jvm.options] [1]: encountered improperly formatted JVM option in [/home/elkf/elasticsearch-7.17.0/config/jvm.options] on line number [33]: [ -Xms1g] [2]: encountered improperly formatted JVM option in [/home/elkf/elasticsearch-7.17.0/config/jvm.options] on line number [34]: [ -Xmx1g]1.3 配置JAVA_HOME
bin/elasticsearch-env
# 2022-02-14（增加ES_JAVA_HOME，es自带jdk 17，需要jdk11以上） ES_JAVA_HOME=/home/elkf/elasticsearch-7.17.0/jdk # now set the path to java if [ ! -z "$ES_JAVA_HOME" ]; then JAVA="$ES_JAVA_HOME/bin/java" JAVA_TYPE="ES_JAVA_HOME" elif [ ! -z "$JAVA_HOME" ]; then # fallback to JAVA_HOME echo "warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME" >&2 JAVA="$JAVA_HOME/bin/java" JAVA_TYPE="JAVA_HOME"1.4 用户权限
# 增加用户组 groupadd elkf # 增加用户，-g 指定组 -p 指定密码 useradd elkf -g elkf -p elkf123$ # 授权，-R : 处理指定目录下的所有文件 chown -R elkf:elkf /home/elkf chown -R elkf:elkf /home/kfq_gov/tomcat7/webapps/logs chown -R elkf:elkf /root/logs chown -R elkf:elkf /root/logs/kfq_gov.log chmod -R 777 /root/logs/kfq_gov.log tail -f /home/kfq_gov/tomcat7/webapps/logs/kfq_gov.log tail -f /root/logs/kfq_gov.log1.5 配置密码
所有用户：elastic、remote_monitoring_user、beats_system、logstash_system、kibana、kibana_system、apm_system
密码：elkf123456
interactiveller_test elasticsearch-7.17.0]# ./bin/elasticsearch-setup-passwords Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: elkf123456 Reenter password for [elastic]: elkf123456 Enter password for [apm_system]: elkf123456 Reenter password for [apm_system]: elkf123456 Enter password for [kibana_system]: elkf123456 Reenter password for [kibana_system]: elkf123456 Enter password for [logstash_system]: elkf123456 Reenter password for [logstash_system]: elkf123456 Enter password for [beats_system]: elkf123456 Reenter password for [beats_system]: elkf123456 Enter password for [remote_monitoring_user]: elkf123456 Reenter password for [remote_monitoring_user]: elkf123456 Changed password for user [apm_system] Changed password for user [kibana_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic]1.6 启动
当使用root用户执行sh /home/elkf/elasticsearch-start.sh后日志错误：
java.lang.RuntimeException: can not run elasticsearch as root
切换用户为elkf，再执行sh /home/elkf/elasticsearch-start.sh
#! /bin/bash /home/elkf/elasticsearch-7.17.0/bin/elasticsearch -d > /home/elkf/elasticsearch-7.17.0/logs/elasticsearch.log 2>&1 & #参数：-d # 后台方式启动 #elasticsearch.log日志好像没啥作用，查看日志文件elasticsearch_prod.log即可。1.7 权限验证
#切换用户
[root@ser elkf]# su elkf
#启动
bash-4.2$ ./elasticsearch-start.sh
# 无密码测试
bash-4.2$ curl localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
# 密码测试
bash-4.2$ curl localhost:9200 -u elastic:elkf123456
{
"name" : "node-001-data",
"cluster_name" : "elasticsearch_prod",
"cluster_uuid" : "fxP0XBjQRnK3i9atOqPkbg",
"version" : {
"number" : "7.17.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
"build_date" : "2022-01-28T08:36:04.875279988Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
1.8 telnet 9300
bash-4.2$ telnet 127.0.0.1 9300 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'.1.9 日志
bash-4.2$ tail -f elasticsearch-7.17.0/logs/logs/elasticsearch_prod.log # tail -f /home/elkf/elasticsearch-7.17.0/logs/logs/elasticsearch_prod.log二 kibana 2.1 配置kibana.yml
config/kibana.yml
# server.publicbaseUrl 缺失，在生产环境中运行时应配置。某些功能可能运行不正常。 # 这里地址改为你访问kibana的地址，不能以 / 结尾 server.publicbaseUrl: "http://172.23.8.184:5601" server.name: kibana # 默认是localhost或者127.0.0.1，只允许本地访问，设置0.0.0.0可允许远程访问 server.host: "0.0.0.0" server.port: 5601 elasticsearch.hosts: ["http://localhost:9200"] #配置本地索引 kibana.index: ".kibana" #2022-02-14-设置elasticsearch账号和密码 elasticsearch.username: "kibana_system" elasticsearch.password: "elkf123456" # 设置中文 i18n.locale: "zh-CN"2.2 启动
sh /home/elkf/kibana-start.sh
#! /bin/bash nohup /home/elkf/kibana-7.17.0-linux-x86_64/bin/kibana > /home/elkf/kibana-7.17.0-linux-x86_64/logs/kibana.log 2>&1 &
使用root用户启动后，日志错误：
Kibana should not be run as root. Use --allow-root to continue.
授权：
[root@ser_teller_test elkf]# chown -R elkf:elkf /home/elkf
切换用户：
[root@ser_teller_test elkf]# su elkf
启动：
bash-4.2$ ./kibana-start.sh2.3 日志
bash-4.2$ tail -f kibana-7.17.0-linux-x86_64/logs/kibana.log # tail -f /home/elkf/kibana-7.17.0-linux-x86_64/logs/kibana.log2.4 测试端口5601
[root@ser_teller_test elkf]# netstat -an | grep 5601 tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN [root@ser_teller_test elkf]# telnet localhost 5601 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. # 测试本地访问 [root@ser_teller_test elkf]# curl http://localhost:5601/ [root@ser_teller_test elkf]# # 测试远程访问 [root@ser_teller_test elkf]# curl http://172.23.8.184:5601/ curl: (7) Failed connect to 172.23.8.184:5601; 拒绝连接 # kibana.yml加入配置 server.host: "0.0.0.0" 即可 bash-4.2$ curl http://172.23.8.184:5601/ bash-4.2$2.5 浏览器访问
http://172.23.8.184:5601/ # 需要输入账户密码，elastic/elkf123456三 logstash 3.1 配置logstash.conf
将 config下的logstash-sample.conf拷贝一份命名为logstash.conf，配置内容：
input {
beats {
port => 5044 # 设置专用端口用于接收各个来源的日志
client_inactivity_timeout => 36000
}
}
filter {
json {
source => "message"
}
}
output {
if [filetype] == "supervision-info" {
# 184-上报终端信息
elasticsearch {
user => elastic
password => elkf123456
hosts => ["127.0.0.1:9200"]
index => "supervision-info-%{+YYYY.MM.dd}"
}
}
}
3.2 配置JAVA_HOME
LOGSTASH_HOME="$(cd `dirname $SOURCEPATH`/..; pwd)"
export LOGSTASH_HOME
export LS_HOME="${LOGSTASH_HOME}"
SINCEDB_DIR="${LOGSTASH_HOME}"
export SINCEDB_DIR
LOGSTASH_JARS=${LOGSTASH_HOME}/logstash-core/lib/jars
# 2022-02-14-设置JAVA_HOME
LS_JAVA_HOME=/home/elkf/logstash-7.17.0/jdk
3.3 启动
sh /home/elkf/logstash-start.sh
logstash-home=/home/elkf/logstash-7.17.0 nohup $logstash-home/bin/logstash --config.reload.automatic -f $logstash-home/config/logstash.conf > $logstash-homelogs/logstash.log 2>&1 &
切换用户elkf，并执行 ./logstash-start.sh
[root@ser_teller_test elkf]# su elkf bash-4.2$ ./logstash-start.sh3.4 日志
bash-4.2$ tail -f logstash-7.17.0/logs/logstash.log 或者 tail -f /home/elkf/logstash-7.17.0/logs/logstash.log3.5 测试端口5044
[root@ser_teller_test elkf]# telnet 172.23.8.184 5044 Trying 172.23.8.184... Connected to 172.23.8.184. Escape character is '^]'.四 filebeat 4.1 配置filebeat.xml
/home/elkf/filebeat-7.17.0-linux-x86_64/filebeat.xml
# ============================== Filebeat inputs ===============================
## 2022-02-14-上送终端信息
- type: log
enabled: true #开启监视，不开不采集
backoff: "1s"
tail_files: false
paths:
- /home/vtq/logs/supervision-info.log
fields:
filetype: supervision-info # 加这个就是区分不同文件的
#fields:自定义字段
#ields_under_root 为true,则自定义字段将为文档中的顶级字段。
fields_under_root: true # 普通单行文本日志
# ---------------------------- Elasticsearch Output ----------------------------
## 关闭Elasticsearch
##output.elasticsearch:
# Array of hosts to connect to.
## hosts: ["localhost:9200"]
# ------------------------------ Logstash Output -------------------------------
## 开启Logstash
output.logstash:
# The Logstash hosts
hosts: ["172.23.8.184:5044"]
4.2 配置执行文件
创建logs目录：
mkdir -p /home/elkf/filebeat-7.17.0-linux-x86_64/logs
配置启动文件：
vi /home/elkf/filebeat-start.sh
内容：
#! /bin/bash FILEBEAT_HOME=/home/elkf/filebeat-7.17.0-linux-x86_64 nohup $FILEBEAT_HOME/filebeat -e -c $FILEBEAT_HOME/filebeat.yml -d "publish" > $FILEBEAT_HOME/logs/filebeat.log 2>&1 & # -d 后台方式启动
授权文件：
[root@ser_teller_test elkf]# chown -R elkf:elkf /home/elkf [root@ser_teller_test elkf]# chmod -R 777 filebeat-start.sh4.3 启动
切换用户
su elkf
执行
bash-4.2$ ./filebeat-start.sh4.4 日志
bash-4.2$ tail -f filebeat-7.17.0-linux-x86_64/logs/filebeat.log #或 tail -f /home/elkf/filebeat-7.17.0-linux-x86_64/logs/filebeat.log
