| Elasticsearch | 关系型数据库 | 说明 |
| INDEX | database | 数据库 |
| Type | table | 表格 |
| document | row | 行 |
| field | column | 列 |
| IPAddr | HostName | Mem |
| 192.168.1.11 | hd1.com | 3G |
| 192.168.1.12 | hd2.com | 3G |
[root@hd1 ~]# mkdir /opt/elk [root@hd1 ~]# mv elasticsearch-7.9.3-linux-x86_64.tar.gz /opt/elk [root@hd1 ~]# cd /opt/elk [root@hd1 elk]# tar zxvf elasticsearch-7.9.3-linux-x86_64.tar.gz [root@hd1 elk]# mv elasticsearch-7.9.3 elasticsearch [root@hd1 elk]# cd elasticsearch [root@hd1 elasticsearch]# ls bin config jdk lib LICENSE.txt logs modules [root@hd1 elasticsearch]# cd bin 尝试着去启动es [root@hd1 bin]# ./elasticsearch [2021-08-15T10:20:59,642][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [hd1.com] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
2） 启动失败的原因是不能以root去运行 我们创建一个普通用户来执行
[root@hd1 ~]# useradd es [root@hd1 ~]# chown -R es.es /opt/elk [root@hd1 ~]# ulimit -n 1024
3） 调整进程最大打开文件数量
[root@hd1 ~]# ulimit -n 65535
永久修改
[root@hd1 ~]# tailf -3 /etc/security/limits.conf * hard nofile 65535 * soft nofile 65535 * soft nproc 4096 * hard nproc 4096 # End of file
4）调整进程最大虚拟内存区域数量
临时设置 [root@hd1 ~]# sysctl -w vm.max_map_count=262144 vm.max_map_count = 262144 永久设置 [root@hd1 ~]# echo "vm.max_map_count=262144" >>/etc/sysctl.conf [root@hd1 ~]# sysctl -p vm.max_map_count = 262144
5）修改配置文件
[root@hd1 ~]# vi /opt/elk/elasticsearch/config/elasticsearch.yml cluster.name: elk-cluster #集群的名称，两个节点保持一致 node.name: node-1 #集群节点的名字 path.data: /opt/elk/data #数据的路径 path.logs: /opt/elk/logs #日志的路径 network.host: 0.0.0.0 #监听的ip地址 http.port: 9200 discovery.seed_hosts: ["192.168.1.11", "192.168.1.12"] #发现集群中的其他节点 cluster.initial_master_nodes: ["node-1"] #设置主节点
注意，在节点2或节点3不启用cluster.initial_master_nodes 参数，注释掉
6） 设置es的权限
[root@hd1 ~]# mkdir /opt/elk/data [root@hd1 ~]# mkdir /opt/elk/logs [root@hd1 ~]# chown -R es.es /opt/elk
7 ）生成启动脚本文件
[root@hd1 ~]# cat /usr/lib/systemd/system/elasticsearch.service [Unit] Description=elasticsearch [Service] User=es ExecStart=/opt/elk/elasticsearch/bin/elasticsearch ExecReload=/bin/kill -HUP $MAINPID KillMode=process #Restart=on-failure [Install] WantedBy=multi-user.target
9）启动测试
[root@hd1 ~]# systemctl daemon-reload [root@hd1 ~]# systemctl start elasticsearch
查看启动日志有错误不？
[root@hd1 config]# journalctl -u elasticsearch -- Logs begin at 日 2021-08-15 17:20:37 CST, end at 日 2021-08-15 17:36:09 CST. -- 8月 15 17:34:04 hd1.com systemd[1]: Started elasticsearch. 8月 15 17:34:04 hd1.com systemd[1]: Starting elasticsearch... 8月 15 17:34:10 hd1.com elasticsearch[2185]: [2021-08-15T17:34:10,459][INFO ][o.e.n.Node 8月 15 17:34:10 hd1.com elasticsearch[2185]: [2021-08-15T17:34:10,464][INFO ][o.e.n.Node 8月 15 17:34:10 hd1.com elasticsearch[2185]: [2021-08-15T17:34:10,464][INFO ][o.e.n.Node 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,459][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,460][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,460][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,460][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,461][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,461][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,461][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,461][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,462][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,462][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,462][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,463][INFO ][o.e.p.PluginsServi 8月 15 17:34:16 hd1.com elasticsearch[2185]: [2021-08-15T17:34:16,463][INFO ][o.e.p.PluginsServi
查看监听的端口，9300用于内部集群之间的通信
[root@hd1 config]# ss -ant |grep 9300 LISTEN 0 128 :::9300 :::* [root@hd1 config]# ss -ant |grep 9200 LISTEN 0 128 :::9200 :::*
10 ） 同理配置第二个节点192.168.1.12
[root@hd2 ~]# mkdir -p /opt/elk [root@hd2 ~]# useradd es
a 将第一个节点整个目录拷贝过去
[root@hd1 ~]# scp -r /opt/elk/* root@192.168.1.12:/opt/elk/
切换到1.12查看一下，并将data和logs里的数据删除
[root@hd2 ~]# cd /opt/elk [root@hd2 elk]# ls data elasticsearch logs [root@hd2 elk]# rm -rf logs/* [root@hd2 elk]# rm -rf data/*
将配置文件指定master的属性注释掉，将node的名字改成node-2
[root@hd2 elk]#cd /opt/elk/elasticsearch/config/ [root@hd2 config]# grep master_nodes: elasticsearch.yml node.name: node-2 #cluster.initial_master_nodes: ["node-1"]
b 将启动脚本文件拷贝过去
[root@hd1 ~]# scp -r /usr/lib/systemd/system/elasticsearch.service root@192.168.1.12:/usr/lib/systemd/system/
c 切换到1.12启动服务
[root@hd2 ~]# useradd es [root@hd2 ~]# chown -R es.es /opt/elk/ [root@hd2 ~]# ulimit -n 65535 [root@hd2 ~]# vi /etc/security/limits.conf * hard nofile 65535 * soft nofile 65535 [root@hd2 ~]# sysctl -w vm.max_map_count=262144 vm.max_map_count = 262144 [root@hd2 ~]# echo "vm.max_map_count=262144" >>/etc/sysctl.conf [root@hd2 ~]# sysctl -p [root@hd2 ~]# systemctl daemon-reload [root@hd2 ~]# systemctl start elasticsearch [root@hd2 ~]# ps -ef |grep elastic
查看启动日志 ，报错信息如下
root@hd2 ~]# cat /opt/elk/logs/elk-cluster.log elasticsearch：max number of threads [3818] for user [es] is too low, increase to at least [4096]
我们需要修改配置文件/etc/security/limits.conf，将nproc设置为4096。
[root@hd2 ~]# grep ^* /etc/security/limits.conf * hard nofile 65535 * soft nofile 65535 * soft nproc 4096 * hard nproc 4096
解决方法二
我们添加内存到1600m，重新启动系统，再次执行如下命令
[root@hd2 ~]# systemctl start elasticsearch [root@hd2 ~]# ss -ant State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 :::9200 :::* LISTEN 0 128 :::9300
上述结果表示启动成功了，原因是java程序 太耗内存和cpu了 ,尤其是内存
11 ） 查看集群各个节点状态
[root@hd2 config]# curl -XGET "http://127.0.0.1:9200/"
{
"name" : "node-2",
"cluster_name" : "elk-cluster",
"cluster_uuid" : "6Bq-5r02QD2fvGQqGOv4Kg",
"version" : {
"number" : "7.9.3",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
"build_date" : "2020-10-16T10:36:16.141335Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
[root@hd1 ~]# curl -XGET "http://127.0.0.1:9200/"
{
"name" : "node-1",
"cluster_name" : "elk-cluster",
"cluster_uuid" : "6Bq-5r02QD2fvGQqGOv4Kg",
"version" : {
"number" : "7.9.3",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
"build_date" : "2020-10-16T10:36:16.141335Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
查看集群情况，带*号的表示是master
[root@hd1 ~]# curl -XGET 'http://127.0.0.1:9200/_cat/nodes?pretty' 192.168.1.11 10 74 6 0.16 0.29 0.20 dilmrt * node-1 192.168.1.12 16 75 5 0.04 0.20 0.18 dilmrt - node-2 [root@hd2 ~]# curl -XGET 'http://127.0.0.1:9200/_cat/nodes?pretty' 192.168.1.12 7 75 0 0.01 0.16 0.17 dilmrt - node-2 192.168.1.11 15 74 0 0.11 0.24 0.19 dilmrt * node-1
Master和Slave的区别：
Master的职责：
统计各node节点状态信息、集群状态信息统计、索引的创建和删除、索引分配的管理、关闭node节点等
Savle的职责：
同步数据、等待机会成为Master
12 ）图形管理ES
[root@hd1 ~]# cd /opt/elk/ [root@hd1 elk]# rz -y data elasticHD_linux_amd64.zip elasticsearch logs [root@hd1 elk]# unzip elasticHD_linux_amd64.zip Archive: elasticHD_linux_amd64.zip inflating: ElasticHD [root@hd1 elk]# nohup ./ElasticHD & [root@hd1 elk]# tail nohup.out -f To view elasticHD console open http://0.0.0.0:9800 in browser exec: "xdg-open": executable file not found in $PATH
访问页面
四 部署安装logstash
logstash 能够将采集日志、格式化、过滤最后将数据推送到elasticsearch存储
input：输入，可以是stdin file tcp redis syslog等
filter：过滤 ，将日志格式化，有丰富的过滤插件 json解码器，mutate数据修改，date时间处理、grok正则捕获
output：输出，输出目标可以是stdout file tcp redis es等
安装logstash
[root@hd1 elk]# tar -xf logstash-7.9.3.tar.gz [root@hd1 elk]# mv logstash-7.9.3 logstash [root@hd1 elk]# yum install java-1.8.0-openjdk -y
主配置文件修改如下所示：
[root@hd1 elk]# cd logstash/config/
[root@hd1 config]# cat logstash.yml
pipeline: #管道配置
batch:
size: 125
delay: 5
#path.config: /opt/elk/logstash/conf.d #conf.d目录自己创建
#定期检查配置是否修改，并重新加载管道。也可以使用SIGHUP信号手动触发
# config.reload.automatic: false
# config.reload.interval: 3s
# http.enabled: true
http.host: 0.0.0.0
http.port: 9600-9700
log.level: info
path.logs: /opt/elk/logstash/logs
[root@hd1 config]# cd ..
[root@hd1 logstash]# mkdir conf.d
修改服务启动文件如下：
[root@hd1 config]# cat /usr/lib/systemd/system/logstash.service
[Unit]
Description=logstash
[Service]
ExecStart=/opt/elk/logstash/bin/logstash
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
#Restart=on-failure
[Install]
WantedBy=multi-user.target
手工启动一下logstash
root@hd1 logstash]# bin/logstash -e 'input { stdin {} } output { stdout {} }'
Sending Logstash logs to /opt/elk/logstash/logs which is now configured via log4j2.properties
[2021-08-16T22:07:52,756][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.3", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +indy +jit [linux-x86_64]"}
[2021-08-16T22:07:53,516][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-16T22:07:55,353][INFO ][org.reflections.Reflections] Reflections took 42 ms to scan 1 urls, producing 22 keys and 45 values
[2021-08-16T22:07:56,650][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#"}
[2021-08-16T22:07:57,400][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.73}
[2021-08-16T22:07:57,465][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2021-08-16T22:07:57,576][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-08-16T22:07:57,843][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello world
{
"host" => "hd1.com",
"message" => "hello world",
"@version" => "1",
"@timestamp" => 2021-08-16T14:08:10.214Z
}
编辑文件写个简单的测试，在logstash安装目录下新建一个文件myPipeline.conf
[root@hd1 logstash]# cat myPipeline.conf
input {
stdin {
}
}
output {
stdout {
codec => rubydebug
}
}
启动logstash 并输入hello
[root@hd1 logstash]# bin/logstash -f myPipeline.conf
Sending Logstash logs to /opt/elk/logstash/logs which is now configured via log4j2.properties
[2021-08-16T21:55:19,686][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.3", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +indy +jit [linux-x86_64]"}
[2021-08-16T21:55:20,224][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-16T21:55:22,119][INFO ][org.reflections.Reflections] Reflections took 40 ms to scan 1 urls, producing 22 keys and 45 values
[2021-08-16T21:55:23,477][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/opt/elk/logstash/myPipeline.conf"], :thread=>"#"}
[2021-08-16T21:55:24,256][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.76}
[2021-08-16T21:55:24,324][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2021-08-16T21:55:24,430][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-08-16T21:55:24,703][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello
{
"@version" => "1",
"@timestamp" => 2021-08-16T13:55:45.206Z,
"message" => "hello",
"host" => "hd1.com"
}
默认给日志加的三个字段
• "@timestamp" 标记事件发生的时间点
• "host" 标记事件发生的主机
• "type" 标记事件的唯一类型
命令行参数：
• -e 字符串形式写配置
• -f 指定配置文件
• -t 测试配置文件语
重新加载启动文件并启动logstash
[root@hd1 config]# systemctl daemon-reload [root@hd1 config]# systemctl start logstash
启动会报错先不用管
2） 输入阶段：从哪里获取日志 常用插件：
• Stdin（一般用于调试）
• File
• Redis
• Beats（例如filebeat）
File插件：用于读取指定日志文件 常用字段：
• path 日志文件路径，可以使用通配符
• exclude 排除采集的日志文件
• start_position 指定日志文件什么位置开始读，默认从结尾 开始，指定beginning表示从头开始读
案例：读取日志文件并输出到文件
具体步骤如下：
1
[root@hd1 logstash]# cat myPipeline.conf > conf.d/test.conf
2 将logstash.yml中的path.config 属性打开
[root@hd1 logstash]# grep path.config config/logstash.yml
path.config: /opt/elk/logstash/conf.d #conf.d目录自己创建
3 修改test.conf文件 ，输入文件为/var/log/messages 输出到/tmp/test.log 中间省略了filter
[root@hd1 logstash]# cd conf.d/
[root@hd1 conf.d]# cat test.conf
input {
file {
path => "/var/log/messages"
}
}
filter {
}
output {
file {
path => "/tmp/test.log"
}
}
这时候重新启动logstash 查看是否有日志输出
[root@hd1 conf.d]# cat /tmp/test.log
{"path":"/var/log/messages","@version":"1","message":"Aug 17 09:49:13 hd1 logstash: [2021-08-17T09:49:13,335][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}","@timestamp":"2021-08-17T01:49:14.531Z","host":"hd1.com"}
{"path":"/var/log/messages","@version":"1","message":"Aug 17 09:49:14 hd1 logstash: [2021-08-17T09:49:14,635][INFO ][logstash.outputs.file ][main][54e84d12565885a8e106bfb7855d1f835b2d2153a54e139856aaea3c6d728bae] Opening file {:path=>"/tmp/test.log"}","@timestamp":"2021-08-17T01:49:15.578Z","host":"hd1.com"}
{"path":"/var/log/messages","@version":"1","message":"Aug 17 09:49:32 hd1 logstash: [2021-08-17T09:49:32,837][INFO ][logstash.outputs.file ][main][54e84d12565885a8e106bfb7855d1f835b2d2153a54e139856aaea3c6d728bae] Closing file /tmp/test.log","@timestamp":"2021-08-17T01:49:33.652Z","host":"hd1.com"}
[root@hd1 conf.d]# echo 123 >>/var/log/messages
[root@hd1 conf.d]# grep 123 /tmp/test.log
{"path":"/var/log/messages","@version":"1","message":"Aug 17 09:51:48 hd1 logstash: [2021-08-17T09:51:48,131][INFO ][logstash.outputs.file ][main][54e84d12565885a8e106bfb7855d1f835b2d2153a54e139856aaea3c6d728bae] Opening file {:path=>"/tmp/test.log"}","@timestamp":"2021-08-17T01:51:49.123Z","host":"hd1.com"}
{"path":"/var/log/messages","@version":"1","message":"123","@timestamp":"2021-08-17T01:52:07.169Z","host":"hd1.com"}
File插件：用于读取指定日志文件 常用字段：
• path 日志文件路径，可以使用通配符
• exclude 排除采集的日志文件
• start_position 指定日志文件什么位置开始读，默认从结尾 开始，指定beginning表示从头开始读
案例2：排除error.log 日志
[root@hd1 conf.d]# mkdir /var/log/test
[root@hd1 conf.d]# cat test.conf
input {
file {
path => "/var/log/test/*.log"
exclude => "error.log"
start_position => "beginning"
}
}
filter {
}
output {
file {
path => "/tmp/test.log"
}
}
[root@hd1 conf.d]# systemctl restart logstash
测试一下，看能否收到error.log 的日志
[root@hd1 conf.d]# echo nonews >> /var/log/test/access.log
[root@hd1 conf.d]# echo badnews >>/var/log/test/error.log
[root@hd1 ~]# tail -f /tmp/test.log
{"message":"nonews","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:16:53.779Z","@version":"1"}
{"message":"error","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:16:53.777Z","@version":"1"}
{"message":"hello","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:16:53.753Z","@version":"1"}
{"message":"error","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:16:53.778Z","@version":"1"}
{"message":"nonews","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:16:53.779Z","@version":"1"}
{"message":"nonews","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:16:53.779Z","@version":"1"}
{"message":"oktest","path":"/var/log/test/access.log","host":"hd1.com","@timestamp":"2021-08-17T02:17:05.906Z","@version":"1"}
案例3 ：设置日志的来源 ，添加日志的属性
输入插件都支持的字段：
• add_field 添加一个字段到一个事件，放到事件顶部，一般用于标记日志来源。例如属于哪个项目，哪个应用
• tags 添加任意数量的标签，用于标记日志的其他属性，例如表明访问日志还是错误日志
• type 为所有输入添加一个字段，例如表明日志类型
[root@hd1 conf.d]# cat test.conf
input {
file {
path => "/var/log/test/*.log"
exclude => "error.log"
start_position => "beginning"
tags => "web"
tags => "nginx"
type => "access"
add_field => {
"project" => "cloud service"
"app" => "douyu"
}
}
}
filter {
}
output {
file {
path => "/tmp/test.log"
}
}
[root@hd1 conf.d]# systemctl restart logstash
[root@hd1 conf.d]# echo 55555 >> /var/log/test/access.log
查看日志
[root@hd1 ~]# tail -f /tmp/test.log
{"@version":"1","@timestamp":"2021-08-17T02:37:30.220Z","type":"access","tags":["web","nginx"],"message":"55555","project":"cloud service","host":"hd1.com","path":"/var/log/test/access.log","app":"douyu"}
