skywalking内部测试服务器安装记录

1、yum install java-1.8.0-openjdk* -y 安装java1.8包

2、yum install net-tools 目前这个之前已经安装好

3、安装工具与docker-ce（一般docker的早期版本，后面分成ce免费社区版本与ee企业收费版本）

首先yum install -y yum-utils

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum -y install docker-ce-17.12.1.ce

4、这样就安装上了上面版本了

service docker start 同时重新启动dockert

systemctl enable docker.service 让docker的服务系统启动后自动启动

systemctl status docker.service 看状态

5、支持es7

docker pull elasticsearch:7.3.2

6、先建立/mydata目录，同时

useradd elasticsearch -d /mydata/elasticsearch

passwd elasticsearch 修改成密码是nbacheng@es123

id elasticsearch 查看用户id是否是1000，按官方文档最好是1000

若不是可以考虑修改usermod -u 1000 elasticsearch

把elasticsearch的uid修改成10000也不行啊，主要原因是先用第一次不用映射目录启动docker elasticsearch

7、先运行一次，因为需要拷贝相关目录与文件到宿主机里

docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e TZ=Asia/Shanghai -e"discovery.type=single-node" elasticsearch:7.3.2

8、docker exec -it elasticsearch bash 通过这个可以进去看相关内容

有时候上面的命令会出错，那目前只有重新启动或start 容器了

docker exec -it elasticsearch ls -l /usr/share/elasticsearch/config

列出里面的容器目录文件

必须要第一次干净的下面docker里文件拷贝到宿主机对应的目录里

docker cp elasticsearch:/usr/share/elasticsearch/config/ . 目录及相关文件拷贝到宿主目录里

docker cp elasticsearch:/usr/share/elasticsearch/logs/ .

docker cp elasticsearch:/usr/share/elasticsearch/data/ .

chown -R elasticsearch:elasticsearch *

注意权限问题

9、docker stop elasticsearch 并删除

10、运行起来

docker run -d --name elasticsearch -v /mydata/elasticsearch/data/:/usr/share/elasticsearch/data -v /mydata/elasticsearch/logs/:/usr/share/elasticsearch/logs -v /mydata/elasticsearch/config/:/usr/share/elasticsearch/config -e TZ=Asia/Shanghai -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.3.2

11、查看是否正常

http://192.168.33.113:9200/

类似上面应该是正常的

12、安装elasticsearch可视化管理工具

docker pull mobz/elasticsearch-head:5

docker run -d --name elasticsearchhead -p 9100:9100 -e TZ=Asia/Shanghai mobz/elasticsearch-head:5

需要在elasticsearch.yml中添加:

http.cors.enabled: true http.cors.allow-origin: "*"

重启 elasticsearch容器

docker restart elasticsearch

否则好像有问题，访问不了,用下面的地址进行访问

http://192.168.33.113:9100/

开始查询与浏览数据都没有，需要修改一个文件如下

docker cp elasticsearchhead:/usr/src/app/_site/vendor.js .

vi vendor.js

共有两处

1）6886行

contentType: "application/x-www-form-urlencoded

改成

contentType: "application/json;charset=UTF-8"

2）7573行

var inspectData = s.contentType === "application/x-www-form-urlencoded" &&

改成

var inspectData = s.contentType === "application/json;charset=UTF-8" &&

补充说明

vi中显示行号的命令为

:set nu

vi中跳转到指定行的命令为

:行号

13、安装启动skywalking-oap-server

docker pull apache/skywalking-oap-server:6.6.0-es7

docker run --name oap -d -p 11800:11800 -p 12800:12800 --link elasticsearch:elasticsearch -e SW_STORAGE=elasticsearch -e TZ=Asia/Shanghai -e SW_STORAGE_ES_CLUSTER_NODES=elasticsearch:9200 apache/skywalking-oap-server:6.6.0-es7

好像这样设置时区不起作用啊

docker run --name oap -d -p 11800:11800 -p 12800:12800 -v /etc/localtime:/etc/localtime --link elasticsearch:elasticsearch -e SW_STORAGE=elasticsearch -e SW_STORAGE_ES_CLUSTER_NODES=elasticsearch:9200 apache/skywalking-oap-server:6.6.0-es7

好像这样设置时区起作用，但docker logs日志时间好像又不对

只有想办法容器启动文件里docker-entrypoint.sh修改里面的java启动代码里增加时区了，如增加 -Duser.timezone=Asia/Shanghai

# add by lqy -Duser.timezone=Asia/Shanghai

exec java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Duser.timezone=Asia/Shanghai

${JAVA_OPTS} -classpath ${CLASSPATH} org.apache.skywalking.oap.server.starter.OAPServerStartUp "$@"

docker启动后还是要注意时区问题，要检查

第一次启动oap服务需要建立大量的存储表等信息，所以时间比较长，最好等它完成后再进行下一步

14、安装启动skywalking-ui

docker pull apache/skywalking-ui:6.6.0

docker run --name oap-ui -d -p 8088:8080 --link oap:oap -e TZ=Asia/Shanghai -e SW_OAP_ADDRESS=oap:12800 apache/skywalking-ui:6.6.0 --security.user.admin.password=swmonitor@admin

好像安全认证在6.2以后被去掉了好像这样设置时区不起作用啊

docker run --name oap-ui -d -p 8088:8080 --link oap:oap -v /etc/localtime:/etc/localtime -e SW_OAP_ADDRESS=oap:12800 apache/skywalking-ui:6.6.0

好像这样设置时区起作用，但docker logs日志时间好像又不对

只有想办法容器启动文件里docker-entrypoint.sh修改里面的java启动代码里增加时区了，如增加 -Duser.timezone=Asia/Shanghai

docker启动后还是要注意时区问题，要检查

用我自己编译的覆盖掉

docker cp skywalking-webapp.jar oap-ui:/skywalking/webapp/

在/skywalking/webapp里的webapp.yml增加下面一项也没有用

security:

user:

admin:

password: swmonitor@admin

du -sh * 可以定期看一下 elasticsearch目录下的logs与data容量问题

docker logs -f --tail=1000 elasticsearch 看最后的日志信息

Linux Tomcat 7, Tomcat 8

修改tomcat/bin/catalina.sh，在首行加入如下信息

CATALINA_OPTS="$CATALINA_OPTS -javaagent:/mydata/appuser/tools/agent/skywalking-agent.jar=agent.service_name=tomcat-5110,collector.backend_service=192.168.33.113:11800"; export CATALINA_OPTS

nohup java -Xms128m -Xmx256m -javaagent:/mydata/appuser/tools/agent/skywalking-agent.jar=agent.service_name=hospital-appService,collector.backend_service=192.168.33.113:11800 -Duser.timezone=Asia/Shanghai -jar hospital-appService.jar --env=prodext > /dev/null 2>&1 &

对spring-data-redis:2.2.1 default use lettuce(version:5.2.1). I found that redis tracking dosen't work in skywalking(version: 6.6.0), but when i manually exclude the lettuce and replace with jedis, it works normally.

Because of JRE 1.8 requirements, it is an optional plugin only. https://github.com/apache/skywalking/blob/6.x/docs/en/setup/service-agent/java-agent/README.md#optional-plugins

说明用spring-data-redis:2.2.1 lettuce(version:5.2.1) 在6.6.0里是放到可选插件里的，说明要移到sdk插件里才能跟踪到，在代理里面进行移动

后来查询了一下，docker的日志文件应该是在/var/log下的messages文件里

ui主页上的显示功能图表的时间范围在右下角可以进行选择，有时候需要往下拉才能看到

运行一段时间后出现docker logs -f --tail=1000 oap 看oap出现下面错误，感觉 elasticsearch 分片超过1000了，但默认好像就是1000的，所以就没有办法写过去了，通过网上查询用下面办法可以增加分片数量到10000，结果一修改完就正常了

curl -X PUT -H "Content-Type:application/json" -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}' 'http://192.168.33.113:9200/_cluster/settings'

curl -X GET 'http://192.168.33.113:9200/_nodes/stats'

获得相关参数吧

运行一段时间后又出现下面的问题

内存使用情况，先在 elasticsearch容器里的jvm.options 配置文件内存先都修改成4G内存

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

2020-03-03 09:00:52,542 - org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker -168865433 [pool-12-thread-1] ERROR [] - Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [1056273900/1007.3mb], which is larger than the limit of [1003493785/957mb], real usage: [1056273728/1007.3mb], new bytes reserved: [172/172b], usages [request=0/0b, fielddata=82095/80.1kb, in_flight_requests=172/172b, accounting=12929020/12.3mb]]

org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [1056273900/1007.3mb], which is larger than the limit of [1003493785/957mb], real usage: [1056273728/1007.3mb], new bytes reserved: [172/172b], usages [request=0/0b, fielddata=82095/80.1kb, in_flight_requests=172/172b, accounting=12929020/12.3mb]]

at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1706) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1683) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:915) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]

at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.ids(ElasticSearch7Client.java:197) ~[storage-elasticsearch7-plugin-6.6.0.jar:6.6.0]

at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.dao.MetricsEs7DAO.multiGet(MetricsEs7DAO.java:45) ~[storage-elasticsearch7-plugin-6.6.0.jar:6.6.0]

at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.syncStorageToCache(MetricsPersistentWorker.java:185) ~[server-core-6.6.0.jar:6.6.0]

at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:121) ~[server-core-6.6.0.jar:6.6.0]

at org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:76) ~[server-core-6.6.0.jar:6.6.0]

at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:95) ~[server-core-6.6.0.jar:6.6.0]

at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_212]

at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:89) ~[server-core-6.6.0.jar:6.6.0]

at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:67) ~[server-core-6.6.0.jar:6.6.0]

at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.6.0.jar:6.6.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/service_instance_cpm/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 429 Too Many Requests]

{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [] would be [1056273900/1007.3mb], which is larger than the limit of [1003493785/957mb], real usage: [1056273728/1007.3mb], new bytes reserved: [172/172b], usages [request=0/0b, fielddata=82095/80.1kb, in_flight_requests=172/172b, accounting=12929020/12.3mb]","bytes_wanted":1056273900,"bytes_limit":1003493785,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [] would be [1056273900/1007.3mb], which is larger than the limit of [1003493785/957mb], real usage: [1056273728/1007.3mb], new bytes reserved: [172/172b], usages [request=0/0b, fielddata=82095/80.1kb, in_flight_requests=172/172b, accounting=12929020/12.3mb]","bytes_wanted":1056273900,"bytes_limit":1003493785,"durability":"PERMANENT"},"status":429}

at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]

at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]