[JMX]jmx

• 背景
• 实现过程
• • 1 环境的准备
  • 2 jmx在hadoop中的编写（或见6）
  • • 2.1 hdfs——jmx
    • 2.2 yarn——jmx
  • 3.jmx相关参数的yaml文件编写
  • 4.Prometheus部分
  • 5.最后操作启停
• 6 jmx配置在hadoop-env.sh同样可以！
• 其他

Prometheus+jmx_prometheus_javaagent-0.16.1.jar+hadoop3.2.1

jmx下载地址：https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

Prometheus下载地址：
https://prometheus.io/download/

hadoop下载地址：
https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/
安装过程就不一一赘述了，本文着重jmx使用

[root@bdp01 bin]# pwd
/opt/hadoop/bin
[root@bdp01 bin]# ll
total 1040
-rwxrwxrwx. 1 1001 mysql 442480 Sep 11  2019 container-executor
-rwxrwxrwx. 1 1001 mysql   8707 Sep 10  2019 hadoop
-rwxrwxrwx. 1 1001 mysql  11265 Sep 10  2019 hadoop.cmd
-rwxrwxrwx. 1 1001 mysql  12573 May  7 14:36 hdfs
-rwxrwxrwx. 1 1001 mysql   8081 Sep 11  2019 hdfs.cmd
-rw-r--r--. 1 root root       0 May  5 17:45 kill
-rwxrwxrwx. 1 1001 mysql   6240 May  7 10:59 mapred
-rwxrwxrwx. 1 1001 mysql   6311 Sep 11  2019 mapred.cmd
-rw-------. 1 root root       0 May  5 17:45 nohup.out
-rwxrwxrwx. 1 1001 mysql  29184 Sep 11  2019 oom-listener
-rwxrwxrwx. 1 1001 mysql 484264 Sep 11  2019 test-container-executor
-rwxrwxrwx. 1 1001 mysql  13157 May  7 15:34 yarn
-rwxrwxrwx. 1 1001 mysql  12840 Sep 11  2019 yarn.cmd

vi hdfs

如图

## hdfs  monitor
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9221 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5221:/opt/hadoop/nn.yaml $HADOOP_NAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9222 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5222:/opt/hadoop/dn.yaml $HADOOP_DATANODE_OPTS"

export HADOOP_JOURNALNODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9225 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5225:/opt/hadoop/journalnode.yaml $HADOOP_JOURNALNODE_OPTS"

export HADOOP_ZKFC_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9224 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5224:/opt/hadoop/zkfc.yaml $HADOOP_ZKFC_OPTS"

[root@bdp01 bin]# ll
total 1040
-rwxrwxrwx. 1 1001 mysql 442480 Sep 11  2019 container-executor
-rwxrwxrwx. 1 1001 mysql   8707 Sep 10  2019 hadoop
-rwxrwxrwx. 1 1001 mysql  11265 Sep 10  2019 hadoop.cmd
-rwxrwxrwx. 1 1001 mysql  12573 May  7 14:36 hdfs
-rwxrwxrwx. 1 1001 mysql   8081 Sep 11  2019 hdfs.cmd
-rw-r--r--. 1 root root       0 May  5 17:45 kill
-rwxrwxrwx. 1 1001 mysql   6240 May  7 10:59 mapred
-rwxrwxrwx. 1 1001 mysql   6311 Sep 11  2019 mapred.cmd
-rw-------. 1 root root       0 May  5 17:45 nohup.out
-rwxrwxrwx. 1 1001 mysql  29184 Sep 11  2019 oom-listener
-rwxrwxrwx. 1 1001 mysql 484264 Sep 11  2019 test-container-executor
-rwxrwxrwx. 1 1001 mysql  13157 May  7 15:34 yarn
-rwxrwxrwx. 1 1001 mysql  12840 Sep 11  2019 yarn.cmd
[root@bdp01 bin]# pwd
/opt/hadoop/bin

如图

vi yarn

export YARN_RESOURCEMANAGER_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6464 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=17104:/opt/hadoop/rm.yaml $YARN_RESOURCEMANAGER_OPTS"

export YARN_NODEMANAGER_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6474 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=17105:/opt/hadoop/nm.yaml $YARN_NODEMANAGER_OPTS"

如图把jar包放到hdfs和yarn配置的相同路径位置，并编写yaml文件。

举两个例子，其实只要端口号修改一下，跟上面hdfs和yarn端口对应就行

vi nn.yaml

startDelaySeconds: 0
hostPort: 192.168.x.xxx:9221
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

 vi dn.yaml

startDelaySeconds: 0
hostPort: 192.168.0.101:9222
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

 vi prometheus.yml

这样写的好处是今后可以动态修改Prometheus配置，无需重启Prometheus

 - job_name: 'hdfs'
    scrape_interval: 80s
    scrape_timeout: 60s
    file_sd_configs:
     - files:
       - /opt/prometheus/prom/hdfs.json
       refresh_interval: 5s

上文中有个json文件，我编写了一个json，举个hdfs的例子。hdfs包括namenode，datanode，journalnode和zkfc

vi hdfs.json

[
  {
      "targets": [
         "192.168.x.xxx:5221"
      ],
      "labels": {
         "project_name": "namenode"
      }
   },
 {
      "targets": [
         "192.168.x.xxx:5222"
      ],
      "labels": {
         "project_name": "datanode"
      }
   },
   {
      "targets": [
         "192.168.x.xxx:5225"
      ],
      "labels": {
         "project_name": "historyserver"
      }
   },
   {
      "targets": [
         "192.168.x.xxx:5224"
      ],
      "labels": {
         "project_name": "zkfc"
      }
   }
]

hdfs yarn Prometheus 重启
查看Prometheus

如果安装成功，http://ip:port也会有数据，如图

export HADOOP_NAMENODE_OPTS=" -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5229:/opt/hadoop/nn.yaml $HADOOP_NAMENODE_OPTS"

bash -x  hdfs --daemon stop datanode