栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

docekr-compose 搭建 mysql + zookeeper + kafka + kafka-eagle

docekr-compose 搭建 mysql + zookeeper + kafka + kafka-eagle

一、docker-compose 搭建 mysql 1、拉取镜像
docker pull mysql:5.7
2、编写mysql的 .cnf 配置文件
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0

[mysqld]
#
# * Basic Settings
#
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
lc-messages-dir	= /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
# bind-address		= 0.0.0.0
#
# * Fine Tuning
#
key_buffer_size		= 16M
max_allowed_packet	= 16M
thread_stack		= 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options  = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit	= 1M
query_cache_size        = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries	= /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id		= 1
#log_bin			= /var/log/mysql/mysql-bin.log
expire_logs_days	= 10
max_binlog_size   = 100M
#binlog_do_db		= include_database_name
#binlog_ignore_db	= include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
3、编写docker-compose.yml 文件
version: '3'
services:
  mysql:
    image: mysql:5.7
    container_name: test_mysql
    ports:
      - "3306:3306"
    volumes:
      - ~/docker/mysql/data:/lib/mysql
      - ~/docker/mysql/conf/my.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf
    environment:
      - "MYSQL_ROOT_PASSWORD=123456"  # 设置密码

4、注意事项
    docker-compose.yml 文件名是默认的,如果自己改变了
    需要启动 docker-compose.yml up -d 后台启动
二、docker-compose 搭建 zookeeper kafka 1、编写zk的 docker-compose.yml 文件
version: '3'
services:
  zoo1:
    image: wurstmeister/zookeeper:latest
    container_name: zoo1
    ports:
      - "2181:2181"
    volumes:
      - ./zoo1/data:/data
      - ./zoo1/datalog:/datalog
    environment:
      ZOO_MY_ID: 1 # 节点ID
      ZOO_PORT: 2181 # zookeeper端口号
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper节点列表
 
  zoo2:
    image: wurstmeister/zookeeper:latest
    container_name: zoo2
    ports:
      - "2182:2181"
    volumes:
      - ./zoo2/data:/data
      - ./zoo2/datalog:/datalog
    environment:
      ZOO_MY_ID: 2 # 节点ID
      ZOO_PORT: 2181 # zookeeper端口号
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper节点列表
 

  zoo3:
    image: wurstmeister/zookeeper:latest
    container_name: zoo3
    ports:
      - "2183:2181"
    volumes:
      - ./zoo3/data:/data
      - ./zoo3/datalog:/datalog
    environment:
      ZOO_MY_ID: 3 # 节点ID
      ZOO_PORT: 2181 # zookeeper端口号
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper节点列表
 
# 统一指定网桥
networks:
  default:
    external:
      name: mynet
2、编写kafka的 docker-compose.yml 文件
version: '3'

services:
  broker1:
    image: wurstmeister/kafka
    container_name: broker1
    hostname: broker1
    ports:
      - "9091:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9091
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.150:9091
      KAFKA_ADVERTISED_HOST_NAME: broker1
      KAFKA_ADVERTISED_PORT: 9091
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
    volumes:
      - ./kafka-log1:/kafka
    external_links:
      - zoo1

  broker2:
    image: wurstmeister/kafka
    container_name: broker2
    hostname: broker2
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.150:9092
      KAFKA_ADVERTISED_HOST_NAME: broker2
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
    volumes:
      - ./kafka-log2:/kafka
    external_links:
      - zoo1

  broker3:
    image: wurstmeister/kafka
    container_name: broker3
    hostname: broker3
    ports:
      - "9093:9092"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.150:9093
      KAFKA_ADVERTISED_HOST_NAME: broker3
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
    volumes:
      - ./kafka-log3:/kafka
    external_links:
      - zoo1

networks:
  default:
    external:
      name: mynet
三、docker-compose 搭建 kafka-eagle 1、配置环境变量
export KE_HOME=/opt/software/kafka-eagle/efak-web-2.0.9
export PATH=$PATH:$KE_HOME/bin
2、配置kafka-eagle的配置文件 system-config.properties
# 配置zk
efak.zk.cluster.alias=cluster1
cluster1.zk.list=192.168.1.150:2181

# 配置数据库
# kafka mysql jdbc driver address
######################################
efak.driver=com.mysql.cj.jdbc.Driver
efak.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.username=root
efak.password=123456
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/710425.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号