栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

kafka-connect mysql debeziem时区问题,时间格式化,时间转换

kafka-connect mysql debeziem时区问题,时间格式化,时间转换

使用功能debeziem的kafka-connect-mysql进行抽取数据时,对于时间字段 date,dateTIme,默认是将字段的值转成时间戳。这样某些情况可能不符合需求。

可以使用TimestampConverter进行时间格式转换。

  "transforms.tsFormat1.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
  "transforms.tsFormat1.target.type": "string",
  "transforms.tsFormat1.field": "transaction_time",
  "transforms.tsFormat1.format": "yyyy-MM-dd HH:mm:ss",

对于date类型还需要添加下面的参数。

"time.precision.mode": "connect",

完整的配置文件如下

{
  "connector.class": "io.debezium.connector.mysql.MySqlConnector",
  "tasks.max": "1",
  "database.hostname": "127.0.0.1",
  "database.port": "3306",
  "database.user": "root",
  "database.password": "123456",
  "database.server.id": "1",
  "database.server.name": "localhostdb",
  "database.whitelist": "databasetest",
  "include.schema.changes": "false",
  "snapshot.mode": "initial",
  "snapshot.locking.mode": "none",
  "database.history.kafka.bootstrap.servers": "127.0.0.1:9092",
  "database.history.kafka.topic": "points_his",
  "decimal.handling.mode": "string",
  "table.include.list": "databasetest.tabletest",
  "transforms": "unwrap,createKey,extractInt,tsFormat1,tsFormat2",
  "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
  "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
  "transforms.createKey.fields": "member_id",
  "transforms.extractInt.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
  "transforms.extractInt.field": "member_id",
  "topic.creation.default.replication.factor": -1,
  "topic.creation.default.partitions": 10,
  "topic.creation.default.cleanup.policy": "compact",
  "topic.creation.default.compression.type": "lz4",
  "time.precision.mode": "connect",
  "transforms.tsFormat1.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
  "transforms.tsFormat1.target.type": "string",
  "transforms.tsFormat1.field": "transaction_time",
  "transforms.tsFormat1.format": "yyyy-MM-dd HH:mm:ss",
  "transforms.tsFormat2.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
  "transforms.tsFormat2.target.type": "string",
  "transforms.tsFormat2.field": "send_time",
  "transforms.tsFormat2.format": "yyyy-MM-dd HH:mm:ss"
}
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/753400.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号