数据迁移工具之DataX

一、DataX

1、DataX框架2、DataX运行原理 二、安装DataX

1、DataX的下载安装地址2、编译 三、配置模板

1、从Stream流到控制台2、从MYSQL到HDFS3、从HDFS到MySQL

DataX 是阿里巴巴开源的一个异构数据源离线同步工具，致力于实现包括关系型数据库(MySQL、Oracle 等)、HDFS、Hive、ODPS、Hbase、FTP 等各种异构数据源之间稳定高效的数据同步功能。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Z79kVoKg-1645612858233)(.DataXDataX框架.png)]

**Reader：**数据采集模块，负责采集数据源的数据，将数据发送给framework。**Writer：**数据写入模块，负责不断向framework取数据，并将数据写入到目的端。**framework：**用于连接reader和writer，作为两者的数据传输通道，并处理缓冲， 流控，并发，数据转换等核心技术问题。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xv6POGPK-1645612858234)(.DataXDataX运行原理.png)]

**Job：**单个作业的管理节点，负责数据清理、子任务划分、TaskGroup监控管理。**Task：**由Job切分而来，是DataX作业的最小单元，每个Task负责一部分数据的同步工作。**Schedule：**将Task组成TaskGroup，单个TaskGroup的并发数量为5。**TaskGroup：**负责启动Task。

下载地址：http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz

源码地址：https://github.com/alibaba/DataX

#解压与安装
tar -zxvf datax.tar.gz -C /opt/software/
#切换到指定文件目录
cd /opt/software/datax/bin/ 
#运行自检脚本
python datax.py /opt/module/datax/job/job.json

#查看DataX配置模板
python datax.py -r streamreader -w streamwriter 
#------------------------------------------------------------------------------
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


Please refer to the streamreader document: https://github.com/alibaba/DataX/blob/master/streamreader/doc/streamreader.md

Please refer to the streamwriter document: https://github.com/alibaba/DataX/blob/master/streamwriter/doc/streamwriter.md

Please save the following configuration as a json file and	use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json to run the job.

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "streamreader", "parameter": {
                        "column": [], "sliceRecordCount": ""
                    }
                },
                "writer": {
                    "name": "streamwriter", "parameter": {
                        "encoding": "", "print": true
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": ""
            }
        }
    }
}
#------------------------------------------------------------------------------
#执行配置
.../datax/bin/datax.py  .../job/XXX.json

#查看模板
python /opt/module/datax/bin/datax.py -r mysqlreader -w hdfswriter
#------------------------------------------------------------------------------
{
    "job":{
        "content":[
            {
                "reader":{
                    "name":"mysqlreader",
                    "parameter":{
                        "column":[

                        ],
                        "connection":[
                            {
                                "jdbcUrl":[

                                ],
                                "table":[

                                ]
                            }
                        ],
                        "password":"",
                        "username":"",
                        "where":""
                    }
                },
                "writer":{
                    "name":"hdfswriter",
                    "parameter":{
                        "column":[

                        ],
                        "compress":"",
                        "defaultFS":"",
                        "fieldDelimiter":"",
                        "fileName":"",
                        "fileType":"",
                        "path":"",
                        "writeMode":""
                    }
                }
            }
        ],
        "setting":{
            "speed":{
                "channel":""
            }
        }
    }
}
#------------------------------------------------------------------------------
#执行配置
.../datax/bin/datax.py  .../job/XXX.json

#查看模板
python bin/datax.py -r hdfsreader -w mysqlwriter
#------------------------------------------------------------------------------
{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "hdfsreader", 
                    "parameter": {
                        "column": [ ], 
                        "defaultFS": "", 
                        "encoding": "UTF-8", 
                        "fieldDelimiter": ",", 
                        "fileType": "orc", 
                        "path": ""
                    }
                }, 
                "writer": {
                    "name": "mysqlwriter", 
                    "parameter": {
                        "column": [ ], 
                        "connection": [
                            {
                                "jdbcUrl": "", 
                                "table": [ ]
                            }
                        ], 
                        "password": "", 
                        "preSql": [ ], 
                        "session": [ ], 
                        "username": "", 
                        "writeMode": ""
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": ""
            }
        }
    }
}
#------------------------------------------------------------------------------
#执行配置
.../datax/bin/datax.py  .../job/XXX.json