Flink通过自定义Sink向MySql写入数据

环境:

开发工具 IDEA

JDK版本 1.8

MySQL版本 5.1.47

Flink版本 1.13.0

首先引入Flink相关依赖:

    org.apache.flink
    flink-java
    ${flink.version}


    org.apache.flink
    flink-streaming-java_${scala.binary.version}
    ${flink.version}


    org.apache.flink
    flink-clients_${scala.binary.version}
    ${flink.version}




    org.apache.flink
    flink-connector-jdbc_${scala.binary.version}
    ${flink.version}


    mysql
    mysql-connector-java
    ${mysql.version}

创建实体类:

public class Book implements Serializable {
    public Long id;
    public String title;
    public String authors;
    public Integer year;

    public Book() {
    }

    public Book(Long id, String title, String authors, Integer year) {
        this.id = id;
        this.title = title;
        this.authors = authors;
        this.year = year;
    }

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    public String getAuthors() {
        return authors;
    }

    public void setAuthors(String authors) {
        this.authors = authors;
    }

    public Integer getYear() {
        return year;
    }

    public void setYear(Integer year) {
        this.year = year;
    }
}

创建自定义Sink用来连接MySQL:

public class MysqlSink extends RichSinkFunction {
    Connection conn = null;
    PreparedStatement ps = null;
    String url = "jdbc:mysql:/
    @Override
    public void open(Configuration parameters) throws Exception {
        // 获取mysql 连接
        conn = DriverManager.getConnection(url, username, password);
        // 关闭自定提交
        conn.setAutoCommit(false);
    }

    
    @Override
    public void close() throws Exception {
        if (conn != null) {
            conn.close();
        }
        if (ps != null) {
            ps.close();
        }
    }

    
    @Override
    public void invoke(Book value, Context context) throws Exception {
        String sql = "insert into book " +
                "values(?,?,?,?)";
        ps = conn.prepareStatement(sql);
        ps.setLong(1, value.id);
        ps.setString(2, value.title);
        ps.setString(3, value.authors);
        ps.setInt(4, value.year);
        // 执行语句
        ps.execute();
        // 提交
        conn.commit();
    }
}

在main方法中创建Flink执行环境并写入相关代码:

注：数据源可以自定义,我这里是读取的文本文件

public class Test01 {
    public static void main(String[] args) throws Exception {
        //创建执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //读取数据源
        DataStreamSource source = env.readTextFile("D:\FlinkTest\src\main\resources\book.txt");

        //进行数据清洗
        SingleOutputStreamOperator map = source.map(new MapFunction() {
            @Override
            public Book map(String s) throws Exception {
                String[] split4 = s.split("\!\|");
                return new Book(Long.valueOf(split4[0]), split4[1], split4[2], Integer.valueOf(split4[3]));
            }
        });

        //数据入库
        map.addSink(new BookSink());

        //开始
        env.execute();
    }
}

然后执行main函数即可,即可完成数据清洗入库

任务在控制台提交时可能会报错,原因是flink中没有mysql相关jar包,把mysql相关jar包上传至flink中的lib目录中即可,注意如果是集群的话需要每台服务都上传