栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

flink 简单写数据到clickhouse

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

flink 简单写数据到clickhouse

完成一个简单的flink写数据到clickhouse demo,简单记录一下,数据源是自定义的,全部是一些没有用的测试数据,所以代码没有贴出来

版本
  • flink 1.13.0
  • java 1.8
  • clickhouse 22.2.2
导入依赖
    
        1.13.0
        1.8
        2.12
        1.7.30
    

        
            org.apache.flink
            flink-java
            ${flink.version}
        
        
            org.apache.flink
            flink-streaming-java_${scala.binary.version}
            ${flink.version}
        
        
            org.apache.flink
            flink-clients_${scala.binary.version}
            ${flink.version}
        

        
            ru.yandex.clickhouse
            clickhouse-jdbc
            0.3.1-patch
        
写入clickhouse实体类

写入clickhouse数据库的bean类

public class ClickhouseAlert {
    public Long timestamp;
    public String src_ip;
    public String src_port;
    public String dest_ip;
    public String dest_port;
    public String sid;
	...
}
clickhosue连接和关闭
public class ClickUtils {
    private static Connection connection;

    public static Connection getConnection() throws ClassNotFoundException, SQLException {
//Class.froName装载一个类并且对其进行实例化
        Class.forName("ru.yandex.clickhouse.ClickHouseDriver");
        String url = "jdbc:clickhouse://192.168.142.130:8123/default";
        connection = DriverManager.getConnection(url);
        return connection;
    }

    public void close() throws SQLException {
        connection.close();
    }
}
RichSinkFunction类

关于RichSinkFunction,属于flink中的知识点,这里不做详细讲述
方法重写:

  • open:连接资源
  • close:关闭释放资源
  • invoke:插入数据操作
public class ClickhouseFunction extends RichSinkFunction {
    Connection connection = null;

    String sql;

    public ClickhouseFunction(String sql) {
        this.sql = sql;
    }

    @Override
    public void open(Configuration parameters) throws Exception {
        super.open(parameters);
        connection = ClickUtils.getConnection();
    }

    @Override
    public void close() throws Exception {
        super.close();
        if(connection!=null){
            connection.close();
        }
    }

    @Override
    public void invoke(Alert value, Context context) throws Exception {
        PreparedStatement preparedStatement = connection.prepareStatement(sql);
        preparedStatement.setLong(1,value.timestamp);
        preparedStatement.setString(2,value.src_ip);
        preparedStatement.setString(3,value.src_port);
        preparedStatement.setString(4, value.dest_ip);
        preparedStatement.setString(5,value.dest_port);
        preparedStatement.setString(6,value.sid);
        preparedStatement.setString(7,value.signature);
        preparedStatement.setString(8,value.category);
        preparedStatement.setInt(9,value.severity);
        preparedStatement.setString(10,value.device_id);
        preparedStatement.addBatch();

        long startTime = System.currentTimeMillis();
        int[] ints = preparedStatement.executeBatch();
//        connection.commit();
        long endTime = System.currentTimeMillis();
//        System.out.println("批量插入完毕用时:" + (endTime - startTime) + " -- 插入数据 = " + ints.length);
        System.out.println("数据插入成功:"+value);
    }
}
主函数
public class StreamToClickhouse {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);


        SingleOutputStreamOperator stram = env.addSource(new AlertSource()).assignTimestampsAndWatermarks(WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(1)).withTimestampAssigner(new SerializableTimestampAssigner() {
            @Override
            public long extractTimestamp(Alert element, long recordTimestamp) {
                return element.timestamp;
            }
        }));

        // sink
        String sql = "INSERT INTO IdsAlert (`timestamp`,src_ip,src_port,dest_ip,dest_port,sid,signature,category,severity,device_id) VALUES (?,?,?,?,?,?,?,?,?,?)";
        stram.addSink(new ClickhouseFunction(sql));
        stram.print();

        env.execute();
    }
}
运行结果

查看clickshouse数据库是否写入数据

参考
  • https://blog.csdn.net/weixin_32265569/article/details/112133937
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/956169.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号