完成一个简单的flink写数据到clickhouse demo,简单记录一下,数据源是自定义的,全部是一些没有用的测试数据,所以代码没有贴出来
版本- flink 1.13.0
- java 1.8
- clickhouse 22.2.2
写入clickhouse实体类1.13.0 1.8 2.12 1.7.30 org.apache.flink flink-java ${flink.version} org.apache.flink flink-streaming-java_${scala.binary.version} ${flink.version} org.apache.flink flink-clients_${scala.binary.version} ${flink.version} ru.yandex.clickhouse clickhouse-jdbc 0.3.1-patch
写入clickhouse数据库的bean类
public class ClickhouseAlert {
public Long timestamp;
public String src_ip;
public String src_port;
public String dest_ip;
public String dest_port;
public String sid;
...
}
clickhosue连接和关闭
public class ClickUtils {
private static Connection connection;
public static Connection getConnection() throws ClassNotFoundException, SQLException {
//Class.froName装载一个类并且对其进行实例化
Class.forName("ru.yandex.clickhouse.ClickHouseDriver");
String url = "jdbc:clickhouse://192.168.142.130:8123/default";
connection = DriverManager.getConnection(url);
return connection;
}
public void close() throws SQLException {
connection.close();
}
}
RichSinkFunction类
关于RichSinkFunction,属于flink中的知识点,这里不做详细讲述
方法重写:
- open:连接资源
- close:关闭释放资源
- invoke:插入数据操作
public class ClickhouseFunction extends RichSinkFunction主函数{ Connection connection = null; String sql; public ClickhouseFunction(String sql) { this.sql = sql; } @Override public void open(Configuration parameters) throws Exception { super.open(parameters); connection = ClickUtils.getConnection(); } @Override public void close() throws Exception { super.close(); if(connection!=null){ connection.close(); } } @Override public void invoke(Alert value, Context context) throws Exception { PreparedStatement preparedStatement = connection.prepareStatement(sql); preparedStatement.setLong(1,value.timestamp); preparedStatement.setString(2,value.src_ip); preparedStatement.setString(3,value.src_port); preparedStatement.setString(4, value.dest_ip); preparedStatement.setString(5,value.dest_port); preparedStatement.setString(6,value.sid); preparedStatement.setString(7,value.signature); preparedStatement.setString(8,value.category); preparedStatement.setInt(9,value.severity); preparedStatement.setString(10,value.device_id); preparedStatement.addBatch(); long startTime = System.currentTimeMillis(); int[] ints = preparedStatement.executeBatch(); // connection.commit(); long endTime = System.currentTimeMillis(); // System.out.println("批量插入完毕用时:" + (endTime - startTime) + " -- 插入数据 = " + ints.length); System.out.println("数据插入成功:"+value); } }
public class StreamToClickhouse {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
SingleOutputStreamOperator stram = env.addSource(new AlertSource()).assignTimestampsAndWatermarks(WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(1)).withTimestampAssigner(new SerializableTimestampAssigner() {
@Override
public long extractTimestamp(Alert element, long recordTimestamp) {
return element.timestamp;
}
}));
// sink
String sql = "INSERT INTO IdsAlert (`timestamp`,src_ip,src_port,dest_ip,dest_port,sid,signature,category,severity,device_id) VALUES (?,?,?,?,?,?,?,?,?,?)";
stram.addSink(new ClickhouseFunction(sql));
stram.print();
env.execute();
}
}
运行结果
查看clickshouse数据库是否写入数据
- https://blog.csdn.net/weixin_32265569/article/details/112133937



