栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

Flink1.14新版kafkaSource和kafkaSink

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

Flink1.14新版kafkaSource和kafkaSink

工具类
public class KafkaUtils {
    
    public static DataStreamSource getNewKafkaSource(StreamExecutionEnvironment env, String topic, String groupId, OffsetsInitializer offsets) {
        // 1.15 之后需要新方法创建kafka source
        KafkaSource source = KafkaSource.builder()
                .setProperty("partition.discovery.interval.ms", "60000")
                .setBootstrapServers(KafkaConfig.bootstrapServers)
                .setTopics(topic)
                .setGroupId(groupId)
                .setValueOnlyDeserializer(new SimpleStringSchema())
                .setStartingOffsets(offsets)
                .build();

        return env.fromSource(source, WatermarkStrategy.noWatermarks(), topic);
    }


    
    public static KafkaSink kafkaSink(String topic, String filed) {
        Properties properties = new Properties();
        properties.setProperty("compression.type", "lz4");
        properties.setProperty("compression.codec", "lz4");
        return KafkaSink.builder()
                .setBootstrapServers(KafkaConfig.bootstrapServers)
                .setRecordSerializer(KafkaRecordSerializationSchema.builder()
                        .setTopic(topic)
                        .setValueSerializationSchema(new SimpleStringSchema())
                        .setPartitioner(new FlinkKafkaPartitioner() {
                            @Override
                            public int partition(String record, byte[] key, byte[] value, String targetTopic, int[] partitions) {
                                JSONObject jsonObject = JSONObject.parseObject(record);
                                Object o = jsonObject.get(filed);
                                return Math.abs(o.hashCode() % partitions.length);
                            }
                        } )
                        .build()
                )
                //setDeliverGuarantee 1.14官方文档有错误,1.15修改过来了
                .setDeliverGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
                .setKafkaProducerConfig(properties)
                .build();

}
使用
// 以下是kafka的offset设置
// OffsetsInitializer.committedOffsets()
// OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST)
// OffsetsInitializer.timestamp(1592323200L)
// OffsetsInitializer.earliest()
// OffsetsInitializer.latest()
KafkaUtils
                .getNewKafkaSource(env, "topic", "groupId", OffsetsInitializer.latest())
                .map(new MapFunction() {
                    @Override
                    public JSONObject map(String value) throws Exception {
                        return JSON.parseObject(value,JSONObject.class);
                    }
                })
                .print("测试");
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/939085.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号