栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

flink 消费kafka数据几种方式

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

flink 消费kafka数据几种方式

先总结三种方式,这三种就是常见的.

@Slf4j
public class KafkaConsumerUtils {

    //简单模式
    public static FlinkKafkaConsumer createKafkaConsumer(StreamExecutionEnvironment env, String topic, String groupID) throws Exception {

        Properties prop  =new Properties();
        prop.setProperty("bootstrap.servers", "192.168.1.178:9092,192.168.1.179:9092,192.168.1.180:9092");
        prop.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        prop.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        prop.setProperty("auto.offset.reset", "latest");


        SimpleStringSchema schema = new SimpleStringSchema();
        FlinkKafkaConsumer consumer = new FlinkKafkaConsumer<>(topic, schema, prop);
        //设置自动提交offset
        consumer.setCommitOffsetsOnCheckpoints(true);
        return  consumer;
    }

    //设置消费位置
    public static FlinkKafkaConsumer flinkKafkaConsumerSerDe (StreamExecutionEnvironment env , String topic , String groupID , KafkaDeserializationSchema schema, StartupMode sm) throws IOException {

        Properties prop  =new Properties();
        prop.setProperty("bootstrap.servers", "192.168.1.178:9092,192.168.1.179:9092,192.168.1.180:9092");
        prop.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        prop.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        prop.setProperty("auto.offset.reset", "latest");



        FlinkKafkaConsumer consumer = new FlinkKafkaConsumer(topic, schema, prop);
        //设置kafka 的消费位置
        if(sm.equals(StartupMode.EARLIEST)){
            consumer.setStartFromEarliest();
        }else if (sm.equals(StartupMode.LATEST)){
            consumer.setStartFromLatest();
        }
        //设置自动提交offset
        consumer.setCommitOffsetsOnCheckpoints(true);
        return  consumer;
    }

    //多个topic
    public static FlinkKafkaConsumer mutiTopickafkaConsumer(StreamExecutionEnvironment env,String groupID) throws IOException {

        Properties prop  =new Properties();
        prop.setProperty("bootstrap.servers", "192.168.1.178:9092,192.168.1.179:9092,192.168.1.180:9092");
        prop.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        prop.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        prop.setProperty("auto.offset.reset", "latest");
        LinkedList topics = new LinkedList<>();
        topics.add("test01");
        topics.add("test02");

        FlinkKafkaConsumer consumer = new FlinkKafkaConsumer<>(topics, new SimpleStringSchema(), prop);
        return  consumer;
    }

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        String topic ="test01";
        String groupId="kafka-group";
        FlinkKafkaConsumer consumer = KafkaConsumerUtils.createKafkaConsumer(env, topic, groupId);
        env.addSource(consumer).print();

        KafkaDeserializationSchemaWrapper schemaWrapper = new KafkaDeserializationSchemaWrapper(new SimpleStringSchema());
        FlinkKafkaConsumer consumer1 = KafkaConsumerUtils.flinkKafkaConsumerSerDe(env, topic, groupId, schemaWrapper, StartupMode.LATEST);
        env.addSource(consumer1).print();


        FlinkKafkaConsumer consumer2 = KafkaConsumerUtils.mutiTopickafkaConsumer(env, groupId);
        env.addSource(consumer2).print();


        env.execute();
    }
}

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/972436.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号