栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

flink Window Join、Interval Join、Window CoGroup (两流匹配 指定key联结,开窗口进行窗口操作)

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

flink Window Join、Interval Join、Window CoGroup (两流匹配 指定key联结,开窗口进行窗口操作)

文章目录
    • 1. 窗口联结(Window Join)
      • 1.1 窗口联结的调用
      • 1.2 窗口联结的处理流程
      • 1.3 窗口联结实例
    • 2.间隔联结(Interval Join)
      • 2.1 间隔联结的原理
      • 2.2 间隔联结的调用
      • 2.3 间隔联结实例
    • 3.窗口同组联结(Window CoGroup)


1. 窗口联结(Window Join)

  Flink 提供了一个窗口联结(window join)算子,可以定义时间窗口,并将两条流中共享一个公共键(key)的数据放在窗口中进行配对处理

1.1 窗口联结的调用

  窗口联结在代码中的实现,首先需要调用 DataStream 的.join()方法来合并两条流,得到一个 JoinedStreams;接着通过.where()和.equalTo()方法指定两条流中联结的 key;然后通过.window()开窗口,并调用.apply()传入联结窗口函数进行处理计算

stream1.join(stream2)
 .where()
 .equalTo()
 .window()
 .apply()

  .where()的参数是键选择器(KeySelector),用来指定第一条流中的 key; 而.equalTo()传入的 KeySelector 则指定了第二条流中的 key。两者相同的元素,如果在同一窗口中,就可以匹配起来

1.2 窗口联结的处理流程

  首先会按照 key 分组、进入对应的窗口中存储;当到达窗口结束时间时,算子会先统计出窗口内两条流的数据的所有组合,也就是对两条流中的数据做一个笛卡尔积(相当于表的交叉连接,cross join),然后进行遍历,把每一对匹配的数据,作为参数(first,second)传入 JoinFunction 的.join()方法进行计算处理,得到的结果直接输出。所以窗口中每有一对数据成功联结匹配,JoinFunction的.join()方法就会被调用一次,并输出一个结果

1.3 窗口联结实例
public class WindowJoinExample {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        DataStream> stream1 = env.fromElements(Tuple2.of("a", 1000L), Tuple2.of("b", 1000L), Tuple2.of("a", 2000L), Tuple2.of("b", 2000L))
                .assignTimestampsAndWatermarks(WatermarkStrategy
                        .>forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner>() {
                                                   @Override
                                                   public long extractTimestamp(Tuple2 stringLongTuple2, long l) {
                                                       return stringLongTuple2.f1;
                                                   }
                                               }
                        )
                );
        DataStream> stream2 = env.fromElements(Tuple2.of("a", 3000L), Tuple2.of("b", 3000L), Tuple2.of("a", 4000L), Tuple2.of("b", 4000L))
                .assignTimestampsAndWatermarks(WatermarkStrategy
                        .>forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner>() {
                                                   @Override
                                                   public long extractTimestamp(Tuple2 stringLongTuple2, long l) {
                                                       return stringLongTuple2.f1;
                                                   }
                                               }
                        )
                );
        stream1.join(stream2)
                .where(new KeySelector, String>() {
                    @Override
                    public String getKey(Tuple2 value) throws Exception {
                        return value.f0;
                    }
                }).equalTo(new KeySelector, String>() {
                               @Override
                               public String getKey(Tuple2 value) throws Exception {
                                   return value.f0;
                               }
                           }
        )
                .window(TumblingEventTimeWindows.of(Time.seconds(5)))
                .apply(new FlatJoinFunction, Tuple2, String>() {
                    @Override
                    public void join(Tuple2 first, Tuple2 second, Collector out) throws Exception {
                        out.collect(first + "=>" + second);
                    }
                })
                .print();
        env.execute();
    }
}

2.间隔联结(Interval Join)

  在电商网站中,用户行为往往会有短时间内的强关联。假设有两条流,一条是下订单的流,一条是浏览数据的流,可针对同一个用户做一个联结,使用户的下订单事件和用户的最近十分钟的浏览数据进行一个联结查询(Interval Join)

2.1 间隔联结的原理

The interval join joins elements of two streams ( A & B ) with a common key and where elements of stream B have timestamps that lie in a relative time interval to timestamps of elements in stream A

  对黄色的流(A)中的任意一个数据元素 a,划分一段时间间隔:[a.timestamp + lowerBound, a.timestamp + upperBound],绿色的流( B)中的数据元素 b, key相同且时间戳在这
个区间范围内,a 和 b 成功配对

where a and b are elements of A and B that share a common key. Both the lower and upper bound can be either negative or positive as long as the lower bound is always smaller or equal to the upper bound. The interval join currently only performs inner joins.
When a pair of elements are passed to the ProcessJoinFunction, they will be assigned with the larger timestamp (which can be accessed via the ProcessJoinFunction.Context) of the two elements.

2.2 间隔联结的调用
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.functions.co.ProcessJoinFunction;
import org.apache.flink.streaming.api.windowing.time.Time;

...

DataStream orangeStream = ...;
DataStream greenStream = ...;

orangeStream
    .keyBy()
    .intervalJoin(greenStream.keyBy())
    .between(Time.milliseconds(-2), Time.milliseconds(1))
    .process (new ProcessJoinFunction

        @Override
        public void processElement(Integer left, Integer right, Context ctx, Collector out) {
            out.collect(left + "," + right);
        }
    });

2.3 间隔联结实例
public class IntervalJoinTest {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        SingleOutputStreamOperator> orderStream = env.fromElements(
                Tuple3.of("Mary", "order-1", 5000L),
                Tuple3.of("Alice", "order-2", 5000L),
                Tuple3.of("Bob", "order-3", 20000L),
                Tuple3.of("Alice", "order-4", 20000L),
                Tuple3.of("Cary", "order-5", 51000L)
        )
                .assignTimestampsAndWatermarks(WatermarkStrategy
                        .>forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner>() {
                            @Override
                            public long extractTimestamp(Tuple3
                                                                 element, long recordTimestamp) {
                                return element.f2;
                            }
                        })
                );
        SingleOutputStreamOperator clickStream = env.fromElements(
                new Event("Bob", "./cart", 2000L),
                new Event("Alice", "./prod?id=100", 3000L),
                new Event("Alice", "./prod?id=200", 3500L),
                new Event("Bob", "./prod?id=2", 2500L),
                new Event("Alice", "./prod?id=300", 36000L),
                new Event("Bob", "./home", 30000L),
                new Event("Bob", "./prod?id=1", 23000L),
                new Event("Bob", "./prod?id=3", 33000L)
        )
                .assignTimestampsAndWatermarks(WatermarkStrategy
                        .forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner() {
                            @Override
                            public long extractTimestamp(Event element, long recordTimestamp) {
                                return element.timestamp;
                            }
                        })
                );
        orderStream
                .keyBy(new KeySelector, String>() {
                    @Override
                    public String getKey(Tuple3 data) throws Exception {
                        return data.f0;
                    }
                })
                .intervalJoin(clickStream.keyBy(new KeySelector() {
                    @Override
                    public String getKey(Event data) throws Exception {
                        return data.user;
                    }
                }))
                .between(Time.seconds(-5), Time.seconds(10)) //之前5秒钟 之后10秒钟 范围
                .process(new ProcessJoinFunction, Event, String>() {
                    @Override
                    public void processElement(Tuple3 left, Event right, Context ctx, Collector out) throws Exception {
                        out.collect(right + " => " + left);
                    }
                })
                .print();

        env.execute();
    }
}

结果:


3.窗口同组联结(Window CoGroup)

  与 window join 非常类似,将两条流合并之后开窗处理匹配的元素,调用时只需要将.join()换为.coGroup()

stream1.coGroup(stream2)
 .where()
 .equalTo()
 .window(TumblingEventTimeWindows.of(Time.hours(1)))
 .apply()

  与 window join 的区别在于,调用.apply()方法定义具体操作时,传入的是一个CoGroupFunction

  传入的三个参数,分别代表两条流中的数据以及用于输出的收集器(Collector)。不同的是,前两个参数不再是单独的每一组配对数据了,而是传入了可遍历的数据集合。即不会再去计算窗口中两条流数据集的笛卡尔积,而是直接把收集到的所有数据一次性传入,至于要怎样配对完全是自定义的。这样.coGroup()方法只会被调用一次,而且即使一条流的数据没有任何另一条流的数据匹配,也可以出现在集合中

public class CoGroupExample {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        DataStream> stream1 = env.fromElements(
                Tuple2.of("a", 1000L),
                Tuple2.of("b", 1000L),
                Tuple2.of("a", 2000L),
                Tuple2.of("b", 2000L)
        )
                .assignTimestampsAndWatermarks(WatermarkStrategy
                        .>forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner>() {
                                                   @Override
                                                   public long extractTimestamp(Tuple2 stringLongTuple2, long l) {
                                                       return stringLongTuple2.f1;
                                                   }
                                               }
                        )
                );
        DataStream> stream2 = env.fromElements(
                Tuple2.of("a", 3000L),
                Tuple2.of("b", 3000L),
                Tuple2.of("a", 4000L),
                Tuple2.of("b", 4000L)
        )
                .assignTimestampsAndWatermarks(
                        WatermarkStrategy
                                .>forMonotonousTimestamps()
                                .withTimestampAssigner(new SerializableTimestampAssigner>() {
                                                           @Override
                                                           public long extractTimestamp(Tuple2 stringLongTuple2, long l) {
                                                               return stringLongTuple2.f1;
                                                           }
                                                       }
                                )
                );
        stream1
                .coGroup(stream2)
                .where(r -> r.f0)
                .equalTo(r -> r.f0)
                .window(TumblingEventTimeWindows.of(Time.seconds(5)))
                .apply(new CoGroupFunction, Tuple2, String>() {
                    @Override
                    public void coGroup(Iterable> iter1, Iterable> iter2, Collector collector) throws Exception {
                        collector.collect(iter1 + "=>" + iter2);
                    }
                })
                .print();
        env.execute();
    }
}
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/952482.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号