-
KakfaConsumerThread 线程不断从Kafka 中消费数据
-
消费的数据存储handover 中
-
kafkaFetch 不断从handover 获取数据进行处理
-
其他流程
nitializeState、snapshotState 这两个方法是实现了CheckpointedFunction接口里面的对应方法,CheckpointedFunction 接口是Flink 提供的两个hook, 任务初始化执行initializeState,用于从状态中恢复数据, 优于open先执行, 用于其恢复offset数据;snapshotState 每次触发checkpoint 时执行,提供用户操作hook, 用于将offset 数据保存在状态中。
consumer offset 提交notifyCheckpointComplete 是实现了CheckpointListener 接口中的方法, checkpoint 完成之后的回调方法, 提交状态中的offset数据至kafka中。
对于整个offset的提交至kafka中, 类似于两阶段的提交过程:
-
第一阶段:执行checkpoint 时即调用snapshotState方法, offset 保存到状态中
-
第二阶段:checkpoint 执行完成时回调notifyCheckpointComplete方法,offset 提交到kafka中
对于第一阶段失败任务直接重启,从最近一次checkpoint记录的位点开始消费,对于第二阶段提交offset至kafka如果失败,并不会导致任务重启,只是做了日志记录,因为提交offset到kafka成功与否并不会影响任务的执行。
启动时offset指定
如果是从checkpoint 恢复,那么就会忽略客户端所指定的startMode , 也就是checkpoint 状态数据优先
producer中反射的用法FlinkKafkaInternalProducer.java
public void resumeTransaction(long producerId, short epoch) {
synchronized (producerClosingLock) {
ensureNotClosed();
Preconditions.checkState(
producerId >= 0 && epoch >= 0,
"Incorrect values for producerId %s and epoch %s",
producerId,
epoch);
LOG.info(
"Attempting to resume transaction {} with producerId {} and epoch {}",
transactionalId,
producerId,
epoch);
Object transactionManager = getField(kafkaProducer, "transactionManager");
synchronized (transactionManager) {
Object topicPartitionBookkeeper =
getField(transactionManager, "topicPartitionBookkeeper");
invoke(
transactionManager,
"transitionTo",
getEnum(
"org.apache.kafka.clients.producer.internals.TransactionManager$State.INITIALIZING"));
invoke(topicPartitionBookkeeper, "reset");
Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
setField(producerIdAndEpoch, "producerId", producerId);
setField(producerIdAndEpoch, "epoch", epoch);
invoke(
transactionManager,
"transitionTo",
getEnum(
"org.apache.kafka.clients.producer.internals.TransactionManager$State.READY"));
invoke(
transactionManager,
"transitionTo",
getEnum(
"org.apache.kafka.clients.producer.internals.TransactionManager$State.IN_TRANSACTION"));
setField(transactionManager, "transactionStarted", true);
}
}
}
protected static Object invoke(Object object, String methodName, Object... args) {
Class>[] argTypes = new Class[args.length];
for (int i = 0; i < args.length; i++) {
argTypes[i] = args[i].getClass();
}
return invoke(object, methodName, argTypes, args);
}
private static Object invoke(
Object object, String methodName, Class>[] argTypes, Object[] args) {
try {
Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
method.setAccessible(true);
return method.invoke(object, args);
} catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
throw new RuntimeException("Incompatible KafkaProducer version", e);
}
}



