使用zmq推送海量数据
例如:项目场景:推送股市的行情实时变化,数据量很大
问题描述
当zmq的消费者消费速度过慢时,会出现导致zmq挂掉
例如:数据传输过程中数据不时出现丢失的情况,偶尔会丢失一部分数据
原因分析:
消息消费方的业务代码效率低,导致速度远远低于消息生产,导致zmq压垮了
解决方案:
在消费者使用线程队列,把接收到的数据放到线程队列中多线程异步消费数据
例如:使用 Disruptor 线程队列:
Disruptor使用请参考:Disruptor详解 - 简书 zmq + disruptor 代码@Order(10)
@Override
public void run(ApplicationArguments args) throws Exception {
try (
ZContext context = new ZContext()) {
ZMQ.Socket subscriber = context.createSocket(SocketType.SUB);
subscriber.connect(zmqPath);
subscriber.subscribe(BsConstant.BOOK_CHANNEL_PRO);
ZMQ.Socket subscriber1 = context.createSocket(SocketType.SUB);
subscriber1.connect(zmqPath);
subscriber1.subscribe(BsConstant.TICKER_CHANNEL_PRO);
logger.info("sub client start!!!");
// 创建bufferSize ,也就是RingBuffer大小,必须是2的N次方
int ringBufferSize = 1024 * 2;
Disruptor disruptor =
new Disruptor<>(new ZmqEventFactory(), ringBufferSize, new CustomizableThreadFactory("zmq-handler"));
disruptor.handleEventsWith(new ZmqDisruptorHandler());
disruptor.start();
ZmqDisruptorProducer producer = new ZmqDisruptorProducer(disruptor.getRingBuffer());
while (true) {
String channel = subscriber.recvStr();
byte[] bytes = subscriber.recv(0);
producer.onData(bytes,channel);
String channel1 = subscriber1.recvStr();
byte[] bytes1 = subscriber1.recv(0);
producer.onData(bytes1,channel1);
}
}catch (Exception e){
logger.error("sub book_p | ticker_p error :", e );
}
}
Disruptor 工厂
package xxx.yyy.direct.factory; import com.lmax.disruptor.EventFactory; import xxx.yyy.direct.dto.ZmqDisruptorDto; public class ZmqEventFactory implements EventFactory{ @Override public ZmqDisruptorDto newInstance() { return new ZmqDisruptorDto(); } }
使用disruptor时,定义的数据格式,这个根据自己的业务决定,比如说你要把从zmq接收到的数据放在distuptor里面消费,那你就定义一个对象接收zmq数据,并把它放进disruptor
我这里使用了protobuf序列化格式:
package xxx.yyy.direct.dto;
import lombok.Data;
@Data
public class ZmqDisruptorDto {
//zmq推送的 buf 序列化结果
private byte[] zmqSource;
}
定义消息队列的消费线程
package xxx.yyy.direct.handler; import cn.hutool.core.util.StrUtil; import com.google.gson.JsonArray; import com.google.protobuf.InvalidProtocolBufferException; import com.lmax.disruptor.EventHandler; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.BeanUtils; import org.springframework.beans.BeansException; import org.springframework.stereotype.Component; import xxx.yyy.direct.constant.BsConstant; import xxx.yyy.direct.dto.ZmqDisruptorDto; import xxx.yyy.direct.dto.market.OrderBookDto; import xxx.yyy.direct.dto.market.TickerDto; import xxx.yyy.direct.dto.protobufDate.OrderBookBuf; import xxx.yyy.direct.dto.protobufDate.TickerBuf; import xxx.yyy.direct.manager.PosCacheManager; import xxx.yyy.direct.utils.ProtobufUtil; import javax.annotation.Resource; import java.math.BigDecimal; @Slf4j @Component public class ZmqDisruptorHandler implements EventHandler{ @Override public void onEvent(ZmqDisruptorDto zmqDisruptorDto, long sequence, boolean endOfBatch) throws Exception { //这里就是把你之前zmq消费者的业务代码搬到这里来就好了 } }



