RocketMQ的编程模型org.apache.rocketmq rocketmq-client 4.9.1 org.apache.rocketmq rocketmq-acl 4.9.1
消息发送者的固定步骤
- 1.创建消息生产者producer,并制定生产者组名
- 2.指定Nameserver地址
- 3.启动producer
- 4.创建消息对象,指定主题Topic、Tag和消息体
- 5.发送消息
- 6.关闭生产者producer
消息消费者的固定步骤
- 1.创建消费者Consumer,制定消费者组名
- 2.指定Nameserver地址
- 3.订阅主题Topic和Tag
- 4.设置回调函数,处理消息
- 5.启动消费者consumer
RocketMQ的推模式也是由拉模式封装出来的
二、顺序消息 RocketMQ保证的是消息的局部有序,而不是全局有序SendResult sendResult = producer.send(msg, new MessageQueueSelector() {....}, orderId);
consumer.registerMessageListener(new MessageListenerOrderly() {...});
三、广播消息 consumer.setMessageModel(MessageModel.BROADCASTING); 四、延迟消息 message.setDelayTimeLevel(3); //messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h 五、批量消息 producer.send(messages);消息大小限制在4M内, 应该有相同的Topic、waitStoreMsgOK、不能是延迟消息、事务消息
六、过滤消息(服务端broker过滤) Tag过滤 和 SQL过滤broker 添加支持: enablePropertyFilter=true
tagconsumer.subscribe("TagFilterTest", "TagA || TagC");
sqlproduct
msg.putUserProperty("a", String.valueOf(i));
consumer
consumer.subscribe("SqlFilterTest", MessageSelector.bySql("(TAGS is not null and TAGS in ('TagA', 'TagB'))" + "and (a is not null and a between 0 and 3)"));
SQL92语法:
RocketMQ只定义了一些基本语法来支持这个特性
数值比较,比如:>,>=,<,<=,BETWEEN,=;
字符比较,比如:=,<>,IN;
IS NULL`` 或者 IS NOT NULL;
逻辑符号 AND,OR,NOT;
常量支持类型为:
数值,比如:123,3.1415;
字符,比如:'abc',必须用单引号包裹起来;
NULL,特殊的常量
布尔值,TRUE 或 FALSE
- 事务消息不支持延迟消息和批量消息
- 单个消息的检查次数限制为 15 次,但是用户可以通过 Broker 配置文件的 transactionCheckMax参数来修改此限制。如果已经检查某条消息超过 N 次的话( N = transactionCheckMax ) 则 Broker 将丢弃此消息,并在默认情况下同时打印错误日志。用户可以通过重写 AbstractTransactionCheckListener 类来修改这个行为
- 事务消息将在 Broker 配置文件中的参数 transactionMsgTimeout 这样的特定时间长度之后被检查。当发送事务消息时,用户还可以通过设置用户属性 CHECK_IMMUNITY_TIME_IN_SECONDS 来改变这个限制,该参数优先于 transactionMsgTimeout 参数
- 事务性消息可能不止一次被检查或消费
- 提交给用户的目标主题消息可能会失败,目前这依日志的记录而定。它的高可用性通过 RocketMQ 本身的高可用性机制来保证,如果希望确保事务消息不丢失、并且事务完整性得到保证,建议使用同步的双重写入机制
- 务消息的生产者 ID 不能与其他类型消息的生产者 ID 共享。与其他类型的消息不同,事务消息允许反向查询、MQ服务器能通过它们的生产者 ID 查询到消费者
SendResult sendResult = producer.sendMessageInTransaction(msg, null);
TransactionListener transactionListener = new TransactionListenerImpl(); ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue(2000), new ThreadFactory() { @Override public Thread newThread(Runnable r) { Thread thread = new Thread(r); thread.setName("client-transaction-msg-check-thread"); return thread; } }); producer.setExecutorService(executorService); producer.setTransactionListener(transactionListener);
public interface TransactionListener {
//执行本地事务
LocalTransactionState executeLocalTransaction(Message var1, Object var2);
//检查本地事务
LocalTransactionState checkLocalTransaction(MessageExt var1);
}
- LocalTransactionState.COMMIT_MESSAGE;
- LocalTransactionState.ROLLBACK_MESSAGE;
- LocalTransactionState.UNKNOW;
事务消息机制的关键是在发送消息时,会将消息转为一个half半消息,并存入RocketMQ内部的一个 RMQ_SYS_TRANS_HALF_TOPIC 这个Topic,这样对消费者是不可见的。再经过一系列事务检查通过后,再将消息转存到目标Topic,这样对消费者就可见了
权限控制(ACL)主要为RocketMQ提供Topic资源级别的用户访问控制
用户在使用RocketMQ权限控制时,可以在Client客户端通过 RPCHook注入AccessKey和SecretKey签名;
将对应的权限控制属性(包括Topic访问权限、IP白名单和AccessKey和SecretKey签名等)设置在$ROCKETMQ_HOME/conf/plain_acl.yml的配置文件中
Broker端对AccessKey所拥有的权限进行校验,校验不过,抛出异常
Broker端具体的配置信息可以参见 https://github.com/apache/rocketmq/blob/develop/docs/cn/acl/user_guide.md
-
主要是在broker.conf中打开acl的标志:aclEnable=true
-
权限配置 plain_acl.yml, 配置文件是热加载, 不用重启Broker服务
#全局白名单,不受ACL控制 #通常需要将主从架构中的所有节点加进来 globalWhiteRemoteAddresses: - 10.10.103.* - 192.168.0.* accounts: #第一个账户 - accessKey: RocketMQ secretKey: 12345678 whiteRemoteAddress: admin: false defaultTopicPerm: DENY #默认Topic访问策略是拒绝 defaultGroupPerm: SUB #默认Group访问策略是只允许订阅 topicPerms: - topicA=DENY #topicA拒绝 - topicB=PUB|SUB #topicB允许发布和订阅消息 - topicC=SUB #topicC只允许订阅 groupPerms: # the group should convert to retry topic - groupA=DENY - groupB=PUB|SUB - groupC=SUB #第二个账户,只要是来自192.168.1.*的IP,就可以访问所有资源 - accessKey: rocketmq2 secretKey: 12345678 whiteRemoteAddress: 192.168.1.* # if it is admin, it could access all resources admin: true
static RPCHook getAclRPCHook() {
return new AclClientRPCHook(new SessionCredentials(ACL_ACCESS_KEY,ACL_SECRET_KEY));
}
//producer
DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName", getAclRPCHook());
//consumer
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_5", getAclRPCHook(), new AllocateMessageQueueAveragely());
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("please_rename_unique_group_name_6", getAclRPCHook());
SpringBoot
https://github.com/apache/rocketmq-spring
pom.xmlapplication.propertiesorg.apache.rocketmq rocketmq-spring-boot-starter 2.2.1 org.springframework.boot spring-boot-starter org.springframework spring-core org.springframework spring-webmvc
rocketmq.name-server=rocketmq-2.localhost.com:9876 rocketmq.producer.group=springBootGroup消息生产者 RocketMQTemplate
@Component
public class SpringProducer {
@Resource
private RocketMQTemplate rocketMQTemplate;
public void sendMessage(String topic,String msg){
this.rocketMQTemplate.convertAndSend(topic,msg);
}
public void sendMessageInTransaction(String topic,String msg) throws InterruptedException {
String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
for (int i = 0; i < 10; i++) {
//尝试在Header中加入一些自定义的属性。
Message message = MessageBuilder.withPayload(msg)
.setHeader(RocketMQHeaders.TRANSACTION_ID,"TransID_"+i)
//发到事务监听器里后,这个自己设定的TAGS属性会丢失。但是上面那个属性不会丢失。
.setHeader(RocketMQHeaders.TAGS,tags[i % tags.length])
//MyProp在事务监听器里也能拿到,为什么就单单这个RocketMQHeaders.TAGS拿不到?这只能去调源码了。
.setHeader("MyProp","MyProp_"+i)
.build();
String destination =topic+":"+tags[i % tags.length];
//这里发送事务消息时,还是会转换成RocketMQ的Message对象,再调用RocketMQ的API完成事务消息机制。
SendResult sendResult = rocketMQTemplate.sendMessageInTransaction(destination, message,destination);
System.out.printf("%s%n", sendResult);
Thread.sleep(10);
}
}
}
事务监听器 SpringBoot的自带的事务ID不一定会一直 需要自己实现通过 setHeader("MyProp","MyProp_"+i) 设置, msg.getHeaders().get("MyProp") 获取
@RocketMQTransactionListener(rocketMQTemplateBeanName = "rocketMQTemplate")
public class MyTransactionImpl implements RocketMQLocalTransactionListener {
private ConcurrentHashMap
由于事件监听器需要通过rocketMQTemplateBeanName名字来识别不同Bean
@ExtRocketMQTemplateConfiguration()
public class ExtRocketMQTemplate extends RocketMQTemplate {}
消息消费者 @RocketMQMessageListener
@Component @RocketMQMessageListener(consumerGroup = "MyConsumerGroup", topic = "TestTopic",consumeMode= ConsumeMode.CONCURRENTLY) public class SpringConsumer implements RocketMQListener{ @Override public void onMessage(String message) { System.out.println("Received message : "+ message); } }
RocketMQMessageListener
public @interface RocketMQMessageListener {
String NAME_SERVER_PLACEHOLDER = "${rocketmq.name-server:}";
String ACCESS_KEY_PLACEHOLDER = "${rocketmq.consumer.access-key:}";
String SECRET_KEY_PLACEHOLDER = "${rocketmq.consumer.secret-key:}";
String TRACE_TOPIC_PLACEHOLDER = "${rocketmq.consumer.customized-trace-topic:}";
String ACCESS_CHANNEL_PLACEHOLDER = "${rocketmq.access-channel:}";
String consumerGroup();
String topic();
//TAG, SQL92
SelectorType selectorType() default SelectorType.TAG;
String selectorExpression() default "*";
//CONCURRENTLY,ORDERLY;
ConsumeMode consumeMode() default ConsumeMode.CONCURRENTLY;
//BROADCASTING,CLUSTERING;
MessageModel messageModel() default MessageModel.CLUSTERING;
int consumeThreadMax() default 64;
int maxReconsumeTimes() default -1;
long consumeTimeout() default 15L;
int replyTimeout() default 3000;
String accessKey() default "${rocketmq.consumer.access-key:}";
String secretKey() default "${rocketmq.consumer.secret-key:}";
boolean enableMsgTrace() default false;
String customizedTraceTopic() default "${rocketmq.consumer.customized-trace-topic:}";
String nameServer() default "${rocketmq.name-server:}";
String accessChannel() default "${rocketmq.access-channel:}";
}
Test
@RunWith(SpringRunner.class)
@SpringBootTest
public class SpringRocketTest {
@Resource
private RocketMQTemplate rocketMQTemplate;
@Test
public void sendMessageTest(){
String springTopic="TestTopic";
//发送字符消息
SendResult sendResult = rocketMQTemplate.syncSend(springTopic, "Hello, World!");
System.out.printf("syncSend1 to topic %s sendResult=%s %n", springTopic, sendResult);
sendResult = rocketMQTemplate.syncSend(springTopic, new User().setUserAge((byte) 18).setUserName("Kitty"));
System.out.printf("syncSend1 to topic %s sendResult=%s %n", springTopic, sendResult);
sendResult = rocketMQTemplate.syncSend(springTopic, MessageBuilder.withPayload(
new User().setUserAge((byte) 21).setUserName("Lester")).setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON_VALUE).build());
System.out.printf("syncSend1 to topic %s sendResult=%s %n", springTopic, sendResult);
//发送对象消息
rocketMQTemplate.asyncSend(springTopic, new OrderPaidEvent("T_001", new BigDecimal("88.00")), new SendCallback() {
@Override
public void onSuccess(SendResult var1) {
System.out.printf("async onSucess SendResult=%s %n", var1);
}
@Override
public void onException(Throwable var1) {
System.out.printf("async onException Throwable=%s %n", var1);
}
});
//发送指定TAG的消息
rocketMQTemplate.convertAndSend(springTopic + ":tag0", "I'm from tag0"); // tag0 will not be consumer-selected
System.out.printf("syncSend topic %s tag %s %n", springTopic, "tag0");
rocketMQTemplate.convertAndSend(springTopic + ":tag1", "I'm from tag1");
System.out.printf("syncSend topic %s tag %s %n", springTopic, "tag1");
//同步发送消息并且返回一个String类型的结果。
String replyString = rocketMQTemplate.sendAndReceive(springTopic, "request string", String.class);
System.out.printf("send %s and receive %s %n", "request string", replyString);
//同步发送消息并且返回一个Byte数组类型的结果。
byte[] replyBytes = rocketMQTemplate.sendAndReceive(springTopic, MessageBuilder.withPayload("request byte[]").build(), byte[].class, 3000);
System.out.printf("send %s and receive %s %n", "request byte[]", new String(replyBytes));
//同步发送一个带hash参数的请求(排序消息),并返回一个User类型的结果
User requestUser = new User().setUserAge((byte) 9).setUserName("requestUserName");
User requestUser2 = new User().setUserAge((byte) 9).setUserName("requestUserName");
User requestUser3 = new User().setUserAge((byte) 9).setUserName("requestUserName");
User requestUser4 = new User().setUserAge((byte) 9).setUserName("requestUserName");
User replyUser = rocketMQTemplate.sendAndReceive(springTopic, requestUser, User.class, "order-id");
User replyUser2 = rocketMQTemplate.sendAndReceive(springTopic, requestUser2, User.class, "order-id");
User replyUser3 = rocketMQTemplate.sendAndReceive(springTopic, requestUser3, User.class, "order-id");
User replyUser4 = rocketMQTemplate.sendAndReceive(springTopic, requestUser4, User.class, "order-id");
System.out.printf("send %s and receive %s %n", requestUser, replyUser);
//同步发送一个带延迟级别的消息(延迟消息),并返回一个泛型结果
ProductWithPayload replyGenericObject = rocketMQTemplate.sendAndReceive(springTopic, "request generic",
new TypeReference>() {
}.getType(), 30000, 2);
System.out.printf("send %s and receive %s %n", "request generic", replyGenericObject);
//异步发送消息,返回String类型结果。
rocketMQTemplate.sendAndReceive(springTopic, "request string", new RocketMQLocalRequestCallback() {
@Override public void onSuccess(String message) {
System.out.printf("send %s and receive %s %n", "request string", message);
}
@Override public void onException(Throwable e) {
e.printStackTrace();
}
});
//异步发送消息,并返回一个User类型的结果。
rocketMQTemplate.sendAndReceive(springTopic, new User().setUserAge((byte) 9).setUserName("requestUserName"), new RocketMQLocalRequestCallback() {
@Override public void onSuccess(User message) {
System.out.printf("send user object and receive %s %n", message.toString());
}
@Override public void onException(Throwable e) {
e.printStackTrace();
}
}, 5000);
//发送批量消息
List msgs = new ArrayList();
for (int i = 0; i < 10; i++) {
msgs.add(MessageBuilder.withPayload("Hello RocketMQ Batch Msg#" + i).
setHeader(RocketMQHeaders.KEYS, "KEY_" + i).build());
}
SendResult sr = rocketMQTemplate.syncSend(springTopic, msgs, 60000);
System.out.printf("--- Batch messages send result :" + sr);
}
}



