栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

【牛客讨论区】第五章:kafka

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

目录
  • 1. 阻塞队列
  • 2. Kafka入门
  • 3. Spring整合Kafka
  • 4. 发送系统通知
    • 4.1 entity
    • 4.2 controller
  • 5. 显示系统通知
    • 5.1 通知列表
    • 5.2 通知详情
    • 5.3 未读消息

1. 阻塞队列

BlockingQueue

  • 解决线程通信的问题。
  • 阻塞方法:put、take。

生产者消费者模式

  • 生产者:产生数据的线程。
  • 消费者:使用数据的线程。

实现类

  • ArrayBlockingQueue
  • LinkedBlockingQueue
  • PriorityBlockingQueue、SynchronousQueue、DelayQueue等


test 目录下新建测试类

package com.nowcoder.community;

import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;

public class BlockingQueueTests {

    public static void main(String[] args) {
        BlockingQueue queue = new ArrayBlockingQueue<>(10);
        new Thread(new Producer(queue)).start();
        new Thread(new Consumer(queue)).start();
        new Thread(new Consumer(queue)).start();
        new Thread(new Consumer(queue)).start();
    }
}
class Producer implements Runnable{

    private BlockingQueue queue;

    public Producer(BlockingQueue queue) {
        this.queue = queue;
    }

    @Override
    public void run() {
        try {
            for (int i = 0; i < 100; i++) {
                queue.put(i);
                System.out.println(Thread.currentThread().getName() + "生产:" + queue.size());
                Thread.sleep(20);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

class Consumer implements Runnable {

    private BlockingQueue queue;

    public Consumer(BlockingQueue queue) {
        this.queue = queue;
    }

    @Override
    public void run() {
        try {
            while (true) {
                queue.take();
                System.out.println(Thread.currentThread().getName() + "消费了:" + queue.size());
                Thread.sleep(new Random().nextInt(1000));
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

运行 main 方法查看结果

2. Kafka入门

Kafka简介

  • Kafka是一个分布式的流媒体平台。
  • 应用:消息系统、日志收集、用户行为追踪、流式处理。

Kafka特点

  • 高吞吐量、消息持久化、高可靠性、高扩展性。

Kafka术语

  • Broker、Zookeeper
  • Topic、Partition、Offset
  • Leader Replica 、Follower Replica

http://kafka.apache.org


小知识:对硬盘的顺序读取,速度是很快的,甚至要快于对内存的随机读取,这也是 kafka 既能持久化又能高性能的原因

【配置】
将素材中的 kafka_2.12-2.2.0.tgz 解压到某个目录即可
配置一下 E:kafka_2.12-2.2.0config 目录下的 zookeeper.properties


dataDir=d:/work/data/zookeeper

配置 server.properties
60行

log.dirs=d:/work/data/kafka-logs

【启动】
打开一个 cmd ,启动 zookeeper

C:Users15642>e:

E:>cd e:/kafka_2.12-2.2.0

e:kafka_2.12-2.2.0>binwindowszookeeper-server-start.bat configzookeeper.properties

再打开一个 cmd,启动 kafka

C:Users15642>e:

E:>cd e:/kafka_2.12-2.2.0

e:kafka_2.12-2.2.0>binwindowskafka-server-start.bat configserver.properties

自动创建了对应的文件夹

再打开一个 cmd
使用 kafka 的生产者,创建主题,发送消息

C:Users15642>e:

E:>cd e:kafka_2.12-2.2.0binwindows

e:kafka_2.12-2.2.0binwindows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

e:kafka_2.12-2.2.0binwindows>kafka-topics.bat --list --bootstrap-server localhost:9092
test

e:kafka_2.12-2.2.0binwindows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
>hello
>word

再打开一个 cmd
使用 kafka 的消费者,读取 test 主题下的消息

C:Users15642>e:

E:>cd kafka_2.12-2.2.0binwindows

E:kafka_2.12-2.2.0binwindows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
hello
word

关于Kafka使用的重要提示

现象:在windows的命令行里启动kafka之后,当关闭命令行窗口时,就会强制关闭kafka。这种关闭方式为暴力关闭,很可能会导致kafka无法完成对日志文件的解锁。届时,再次启动kafka的时候,就会提示日志文件被锁,无法成功启动。
方案:将kafka的日志文件全部删除,再次启动即可。
建议:不要暴力关闭kafka,建议通过在命令行执行kafka-server-stop命令来关闭它。
其他:将来在Linux上部署kafka之后,采用后台运行的方式,就会避免这样的问题。

【关闭】
新打开一个 cmd,先关闭 kafka,再关闭 zookeeper

C:Users15642>e:

E:>cd kafka_2.12-2.2.0binwindows

E:kafka_2.12-2.2.0binwindows>kafka-server-stop.bat
删除实例 \DESKTOP-R3O58DJROOTCIMV2:Win32_Process.Handle="16672"
实例删除成功。

E:kafka_2.12-2.2.0binwindows>zookeeper-server-stop.bat
删除实例 \DESKTOP-R3O58DJROOTCIMV2:Win32_Process.Handle="9048"
实例删除成功。
删除实例 \DESKTOP-R3O58DJROOTCIMV2:Win32_Process.Handle="8956"
实例删除成功。
删除实例 \DESKTOP-R3O58DJROOTCIMV2:Win32_Process.Handle="7388"
实例删除成功。
3. Spring整合Kafka

引入依赖

  • spring-kafka

配置Kafka

  • 配置server、consumer

访问Kafka

  • 生产者
kafkaTemplate.send(topic, data); 
  • 消费者
@KafkaListener(topics = {"test"}) 
public void handleMessage(ConsumerRecord record) {}

pom.xml


    org.springframework.kafka
    spring-kafka

E:kafka_2.12-2.2.0config 下的 consumer.properties
改一下 group.id

group.id=community-consumer-group

application.properties

# KafkaProperties
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=community-consumer-group
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=3000

新建测试类

package com.nowcoder.community;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.junit.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;

@SpringBootTest
public class KafkaTests {

    @Resource
    private KafkaProducer kafkaProducer;

    @Test
    public void testKafka() {
        kafkaProducer.sendMessage("test", "你好");
        kafkaProducer.sendMessage("test", "在吗");

        try {
            Thread.sleep(1000 * 10);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

@Component
class KafkaProducer {

    @Resource
    private KafkaTemplate kafkaTemplate;

    public void sendMessage(String topic, String content) {
        kafkaTemplate.send(topic, content);
    }
}

@Component
class KafkaConsumer {

    @KafkaListener(topics = {"test"})
    public void handleMessage(ConsumerRecord record) {
        System.out.println(record.value());
    }
}

先启动 zookeeper,再启动 kafka,最后执行测试方法看看能不能读取到信息

4. 发送系统通知

触发事件

  • 评论后,发布通知
  • 点赞后,发布通知
  • 关注后,发布通知

处理事件

  • 封装事件对象
  • 开发事件的生产者
  • 开发事件的消费者

4.1 entity
package com.nowcoder.community.entity;

import java.util.HashMap;
import java.util.Map;

public class Event {

    private String topic;
    private int userId;
    private int entityType;
    private int entityId;
    private int entityUserId;
    private Map data = new HashMap<>();

    public String getTopic() {
        return topic;
    }
    //改动所有 set 方法,方便连续调用 set.set.set
    public Event setTopic(String topic) {
        this.topic = topic;
        return this;
    }

    public int getUserId() {
        return userId;
    }

    public Event setUserId(int userId) {
        this.userId = userId;
        return this;
    }

    public int getEntityType() {
        return entityType;
    }

    public Event setEntityType(int entityType) {
        this.entityType = entityType;
        return this;
    }

    public int getEntityId() {
        return entityId;
    }

    public Event setEntityId(int entityId) {
        this.entityId = entityId;
        return this;
    }

    public int getEntityUserId() {
        return entityUserId;
    }

    public Event setEntityUserId(int entityUserId) {
        this.entityUserId = entityUserId;
        return this;
    }

    public Map getData() {
        return data;
    }

    public Event setData(String key, Object value) {
        this.data.put(key, value);
        return this;
    }
}

新建一个包 event,封装生产者和消费者

生产者

package com.nowcoder.community.event;

import com.alibaba.fastjson.JSONObject;
import com.nowcoder.community.entity.Event;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;

@Component
public class EventProducer {

    @Resource
    private KafkaTemplate kafkaTemplate;

    // 处理事件
    public void fireEvent(Event event) {
        // 将事件发布到指定的主题
        kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event));
    }
}

消费者

CommunityConstant

//主题:评论
String TOPIC_COMMENT = "comment";
//主题:点赞
String TOPIC_LIKE = "like";
//主题:关注
String TOPIC_FOLLOW = "follow";
//系统用户ID
int SYSTEM_USER_ID = 1;
package com.nowcoder.community.event;

import com.alibaba.fastjson.JSONObject;
import com.nowcoder.community.entity.Event;
import com.nowcoder.community.entity.Message;
import com.nowcoder.community.service.MessageService;
import com.nowcoder.community.util.CommunityConstant;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

@Component
public class EventConsumer implements CommunityConstant {

    public static final Logger logger = LoggerFactory.getLogger(EventConsumer.class);

    @Resource
    private MessageService messageService;

    @KafkaListener(topics = {TOPIC_COMMENT, TOPIC_LIKE, TOPIC_FOLLOW})
    public void handleCommentMessage(ConsumerRecord record) {
        if (record == null || record.value() == null) {
            logger.error("消息的内容为空!");
            return;
        }
        Event event = JSONObject.parseObject(record.value().toString(), Event.class);
        if (event == null) {
            logger.error("消息格式错误!");
            return;
        }
        //发送站内通知
        Message message = new Message();
        message.setFromId(SYSTEM_USER_ID);
        message.setToId(event.getEntityUserId());
        message.setConversationId(event.getTopic());
        message.setCreateTime(new Date());

        Map content = new HashMap<>();
        content.put("userId", event.getUserId());
        content.put("entityType", event.getEntityType());
        content.put("entityId", event.getEntityId());

        if (!event.getData().isEmpty()) {
            for (Map.Entry entry : event.getData().entrySet()) {
                content.put(entry.getKey(), entry.getValue());
            }
        }

        message.setContent(JSONObject.toJSONString(content));
        messageService.addMessage(message);
    }

}
4.2 controller

CommentMapper 补充方法

Comment selectCommentById(int id);

comment-mapper.xml 实现该方法


    select 
    from message
    where id in (
        select max(id) from message
        where status != 2
        and from_id = 1
        and to_id = #{userId}
        and conversation_id = #{topic}
    )



    select count(id) from message
    where status = 0
        and from_id = 1
        and to_id = #{userId}
        
            and conversation_id = #{topic}
        

MessageService
增加方法

public Message findLatestNotice(int userId, String topic) {
    return messageMapper.selectLatestNotice(userId, topic);
}

public int findNoticeCount(int userId, String topic) {
    return messageMapper.selectNoticeCount(userId, topic);
}

public int findNoticeUnreadCount(int userId, String topic) {
    return messageMapper.selectNoticeUnreadCount(userId, topic);
}

MessageController
增加方法

public class MessageController implements CommunityConstant {

	@GetMapping("/notice/list")
    public String getNoticeList(Model model) {
        User user = hostHolder.getUser();
        //查询评论类的通知
        Message message = messageService.findLatestNotice(user.getId(), TOPIC_COMMENT);
        Map messageVO = new HashMap<>();
        if (message != null) {
            messageVO.put("message", message);

            String content = HtmlUtils.htmlUnescape(message.getContent());
            HashMap data = JSONObject.parseObject(content, HashMap.class);

            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
            messageVO.put("postId", data.get("postId"));

            int count = messageService.findNoticeCount(user.getId(), TOPIC_COMMENT);
            messageVO.put("count", count);

            int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_COMMENT);
            messageVO.put("unread", unread);
        }
        model.addAttribute("commentNotice", messageVO);

        //查询点赞类的通知
        message = messageService.findLatestNotice(user.getId(), TOPIC_LIKE);
        messageVO = new HashMap<>();
        if (message != null) {
            messageVO.put("message", message);

            String content = HtmlUtils.htmlUnescape(message.getContent());
            HashMap data = JSONObject.parseObject(content, HashMap.class);

            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
            messageVO.put("postId", data.get("postId"));

            int count = messageService.findNoticeCount(user.getId(), TOPIC_LIKE);
            messageVO.put("count", count);

            int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_LIKE);
            messageVO.put("unread", unread);
        }
        model.addAttribute("likeNotice", messageVO);

        //查询关注类的通知
        message = messageService.findLatestNotice(user.getId(), TOPIC_FOLLOW);
        messageVO = new HashMap<>();
        if (message != null) {
            messageVO.put("message", message);

            String content = HtmlUtils.htmlUnescape(message.getContent());
            HashMap data = JSONObject.parseObject(content, HashMap.class);

            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));

            int count = messageService.findNoticeCount(user.getId(), TOPIC_FOLLOW);
            messageVO.put("count", count);

            int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_FOLLOW);
            messageVO.put("unread", unread);
        }
        model.addAttribute("followNotice", messageVO);

        //查询未读消息数量
        int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
        model.addAttribute("letterUnreadCount", letterUnreadCount);
        int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
        model.addAttribute("noticeUnreadCount", noticeUnreadCount);

        return "/site/notice";
    }

getLetterList 方法的 return 之前加上两行:

int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
model.addAttribute("noticeUnreadCount", noticeUnreadCount);

letter.html
74行


	系统通知27

notice.html
2行


8-9行



15行


197行


67行 ul 标签



82行 ul 标签



5.2 通知详情

MessageMapper

//查询某个主题所包含的通知列表
List selectNotices(@Param("userId") int userId,
                            @Param("topic") String topic,
                            @Param("offset") int offset,
                            @Param("limit") int limit);

message-mapper.xml