springboot 整合 kafka 批量处理 - 问答

在 Spring Boot 中整合 Kafka 进行批量处理，可以通过以下步骤实现：

添加依赖

在 pom.xml 文件中添加 Kafka 和 Spring Boot Kafka 相关的依赖：

<dependencies>
    <!-- Spring Boot Kafka Starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-kafka</artifactId>
    </dependency>
</dependencies>

配置 Kafka

在 application.yml 或 application.properties 文件中配置 Kafka 相关参数：

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: my-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

创建 Kafka 配置类

创建一个配置类，用于设置 Kafka 生产者和消费者的属性：

@Configuration
public class KafkaConfig {

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return new DefaultKafkaProducerFactory<>(configProps);
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        return new DefaultKafkaConsumerFactory<>(props);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

创建 Kafka 监听器

创建一个监听器类，用于处理接收到的 Kafka 消息：

@Service
public class KafkaListener {

    @KafkaListener(topics = "my-topic", groupId = "my-group")
    public void listen(List<String> messages) {
        for (String message : messages) {
            System.out.println("Received message: " + message);
            // 在这里进行批量处理
        }
    }
}

发送批量消息

创建一个生产者类，用于向 Kafka 发送批量消息：

@Service
public class KafkaProducer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessages(List<String> messages) {
        kafkaTemplate.send("my-topic", messages);
    }
}

在主应用中调用

在主应用中，可以调用 KafkaProducer 发送批量消息，并监听这些消息：

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Autowired
    private KafkaProducer kafkaProducer;

    @Autowired
    private KafkaListener kafkaListener;

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);

        Application application = new Application();
        application.kafkaProducer.sendMessages(Arrays.asList("message1", "message2", "message3"));
    }
}

这样，当发送批量消息到 my-topic 主题时，KafkaListener 将接收到这些消息并进行批量处理。

0 赞

0 踩