Yuri Mizushima, Software Engineer @ Yahoo Japan Corporation

Yuri Mizushima is working at Yahoo Japan Corp as a software engineer. He has been joined messaging platform team since 2019.

Reduce Redundant Producers from Partitioned Producer

Thur Jun 17, 2:00 PM - 2:35 PM, PT

Yahoo Japan Corporation has been using Apache Pulsar as a centralized messaging platform.
Recently, we have found an efficiency issue caused by an unspecified number of producers. Morespecifically, in some situations partitioned producer doesn't need to connect to all partitions.
A couple of cases as below:

1. different rate producers
- we should set the number of partitions according to the highest-rate producer
- however, this number is excessively large for low-rate producers
2. SinglePartition routing mode
- each producer uses only one partition

Currently we are planning to make partitioned producer able to connect to not only all partitions, but also part of partitions. We expect that this feature allows us to use system resources more efficiently.
In this session we will explain more detail and some experimental results.