Yunze Xu works at StreamNative as a software engineer. He is the core maintainer of KoP and a committer of Apache Pulsar. He worked at Qihoo 360 previously for developing the internal messaging system QBus, which is based on Apache Kafka.
Software Engineer @ StreamNative
Senior Software Engineer @ Tencent
Aloys Zhang works at Tencent as a senior software engineer and focus on real time processing and messaging system. He is the maintainer of KoP and a contributor of Apache Pulsar.
Take Kafka-on-Pulsar to Production at Internet Scale: Improvements Made for Pulsar 2.8.0
Wed Jun 16, 1:45 PM - 2:20 PM, PT
KoP is a protocol handler that provides a convenient method for Kafka clients to connect to Pulsar without any change.
Before Pulsar 2.8.0, KoP was not used for production due to some flows. On the one hand, KoP simulated Kafka’s offset by using a message id conversion, which limited the max count of ledgers and caused other problems. On the other hand, the conversion of message format between Kafka and Pulsar had a significant impact on the performance.
In this talk, I’ll introduce how we improve KoP for Pulsar 2.8.0, including the new implementation of KoP offset, the Kafka entry formatter that improves the performance, and some other changes that improve KoP’s stability and Kafka protocol compatibilities. After that, KoP will be ready for production.