Tom Wang is a Staff Software Engineer at Iterable working on workflows backend. As one of the experts on workflows backend, Tom has developed many backend features and fixes to ensure workflows continue to scale to billions of transitions a day.

Tom Wang Head Shot.jpg

Tom Wang

Staff Software Engineer @ Iterable
Thomas Kim Photo.jpg

Thomas Kim

Principal Engineer @ Iterable

Thomas Kim is a Software Architect at Iterable, a marketing automation SaaS that serves some of the world's largest consumer and growth companies. At Iterable, he focuses on building data systems that scale to billions of transactions. He was formerly a tech lead on Workday Search. Prior to that, he was the CTO of a small BI startup and an early engineer at Salesforce. He loves chasing around his two young daughters, and statically typed functional programming. He has been living and working in San Francisco since 2001.

Migrating a Billion Transactions a Day To Apache Pulsar

Wed Jun 16, 10:45 AM - 11:20 AM, PT

Iterable is a marketing automation SaaS that operates at vast scale. Iterable processes a billion queue transactions per day representing the flow of users through programmatic marketing campaigns, which we call Workflow Campaigns. We're currently migrating this workload from RabbitMQ to Apache Pulsar. In this talk we'll go over how Workflow Campaigns work, and the features of Apache Pulsar that make it well-suited to this workload. Pulsar transactions ensure exactly once processing of these campaigns. Tiered storage allows us to keep a permanent record of every users' campaign history. The log of state transitions that we store in Pulsar topics allows us to create derived metrics for near-real-time analytics. Stream processing with Flink allows us to enforce simple count-based application limits on Workflows. Finally, we'll discuss the process of migration with no downtime.