How LivePerson optimized Logstash and Kafka performance on GCP through benchmarking

**LivePerson** published findings on the **Elastic Blog** detailing how benchmarking **Google Cloud Platform** (**GCP**) machine types optimized their **Logstash** and **Kafka** infrastructure. By evaluating different instance options, the company cut **Logstash** costs by **over half** using **AMD Milan** instances. Additionally, their evaluation of **Kafka** **compression codec selection** significantly boosted pipeline throughput.

The performance optimization targeted two distinct components of the data pipeline: log processing with **Logstash** and message streaming with **Kafka**. Technical benchmarking on **GCP** showed that **AMD Milan** machine types provided the specific compute performance needed to lower **Logstash** operational costs by **over half**. On the messaging layer, systematic testing of **Kafka** **compression codec selection** optimized payload processing, resulting in the recorded throughput increase.

For infrastructure engineers and builders, these benchmark results demonstrate that default cloud configurations can lead to unnecessary spending and underutilized throughput. The technical findings show that matching workload requirements to hardware architectures like **AMD Milan** instances yields immediate cost reductions for **Logstash**. Simultaneously, fine-tuning **compression codec selection** in **Kafka** proves that software-level codec choices can unlock substantial throughput improvements on existing cloud infrastructure.

In the broader market context of cloud infrastructure management on **GCP**, benchmarking specific machine types remains a critical step for data-intensive applications. **LivePerson**'s results demonstrate how optimizing open-source tools like **Logstash** and **Kafka** through tailored hardware selection (**AMD Milan**) and codec configuration provides a repeatable framework for managing cloud expenditure and system capacity.

The practical takeaway for technical teams is to conduct empirical benchmarks on **GCP** machine types and software settings rather than relying on baseline defaults. Engineers should evaluate **AMD Milan** instances for **Logstash** workloads to target cost cuts of **over half**, while testing **Kafka** **compression codec selection** to maximize throughput across data streaming pipelines.