~/series/kafka-connect

Introduction # In this post, we will walk through the process of setting up an Apache Kafka cluster without using Zookeeper, leveraging Kafka’s KRaft mode …

In this tutorial, we’ll walk through the process of setting up a Kafka connector using Docker and docker-compose.yml. We’ll focus on configuring a …

A practical guide to creating tombstone records for Kafka compacted topics using the kafka-console-producer.sh command-line tool with the null marker feature

Kafka Connect provides a flexible way to process streaming data using Single Message Transforms (SMTs). If you need to filter out tombstone records (records …

Apache Kafka is a powerful distributed event streaming platform, and **Kafka Connect** makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need **custom connectors** tailored for proprietary systems, custom logic, or advanced error handling. That’s where this **production-ready template** comes in—it removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.

Integration testing is crucial for Kafka Connectors to ensure they work correctly with external systems. This guide provides a comprehensive approach to setting up and executing integration tests for Kafka Connectors, covering everything from environment setup to best practices for testing source and sink connectors.

The Problem: Getting Real-time Data into Kafka # Have you ever tried to stream Wikipedia changes or other real-time Server-Sent Events (SSE) updates into Kafka? …