Kafka Connect on Joel Hanson

Configuring Kafka Connector with Docker: A Step-by-Step Guide

joelhanson025@gmail.com (Joel Hanson) — Sat, 12 Oct 2024 12:17:49 +0530

In this tutorial, we’ll walk through the process of setting up a Kafka connector using Docker and docker-compose.yml. We’ll focus on configuring a file connector, which is useful for reading data from files and writing data to files using Kafka.

Setting Up a Kafka Cluster Without Zookeeper Using Docker

joelhanson025@gmail.com (Joel Hanson) — Sat, 05 Oct 2024 23:14:04 +0530

Introduction
#

In this post, we will walk through the process of setting up an Apache Kafka cluster without using Zookeeper, leveraging Kafka’s KRaft mode for metadata management. Kafka no longer requires a Zookeeper instance, simplifying the cluster setup. We will use Docker to deploy this Kafka cluster with the latest version of Kafka.

SSE Kafka Connector: Stream Real-time Events into Kafka

joelhanson025@gmail.com (Joel Hanson) — Wed, 27 Aug 2025 21:15:47 +0530

The Problem: Getting Real-time Data into Kafka
#

Have you ever tried to stream Wikipedia changes or other real-time Server-Sent Events (SSE) updates into Kafka? If you have, you probably wrote custom Kafka producer and consumer code to handle these events, managed reconnections, and built your own error handling.

Mastering Integration Testing for Kafka Connectors: A Complete Guide

joelhanson025@gmail.com (Joel Hanson) — Sun, 13 Jul 2025 21:40:00 +0530

Integration testing is crucial for Kafka Connectors to ensure they work correctly with external systems. This guide provides a comprehensive approach to setting up and executing integration tests for Kafka Connectors, covering everything from environment setup to best practices for testing source and sink connectors.

Creating Tombstone Records Using kafka-console-producer.sh: A Quick Guide

joelhanson025@gmail.com (Joel Hanson) — Tue, 22 Apr 2025 16:03:27 +0530

A practical guide to creating tombstone records for Kafka compacted topics using the kafka-console-producer.sh command-line tool with the null marker feature

Build Custom Kafka Connectors Fast with This Open-Source Template

joelhanson025@gmail.com (Joel Hanson) — Mon, 14 Apr 2025 16:19:42 +0530

Apache Kafka is a powerful distributed event streaming platform, and Kafka Connect makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need custom connectors tailored for proprietary systems, custom logic, or advanced error handling. That’s where this production-ready template comes in—it removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.

Filtering Tombstone Records in Kafka Connect

joelhanson025@gmail.com (Joel Hanson) — Tue, 11 Mar 2025 12:19:08 +0530

Kafka Connect provides a flexible way to process streaming data using Single Message Transforms (SMTs). If you need to filter out tombstone records (records with null values), you should use the generic Filter transformation along with the RecordIsTombstone predicate.