The Problem: Getting Real-time Data into Kafka Have you ever tried to stream Wikipedia changes or other real-time Server-Sent Events (SSE) updates into Kafka? If you have, you probably wrote custom Kafka producer and consumer code to handle these events, managed reconnections, and built your own error handling. There had to be a better way. Introducing the SSE Kafka Connector I鈥檓 excited to share my new open-source project: a generic Kafka Connect connector for Server-Sent Events. It lets you stream data from any SSE endpoint straight into Kafka topics鈥攏o custom code required. ...
The Problem: When APIs Meet AI Assistants A few weeks ago, I was experimenting with Model Context Protocol (MCP) for a Kafka integration project. The experience was eye-opening - being able to interact with complex systems through natural language felt like a glimpse into the future of development workflows. After successfully getting MCP working with Kafka, I started thinking about other systems I could connect. That鈥檚 when I looked at one of my old Django projects. ...
What if I told you that you could build, host, and maintain data APIs completely free using just GitHub? Here鈥檚 how I built a production-ready data platform without spending a penny on hosting. The Problem: APIs Are Expensive (Or Are They?) Building data-driven applications usually means expensive cloud hosting, database costs, and server maintenance. Most developers assume they need AWS, Google Cloud, or Azure to run automated data collection and serve APIs. But what if there was a better way? ...
Ever wondered where those massive icebergs the size of cities are drifting? I built a system to track them in real-time using NASA satellite data. The Problem: Lost Giants in the Southern Ocean Antarctic icebergs are some of the most fascinating phenomena on Earth. These floating ice mountains鈥攕ome larger than entire countries鈥攂reak off from ice shelves and drift across the Southern Ocean for years. But tracking their movements has always been challenging. ...
Integration testing is crucial for Kafka Connectors to ensure they work correctly with external systems. This guide provides a comprehensive approach to setting up and executing integration tests for Kafka Connectors, covering everything from environment setup to best practices for testing source and sink connectors.
A step鈥慴y鈥憇tep guide to using FastMCP and the MCP protocol to expose Kafka operations (topic management, produce/consume, troubleshooting) as LLM鈥慳ccessible tools.
Tired of manually reformatting Markdown for Medium? Discover my open-source tool that converts Hugo-friendly Markdown into perfectly formatted Medium posts instantly.
A practical guide to creating tombstone records for Kafka compacted topics using the kafka-console-producer.sh command-line tool with the null marker feature
Apache Kafka is a powerful distributed event streaming platform, and Kafka Connect makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need custom connectors tailored for proprietary systems, custom logic, or advanced error handling. That鈥檚 where this production-ready template comes in鈥攊t removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.
Kafka Connect provides a flexible way to process streaming data using Single Message Transforms (SMTs). If you need to filter out tombstone records (records with null values), you should use the generic Filter transformation along with the RecordIsTombstone predicate. Here鈥檚 the correct configuration: # Define the predicate to detect tombstone records (i.e., records with null values) predicates=dropTombstone predicates.dropTombstone.type=org.apache.kafka.connect.transforms.predicates.RecordIsTombstone # Configure the Filter transformation to drop records that match the predicate transforms=dropTombstone transforms.dropTombstone.type=org.apache.kafka.connect.transforms.Filter transforms.dropTombstone.predicate=dropTombstone Explanation What is a Predicate? A predicate in Kafka Connect is a condition that evaluates whether a given record meets certain criteria. It returns either true or false. If true, the transformation (such as filtering) is applied. In this case, the predicate named dropTombstone uses the built-in class RecordIsTombstone, which evaluates to true when a record鈥檚 value is null. ...