Posts

Ever wondered where those massive icebergs the size of cities are drifting? I built a system to track them in real-time using NASA satellite data. The Problem: Lost Giants in the Southern Ocean # Antarctic icebergs are some of the most fascinating phenomena on Earth. These floating ice mountains—some larger than entire countries—break off from ice shelves and drift across the Southern Ocean for years. But tracking their movements has always been challenging.

Integration testing is crucial for Kafka Connectors to ensure they work correctly with external systems. This guide provides a comprehensive approach to setting up and executing integration tests for Kafka Connectors, covering everything from environment setup to best practices for testing source and sink connectors.

A step‑by‑step guide to using FastMCP and the MCP protocol to expose Kafka operations (topic management, produce/consume, troubleshooting) as LLM‑accessible tools.

Tired of manually reformatting Markdown for Medium? Discover my open-source tool that converts Hugo-friendly Markdown into perfectly formatted Medium posts instantly.

A practical guide to creating tombstone records for Kafka compacted topics using the kafka-console-producer.sh command-line tool with the null marker feature

Apache Kafka is a powerful distributed event streaming platform, and Kafka Connect makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need custom connectors tailored for proprietary systems, custom logic, or advanced error handling. That’s where this production-ready template comes in—it removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.

Kafka Connect provides a flexible way to process streaming data using Single Message Transforms (SMTs). If you need to filter out tombstone records (records with null values), you should use the generic Filter transformation along with the RecordIsTombstone predicate.

In this technical deep-dive, we unravel the power of Server-Sent Events (SSE), a game-changing web technology that simplifies real-time communication. Developers often struggle with complex, resource-intensive methods for creating live updates, but SSE offers an elegant solution that’s both lightweight and powerful.

In this tutorial, we’ll walk through the process of setting up a Kafka connector using Docker and docker-compose.yml. We’ll focus on configuring a file connector, which is useful for reading data from files and writing data to files using Kafka.

Introduction # In this post, we will walk through the process of setting up an Apache Kafka cluster without using Zookeeper, leveraging Kafka’s KRaft mode for metadata management. Kafka no longer requires a Zookeeper instance, simplifying the cluster setup. We will use Docker to deploy this Kafka cluster with the latest version of Kafka.

This blog post explores online machine learning through a practical example using the CIFAR10 dataset and Apache Kafka.

Learn how to implement a powerful Retrieval-Augmented Generation (RAG) system using PostgreSQL and pgvector. This comprehensive guide covers everything from setting up a custom PostgreSQL Docker image to creating a fully functional RAG query system with vector embeddings and language model inference.