Build Custom Kafka Connectors Fast with This Open-Source Template

🔗 GitHub Repository: github.com/Joel-hanson/kafka-connector-template
⭐ Star the repo to stay updated!

Apache Kafka is a powerful distributed event streaming platform, and Kafka Connect makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need custom connectors tailored for proprietary systems, custom logic, or advanced error handling.

That’s where this production-ready template comes in—it removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.

Use This Template to Kickstart Your Own Connector

#

Instead of cloning and manually renaming everything, you can use the GitHub Template feature to instantly create your own connector project with this structure.

Steps to Create Your Project from This Template

#

1. Go to the Template Repo
   github.com/Joel-hanson/kafka-connector-template

2. Click “Use this template”
   At the top of the repo, hit the green Use this template button.

3. Fill in Your Project Details

   • Set the repository name
   • Add a description
   • Choose if you want it to be public or private

4. Click “Create repository from template”

5. Clone Your New Repo

   git clone https://github.com/your-username/your-connector-repo.git
   cd your-connector-repo

6. Start Building

   • Modify the connector classes under src/main/java
   • Update your pom.xml with any new dependencies
   • Customize Docker configs and connector configs in /config

Tip: Don’t forget to replace package names (com.example.kafka.connect) with your own!

Why Use This Template?

#

Building a Kafka connector from scratch can take days or even weeks. This open-source template gives you a 5-minute head start:

✅ Skip complex setup – Docker, Kafka (KRaft), and CI/CD are preconfigured
✅ Focus on logic – Just implement your business-specific code
✅ Test confidently – Comes with comprehensive integration tests using TestContainers
✅ Go production – Includes Docker health checks, error handling patterns, and more

The project includes extensive integration tests using TestContainers that verify end-to-end functionality with real Kafka infrastructure. These tests are based on the testing framework from the Aiven JDBC Connector for Apache Kafka, providing a proven foundation for reliable connector testing.

Tech Stack & Project Structure

#

This template includes:

• Apache Kafka (KRaft mode) – No ZooKeeper needed
• Kafka Connect with both source and sink examples
• Docker for running Kafka, Connect, and ZooKeeper-free clusters
• Maven for dependency management
• GitHub Actions for automated testing

Directory Layout:

kafka-connector-template/
├── .github/           # CI/CD workflows
├── config/            # Connector configs
├── docker/            # Docker setup (KRaft mode)
├── src/               # Java source and test code
│   ├── main/
│   └── test/
├── pom.xml            # Maven config
└── README.md

Quick Start in 5 Minutes

#

1. Clone and Build the Project

#

git clone https://github.com/Joel-hanson/kafka-connector-template.git
cd kafka-connector-template
mvn clean package

2. Start Kafka + Connect (KRaft Mode)

#

cd docker
docker-compose up -d

3. Deploy the Example Source Connector

#

curl -X POST -H "Content-Type: application/json" \
  --data @config/source-connector.json \
  http://localhost:8083/connectors

4. Validate It’s Working

#

docker exec -it kafka kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic example-source-topic

Building Your Own Connector

#

Implementing a Source Connector

#

A source connector pulls data from external systems into Kafka.

Edit:

src/main/java/com/example/kafka/connect/source/ExampleSourceConnector.java
src/main/java/com/example/kafka/connect/source/ExampleSourceTask.java

Sample:

public class MySourceConnector extends SourceConnector {
    @Override
    public void start(Map<String, String> props) {
        // Initialization logic
    }

    @Override
    public Class<? extends Task> taskClass() {
        return MySourceTask.class;
    }

    @Override
    public List<Map<String, String>> taskConfigs(int maxTasks) {
        // Configure tasks
    }
}

Implementing a Sink Connector

#

A sink connector moves data from Kafka to an external system.

Edit:

src/main/java/com/example/kafka/connect/sink/ExampleSinkConnector.java
src/main/java/com/example/kafka/connect/sink/ExampleSinkTask.java

Sample:

public class MySinkTask extends SinkTask {
    @Override
    public void put(Collection<SinkRecord> records) {
        // Write to destination
    }
}

Docker Highlights

#

Running Kafka with KRaft (no ZooKeeper) is easier than ever.

docker/docker-compose.yml includes:

kafka:
  image: apache/kafka:3.6.0
  environment:
    KAFKA_PROCESS_ROLES: controller,broker
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    ...

Start it with:

docker-compose up -d

Testing Connectors

#

Run Unit Tests

#

mvn test

Run Integration Tests

#

mvn verify

The template includes:

• Validation of connector configs
• Sample source/sink connector tests
• Comprehensive integration tests with TestContainers
• End-to-end functionality verification with real Kafka clusters

Deploying the Connector via REST API

#

Once Kafka Connect is running, deploy your connector:

POST the Configuration

#

curl -X POST -H "Content-Type: application/json" \
  --data '{
    "name": "source-connector",
    "config": {
      "connector.class": "com.example.kafka.connect.source.SourceConnector",
      "tasks.max": "1",
      "topic": "test-topic",
      "poll.interval.ms": "5000"
    }
  }' \
  http://localhost:8083/connectors

Check Connector Status

#

curl http://localhost:8083/connectors/source-connector/status | jq

Common Issues & Fixes

#

Docker Volume Issues:

docker-compose down -v && docker-compose up --build

Connector Doesn’t Load?

• Ensure your .jar is in the right path
• Restart Kafka Connect:

docker restart connect

Ready for Production?

#

Here’s how to take it to the next level:

✅ Enable SSL and Auth in connect-distributed.properties
✅ Add Prometheus metrics for monitoring
✅ Use dead-letter queues and retry logic for error handling
✅ Set config.storage.replication.factor=3 for HA setups

Contributing

#

Want to improve the template or report a bug?

1. Fork the repo
2. Make your changes
3. Open a PR!

Further Reading

#

• Kafka Connect Documentation

Star the template → github.com/Joel-hanson/kafka-connector-template

What’s Next? Level Up Your Testing Game

#

Follow-up Reading: I’ve written a comprehensive follow-up post on how to automate integration testing for Kafka connectors, which complements this template perfectly.

Mastering Integration Testing for Kafka Connectors: A Complete Guide

Learn how to build robust test suites that catch bugs before they hit production!