Kafka Connect

Kafka Connect

Kafka Connect is a framework for streaming data between Kafka and external systems (databases, S3, Elasticsearch) using pre-built connectors.

# Source connector — pull from MySQL into Kafka
POST http://localhost:8083/connectors
{
  "name": "mysql-source",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "mysql",
    "database.port": "3306",
    "database.user": "kafka",
    "database.password": "kafka",
    "database.server.id": "1",
    "database.server.name": "mydb",
    "table.include.list": "mydb.orders",
    "database.history.kafka.topic": "schema-changes"
  }
}

# Sink connector — push from Kafka to Elasticsearch
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector"
"connection.url": "http://elasticsearch:9200"
"topics": "mydb.orders"