Kafka ksqlDB

ksqlDB is a database that's purpose-built for stream processing applications.

⚠️ ksqlDB it is not a SQL database, it provides an extra layer for implementing kstream, ktable and connectors through a language (ksql) based on sql.

Create an alias for ksqldb-cli:

⚠️ Run alias commands inside the root folder.

alias ksqldb-cli="docker run --rm -it --network kafka-sandbox_network --workdir /ksqldb -v $PWD/kafka-ksqldb/tests:/ksqldb/tests -v $PWD/kafka-ksqldb/statements:/ksqldb/statements -v $PWD/kafka-ksqldb-extensions/extensions:/ksqldb/extensions kafka-cli:latest "

To permanently add the alias to your shell (~/.bashrc or ~/.zshrc file):

echo "alias ksqldb-cli='docker run --rm -it --network kafka-sandbox_network --workdir /ksqldb -v $PWD/kafka-ksqldb/tests:/ksqldb/tests -v $PWD/kafka-ksqldb/statements:/ksqldb/statements -v $PWD/kafka-ksqldb-extensions/extensions:/ksqldb/extensions kafka-cli:latest '" >> ~/.zshrc

Run ksqlDB:

cd kafka-ksqldb
docker compose up -d
cd ..

After a few seconds:

http :8088/info

One line shell interaction:

ksqldb-cli ksql -e "SHOW STREAMS;" http://ksqldb:8088

Interactive ksqlDB shell:

ksqldb-cli ksql http://ksqldb:8088
SHOW STREAMS;

Docker Compose

services:
  ksqldb:
    image: confluentinc/cp-ksqldb-server:${VERSION}
    environment:
      KSQL_KSQL_SERVICE_ID: kafka-ksqldb.
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KSQL_KSQL_CONNECT_URL: http://kafka-connect:8083
      KSQL_KSQL_EXTENSION_DIR: /ksqldb/extensions
    ports:
      - "8088:8088"
    restart: on-failure
    volumes:
      - ../kafka-ksqldb-extensions/extensions:/ksqldb/extensions
    healthcheck:
      test: curl http://localhost:8088
      interval: 30s
      timeout: 30s
      retries: 5
      start_period: 30s

networks:
  default:
    external: true
    name: kafka-sandbox_network