articles
Kafka

Kafka

misckafkamessagingstreaming

Published on 2023-02-14

Get Started

Create network:

docker network create -d bridge bigdata --subnet 10.10.0.0/16

Create compose file:

version: '2'
 
networks:
  bigdata:
    external: true
 
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
      bigdata:
        ipv4_address: 10.10.2.1
 
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
    networks:
      bigdata:
        ipv4_address: 10.10.2.2

Tools

Create Topic

topic: test
partitions: 9
replication-factor: 3

kafka-topics.sh --bootstrap-server 10.10.2.1:9092,10.10.2.2:9092,10.10.2.3:9092 --topic test --create --partitions 9 --replication-factor 3

topic describe:

kafka-topics.sh --bootstrap-server 10.10.2.1:9092,10.10.2.2:9092,10.10.2.3:9092 --topic test --describe
 
Topic: test	TopicId: 3jstEOR0Rc2cqyAJoUtTtA	PartitionCount: 9	ReplicationFactor: 3	Configs:
	Topic: test	Partition: 0	Leader: 1003	Replicas: 1003,1002,1001	Isr: 1003,1002,1001
	Topic: test	Partition: 1	Leader: 1002	Replicas: 1002,1001,1003	Isr: 1002,1001,1003
	Topic: test	Partition: 2	Leader: 1001	Replicas: 1001,1003,1002	Isr: 1001,1003,1002
	Topic: test	Partition: 3	Leader: 1003	Replicas: 1003,1001,1002	Isr: 1003,1001,1002
	Topic: test	Partition: 4	Leader: 1002	Replicas: 1002,1003,1001	Isr: 1002,1003,1001
	Topic: test	Partition: 5	Leader: 1001	Replicas: 1001,1002,1003	Isr: 1001,1002,1003
	Topic: test	Partition: 6	Leader: 1003	Replicas: 1003,1002,1001	Isr: 1003,1002,1001
	Topic: test	Partition: 7	Leader: 1002	Replicas: 1002,1001,1003	Isr: 1002,1001,1003
	Topic: test	Partition: 8	Leader: 1001	Replicas: 1001,1003,1002	Isr: 1001,1003,1002

Producer Console

kafka-console-producer.sh --bootstrap-server 10.10.2.1:9092,10.10.2.2:9092,10.10.2.3:9092 --topic test

Consumer Console

group: test1

kafka-console-consumer.sh --bootstrap-server 10.10.2.1:9092,10.10.2.2:9092,10.10.2.3:9092 --topic test --group test1

Consumer Group Describe

kafka-consumer-groups.sh --bootstrap-server 10.10.2.1:9092,10.10.2.2:9092,10.10.2.3:9092 --describe --group test1
 
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                           HOST            CLIENT-ID
test1           test            6          11              11              0               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            0          0               2               2               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            7          0               0               0               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            5          3               3               0               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            8          0               0               0               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            1          0               0               0               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            4          0               3               3               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            3          0               1               1               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer
test1           test            2          5               5               0               console-consumer-a4b78ca7-b16f-4466-ae94-4f03c95cf5ce /10.10.2.1      console-consumer

Bibliography

Archive