Apache Kafka – Terefas básicas admin, Troubleshooting e Logs (Parte 2)

Fundamentos Apache Kafka na visão Admin.

O vídeo foi feito com a idéia de compartilhar o meu conhecimento adquirido, trabalhando com Kafka, para colegas de trabalho e alguns amigos.
Eu mesmo editei o vídeo, sem muito conhecimento na ferramenta.

Entendo na pratica como realizar Troubleshooting, quais logs devemos olhar no nosso Cluster do Kafka, por onde devemos começar.

Gravado com OBS.
SO: Ubuntu 22.04.2 LTS
Editor: Kdenlive

Para facilitar, eu estou publicando todo material no meu github:

https://github.com/mordp1/espetinhodekafka/tree/main/Day10

## Comandos:

### Criando Cluster

Utilizaremos o exemplo da Conduktor.

https://www.conduktor.io/kafka/how-to-start-kafka-using-docker/

### Subindo
“`bash
$ docker-compose -f zk-multiple-kafka-multiple.yml up -d
“`
### Conectando no shell do container kafka1
“`bash
$ docker exec -it kafka1 /bin/bash
“`
### Criando os 3 topicos dentro do container
“`bash
$ kafka-topics –bootstrap-server kafka1:19092 –topic topic1 –create –partitions 1 –replication-factor 1

$ kafka-topics –bootstrap-server kafka1:19092 –topic topic2 –create –partitions 2 –replication-factor 2

$ kafka-topics –bootstrap-server kafka1:19092 –topic topic3 –create –partitions 3 –replication-factor 3

$ kafka-topics –bootstrap-server kafka1:19092 –topic topic10 –create –partitions 10 –replication-factor 3
“`
### Para executar os comandos fora do container, precisa ter os binários do Kafka:
“`bash
$ wget -c https://dlcdn.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz -O – | tar -xz
$ cd kafka_2.13-3.4.0
“`
### Comandos para criar topicos fora do container
“`bash
$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –topic topic1 –create –partitions 1 –replication-factor 1

$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –topic topic2 –create –partitions 2 –replication-factor 2

$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –topic topic3 –create –partitions 3 –replication-factor 3

$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –topic topic10 –create –partitions 10 –replication-factor 3
“`

### Describe todos os topicos
“`bash
$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –describe
“`

### Producer test
“`bash
$ bin/kafka-producer-perf-test.sh –topic topic10 –throughput -1 –num-records 30000 –record-size 1024 –producer-props acks=all bootstrap.servers=localhost:9092
“`

### Consumer test
“`bash
$ bin/kafka-consumer-perf-test.sh –topic topic10 –broker-list localhost:9092 –messages 30000 –group test-group | jq -R .|jq -sr ‘map(./”,”)|transpose|map(join(“: “))[]’
“`

### Listar todos os grupos de consumidores
“`bash
$ bin/kafka-consumer-groups.sh –bootstrap-server localhost:9092 –list
“`

### Detalhes de um grupo

“`bash
$ bin/kafka-consumer-groups.sh –bootstrap-server localhost:9092 –group test-group –describe
“`

### Remover container:
“`bash
$ docker stop kafka2
“`

### Verificar os logs do container

“`bash
$ docker logs -f kafka1
“`

### Describe todos os topicos
“`bash
$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –describe
“`

### Describe -under-replicated-partitions
“`bash
$ bin/kafka-topics.sh –bootstrap-server localhost:9092 –describe –under-replicated-partitions
“`

#apache #kafka #apachekafka #data #datastreams #kafkaadministration #kafkaadmin
#stream #technology #linux #devops #sre #engenharia #engenhariadesoftware #dev #kafkaconnect #troubleshooting