Podłączenie producera kafki z innego kontenera

0

Mam problem z wrzucaniem producerem wiadomości na topic kafki. Sytuacja wygląda tak (aplikacje spring boot):

processor-service i kafka są w jednym docker-compose. Consumer będący w processor-service udanie subskrybuje topic.
client-service w drugim docker-compose nie może wrzucić message na topic. Otrzymany błąd to:

2023-03-28T18:29:59.611Z  INFO 1 --- [   scheduling-1] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
2023-03-28T18:29:59.673655000Z 2023-03-28T18:29:59.672Z  INFO 1 --- [   scheduling-1] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.3.2
2023-03-28T18:29:59.674622100Z 2023-03-28T18:29:59.673Z  INFO 1 --- [   scheduling-1] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: b66af662e61082cb
2023-03-28T18:29:59.675266800Z 2023-03-28T18:29:59.675Z  INFO 1 --- [   scheduling-1] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1680028199670
2023-03-28T18:29:59.999613100Z 2023-03-28T18:29:59.999Z  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Node -1 disconnected.
2023-03-28T18:30:00.009419500Z 2023-03-28T18:30:00.005Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/127.0.0.1:9092) could not be established. Broker may not be available.
2023-03-28T18:30:00.009451000Z 2023-03-28T18:30:00.006Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected

W processor-service wygląda ok:

2023-03-28T18:28:23.894Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.3.2
2023-03-28T18:28:23.895381800Z 2023-03-28T18:28:23.895Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: b66af662e61082cb
2023-03-28T18:28:23.895458800Z 2023-03-28T18:28:23.895Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1680028103893
2023-03-28T18:28:23.899088100Z 2023-03-28T18:28:23.898Z  INFO 1 --- [           main] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-firstGroup-1, groupId=firstGroup] Subscribed to topic(s): topic-1
2023-03-28T18:28:23.948411700Z 2023-03-28T18:28:23.948Z  INFO 1 --- [           main] c.m.p.ProcessorServiceApplication        : Started ProcessorServiceApplication in 8.57 seconds (process running for 10.654)
...
2023-03-28T18:28:27.102Z  INFO 1 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : firstGroup: partitions assigned: [topic-1-0]

oraz oba docker-compose

version: '2'

services:
  app:
    image: 'processor-service-image'
    ports:
      - "8080:8080"
    build:
      context: .
    container_name: processor-service
    depends_on:
      - db
    environment:
      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/processor-service-database
      - SPRING_DATASOURCE_USERNAME=docker_user
      - SPRING_DATASOURCE_PASSWORD=pass
      - SPRING_JPA_HIBERNATE_DDL_AUTO=create-drop

  db:
    image: 'postgres:latest'
    container_name: postgres-container
    environment:
      - POSTGRES_USER=docker_user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=processor-service-database
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST: localhost
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ADVERTISED_HOST_PORT: 9092


version: '2'

services:
  app:
    image: 'client-service-image'
    ports:
      - "8081:8081"
    build:
      context: .
    container_name: client-service
    environment:
      SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092

λ docker ps -a
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                  PORTS                                                NAMES
41f146c81cb7        processor-service-image   "java -jar /processo…"   24 seconds ago      Up 22 seconds           0.0.0.0:8080->8080/tcp                               processor-service
589800857a7f        wurstmeister/kafka        "start-kafka.sh"         27 seconds ago      Up 24 seconds           0.0.0.0:9092->9092/tcp                               kafka
7d66e0804386        wurstmeister/zookeeper    "/bin/sh -c '/usr/sb…"   27 seconds ago      Up 24 seconds           22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   zookeeper
d06af49b1d37        postgres:latest           "docker-entrypoint.s…"   27 seconds ago      Up 24 seconds           5432/tcp                                             postgres-container
54299e81a0c1        client-service-image      "java -jar /client-s…"   6 minutes ago       Up 6 minutes            0.0.0.0:8081->8081/tcp                               client-service

Serwis client-service mogę odpalić w kontenerze i lokalnie, ale w obu przypadkach nie działa. Ktoś wie jak to można rozwiązać?

0

Co znaczy nie działa?

1

Znalazłem rozwiązanie. Okazało się, że każdy docker compose tworzy swój network. Co ciekawe nie mogłem ustawić customowego dla obu bo i tak oba compose tworzyły swoje.
Dodałem więc kontenery z drugiego docker-compose do network tego pierwszego czyli:

  1. Poprzez docker network ls sprawdziłem jaki network mają kontenery z pierwszego compose. Był to: processor-service_network-1.
  2. Dodałem do drugiego compose:
networks:
  processor-service_network-1:
    external: true

a więc cały compose wygląda tak:

version: '2'

services:
  app:
    image: 'client-service-image'
    ports:
      - "8081:8081"
    build:
      context: .
    container_name: client-service
    networks:
      - processor-service_network-1
    environment:
      SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092

networks:
  processor-service_network-1:
    external: true

1 użytkowników online, w tym zalogowanych: 0, gości: 1