.NetKafka

[.Net] Connection to Kafka – create Kafka producer for dummies

Adam Majstrak4 Comments on [.Net] Connection to Kafka – create Kafka producer for dummies

In this article you can find out how to create Kafka producer using .Net 8

All examples you can find on my github

First thing you have to do, is install kafka with zookeeper on your computer. To do it, I recommend start docker-compose according to instruction here.
We are creating new docker-compose.yml file like this:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
  
  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

You need to be in folder with docker.compose.yml file and execute command: docker-compose up.

To test connection to our kafka we need to download offset explorer from here
After open offset explore you need to right mouse click on cluster folder, define “Cluster name” in Properties tab (can be any) an “Bootstrap server” in Advanced tab – “localhost:29092”. After that you can click “Connect”.
After successful connection you can create new topic “Test”

Click on new created topic and change value of “key” in Properties tab to “No Key” and click Update button. Now we are ready to codding!

Go to Visual Studio and create new project in .net 8.
Right click on you project and create add new package

Install Confluent.Kafka lib

Go to Program.cs and enter following code:

using Confluent.Kafka;
using System.Text.Json;

var config = new ProducerConfig { BootstrapServers = "localhost:29092" };

using (var p = new ProducerBuilder<Null, string>(config).Build())
{
    try
    {
        var objToSend = new
        {
            name = "adam"
        };

        var message = JsonSerializer.Serialize(objToSend);

        var dr = await p.ProduceAsync("Test", new Message<Null, string> 
        {
            Value = message 
        });

        Console.WriteLine($"Delivered '{dr.Value}' to '{dr.TopicPartitionOffset}'");
    }
    catch (ProduceException<Null, string> e)
    {
        Console.WriteLine($"Delivery failed: {e.Error.Reason}");
    }
}

Console.ReadKey();

Run the program – new message should be send to your topic.

Go to offset explorer and check if message has been stored to your partition.

That’s it!

4 Comments

  1. Do it for java. Kafka seamlessly integrates with Java, offering a robust and efficient environment for building scalable, distributed systems. The native support for Kafka in Java provides a smoother and more natural experience, enabling developers to leverage its full potential effortlessly. While .NET support for Kafka exists, the Java ecosystem seems to have a more mature and well-established connection with Kafka, making it the preferred choice for many developers aiming for seamless integration and optimal performance in their distributed applications.

    Reply

    1. Neither in Java nor in .NET kafka is “native supported”. To provide kafka in Java you also need third party libs like org.apache.kafka. There is no big different between implementation Kafka in .Net and Java.

      Reply

Leave a Reply