{"id":26811,"date":"2020-04-22T00:00:00","date_gmt":"2020-04-21T22:00:00","guid":{"rendered":"https:\/\/blexin.com\/la-metamorfosi-della-comunicazione-con-kafka\/"},"modified":"2021-05-21T20:28:29","modified_gmt":"2021-05-21T18:28:29","slug":"the-metamorphosis-of-communication-with-kafka","status":"publish","type":"post","link":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/","title":{"rendered":"“The Metamorphosis” of communication with Kafka"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

In my previous articles, we have seen how useful it is to use communication technologies and solutions such as RabbitMQ<\/a> or an ESB (Enterprise Service Bus<\/em>) such as Rebus<\/a>, whether they are on-premise or cloud, to decouple communication in software systems.<\/p>\n\n\n\n

Not many years ago, in Linkedin they used ActiveMQ<\/a> to try to solve a problem related to real-time access to monitoring and user activities traceability data, which  were also related each other. The try was not successful because with ActiveMQ<\/em> they could not scale-out to manage the huge amount of traffic. Therefore, they decided to develop a messaging system that would provide both the persistence of the messages, in order to make them usable for multiple applications, and the chance to scale horizontally in order to manage a large amount of data. Finally, they decided to give it a name of a writer who sounded good, and that was how Kafka<\/em> was born (Kafka: The Definitive Guide<\/a>).<\/p>\n\n\n\n

Fundamental concepts<\/strong><\/h2>\n\n\n\n

It was initially created as a “distributed publish\/subscribe messaging system” until it evolved into a distributed streaming platform. Basically, we will have a Publisher<\/em> (Producer<\/em>) who publishes messages on a Broker<\/em> (Kafka Broker<\/em>) and a Subscriber<\/em> (Consumer<\/em>) who subscribes to them.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The message<\/em> is the information unit that travels in Kafka and is nothing more than an array of bytes, which, optionally, can be accompanied by a key<\/em> used to manage the writing of the message. For reasons of efficiency, in Kafka it is generally preferred to transmit messages in batches<\/em>.<\/p>\n\n\n\n

When a broker receives a message from a producer, he categorizes it into a Topic<\/em>. The topic is the actual representation of the data stream that passes from the producer to the consumer. The topics are, in turn, divided into partitions<\/em>. A partition is an ordered and immutable sequence of messages that are stored one after the other, and that can only be read in the order in which they were written. In fact, the messages will not be sorted within the topic but will be sorted at the partition level, assigning each message a unique id<\/em> called Offset<\/em>. In order to ensure the order of the messages, the partitions are also a tool for redundancy and scalability as they can be replicated within Kafka.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

In fact, the basic idea is that a Kafka broker operates as part of a Cluster<\/em> of brokers running on one or more servers. A cluster is responsible for receiving messages, saving them in topics for a configurable retention period<\/em> and assign the offsets within partitions. The partitions can be replicated within the cluster and for each partition there will be only one broker who will act as leader<\/em>. This last will manage the messages for that partition, while the other brokers on which the partition is replicated will act as followers<\/em> by saving copies passively than the leader. This is very important because if the broker leader fails, one of the followers can be elected leader and thus allow the cluster to run properly.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

We said that the messages are published by the Producers<\/em>. In addition to creating the data stream, a Producer can choose which partition to write a particular message through a partitioner<\/em>. The choice is usually based on the message key: if the key is not specified, the partitioner will choose the partition according to a round-robin<\/em> (RR<\/em>) scheduling to balance the load; otherwise, if there is a key, the partitioner will be able to route messages to specific partitions, for example, by hashing the key so that messages with the same hash value are written in the same partition.<\/p>\n\n\n\n

As an additional fault-tolerance<\/em> mechanism, one of the brokers within the cluster will play the role of controller<\/em> and will take care of assigning the partitions to the other brokers and supervising the health of the cluster components.<\/p>\n\n\n\n

The stream of messages will be consumed by those softwares, the consumers<\/em>, who subscribe to a specific topic and take their messages ordered thanks to the use of the offset we have talked about. In fact, when a consumer starts to consume the messages of a partition, a particular topic is created called _consumer_offsets<\/em>, where he saves the offset of the last message consumed and thus maintaining a state of his reading.<\/p>\n\n\n\n

Consumers are also designed to operate as part of a whole, called the Consumer Group<\/em>.<\/p>\n\n\n\n

A Consumer Group is made up of one or more consumers who work together to consume messages more efficiently within a topic by dividing the partitions equally. In fact, if one of the group’s consumers fails, its part is distributed among the other consumers. In addition to having multiple consumers within a group, you can also have multiple groups consuming messages from a topic. This fact is very useful when there are multiple applications interested in the messages of a certain topic. The Consumer Group concept allows you to generalize the advantages of traditional messaging systems. As in a queue system, it is possible to divide the processing among various consumers: the various consumer instances within the group do that. As in a publish\/subscribe system, messages can be read by multiple Consumer Groups.<\/p>\n\n\n\n

In this way, it is possible to guarantee both scalability for processing and reliability for delivery to multiple User Groups and therefore to multiple interested applications. In addition, with the concept of partition, Kafka guarantees the order of messages even with multiple consumers, unlike a more traditional system such as the queue system in which messages are delivered asynchronously and therefore in presence of multiple consumers is not possible to guarantee the order.<\/p>\n\n\n\n

Let’s take an example<\/strong><\/h2>\n\n\n\n

Introduced all the key concepts of Kafka, we see now an example of publish\/subscribe developed in .NET Core<\/em>. For the kafka-server, we will use an image instantiated on a Docker<\/em> container.<\/p>\n\n\n\n

In order to let a kafka server run properly, we must first make sure that it is connected to a Zookeeper<\/em> instance. Zookeeper is a centralized service that manages configurations and synchronizations in distributed applications. Kafka uses Zookeeper to save metadata related to brokers, topics and partitions. So let’s create a container docker for Zookeeper by running the following command from the terminal where we go to specify the port and a server id<\/em>:<\/p>\n\n\n

\ndocker run --net=kafka \\\n-d \\\n--name=zookeeper \\\n-e ZOOKEEPER_CLIENT_PORT=2181 \\\n-e ZOOKEEPER_SERVER_ID=1 \\\nconfluentinc\/cp-zookeeper:5.4.1\n<\/pre><\/div>\n\n\n
After that, we create the container for the Kafka server by specifying the Zookeeper address, an address where to reach Kafka outside the container and the replication value of the topics to be specified in the case of single-node clusters:<\/p>\n\n\n
\ndocker run --net=kafka \\\n-d \\\n-p 9092:9092 \\\n--name=kafka \\\n-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \\\n-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT:\/\/localhost:9092 \\\n-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \\\nconfluentinc\/cp-kafka:5.4.\n<\/pre><\/div>\n\n\n
Let’s create a topic with three partitions with the following command:<\/p>\n\n\n
\ndocker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic demo-topic --partitions 3\n<\/pre><\/div>\n\n\n
to see the result we use instead:<\/p>\n\n\n
\ndocker exec -it kafka kafka-topics --describe --bootstrap-server localhost:9092\n<\/pre><\/div>\n\n\n
\"\"<\/figure>\n\n\n\n
Once the topic has been created, we create a console application<\/em> that will perform the role of Producer<\/em>. We will use the Kafka client<\/em> for .NET<\/em>, Confluent.Kafka<\/em>, made available by Confluent<\/a>.<\/p>\n\n\n
\nusing System;\nusing System.Threading.Tasks;\nusing Confluent.Kafka;\n\u00a0\nnamespace Kafka.Producer\n{\n\u00a0\u00a0\u00a0\u00a0class Program\n\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0static async Task Main(string[] args)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0var config = new ProducerConfig { BootstrapServers = "localhost:9092", Partitioner = Partitioner.Random };\n\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0using (var producer = new ProducerBuilder<Null, string>(config).Build())\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0for (int i = 0; i < 5; i++)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0var deliveryResult = await producer.ProduceAsync("demo-topic", new Message<Null, string> { Value = $"DemoMessage '{i}'" });\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Console.WriteLine($"Published message to '{deliveryResult.TopicPartitionOffset}'");\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0catch (ProduceException<Null, string> e)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Console.WriteLine($"Error publishing message: {e.Error.Reason}");\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0}\n}\n<\/pre><\/div>\n\n\n
We configure the Producer indicating the address of the Kafka server and the type of Partitioner (in this case Random<\/em> will implement an RR scheduling). Then we create a Producer using the ProducerBuilder<\/em> with null key, since we chose a non-key based partitioner. Using the ProduceAsync<\/em>() method we send the messages on “demo-topic” and print the result of the writing in the console in order to display in which partition and in which offset the messages have been saved.<\/p>\n\n\n\n
Running the program we get the following result:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
With the following command we go to use the Kafka console consumer, a tool that reads the messages within the topic and reports them on the console:<\/p>\n\n\n
\ndocker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic demo-topic --from-beginning\n<\/pre><\/div>\n\n\n
\"\"<\/figure>\n\n\n\n
As we can see, the messages are not shown in order because, as we said, the order is at the partition level, not at the topic level. We can see this with the following command related to partition 2:<\/p>\n\n\n
\ndocker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic demo-topic --from-beginning --partition 2\n<\/pre><\/div>\n\n\n
\"\"<\/figure>\n\n\n\n
As we can see they are ordered. Let’s now create the consumer application:<\/p>\n\n\n
\nusing System;\nusing Confluent.Kafka;\n\u00a0\nclass Program\n{\n\u00a0\u00a0\u00a0\u00a0public static void Main(string[] args)\n\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0var conf = new ConsumerConfig\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0GroupId = "demo-consumer-group",\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0BootstrapServers = "localhost:9092",\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0AutoOffsetReset = AutoOffsetReset.Earliest\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0};\n\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0using (var consumer = new ConsumerBuilder<Ignore, string>(conf).Build())\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0consumer.Subscribe("demo-topic");\n\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0var readingMessage = true;\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0while (readingMessage)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0var consumeResult = consumer.Consume(10000);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if (consumeResult == null)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Console.WriteLine("There are no messages in topic");\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0readingMessage = false;\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0else\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Console.WriteLine($"Consumed message '{consumeResult.Message.Value}' at: '{consumeResult.TopicPartitionOffset}'.");\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0catch (ConsumeException e)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Console.WriteLine($"Error occured: {e.Error.Reason}");\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0catch (OperationCanceledException)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\n\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0consumer.Close();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\n\u00a0\u00a0\u00a0\u00a0}\n}\n<\/pre><\/div>\n\n\n
We configure the consumer by indicating the Consumer Group, the address of the Kafka server and specifying what action to take when there is no initial offset. We create the consumer and subscribe it to our topic, then we start reading the topic until the timeout<\/em> specified through the Consume<\/em> method. We print the result of the reading in the console and finally with the Close<\/em> method we commit the offsets on Kafka, alerting that the consumer is leaving the group.<\/p>\n\n\n\n
By running the Consumer, we can see that all published messages have been consumed correctly.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The Kafka ecosystem<\/strong><\/p>\n\n\n\n
This is just an overview of the core features of Kafka, which has become a real ecosystem over the years. In fact, in addition to the exchange of messages between producer and consumer, there are two further features: Kafka Connect<\/em> and Kafka Stream<\/em>.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The first is a framework for the data transfer of heterogeneous systems through Kafka which provides well-defined APIs to implement so-called connectors<\/em>. Think about the variation of the field of a table of a database notified through Kafka: this is possible through a database connector to Kafka.<\/p>\n\n\n\n
Instead, the second feature is a library for creating streaming applications, that is applications that transform Kafka input topics into output topics.<\/p>\n\n\n\n
Conclusions<\/strong><\/h2>\n\n\n\n
Hoping that this introduction has been interesting to enter the Kafka ecosystem, I leave you the link of the repository<\/a> where you can find the producer and consumer implementation, as well as the commands to create the containers for Kafka and Zookeeper.<\/p>\n\n\n\n
See you next time!<\/p>\n\n\n\n\n","protected":false},"excerpt":{"rendered":"
Let\u2019s discover Kafka by introducing the fundamental concepts and showing an example in .NET Core<\/p>\n","protected":false},"author":196716251,"featured_media":26783,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"off","_et_pb_old_content":"","_et_gb_content_width":"","_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","inline_featured_image":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[688637524],"tags":[688637542,688637546,688637438,688637537],"class_list":["post-26811","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog-en","tag-dotnetcore-en","tag-docker-en","tag-kafka-en","tag-swdesign-en"],"yoast_head":"\n"The Metamorphosis" of communication with Kafka - Blexin<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\""The Metamorphosis" of communication with Kafka - Blexin\" \/>\n<meta property=\"og:description\" content=\"Let\u2019s discover Kafka by introducing the fundamental concepts and showing an example in .NET Core\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/\" \/>\n<meta property=\"og:site_name\" content=\"Blexin\" \/>\n<meta property=\"article:published_time\" content=\"2020-04-21T22:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-21T18:28:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i1.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"608\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Genny Paudice\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Genny Paudice\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/\"},\"author\":{\"name\":\"Genny Paudice\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/#\\\/schema\\\/person\\\/ae0e97a3ebb79d2f73e4411c1eb28973\"},\"headline\":\"“The Metamorphosis” of communication with Kafka\",\"datePublished\":\"2020-04-21T22:00:00+00:00\",\"dateModified\":\"2021-05-21T18:28:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/\"},\"wordCount\":1643,\"image\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/blexin.com\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1\",\"keywords\":[\".NetCore\",\"Docker\",\"Kafka\",\"softwaredesign\"],\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/\",\"url\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/\",\"name\":\"\\\"The Metamorphosis\\\" of communication with Kafka - Blexin\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/blexin.com\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1\",\"datePublished\":\"2020-04-21T22:00:00+00:00\",\"dateModified\":\"2021-05-21T18:28:29+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/#\\\/schema\\\/person\\\/ae0e97a3ebb79d2f73e4411c1eb28973\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/blexin.com\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/blexin.com\\\/wp-content\\\/uploads\\\/2020\\\/12\\\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1\",\"width\":1024,\"height\":608},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/blog-en\\\/the-metamorphosis-of-communication-with-kafka\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/blexin.com\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"“The Metamorphosis” of communication with Kafka\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/blexin.com\\\/en\\\/\",\"name\":\"Blexin\",\"description\":\"Con noi \u00e8 semplice\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/blexin.com\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/blexin.com\\\/en\\\/#\\\/schema\\\/person\\\/ae0e97a3ebb79d2f73e4411c1eb28973\",\"name\":\"Genny Paudice\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3093a6ee7b7f555e94e4a5d6f19cb09f3af33635007643b2076eb89329789828?s=96&d=identicon&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3093a6ee7b7f555e94e4a5d6f19cb09f3af33635007643b2076eb89329789828?s=96&d=identicon&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3093a6ee7b7f555e94e4a5d6f19cb09f3af33635007643b2076eb89329789828?s=96&d=identicon&r=g\",\"caption\":\"Genny Paudice\"},\"url\":\"https:\\\/\\\/blexin.com\\\/en\\\/author\\\/genny-paudiceblexin-com\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\"The Metamorphosis\" of communication with Kafka - Blexin","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/","og_locale":"en_US","og_type":"article","og_title":"\"The Metamorphosis\" of communication with Kafka - Blexin","og_description":"Let\u2019s discover Kafka by introducing the fundamental concepts and showing an example in .NET Core","og_url":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/","og_site_name":"Blexin","article_published_time":"2020-04-21T22:00:00+00:00","article_modified_time":"2021-05-21T18:28:29+00:00","og_image":[{"width":1024,"height":608,"url":"https:\/\/i1.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1","type":"image\/png"}],"author":"Genny Paudice","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Genny Paudice","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#article","isPartOf":{"@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/"},"author":{"name":"Genny Paudice","@id":"https:\/\/blexin.com\/en\/#\/schema\/person\/ae0e97a3ebb79d2f73e4411c1eb28973"},"headline":"“The Metamorphosis” of communication with Kafka","datePublished":"2020-04-21T22:00:00+00:00","dateModified":"2021-05-21T18:28:29+00:00","mainEntityOfPage":{"@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/"},"wordCount":1643,"image":{"@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1","keywords":[".NetCore","Docker","Kafka","softwaredesign"],"articleSection":["Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/","url":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/","name":"\"The Metamorphosis\" of communication with Kafka - Blexin","isPartOf":{"@id":"https:\/\/blexin.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#primaryimage"},"image":{"@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1","datePublished":"2020-04-21T22:00:00+00:00","dateModified":"2021-05-21T18:28:29+00:00","author":{"@id":"https:\/\/blexin.com\/en\/#\/schema\/person\/ae0e97a3ebb79d2f73e4411c1eb28973"},"breadcrumb":{"@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#primaryimage","url":"https:\/\/i0.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1","contentUrl":"https:\/\/i0.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1","width":1024,"height":608},{"@type":"BreadcrumbList","@id":"https:\/\/blexin.com\/en\/blog-en\/the-metamorphosis-of-communication-with-kafka\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blexin.com\/en\/"},{"@type":"ListItem","position":2,"name":"“The Metamorphosis” of communication with Kafka"}]},{"@type":"WebSite","@id":"https:\/\/blexin.com\/en\/#website","url":"https:\/\/blexin.com\/en\/","name":"Blexin","description":"Con noi \u00e8 semplice","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blexin.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blexin.com\/en\/#\/schema\/person\/ae0e97a3ebb79d2f73e4411c1eb28973","name":"Genny Paudice","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/3093a6ee7b7f555e94e4a5d6f19cb09f3af33635007643b2076eb89329789828?s=96&d=identicon&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/3093a6ee7b7f555e94e4a5d6f19cb09f3af33635007643b2076eb89329789828?s=96&d=identicon&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3093a6ee7b7f555e94e4a5d6f19cb09f3af33635007643b2076eb89329789828?s=96&d=identicon&r=g","caption":"Genny Paudice"},"url":"https:\/\/blexin.com\/en\/author\/genny-paudiceblexin-com\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/blexin.com\/wp-content\/uploads\/2020\/12\/a02436f4-6edd-4491-856a-c1985e9aa754.png?fit=1024%2C608&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/pcyUBx-6Yr","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/posts\/26811","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/users\/196716251"}],"replies":[{"embeddable":true,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/comments?post=26811"}],"version-history":[{"count":10,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/posts\/26811\/revisions"}],"predecessor-version":[{"id":32929,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/posts\/26811\/revisions\/32929"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/media\/26783"}],"wp:attachment":[{"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/media?parent=26811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/categories?post=26811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blexin.com\/en\/wp-json\/wp\/v2\/tags?post=26811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}