Public Cloud – Flexible Engine

Distributed Message Service for Kafka

[Features – Applications Services]

A message queuing service based on Apache Kafka and more specifically Kafka premium instances

logo distributed message service for Kafka

Distributed Message Service (DMS) for Kafka is a message queuing service based on Apache Kafka and more specifically Kafka premium instances. The computing, storage, and bandwidth resources used by an instance are exclusively occupied by the user. 

Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/offline system analytics, and real-time monitoring.

Scenarios 1: Asynchronous Communication

Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.

For example, Kafka can be used to send a notification email and SMS message after a user has registered with a website, providing fast responses throughout the registration process.

Figure 1 Serial registration and notification

Distributed message service for Kafka - Figure 1 (Serial registration and notification)

Figure 2 Asynchronous registration and notification using message queues

Scenario 2: Log Synchronization

Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or offline. In addition, Kafka can collect key log information to monitor applications.

Log synchronization process :

distributed message service for Kafka - log synchronization process

Benefits

Rapid deployment

Simply set instance information on the DMS for Kafka console, submit your order, and a complete Kafka premium instance will be automatically created and deployed.

Service migration without modifications

DMS for Kafka is compatible with open-source Kafka APIs and supports all message processing functions of open-source Kafka. If your application services are developed based on open-source Kafka, you can easily migrate them to DMS for Kafka after specifying a few authentication configurations.

Security

Operations on Kafka premium instances are recorded and can be audited. Messages can be encrypted before storage.In addition to SASL, Virtual Private Clouds (VPCs) and security groups also provide security controls on network access.

Data reliability

Kafka premium instances support data persistence and replication. Messages can be replicated synchronously or asynchronously between replicas.

High availability

Kafka runs in clusters, enabling failover and fault tolerance so that services can run smoothly. Kafka premium instances can be deployed across AZs to further enhance service availability.

Simple O&M

The public cloud provides a whole set of monitoring and alarm services, eliminating the need for 24/7 attendance. A set of Kafka premium instance metrics are monitored and reported, including the number of partitions, topics, and accumulated messages. You can configure alarm rules and receive SMS or email notifications on how your services are running in real time.

Flexible specifications

You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.

Functions

The following table describes the functions visible to tenants. These functions do not include O&M features.

CategoryFunctionDescription

Basic functions

Creating a queue

Creates a queue for receiving and sending messages.
Two options are provided at the same time: one is selecting queue type for standard or FIFO.


Viewing queues

Lists the brief information about all queues of a tenant.

Viewing queue details

Lists the details of a queue, and the details of the groups that consume messages in this queue.

Deleting queues

Deletes one or multiple specified queues.

Sending messages to a specified queue

Sends messages (including message content and user-defined message attributes) to a specified queue.

Creating a message consumer group for a specified queue

Creates a message consumer group with a specified name. Application can consume message from a queue using a message consumer group.


Consuming messages in a specified queue

With this function, you can choose the type of message, including normal, the number of automatically consumed messages and the interval between consuming messages. You can also view the content, size, and sending time of each message, and stop message consumption.

Confirming message consumption

Submits message consumption confirmation to a message queue after each instance of message consumption.


Managing queues

Manages message queues through the web-based DMS console.


Accessing message queues

Accesses message queues through RESTful APIs for standard queue and FIFO queue.

Monitoring

CES

For details, see the CES Product Description.

diam Phasellus sit quis, id vel, mattis fringilla ut