As supply chains embark on their objective to become real-time planning and operations entities, building the technology architecture to support this objective becomes paramount. This is why real-time ecosystems like Kafka can play a critical role in the digital transformation journey to become real-time.

This three-part article will explore how the Kafka ecosystem can help supply chains attain realistic real-time capabilities and generate innovation.

The first part of this article series introduced Kafka. In this second part of the article, we will start exploring how Kafka ecosystems can help build real-time capabilities in the supply chain context.

We will use the traditional end-to-end view of the supply chain to highlight how Kafka can help real-time analytics capabilities across the supply chain. While we do not believe supply chains are simple end-to-end entities, this is a view that most of us are familiar with and will be intuitive for readers to follow. The example sub-functions in the supply chain of our hypothetical entity, ACME Corp. are shown in the figure below:

Figure 1: A high-level overview of ACME’s supply chain flow

Kafka for sourcing and procurement data science solutions

Stephanie, the manager of the data science team at ACME keeps exploring the best practices across the data science ecosystem. That means she is constantly consuming knowledge, reading white papers, pursuing certifications, and attending conferences whenever she gets a chance. In 2019, she attended the Kafka summit in London, where she learned how BMW leveraged Kafka and Deep Learning/NLP for contract management.

Over the next few years, her team assisted in replicating a Kafka-enabled deep learning-based contract intelligence solution that can analyze contracts in real-time. Like the BMW solution, this solution performs innovative information extraction, automated risk assessment, plausibility checks, and negotiation support. But Stephanie also wanted to develop her unique sourcing and procurement solutions.

One of the solutions she developed consumes real-time order entries from ACME’s operations across the globe. A deep learning solution then leverages this to identify procurement-related challenges and bottlenecks that may emerge as the order moves forward in the pipeline. For example, if a customer orders a product, and the supplier for that product just failed the quality reliability assessment, the algorithm will flag all orders for that product as those at risk.

As we will see in subsequent sections, this capability to read streams of orders in real-time in a high order volume environment like ACME can be leveraged extensively.

Kafka for manufacturing

At the core of the data science initiatives that Stephanie has led are Kafka & MQTT. The genesis of these initiatives was one of the videos that Stephanie watched during her leisure time.

Understanding MQTT

Before we explore the content of the video, let us understand briefly what MQTT is. MQTT stands for Message Queuing Telemetry Transport. Let us start with the official definition of MQTT:

“MQTT is a Client Server publish/subscribe messaging transport protocol. It is lightweight, open, simple, and designed so as to be easy to implement. These characteristics make it ideal for use in many situations, including constrained environments such as for communication in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a small code footprint is required and/or network bandwidth is at a premium.”

– MQTT Documentation

The reason I have emphasized IoT in bold in the definition is because IoT devices are/will be critical sources of streaming data in manufacturing plants.

As you may already know, Kafka and MQTT are complementary technologies. Together, these two technologies enable end-to-end integration of IoT data. By integrating Kafka and MQTT, organizations can design robust IoT architectures that have reliable connectivity and can execute fast & efficient data exchange between devices and IoT platforms.

There are multiple products available in the market that leverage MQTT protocol. One of the widely used ones is a MQTT broker.

MQTT Broker

The MQTT broker is the backend system that coordinates messages between clients. Responsibilities of the broker include receiving and filtering messages, identifying clients subscribed to each message, and sending them the messages.

Figure 2: High-level overview of MQTT broker

The MQTT broker in Figure 2 performs data extraction, filtering, enrichment, and transformation. In another article series, we will explore the MQTT protocol in detail. But let us understand how Stephanie built a manufacturing data science solution leveraging MQTT broker and Kafka. The high-level elements of the solution are shown in Figure 3.

Figure 3: MQTT and Kafka for manufacturing data science

As you can see, to develop real-time data science solutions, Stephanie has to leverage an architecture that can effectively capture data in real time from the floor of ACME’s intelligent factories worldwide. And Kafka plays a crucial role in enabling that architecture. We will not get into details of the different types of analytics Stephanie’s team performs. The key for this article is to understand that once you have built this capability, the opportunities to experiment with data are limitless.

This article’s third and final part will explore how Kafka helps transform visibility and analytics in transportation, warehousing, and end-to-end visibility and planning (control towers). The third and final part can be found here.

References:

• Querying Kafka with Kafka Streams, Oreilly
• Getting Started with Kafka Streams, Oreilly
• Kafka Streams: Deploy a Local Kafka Cluster, Oreilly Lab
• Unlocking the Value of Real-Time Analytics, Oreilly
• https://www.confluent.io/resources/kafka-summit-london-2019/
• https://ieeexplore.ieee.org/document/10220093
• https://www.confluent.io/online-talks/modernizing-the-manufacturing-industry-with-kafka-and-mqtt/
• https://mqtt.org
• https://www.emqx.com/en/blog/mqtt-and-kafka
• https://info.hivemq.com/mqtt-broker?