As supply chains embark on their objective to become real-time planning and operations entities, building the technology architecture to support this objective becomes paramount. This is why real-time ecosystems like Kafka can play a critical role in the digital transformation journey to become real-time.

This three-part article will explore how the Kafka ecosystem can help supply chains attain realistic real-time capabilities and generate innovation. 

The first part of this article series introduced Kafka. In the second part of the article, we started exploring how Kafka ecosystems can help build real-time capabilities in the supply chain context, starting with procurement and manufacturing. In this third and final part, we will explore Kafka based architectures for inventory management, warehousing and control towers.

Kafka and inventory management

During her research, Stephanie came across the real-time inventory management system implemented by Walmart. The gist of this implementation was to build a centralized, real-time view of the inventory based on events across the supply chain, ranging from warehouses to stores. Each event source has its own Kafka cluster and corresponding stream, which are then centralized into one normalized Kafka cluster. From the reading, it seemed like this was done at an item level. The architecture has been reproduced in Figure 1.

Figure 1: Walmart’s real-time replenishment architecture. Source: https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/

In Stephanie’s mind, the secret ingredient in the architecture lay in the normalization to build one single Kafka stream. Since the purpose of this centralized view is not just a status of inventory but also to leverage it for predictive and prescriptive analytics, this architecture can not be developed in silo for an organization like ACME, which also has manufacturing locations. There were not enough details on how this architecture works with other existing systems.

Stephanie’s goal was not just inventory data centralization. While the centralized real-time data view definitely provides real-time tactical planning capabilities, she wanted to leverage it for operational and strategic planning (Though the real-time may not be an imperative requirement for the strategic view). She wanted to build analytics and data science capabilities that leverage centralized clusters for each data source. This will ensure that data and analytics are harmonized and standardized across the enterprise.

After all, the real power of building this type of architecture is leveraging it beyond vanilla applications like real-time product availability scenarios like buy online and pick at store and replenishment. Once you have built a foundation like this, the opportunities to leverage that real-time data are limitless.

Kafka architecture for supply chain control tower

And that is why Stephanie knew that there are two key Kafka-enabled capabilities that need to be built:

• Develop a centralized view of real-time streams for inventory and relevant events from the end-to-end supply chain. This is what is wrongly perceived as a supply chain control tower.
• Generate real-time insights from that real-time data. The word “control” in supply chain control tower originates from the fact that you should be able to leverage that real-time data to control aspects of your supply chain, proactively or reactively.

No matter what type of data architecture or view your organization has, including the data mesh logical view, Kafka clusters can still help build the centralization for supply chain control towers, as shown in Figure 2.

Figure 2: Integrating data products in a data mesh with Kafka clusters. Based on : https://www.kai-waehner.de/blog/2022/09/23/supply-chain-control-tower-for-end-to-end-visibility-using-apache-kafka/

This is where Stephanie believes her team can help with the magic. Leveraging analytics, ACME can control its supply chain proactively and reactively using its supply chain control tower.

However this also generated a sidebar question in Stephanie’s mind-“If I can build a centralized view leveraging Kafka clusters from across the organization, why would I need rigidly defined control tower applications? If I have done the hard work of building the data foundation, I can build a customized front-end application embedded with analytics within weeks. With testing incorporated in the timeline, I can produce that customized control tower application in less than three months.” But that was not within the context of what Stephanie is strategizing at this point, so she let it slip.

Kafka and warehouse management

Last but not least, Stephanie has some fantastic ideas about how she can leverage real-time data powered by Kafka to transform warehouse operations beyond inventory management.

Warehouse activities are event-driven, and if you consider every time a barcode is scanned as an event, it can span into millions of events for large warehouses and fulfillment centers. Add to that the data streams generated by a plethora of automation equipment in the warehouse.

Leveraging a Kafka-enabled architecture, Stephanie can build precisely the view a centralized deep learning algorithm would need, to perform analytics like:

• Real-time inbound and outbound load scheduling
• Real-time flow optimization
• Real-time workforce optimization
• Real-time warehouse asset utilization tracking

Conclusion

It is evident that Kafka, or similar technologies, can play a crucial role in building the real-time capabilities many supply chain executives fancy. Yet, very few supply chains have been able to implement real-time streaming data architectures. I consistently emphasize that to become a driven, digitally savvy organization, your key focus should be building a data foundation. Once you attain that objective, building other capabilities, like centralized deep learning models, will be much faster and more productive.

References:

• Querying Kafka with Kafka Streams, Oreilly
• Getting Started with Kafka Streams, Oreilly
• Kafka Streams: Deploy a Local Kafka Cluster, Oreilly Lab
• Unlocking the Value of Real-Time Analytics, Oreilly
• https://www.confluent.io/resources/kafka-summit-london-2019/
• https://ieeexplore.ieee.org/document/10220093
• https://www.confluent.io/online-talks/modernizing-the-manufacturing-industry-with-kafka-and-mqtt/
• https://mqtt.org
• https://www.emqx.com/en/blog/mqtt-and-kafka
• https://info.hivemq.com/mqtt-broker?
• https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
• https://www.kai-waehner.de/blog/2022/09/23/supply-chain-control-tower-for-end-to-end-visibility-using-apache-kafka/