Real-time integration of IoT sensor data with automated data pipelines for effective logistics management
Sigmoid optimized logistics management by re-architecting the data platform and building robust and scalable data pipelines which resulted in faster data collection, improved data quality, and seamless pipeline extensibility.
Business Challenge
The client supplies industrial gases and related equipment for the refining, chemical, metals, and electronics industries. They rely heavily on data from IoT sensors installed in trucks to monitor the transportation of their hazardous materials. However, data quality from the third-party IoT sensors has been inconsistent due to the limitations of the current data pipeline architecture such as missing data quality checks. Monitoring drivers' road safety, driving behavior, and the temperature of the container while transporting gases and chemicals is also critical, but their existing system has a delayed response time of up to 48 hours. By establishing a stronger data foundation they wanted to develop a near real time internal dashboard for fleet managers, data analysts, and enterprise architects, to process data as per their needs.
Sigmoid Solution
Sigmoid re-architected the IoT data pipeline for devices installed in the client’s trucks to meet data quality, security, and governance standards. We audited the existing data platform, the data acquisition process, and data from IoT sensors while providing recommendations on best data engineering practices, standards, and guidelines. The solution helped build a robust and scalable data pipeline for multiple data sources with code reusability, modularization, and easy maintenance for adding new data sources in the future. Both master data and transaction type input APIs were considered for creating the data pipeline and various technology options were presented to the client. Glue was used for data prep and data transformation, Redshift for query platform/reporting, Step Functions for workflow management, and PyDeque for data quality jobs.
Business Impact
Sigmoid's solution provided benefits such as extensible data pipelines, faster and more reliable data collection, and improved data quality through strategic checks. The data pipeline facilitated extended analysis to gain deeper insights by conducting time-based comparison of KPIs such as fuel efficiency, safety compliance, and truck maintenance. We also developed a scorecard to evaluate the performance of the drivers.
7 APIs
integrated for new data pipeline development
3 hr
daily manual data preparation time eliminated through automation
Extensible
pipeline for quick integration of new IoT sensors
