Accelerated Warehouse Automation with Digital Twins and Synthetic Data

Executive Summary

Toyota Material Handling Europe is a leader in the material handling business, offering AGVs (automated guided vehicles) in addition to other logistics solutions and services. As a premier manufacturer and distributor in the sector, Toyota recognized the urgency to enhance its warehouse automation processes amidst evolving market demands. To address this challenge, Toyota sought to leverage cutting-edge technology through a collaboration with NVIDIA and SoftServe, focusing on an innovative digital twin initiative that promises to revolutionize material handling.

To begin, Toyota Material Handling Europe selected a transformative aspect of warehouse logistics for simulation: collaborative case picking. As automation increases, the workflows involving interactions between pickers and AGVs have grown increasingly important. Simulating the complex interactions took two parallel streams of work:

1. A digital twin built using the NVIDIA Omniverse to simulate real-world scenarios
2. Training of AI models on synthetic data enhanced with NVIDIA Cosmos-Transfer

Together, these two initiatives have accelerated Toyota’s ability to develop safe automated guided vehicles for warehouse operations and positioned the company at the forefront of warehouse automation.

The Goal: Rapid Development of Autonomous Forklifts

The primary challenge in the domain of robotics and automation lies in the ability to effectively simulate real-world scenarios. Traditional testing methods were slow, expensive, and limited — slowing innovation and extending time-to-market.

To accelerate development of autonomous forklifts, Toyota needed a solution that could:

• Reduce reliance on costly, time-consuming physical tests
• Simulate intricate warehouse environments to enhance navigation and interaction
• Integrate advanced technologies to proactively solve material handling challenges

To stay competitive, Toyota turned to high-fidelity simulation and digital twin technology — enabling rapid, low-risk testing and optimization before physical deployment.

Stream 1: High-Fidelity Digital Twin for Forklift Simulations

In collaboration with SoftServe and NVIDIA, Toyota Material Handling launched a digital twin initiative that leverages physical AI and synthetic data generation (SDG) to simulate, validate, and optimize autonomous forklift operations within high-fidelity virtual replicas of warehouses and production environments. This system utilizes AI to detect pallet components, interpret depth information from labels, and orchestrate the full manipulation sequence — including precise fork positioning and the complete movement required to safely pick up or place down pallets with payloads.

By doing so, it significantly reduces the risk of damage and enhances operational efficiency in dynamic, real-world logistics scenarios.

Results:

At the core of this solution is the NVIDIA Omniverse platform, a real-time simulation and collaboration framework for creating high-fidelity, physics-accurate digital twins. SoftServe collaborated with Toyota to develop and integrate:

• Digital Replicas of Autonomous Forklifts: High-fidelity digital twins of Toyota's forklift models were created to mirror real-world physics and operational timelines.
• AI-Driven Simulations: The initiative integrated real software-in-the-loop capabilities, enabling AI systems to simulate actual operational conditions and assess the forking and material handling tasks.
• Human-Robot Interaction Scenarios: Collaborative scenarios, such as case picking, were tested to ensure safe and efficient interactions between human operatives and robotic systems.

Stream 2: AI Model Improvement With Enhanced Synthetic Data

The AI models at the heart of AGVs must be trained to enhance their precision (percentage of correct positive predictions a model makes) and recall (percentage of the actual positive instances the model correctly identifies). To minimize cost and time spent on gathering scarce real-world data, Toyota worked with SoftServe to enhance synthetic training data for warehouse object detection tasks. Leveraging NVIDIA Cosmos-Transfer, the teams demonstrated that AI models trained on Cosmos-processed data significantly outperform simulator-trained models on real-world data while maintaining comparable performance on synthetic datasets.

The project contained two steps:

1. NVIDIA Cosmos-Transfer: Deployed NVIDIA Cosmos-Transfer on OVX systems to generate ultra-realistic synthetic data, bridging the simulation-to-reality gap for Toyota's autonomous forklifts.
2. NVIDIA Cosmos Post-Training: Further bridged the sim-to-real gap by post-training the Cosmos model with just 15 minutes of client warehouse videos on the NVIDIA DGX Cloud Lepton platform.

Training and Evaluation on Three Datasets

Toyota Material Handling Europe and SoftServe used specific datasets to evaluate the models’ performance.

• Simulated dataset: Generated using the NVIDIA Isaac Sim™ simulator. It consisted of 50 videos showing raw, unwrapped boxes. This dataset served as a baseline for synthetic training, offering perfect labeling in a controlled environment.
• Cosmos-enhanced dataset: Created by processing the original 50 videos through Cosmos-Transfer. Each video was processed three times, producing 150 videos that simulated realistic stretch film and foil wrappings to mimic real-world conditions.
• Real-world dataset: Compiled from client-provided warehouse footage. This dataset included 137 hand-labeled images of actual boxes with stretch wrap, capturing authentic lighting and genuine warehouse environments.
• Cosmos-Transfer post-training dataset: Contained 141 videos (5 seconds each) captured directly from the actual warehouse environment. The dataset improved label accuracy, color fidelity, floor characteristics, and shadow rendering.

Results:

The NVIDIA Cosmos™-trained model achieved 89.6% precision and 84.7% recall on real-world datasets on the DGX Cloud Lepton platform with Cosmos-trained models, compared to just 49.4% recall for simulator-only models. Moreover, the post-trained Cosmos model demonstrated dramatic gains — achieving 99.5% precision and 92.8% recall on real-world data by adapting visuals such as labels, colors, flooring, and shadows to match the client’s environment.

This advanced customization resulted in superior detection quality and alignment — outperforming both simulator-only and standard Cosmos-trained models. Thanks to the initiative, Toyota is now in the position to leverage AI models in its AGVs with demonstrably greater ability to perform critical tasks in AGV-human interactions.

Impact: Rapid and Affordable AGV Commissioning

The implementation of this advanced simulation framework will yield considerable benefits for Toyota:

• Enhanced operational efficiency: The predictive simulations will greatly reduce the risks of damage during forklift operations, increasing overall safety and workflow efficiency.
• Faster time-to-market: By simulating complex scenarios in a virtual environment, Toyota will accelerate the development cycle of autonomous forklifts, allowing them to respond swiftly to market demands.
• Cost reductions: The digital twin initiative will allow Toyota to minimize costs associated with physical testing, failures, and damages by testing in a risk-free environment before deployment.

Tech Stack

Conclusion

With the help of NVIDIA's innovative technologies and SoftServe’s expertise, Toyota will position itself as a leader in warehouse automation, ready to tackle the dynamics of modern logistics. Through the strategic collaboration with SoftServe and NVIDIA, Toyota Material Handling Europe has not only addressed its initial operational challenges but has also set a new benchmark in the field of warehouse automation.

The digital twin initiative illustrates the potential of combining innovative technologies with operational expertise, while the enhancement of AI models with SDG demonstrates the possibilities to bridge the gap between the virtual and physical world — paving the way for a future of automated logistics that is more efficient, safe, and responsive to market needs.

Are you interested in developing advanced automation solutions? SoftServe can help you use advanced digital twin technology and AI-driven simulations to set a new standard for safety and productivity — before deploying a single robot.