NVIDIA Research Breakthroughs Put Advanced Robots in Motion

[UHQBot 24]

Across robot training and development, NVIDIA Research is uncovering breakthroughs in areas such as multimodal generative AI and synthetic data generation.

The team’s latest innovations will be spotlighted at the International Conference on Robotics and Automation (ICRA), running May 19-23 in Atlanta.

“ICRA has played a pivotal role in shaping the direction of robotics and automation, marking key milestones in the field’s evolution and celebrating achievements that have had a lasting impact on technology and society,” said Dieter Fox, senior director of robotics research at NVIDIA. “The research we’re contributing this year will further advance the development of autonomous vehicles and humanoid robots by helping close the data gap and improve robot safety and control.”

Generative AI for Scalable Robotic Learning

NVIDIA-authored papers showcased at ICRA give a glimpse into the future of robotics. They include:

• DreamDrive: This 4D spatial-temporal scene generation approach creates realistic, controllable 4D driving scenes using video diffusion and 3D Gaussian splatting for autonomous vehicles.
• System-Level Safety Monitoring and Recovery for Perception Failures in Autonomous Vehicles: This real-time Q-network can detect and help autonomous vehicles account for perception failures, bolstering AV planning and safety.
• Inference-Time Policy Steering Through Human Interactions: This framework enables human-guided adjustments to generative policies during inference for better alignment and safety.
• DexMimicGen: This system can generate large-scale bimanual dexterous manipulation datasets from just a few human demonstrations.
• HOVER: A unified neural controller for humanoid robots that seamlessly transitions between locomotion, manipulation and other modes.
• MatchMaker: This pipeline automates generation of diverse 3D assembly assets for simulation-based training, enabling robots to learn insertion tasks without manual asset curation.
• SPOT: This learning framework uses SE(3) pose trajectory diffusion for object-centric manipulation, enabling cross-embodiment generalization.
• System-Level Safety Monitoring and Recovery for Perception Failures in Autonomous Vehicles: This real-time system monitors perception reliability and triggers recovery strategies to ensure robust planning and safety in autonomous vehicles.
• Inference-Time Policy Steering through Human Interactions: This framework enables real-time human-guided adjustments to model outputs during inference, improving policy alignment without retraining.

Robotics workshops at ICRA featuring NVIDIA speakers include:

• Pioneering the Future of Human Motion Prediction: Explore how advancements in AI, robotics and biomechanics can improve human-computer interaction and safety.
• Towards Reliable and Trustworthy Embodied AI in Everyday Scenarios: Learn how to address the challenges of developing reliable and trustworthy embodied AI systems for seamless integration into everyday environments.
• Beyond the Lab: Robust Planning and Control in Real-World Scenarios: Gather insights on robust planning and control techniques that can help robots adjust to the complexities of real-world scenarios.
• Safely Leveraging Vision-Language Foundation Models in Robotics: Explore strategies for safely integrating vision-language foundation models into robotic applications to bolster performance and reliability.
• Human-Centered Robot Learning in the Era of Big Data and Large Models: Learn human-centered approaches to robot learning, using big data and large models for enhanced adaptability and intelligence.
• RoboARCH: Robotics Acceleration with Computing Hardware and Systems: Hear about innovations in computing hardware and systems that are driving rapid progress and acceleration in robotics research and applications.

Explore the latest work from NVIDIA Research, and check out the Robotics Research and Development Digest (R²D²), which gives developers deeper insight into the latest physical AI and robotics breakthroughs.

View the full article