Dall-e-bot: Introducing Web-scale Diffusion Models To Robotics | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A continuously updated collection of research papers on LLMs.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Dall-e-bot: Introducing Web-scale Diffusion Models To Robotics

Ivan Kapelyukh, Vitalis Vosylius, Edward Johns . IEEE Robotics and Automation Letters 2023 – 46 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Compositional Generalization Datasets Evaluation Interdisciplinary Approaches Productivity Enhancement Training Techniques

We introduce the first work to explore web-scale diffusion models for robotics. DALL-E-Bot enables a robot to rearrange objects in a scene, by first inferring a text description of those objects, then generating an image representing a natural, human-like arrangement of those objects, and finally physically arranging the objects according to that goal image. We show that this is possible zero-shot using DALL-E, without needing any further example arrangements, data collection, or training. DALL-E-Bot is fully autonomous and is not restricted to a pre-defined set of objects or scenes, thanks to DALL-E’s web-scale pre-training. Encouraging real-world results, with both human studies and objective metrics, show that integrating web-scale diffusion models into robotics pipelines is a promising direction for scalable, unsupervised robot learning.

Similar Work