The LEGOGPT system works in the three parts shown in this diagram.
Credit: Pun et al.
The researchers also expanded the system’s capabilities by adding textures and color options. Using a look prompt, such as “Metallic Purple Electric Guitar,” LEGO Gupto generates a guitar model, and assigns the bricks a purple color.
Testing with robots and humans
To prove that their designs work in real life, researchers have robots assemble AI-created LEGO models. They used a dual robotic arm system with force sensors to pick up and place the bricks according to instructions generated by the AI.
Human testers also built parts of the design with their hands that demonstrate that AI creates truly constructable models. “Our experiments show that LEGOGPT produces stable, diverse, aesthetically pleasing LEGO designs that closely match the input text prompts,” the team states in their paper.
When tested against other AI systems for 3D creation, LegoGPT stands out with a focus on structural integrity. The team tested against several alternatives, including Llama-Mesh and other 3D generation models, and found that their approach produced the highest percentage of stable structures.
A video provided by researchers in which two robot arms build Legoput creation.
A video provided by researchers in which two robot arms build Legoput creation.
Still, there are some limitations. The current version of LEGOGPT works only within 20x20x20 building spaces and uses only eight standard brick type types. “Our methods now support a fixed set of LEGO Bricks, which are commonly used,” the team admitted. “In future work, I plan to expand my brick library to include a wider range of dimensions and brick types, such as slopes and tiles.”
Researchers also hope to expand their training dataset to include more objects than the 21 categories currently available. Meanwhile, others can literally build on their work. The researchers have released datasets, code and models on the project website and GitHub.
