.png)
.png)
.png)
What if robots could collaborate with artists?
Feb. 09, 2023.
2 mins. read.
7 Interactions
About the Writer
Carnegie Mellon University’s Robotics Institute is exploring that idea
Their FRIDA (Framework and Robotics Initiative for Developing Arts, named for Mexican painter Frida Kahlo), uses a robotic arm with a paintbrush taped to it and AI to collaborate with humans on works of art.
FRIDA is a robotic painting system, “but FRIDA is not an artist,” says Peter Schaldenbrand, a School of Computer Science Ph.D. student in the Robotics Institute, who is working with FRIDA and exploring AI and creativity. “Instead, FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA and then FRIDA can execute them.” Artists can direct FRIDA by inputting a text description, submitting other works of art to inspire its style; or by uploading a photograph and asking Frida to paint a representation of it.
An artist can direct FRIDA by inputting a text description, submitting other works of art to inspire its style, or uploading a photograph and asking it to paint a representation of it.
One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap — the difference between what FRIDA composes in simulation and what it paints on the canvas. FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.
Beyond DALL-E-2
The robot uses AI models similar to those powering tools like OpenAI’s ChatGPT and DALL-E 2, which generate text or an image, respectively, in response to a prompt. But instead, FRIDA simulates how it would paint an image with brush strokes and uses machine learning to evaluate its progress as it works.
DALL-E 2 uses large-vision language models to produce digital images. FRIDA takes that a step further and uses its embodied robotic system to produce physical paintings. One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap, the difference between what FRIDA composes in simulation and what it paints on the canvas. So FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.
The team is also experimenting with other inputs, including audio. They played ABBA’s “Dancing Queen” and asked FRIDA to paint it.
Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter.
2 Comments
2 thoughts on “What if robots could collaborate with artists?”
The writer has done an excellent job capturing the innovative and creative essence of Carnegie Mellon’s FRIDA project, emphasizing the unique intersection of robotics and art. Their exploration of how technology can foster artistic expression is inspiring and thought-provoking. Thank you for such an insightful article that makes complex technology feel accessible and exciting!
🟨 😴 😡 ❌ 🤮 💩
Carnegie Mellon University’s Robotics Institute is pushing the boundaries of what's possible in robotics. Their innovative research and development are not only advancing the field but also opening up new possibilities for real-world applications. The potential impact on industries ranging from healthcare to manufacturing is immense, promising to revolutionize the way we interact with technology and automation. It’s inspiring to see such cutting-edge work being done, highlighting the future of robotics and its potential to improve our daily lives.
🟨 😴 😡 ❌ 🤮 💩