back Back

What if robots could collaborate with artists?

Feb. 09, 2023.
2 min. read Interactions

About the Writer

Amara Angelica

164.11594 MPXR

Senior Editor Amara Angelica, an electrical engineer and inventor, was previously Editor of Kurzweil AI, working with Ray Kurzweil on The Singularity Is Near and other works.

Carnegie Mellon University’s Robotics Institute is exploring that idea

Their FRIDA (Framework and Robotics Initiative for Developing Arts, named for Mexican painter Frida Kahlo), uses a robotic arm with a paintbrush taped to it and AI to collaborate with humans on works of art.

FRIDA’s first paintings (credit: Carnegie Mellon University/The Robotics Institute)

FRIDA is a robotic painting system, “but FRIDA is not an artist,” says Peter Schaldenbrand, a School of Computer Science Ph.D. student in the Robotics Institute, who is working with FRIDA and exploring AI and creativity. “Instead, FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA and then FRIDA can execute them.” Artists can direct FRIDA by inputting a text description, submitting other works of art to inspire its style; or by uploading a photograph and asking Frida  to paint a representation of it.

An artist can direct FRIDA by inputting a text description, submitting other works of art to inspire its style, or uploading a photograph and asking it to paint a representation of it.

One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap — the difference between what FRIDA composes in simulation and what it paints on the canvas. FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

Beyond DALL-E-2

The robot uses AI models similar to those powering tools like OpenAI’s ChatGPT and DALL-E 2, which generate text or an image, respectively, in response to a prompt. But instead, FRIDA simulates how it would paint an image with brush strokes and uses machine learning to evaluate its progress as it works.

DALL-E 2 uses large-vision language models to produce digital images. FRIDA takes that a step further and uses its embodied robotic system to produce physical paintings. One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap, the difference between what FRIDA composes in simulation and what it paints on the canvas. So FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

The team is also experimenting with other inputs, including audio. They played ABBA’s “Dancing Queen” and asked FRIDA to paint it.

Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter

Comment on this content

0 Comments

0 thoughts on “What if robots could collaborate with artists?

Like
Dislike
Share

0

Comments
Reactions
💯 💘 😍 🎉 👏
🟨 😴 😡 🤮 💩

Here is where you pick your favorite article of the month. An article that collected the highest number of picks is dubbed "People's Choice". Our editors have their pick, and so do you. Read some of our other articles before you decide and click this button; you can only select one article every month.

People's Choice
Bookmarks