back Back

What if robots could collaborate with artists?

Feb. 09, 2023.
2 mins. read. 7 Interactions

About the Writer

Amara Angelica

228.34065 MPXR

Amara Angelica, an electronics engineer and inventor, was previously Editor of Kurzweil AI, working with Ray Kurzweil on The Singularity Is Near and other works.

Carnegie Mellon University’s Robotics Institute is exploring that idea

Their FRIDA (Framework and Robotics Initiative for Developing Arts, named for Mexican painter Frida Kahlo), uses a robotic arm with a paintbrush taped to it and AI to collaborate with humans on works of art.

FRIDA’s first paintings (credit: Carnegie Mellon University/The Robotics Institute)

FRIDA is a robotic painting system, “but FRIDA is not an artist,” says Peter Schaldenbrand, a School of Computer Science Ph.D. student in the Robotics Institute, who is working with FRIDA and exploring AI and creativity. “Instead, FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA and then FRIDA can execute them.” Artists can direct FRIDA by inputting a text description, submitting other works of art to inspire its style; or by uploading a photograph and asking Frida  to paint a representation of it.

An artist can direct FRIDA by inputting a text description, submitting other works of art to inspire its style, or uploading a photograph and asking it to paint a representation of it.

One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap — the difference between what FRIDA composes in simulation and what it paints on the canvas. FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

Beyond DALL-E-2

The robot uses AI models similar to those powering tools like OpenAI’s ChatGPT and DALL-E 2, which generate text or an image, respectively, in response to a prompt. But instead, FRIDA simulates how it would paint an image with brush strokes and uses machine learning to evaluate its progress as it works.

DALL-E 2 uses large-vision language models to produce digital images. FRIDA takes that a step further and uses its embodied robotic system to produce physical paintings. One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap, the difference between what FRIDA composes in simulation and what it paints on the canvas. So FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

The team is also experimenting with other inputs, including audio. They played ABBA’s “Dancing Queen” and asked FRIDA to paint it.

Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter

Comment on this content

2 Comments

2 thoughts on “What if robots could collaborate with artists?

  1. The writer has done an excellent job capturing the innovative and creative essence of Carnegie Mellon’s FRIDA project, emphasizing the unique intersection of robotics and art. Their exploration of how technology can foster artistic expression is inspiring and thought-provoking. Thank you for such an insightful article that makes complex technology feel accessible and exciting!

    Like
    Dislike
    Share
    Reply
  2. Carnegie Mellon University’s Robotics Institute is pushing the boundaries of what's possible in robotics. Their innovative research and development are not only advancing the field but also opening up new possibilities for real-world applications. The potential impact on industries ranging from healthcare to manufacturing is immense, promising to revolutionize the way we interact with technology and automation. It’s inspiring to see such cutting-edge work being done, highlighting the future of robotics and its potential to improve our daily lives.

    Like
    Dislike
    Share
    Reply

4

Like

Dislike

Share

2

Comments
Reactions
💯 💘 😍 🎉 👏
🟨 😴 😡 🤮 💩

Here is where you pick your favorite article of the month. An article that collected the highest number of picks is dubbed "People's Choice". Our editors have their pick, and so do you. Read some of our other articles before you decide and click this button; you can only select one article every month.

People's Choice
Bookmarks