Carnegie Mellon University’s Robotics Institute is exploring that idea

Their FRIDA (Framework and Robotics Initiative for Developing Arts, named for Mexican painter Frida Kahlo), uses a robotic arm with a paintbrush taped to it and AI to collaborate with humans on works of art.

FRIDA is a robotic painting system, “but FRIDA is not an artist,” says Peter Schaldenbrand, a School of Computer Science Ph.D. student in the Robotics Institute, who is working with FRIDA and exploring AI and creativity. “Instead, FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA and then FRIDA can execute them.” Artists can direct FRIDA by inputting a text description, submitting other works of art to inspire its style; or by uploading a photograph and asking Frida  to paint a representation of it.

An artist can direct FRIDA by inputting a text description, submitting other works of art to inspire its style, or uploading a photograph and asking it to paint a representation of it.

One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap — the difference between what FRIDA composes in simulation and what it paints on the canvas. FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

Beyond DALL-E-2

The robot uses AI models similar to those powering tools like OpenAI’s ChatGPT and DALL-E 2, which generate text or an image, respectively, in response to a prompt. But instead, FRIDA simulates how it would paint an image with brush strokes and uses machine learning to evaluate its progress as it works.

DALL-E 2 uses large-vision language models to produce digital images. FRIDA takes that a step further and uses its embodied robotic system to produce physical paintings. One of the biggest technical challenges in producing a physical image is reducing the simulation-to-real gap, the difference between what FRIDA composes in simulation and what it paints on the canvas. So FRIDA uses an idea known as real2sim2real. The robot’s actual brush strokes are used to train the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

The team is also experimenting with other inputs, including audio. They played ABBA’s “Dancing Queen” and asked FRIDA to paint it.

Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter.