FRIDA the robot collaborates with humans to create art using artificial intelligence (AI). This robotic arm with a paintbrush is a project by the Framework and Robotics Initiative for Developing Arts (FRIDA) led by Peter Schaldenbrand, with faculty members Jean Oh and Jim McCann. The aim of the project is to explore the intersection of human and robotic creativity, using AI models similar to those powering tools like OpenAI’s ChatGPT and DALL-E 2 to generate text or image in response to a prompt.
FRIDA is not an artist, but a robotic painting system that an artist could collaborate with to specify high-level goals for the painting, and then FRIDA would execute them. Users can direct FRIDA by inputting a text description, submitting other works of art to inspire its style, or uploading a photograph and asking it to paint a representation of it. The team is also experimenting with other inputs, including audio.
FRIDA or Framework and Robotics Initiative for Developing Arts, named after Frida Kahlo, is a project exploring the intersection of human and robot creativity, according to Jim McCann, assistant professor at Carnegie Mellon’s Robotics Institute pic.twitter.com/bKQwTlEp0B
— Reuters (@Reuters) February 25, 2023
How does FRIDA work?
FRIDA simulates how it would paint an image with brush strokes, and its final products are impressionistic and whimsical. FRIDA’s brushstrokes are bold and lack the precision sought so often in robotic endeavors. If FRIDA makes a mistake, it riffs on it, incorporating the errant splotch of paint into the end result.
FRIDA taps into AI and machine learning several times during its artistic process. It spends an hour or more learning how to use its paintbrush and uses large vision-language models trained on massive datasets that pair text and images scraped from the internet to understand the input. FRIDA takes a step further than other image-generating tools and uses its embodied robotic system to produce physical paintings, and its biggest technical challenge is reducing the simulation-to-real gap.
What is next for FRIDA?
FRIDA’s team seeks to address some of the limitations in current large vision-language models by continually refining the ones they use. The team fed the models the headlines from news articles to give it a sense of what was happening in the world and further trained them on images and text more representative of diverse cultures to avoid an American or Western bias.
The project’s main goal is to promote human creativity through FRIDA, and not to take artists’ jobs. It is an opportunity for artists to collaborate with the robot to express their ideas in painting. FRIDA is part of the Bot Intelligence Group (BIG) lab in the Squirrel Hill neighborhood of Pittsburgh.
Image credit: CMU
