Exploration : ComfyUI & TouchDesigner: collaborating with AI in real time
In this exploration, I experiment with a visual dialogue between my hands and artificial intelligence.
I film from above as I manipulate children’s modeling clay on a table. Using a workflow that combines ComfyUI and TouchDesigner, each frame is interpreted in near real time, about one image per second on a single RTX 4070 Ti, sometimes slower if the computational load is higher. What I roughly sculpt becomes the base for a visual proposal, generated from text prompts.
The system doesn’t replace the gesture, it responds to it. The spatial arrangement of elements in the composition is fully taken into account. With certain settings, the forms act like containers ready to receive generated content. With others, I retain more control, allowing the AI to simply clean up or polish what I present as a rough idea.
I guide the process with simple prompts like funny character or fruits character, while using a negative prompt to block unwanted results. Let’s just say plasticine and adult content are not a good mix.
The AI becomes a creative partner, a playful mirror reacting to my gestures with unexpected suggestions.
Plasticine is only a starting point. This protocol also works with quick sketches, various materials… even hands. By adjusting the transformation strength, I can strike a balance between intention and surprise.
I see many possibilities here, especially as a new and stimulating way to brainstorm, alone or in a group. You shape, you suggest (voice commands are my next step), and you observe how the AI interprets your gestures. Somewhere between tinkering and co-creation, this approach opens up playful visual exchanges that can enrich any creative process.
Everything the camera sees becomes material for interpretation.
In this process, nothing escapes the camera’s eye. Every element it captures, like tools, textures, scraps, shadows, or nearby objects — becomes interpretable. The AI doesn’t distinguish between what’s intentional and what’s not. It transforms, reinvents, and reimagines. When my hands appear while manipulating the clay, they can
In this process, nothing escapes the camera’s eye. Every element it captures, like tools, textures, scraps, shadows, or nearby objects — becomes interpretable. The AI doesn’t distinguish between what’s intentional and what’s not. It transforms, reinvents, and reimagines. When my hands appear while manipulating the clay, they can
interfere with the AI’s understanding of the shape. Sometimes, their outlines blend with the object, producing strange or unexpected results. A finger becomes a beak; a palm turns into a face. These visual “accidents” are not mistakes, they open up new creative paths, where ambiguity becomes a playground.
Each frame is interpreted in near real time, about one image / second on my single RTX 4070 Ti - This video are 500% up speed