event
Synthux Hackathon 2024

Cadence

A magic book that connects sound, motion, and language to explore meaning & movement.
HTML/CSS/JS
Camera feed
Online data
Touch sensor
Electret microphone
Text to Speech
prompt

Coaches & collaborators

Jason Freeman
institute
Georgia Institute of Technology
The orator opens the book and begins to speak. The words appear on the page just as they were spoken. But lo, what's this? By simply touching the page, the words may be rearranged, removed, redoubled, revised. And as the words are changed, so too are the sounds and motions of the orator speaking. All three are linked: sound, motion, and language. Ordinarily, editing audio & video affects the words, but here the connection is reversed: editing the words affects the audio & video. In this way, the recording of speech may be dissected, deconstructed, discombobulated and turned into something new.
No items found.
the tech

How does it work?

Cadence is primarily implemented as a web application, consisting of a Python server and TypeScript client. The server runs whisper.cpp (a port of OpenAI's Whisper) to transcribe audio to text & gentle (a forced aligner) to precisely align the text with the recording, while the client that enables the user to splice and sequence the video by manipulating the words and spaces in the transcript, using the Web Audio API for scheduling and playback. To give Cadence a fitting physical form, we turned a hardback book into an enclosure for electronics, with Android devices playing back audio & video (on the left) and enabling the user to drag words in the transcript (on the right), along with an ESP32 to collect sensor data from potentiometers and buttons for parameter control.
Open source code
Click here to grab this project's code
HTML/CSS/JS
Camera feed
Online data
Captures data in real time from web APIs
Touch sensor
A touch capacitance sensor can detect anything that is conductive
Electret microphone
Record or detect audio gain (volume) changes
Text to Speech
About the instrument
we're using cookies to run our site. privacy policy.