Built at JEVATHON (w/ The AI Collective)

Beethoven

A live band you conduct with paintings: arrange artworks on a musical staff, and every bar an AI conductor (TypeSafe Jev) decides how the band plays while Google's Lyria RealTime generates the music. # Painting Band 🎨🎶 **Arrange paintings on a staff. A playhead sweeps across. Every bar, an AI conductor decides how the band plays, and an AI music model plays it live.** Nothing is pre-recorded. The music is generated live from what's on the canvas. ## How it works 1. **See:** Gemini looks at each painting and writes how it should sound: mood words, sound ideas (e.g. "cascading koto arpeggios"), instruments, and the painting's musical tradition (Japanese, Hindustani, Western…). Upload your own image and it's described in about 2–3 s. 2. **Arrange:** Drag paintings onto the staff. - **Left → right** = when a painting plays - **Height** = register (high or low) - **Size** = how strongly its mood and sounds lead 3. **Conduct:** Every bar (~2.7 s), **TypeSafe Jev** answers up to 12 typed questions in parallel in ~150–300 ms: - how loud each painting should be, and the mood and energy - drums, bass and the lead instrument - two extra sounds and the transition - which culture leads when traditions mix Jev never plays notes; it only decides. 4. **Play:** Jev's choices become weighted text prompts plus density, brightness and mute settings for **Google Lyria RealTime**, which streams AI-generated 48 kHz audio and blends toward each new decision. The music changes when you move, resize, add or remove a painting. ## Architecture ``` Canvas → scene as words → Jev (decides) → translator → Lyria RealTime (plays) ↘ Strudel (backup engine) Images → Gemini vision → mood, sound ideas, instruments, culture ``` - **Deciders never make sound; engines never make decisions.** Code sits in between. - **Deadline-safe:** if Jev is later than 150 ms before the next bar, the band reuses the last bar's choices, so the music never stops. - **Always playing:** if Lyria stalls, the app switches to **Strudel** (a browser-based live-coding music engine) mid-song, and you can switch back with one click. - **Explainable:** the `</>` panel shows each bar's questions and Jev's odds for every option, the exact prompts sent to Lyria, what Gemini saw in each painting, and a live log of every model call. ## Features - Drag, resize and delete paintings; draggable playhead (Home = start) - Silence where there's no painting under the playhead - Upload any image; Gemini describes it (with automatic model fallback when busy) - Cultural awareness: each painting's musical tradition shapes the sound, and Jev picks which tradition leads - Jev vs. Rules toggle (keys J/R) to hear the difference judgement makes - Live stats: Jev latency, missed bars, and cost (~$0.00001 per bar) ## Tech stack - **Vite + React + TypeScript** - **TypeSafe Jev:** per-bar decisions (`/v1/systemone`, typed choice answers with confidence) - **Google Gemini:** image understanding (structured JSON output) - **Google Lyria RealTime:** streaming music generation (`@google/genai`) - **Strudel:** fallback engine (`@strudel/web`, General MIDI soundfonts) - API keys stay server-side behind a Vite proxy (except Lyria, which streams from the browser in this local demo) ## Run it ```bash npm install cp .env.example .env.local # add JEV_KEY, JEV_BASE_URL, VITE_GEMINI_KEY npm run dev ``` Open http://localhost:5173 and drag a painting onto the staff. ## Paintings Seven public-domain works from Wikimedia Commons: Hokusai, Monet, Van Gogh, Bruegel, Seurat, Rembrandt and Raja Ravi Varma.