# Multimodal > Treat images, audio, and files as first-class message parts in Sema — one composable Prompt that even a text-only model can see and hear. Modern agents take more than text: an image alongside the prompt, an audio clip, a file attachment. In Sema those are **first-class message parts**, built with small verbs and composed into the same inspectable `Prompt` as your text. The payoff is that a *plain text model* can still "see" and "hear" — the runtime resolves each non-text part to text through on-device models — and a natively multimodal model can take the parts directly. Same code, same seam. This guide covers native multimodal messages and grounds them in the [`sdk-multimodal`](/reference/examples-api/sdk-multimodal/) example, which drives real small models behind clean Sema functions. ## The pieces | Part | Builder | Resolves to (text model) | |---|---|---| | Text | a plain `str` | itself | | Image | `image(path)` | a caption / description | | Audio | `audio(path)` | a transcript | | File | `attachment(path)` | its contents | | A message | `message(role, parts)` | grouped role + parts | | A prompt | `compose(messages)` | an inspectable `Prompt` | ## Building a multimodal message `message(role, parts)` groups a role with a list of parts — strings for text, plus `image`, `audio`, and `attachment` builders: ```sema msgs = [ message("system", ["You are a helpful assistant."]), message("user", ["What do you hear and see?", audio("clip.wav"), image("scene.png")]), ] answer = generate(compose(msgs), 256) # or the SDK's chat_mm(msgs) ``` `compose(messages)` returns a `Prompt` — so it is debuggable ([prompt templates](/neurosymbolic/simulate/) are the same type) — **resolving every non-text modality to text through the config-registry seams**: audio → a native Whisper transcript, image → a native caption, a file → its contents. :::note This is where the framework earns its keep: a plain text model can still hear and see, because the runtime uses small on-device models to resolve modalities the language model itself was never trained on. A natively-multimodal model instead takes the parts directly at the provider boundary — the seam is the same. ::: Because the composed value is a `Prompt`, you can inspect exactly what the model will receive, including how each modality resolved: ```sema composed = compose([sys, msg]) print(composed.debug) # roles, resolved parts, token estimate, warnings ``` The [`ai-console`](/reference/examples-api/ai-console/) example composes a full multimodal message and prints `composed.debug` so you can watch an image and an audio clip fold into the same prompt. ## The SDK: capabilities as Sema functions Rather than call the raw builders everywhere, `sdk-multimodal` exposes each capability as a clean Sema function whose backend is chosen by the config/model registry. The whole surface is ordinary Sema with declared effects: ```sema def caption(image_path: str) -> str !{proc.run, fs.read}: sem "Describe an image in natural language" return python.call("sema_lang_sdk.vision", "caption", [image_path]) def vqa(image_path: str, question: str) -> str !{proc.run, fs.read}: sem "Answer a question about an image" return python.call("sema_lang_sdk.vision", "vqa", [image_path, question]) def ocr(image_path: str) -> str !{proc.run, fs.read}: sem "Extract text from an image (OCR)" return python.call("sema_lang_sdk.ocr", "read", [image_path]) def transcribe(audio_path: str) -> str !{proc.run, fs.read}: sem "Transcribe speech from an audio file to text (STT)" return python.call("sema_lang_sdk.stt", "transcribe", [audio_path]) def speak(text: str, out_path: str) -> str !{proc.run, fs.write}: sem "Synthesize speech audio from text (TTS); returns the output path" return python.call("sema_lang_sdk.tts", "speak", [text, out_path]) ``` Using them is a short program — vision, OCR, and VQA on an image, then a speech round-trip: ```sema from sdk_multimodal.ai import caption, ocr, vqa, transcribe, speak def main() -> None !{proc.run, fs.read, fs.write, observe.record}: log.info("caption", text=caption("text.png")) log.info("ocr", text=ocr("text.png")) log.info("vqa", answer=vqa("text.png", "what color is the box?")) # Speech round-trip: TTS writes audio, STT reads it back. speak("the quick brown fox jumps over the lazy dog", "spoken.wav") log.info("stt (round-trip)", text=transcribe("spoken.wav")) ``` ## Native, on-device backends Every text/vision/speech-in modality now runs **natively on-device via candle, zero Python**: text generation (GGUF), embeddings (BERT), speech-to-text (Whisper), image captioning (BLIP), OCR (TrOCR), and visual question answering (moondream). Each is a config-registry seam — set `[models] ` in `sema.toml` to a Hugging Face repo id and the `real-model` build routes to the native backend: ```toml [models] stt = "whisper-tiny" vision = "blip" ``` With that config, `compose` turns an audio clip into a transcript and an image into a caption, both inline in the composed prompt, on-device. The one modality still on the Python bridge is TTS (no small native model in the size band); the SDK's `speak` remains available for it. See [Packaging & Providers](/guides/packaging/) for the config layer and how to swap a backend. :::caution[Unresolvable modalities never crash] If a part can't be resolved, it composes to a labelled placeholder (visible in `prompt.debug`) rather than crashing the run — Sema degrades safely and always surfaces the degradation. Set `SEMA_STRICT=1` to turn that into a hard error while you debug. ::: ## Run and verify From the `sema/` directory: ```bash sema check examples/sdk-multimodal SEMA_STRICT=1 sema run examples/sdk-multimodal ``` Runs for real once the model extras are installed; otherwise model-backed calls fail with a typed `ModelUnavailable` error — opt into `[engine] deterministic = true` for a hermetic, testable run. ## Variations - **Speech-to-speech.** Chain `transcribe` → `generate` → `speak` for a spoken round trip; the SDK's `voice_reply` wraps exactly this pipeline. - **Override a modality backend.** Register a `@provides("ocr")` or `@provides("caption")` function to swap in your own model — see [Packaging & Providers](/guides/packaging/#custom-capability-providers). - **Inspect before sending.** Read `composed.debug` in a `test` block to assert which parts resolved and how the token estimate came out. ## See also - [sdk-multimodal example (generated)](/reference/examples-api/sdk-multimodal/) - [simulate & Models](/neurosymbolic/simulate/) · [Packaging & Providers](/guides/packaging/) - [Python Interop](/guides/python-interop/)