Multimodal

Modern agents take more than text: an image alongside the prompt, an audio clip, a file attachment. In Sema those are first-class message parts, built with small verbs and composed into the same inspectable Prompt as your text. The payoff is that a plain text model can still “see” and “hear” — the runtime resolves each non-text part to text through on-device models — and a natively multimodal model can take the parts directly. Same code, same seam.

This guide covers native multimodal messages and grounds them in the sdk-multimodal example, which drives real small models behind clean Sema functions.

The pieces

Part	Builder	Resolves to (text model)
Text	a plain str	itself
Image	image(path)	a caption / description
Audio	audio(path)	a transcript
File	attachment(path)	its contents
A message	message(role, parts)	grouped role + parts
A prompt	compose(messages)	an inspectable Prompt

Building a multimodal message

message(role, parts) groups a role with a list of parts — strings for text, plus image, audio, and attachment builders:

msgs = [
    message("system", ["You are a helpful assistant."]),
    message("user", ["What do you hear and see?", audio("clip.wav"), image("scene.png")]),
]
answer = generate(compose(msgs), 256)     # or the SDK's chat_mm(msgs)

compose(messages) returns a Prompt — so it is debuggable (prompt templates are the same type) — resolving every non-text modality to text through the config-registry seams: audio → a native Whisper transcript, image → a native caption, a file → its contents.

Note

This is where the framework earns its keep: a plain text model can still hear and see, because the runtime uses small on-device models to resolve modalities the language model itself was never trained on. A natively-multimodal model instead takes the parts directly at the provider boundary — the seam is the same.

Because the composed value is a Prompt, you can inspect exactly what the model will receive, including how each modality resolved:

composed = compose([sys, msg])
print(composed.debug)     # roles, resolved parts, token estimate, warnings

The ai-console example composes a full multimodal message and prints composed.debug so you can watch an image and an audio clip fold into the same prompt.

The SDK: capabilities as Sema functions

Rather than call the raw builders everywhere, sdk-multimodal exposes each capability as a clean Sema function whose backend is chosen by the config/model registry. The whole surface is ordinary Sema with declared effects:

def caption(image_path: str) -> str !{proc.run, fs.read}:
    sem "Describe an image in natural language"
    return python.call("sema_lang_sdk.vision", "caption", [image_path])

def vqa(image_path: str, question: str) -> str !{proc.run, fs.read}:
    sem "Answer a question about an image"
    return python.call("sema_lang_sdk.vision", "vqa", [image_path, question])

def ocr(image_path: str) -> str !{proc.run, fs.read}:
    sem "Extract text from an image (OCR)"
    return python.call("sema_lang_sdk.ocr", "read", [image_path])

def transcribe(audio_path: str) -> str !{proc.run, fs.read}:
    sem "Transcribe speech from an audio file to text (STT)"
    return python.call("sema_lang_sdk.stt", "transcribe", [audio_path])

def speak(text: str, out_path: str) -> str !{proc.run, fs.write}:
    sem "Synthesize speech audio from text (TTS); returns the output path"
    return python.call("sema_lang_sdk.tts", "speak", [text, out_path])

Using them is a short program — vision, OCR, and VQA on an image, then a speech round-trip:

from sdk_multimodal.ai import caption, ocr, vqa, transcribe, speak

def main() -> None !{proc.run, fs.read, fs.write, observe.record}:
    log.info("caption", text=caption("text.png"))
    log.info("ocr", text=ocr("text.png"))
    log.info("vqa", answer=vqa("text.png", "what color is the box?"))
    # Speech round-trip: TTS writes audio, STT reads it back.
    speak("the quick brown fox jumps over the lazy dog", "spoken.wav")
    log.info("stt (round-trip)", text=transcribe("spoken.wav"))

Native, on-device backends

Every text/vision/speech-in modality now runs natively on-device via candle, zero Python: text generation (GGUF), embeddings (BERT), speech-to-text (Whisper), image captioning (BLIP), OCR (TrOCR), and visual question answering (moondream). Each is a config-registry seam — set [models] <cap> in sema.toml to a Hugging Face repo id and the real-model build routes to the native backend:

[models]
stt    = "whisper-tiny"
vision = "blip"

With that config, compose turns an audio clip into a transcript and an image into a caption, both inline in the composed prompt, on-device. The one modality still on the Python bridge is TTS (no small native model in the size band); the SDK’s speak remains available for it. See Packaging & Providers for the config layer and how to swap a backend.

Unresolvable modalities never crash

If a part can’t be resolved, it composes to a labelled placeholder (visible in prompt.debug) rather than crashing the run — Sema degrades safely and always surfaces the degradation. Set SEMA_STRICT=1 to turn that into a hard error while you debug.

Run and verify

From the sema/ directory:

sema check examples/sdk-multimodal
SEMA_STRICT=1 sema run examples/sdk-multimodal

Runs for real once the model extras are installed; otherwise model-backed calls fail with a typed ModelUnavailable error — opt into [engine] deterministic = true for a hermetic, testable run.

Variations

• Speech-to-speech. Chain transcribe → generate → speak for a spoken round trip; the SDK’s voice_reply wraps exactly this pipeline.
• Override a modality backend. Register a @provides("ocr") or @provides("caption") function to swap in your own model — see Packaging & Providers.
• Inspect before sending. Read composed.debug in a test block to assert which parts resolved and how the token estimate came out.

See also

• sdk-multimodal example (generated)
• simulate & Models · Packaging & Providers
• Python Interop