Creating speech-to-text

Speech-to-text is the mirror image of a voiceover. A voiceover starts from text and gives you audio plus word timestamps; speech-to-text starts from audio you already have, someone talking on camera, a recorded interview, a podcast, and gives you the words plus the timestamp of each one. The result is used the same way: captions, and timing other things to what is said.

Transcribing

from imaginaryarts import VideoFile, create_stt

footage = VideoFile(ROOT / "assets/interview.mp4")
speech = create_stt(footage.audio, project=project)

create_stt takes an AudioFile, and VideoFile.audio is the easiest way to get one from footage. It returns a Transcript.

Transcription is done with ElevenLabs Scribe, which detects the language automatically and returns word-level timestamps. Like every generated thing, the transcript is cached: the same audio is never transcribed twice, and trimming the audio differently produces a new transcript for that window.

Captions from footage

The most common use. The transcript's words are in the audio file's own time, and since the footage is placed at the start of the video, that is also video time:

video.add_video(footage, at=0)
video.add_dynamic_text(DynamicText(
    words=speech.words,
    text_style=TextStyle(font=Fonts.Barlow, size=60, color=Color(system="white"), bold=True),
    flow=DynamicTextFlow.SentenceBySentence(max_lines=2),
    position=Position.BOTTOM,
))

If the footage is placed somewhere other than zero, the words need the same offset. shifted returns a copy of the transcript with every timestamp moved:

video.add_video(footage, at=3000)
video.add_dynamic_text(DynamicText(words=speech.shifted(3000).words, ...))

And if you place a trimmed piece of the footage, transcribe that piece, so the words match what is on screen:

piece = footage.trim([5000, 15000])
speech = create_stt(piece.audio, project=project)   # 10 s of words, starting at 0
video.add_video(piece, at=0)

Finding moments in the speech

A transcript knows when everything was said, which makes it a map of the footage. find looks up a phrase and returns where it was spoken, so b-roll can be placed exactly after a sentence without anyone scrubbing through the file:

matches = speech.find("crossed one trillion dollars")
moment = matches[0]
video.add_image(b_roll, interval=[moment.end, moment.end + 6000], z_index=2)

Matching is fuzzy by default, so it tolerates the small differences between what you type and what the transcriber heard. The full behaviour is described in Timing.

Scenes and clauses in a transcript

A voiceover generated from a list of scenes knows where each scene starts and ends. A transcript can know that too, if you tell it what was said. Pass the same kind of script you would give create_voiceover, and the words are aligned to it:

speech = create_stt(
    footage.audio,
    project=project,
    script=[
        "Most people quit right before it starts working.",
        "I know, because I did, three times.",
    ],
)
speech.scenes[1].start     # when the second sentence begins, in ms

The script must be what the audio actually says; the alignment walks the two side by side and warns if the word counts drift apart.

One interface, two sources

Transcript and the AIVoiceover returned by create_voiceover are both a TimestampedVoice: audio plus timestamped words. Everything that consumes one accepts the other, so captions and find do not care where the words came from.

The full signature

create_stt(
    audio,                 # an AudioFile, or a path to one
    *,
    project,
    script=None,           # str | list[str] | list[list[str]] to get scenes and clauses
    language=None,         # a language code to skip auto-detection, e.g. "en"
    name=None,             # a label for the cached files
    take=1,                # bump to transcribe again
) -> Transcript