Timing

Timing is the craft of putting things where the voice is: cutting to the next picture when the narrator moves on, dropping b-roll in right after a sentence, holding a caption until the next one starts. Because every voice in the library carries word timestamps, none of this is done by ear. This page covers the tools: scenes and clauses, tile(), manual placement, and find().

Two clocks

There are two time bases to keep apart.

File time is the position inside an audio recording. A voiceover from create_voiceover is one continuous recording of the whole script; a word 2.0 seconds into it is at file time 2000, no matter where that word ends up in the video.

Video time is the position on the timeline, what add_image(interval=...) and captions use.

The rule: segments straight from a voiceover (voiceover.scenes[i], voiceover.words) are in file time. Anything placed, the scenes returned by tile() or the result of add_voiceover, is in video time. A transcript from create_stt is in its file's time, which equals video time when that file is placed at zero.

Scenes and clauses

When a voiceover is made from a list, each item is a scene and the voiceover knows where every scene starts and ends. When it is made from a list of lists, each inner string is a clause, and the voiceover knows those boundaries too.

voiceover = create_voiceover(
    [
        ["Thales of Miletus asked what everything is made of."],
        ["His answer was water.", "He was wrong."],
        ["But the question was the beginning of science."],
    ],
    voice=Voices.BritishNarrator,
    project=project,
)

scene = voiceover.scenes[1]
scene.text          # "His answer was water. He was wrong."
scene.start         # first word's start, file time, ms
scene.end           # last word's end
scene.words         # the words, each with .text, .start, .end
scene.clauses       # two segments: "His answer was water." and "He was wrong."
scene.clauses[1].start

Scenes are the natural unit for "one picture per part of the narration". Clauses are for finer decisions inside a scene: pausing between a quote and its attribution, or switching pictures mid-scene.

tile(): the standard layout

tile() lays the whole voiceover out on the timeline in order, inserting silence between scenes and clauses, and tells you where each scene's picture should go. It is the one call most narrated videos need.

placed = voiceover.tile(scene_gap=500, clause_gap=250, lead_ms=500, tail_ms=2500)

for i, scene in enumerate(placed.scenes):
    video.add_image(images[i], interval=[scene.start, scene.end])
    video.add_voiceover(scene)
    video.add_dynamic_text(DynamicText(words=scene.words, ...))

Each returned scene is a TiledScene, in video time:

The picture intervals are designed to tile the video exactly: the first starts at 0, each one ends where the next scene's speech begins, and the last ends tail_ms after the final word. So cuts land on the onset of speech, which is where they feel right, and the pictures cover the gaps between scenes.

The gaps are not in the recording. The voiceover was read in one take with natural pauses; tile() slices it and re-spaces it on the timeline, extending each slice to the middle of the surrounding pauses so no word tail is ever cut. That is why you can change scene_gap and re-render without re-recording anything.

tile() needs every scene to speak. A silent scene (None in the script) has no onset to cut on, so a voiceover with silent scenes is placed manually.

Placing by hand

add_voiceover places one segment at a moment of your choosing and returns the placement, with timestamps converted to video time:

placed = video.add_voiceover(voiceover.scenes[0], at=1000)
placed.start, placed.end       # video time
placed.words                   # for captions

video.add_image(images[0], interval=[0, placed.end + 800])

Any segment works: a scene, a clause, or the whole voiceover. Placements from the same recording may not overlap, and a placement must end before the video does.

Mixing the two styles is fine. A common pattern is tile() for the narration and a manual add_voiceover for a scene you want somewhere unusual.

find(): locating a phrase

find searches the words for a phrase and returns every place it occurs, with timestamps. It works on any voice, generated or transcribed, and on placed scenes.

matches = speech.find("net worth just crossed one trillion dollars")

if len(matches) != 1:
    raise RuntimeError("expected exactly one match")

m = matches[0]
video.add_image(money_pile, interval=[m.end, m.end + 6000], z_index=2)

Each match:

The full signature:

find(text, *, method="fuzzy", threshold=0.7, max_matches=None) -> list[Match]

method="fuzzy", the default, compares the phrase to every run of words of about the same length and keeps the runs that score at least threshold, so a transcript that heard "trillion dollar" instead of "trillion dollars" still matches. A higher threshold is stricter. method="exact" requires the words to be identical, ignoring case and punctuation. Matches are returned in order of time, with overlapping candidates collapsed to the best one.

The timestamps are in the clock of what you searched: file time on a raw AIVoiceover, video time on a TiledScene, a PlacedVoiceover, or a transcript of footage placed at zero. Search the placed thing when you want video time.

Rules of thumb

• Cut on onsets. A picture change on the first word of a sentence reads as intentional. tile() does this for you; do the same by hand with .voice_start.
• B-roll after the phrase, not on it. Let the viewer hear the claim, then show it: place at match.end, not match.start.
• Give the ear a beat. A scene_gap of 400 to 700 ms feels like a narrator pausing; below 200 it feels rushed.
• Hold the last frame. tail_ms of two to three seconds gives the ending room to land, and space for a watermark or call to action.