How I made a talking avatar of myself that speaks Darija

A talking avatar is a picture that moves its mouth, eyes and hands in time with a voice. Mine starts from one drawing of me. Google's Gemini draws 12 face variants and 4 hand gestures from it, a Gemini voice reads the script in Moroccan Darija, and the free Rhubarb Lip Sync tool picks the right mouth for every sound. Chrome and ffmpeg turn it all into a video.

So the presenter in my Reels isn't really me. The face is a drawing, and the voice isn't mine either. Twelve videos on my YouTube channel, uploaded between 27 September and 13 October 2026, were made this way. Here's how the talking avatar works, the mistakes I heard along the way, and what it costs.

What is a talking avatar?#

It's a cartoon trick: you don't animate a mouth, you swap between a few drawn mouths fast enough that the eye believes it. A talking avatar does the same with a portrait. A program listens to the voice, decides which mouth shape fits each moment, and shows that picture.

Online tools sell this as a service: upload a photo, type a text, get a video. I haven't tested them. Mine is a script that runs on my own computer, inside the tool I use for all my explainer videos, and it's built around one constraint you'll see below: the voice has to speak Darija.

One drawing becomes 12 faces and 4 gestures#

Everything starts from the illustration of me on the home page of this site. It's a drawing, not a photo. I send it to Gemini's image model with a strict instruction: change only what I ask, keep the style, the glasses, the beard and the framing exactly the same.

Then I ask for one change at a time:

• 8 mouth shapes: lips pressed (for m, b, p), slightly open, open on "a", wide, "ee", "oh", "oo", and teeth on the lower lip (for f, v).
• 4 face extras: a smile, a blink, a half blink, and raised eyebrows.
• 4 hand gestures: explaining, open hands, pointing up, and counting on fingers.

The trick that makes it hold together: the video never shows a whole generated face. It takes only the mouth area or the eye area from each variant and lays it over the original drawing, so nothing else can drift. For the gestures, the original head is pasted back on each new body, so the mouths still line up.

Which AI voice speaks Moroccan Darija?#

None, it turns out. On 20 October 2026 I listed every voice in Google's Gemini voice library through its API: 2,089 voices. 121 of them are Arabic: 29 in Modern Standard Arabic and 92 in Egyptian Arabic. Not one is Moroccan.

So I take one fixed voice, Orus (Google's text-to-speech docs describe it as "Firm"), from the Gemini 3.8 Flash TTS model, and tell it how to speak. The direction doesn't go inside the text: with these models, an instruction written in the text gets read aloud, and early clips came out about three times too long. It goes in a separate field the docs call a speech_metadata annotation. This is the one in my scripts:

voice: "Orus",
style: "Moroccan Darija as spoken in Marrakech, by a young Moroccan web developer… Moroccan accent only: not Egyptian, not Gulf, not Levantine, not Modern Standard Arabic.",

The words matter even more than the direction. Text that looks like Standard Arabic gets read like the news. So the scripts are written for the ear: the French words Moroccans actually use (السيرفور، البلاتفورم), ث written as ت the way it's said, ك before verbs. A small checker flags Standard Arabic, Egyptian and Gulf words before any voice is made.

And why not my own voice? Gemini can copy a real voice, but it asks for a recorded consent sentence first. I tried a shortcut without one, and the copies came back as other people, often a woman's voice (200–290 Hz, where mine sits around 118 Hz). Until I record that consent properly, the voice in my videos is Orus with a Marrakech accent in words.

How does lip sync work?#

The mouth is the job of Rhubarb Lip Sync, a free, open-source command-line tool under the MIT license. It listens to the recording and returns a list of mouth cues: which shape, at which moment. It has a phonetic mode that doesn't try to understand English words, only sounds, and its docs say to use it "if your recordings are not in English". That's what makes it work on Darija.

The shapes themselves have a fun history. From Rhubarb's README:

These six mouth shapes were invented at the Hanna-Barbera studios for shows such as Scooby-Doo and The Flintstones.

Each Rhubarb shape points to one of my faces:

A mouth alone still looks like a passport photo that learned to talk. So the face also lives on its own: a blink every 2 to 6 seconds at random, a small nod on stressed syllables, eyebrows up on the strongest beats, and a hand gesture held at least 3.5 seconds so the arms don't flap around.

Every frame of the talking avatar is a web page#

Here's the part that surprises developers. The video is a web page. One HTML file draws the cards, the captions, the logos and the talking avatar for any given moment. Chrome takes a picture of that page 30 times per second of video, and ffmpeg joins the pictures and the voice into an MP4: 1080×1920 for Reels and Shorts, 1920×1080 for YouTube.

It's not fast: about 3 to 4 minutes of rendering for 2 minutes of video, on my own computer. But it means every card in the video is just HTML and CSS, the same things I build websites with.

What went wrong, and the checks I added#

• "Matawa". The Darija word for developer, مطور, written without vowel marks, came out as "matawa". With the marks (مُطَوِّرْ), it's right.
• "Easy". CCL written as سيسيإل came out as "سي سهل", sahl, "easy". Spelled letter by letter with vowel marks (سِي سِي إِلْ), it's fine.
• A second listener. Now a second Gemini model listens to every line and compares it with the script. A line that doesn't match is made again, up to 3 tries, then flagged for me to hear.
• The AI judge. I once asked an AI to rate how Moroccan each voice sounded. It gave every voice 10/10. Since then, the only judge is my ear.
• Numbers. Before any voice is made, the tool reads every number on screen and in the subtitles and stops if it isn't in the fact sheet for that video.

How much does a talking avatar cost to run?#

Very little, by my own notes from September 2026:

The real cost is the time it took to build and tune the tool, and the time I still spend listening to every line.

Is a talking avatar allowed on YouTube?#

Yes, if you say so. YouTube's policy requires creators to disclose realistic AI content, for example content that makes a real person appear to say something they didn't. You do it with the "AI use" setting at upload; for animated, non-photorealistic content, the label may appear in the expanded description.

My avatar is a drawing, but it's a drawing of a real person, with an AI voice. So my upload tool turns the AI label on, and the Reel about this method opens with the plainest disclosure I could think of: this video isn't me.

Could a talking avatar work for your courses?#

The same steps work for any script: a lesson, a product tour, a welcome video for new students. If you're planning an e-learning platform with video lessons in Darija, or you want a tool like this inside your own platform, tell me about it. For the other side of AI and Darija, speech recognition instead of speech, see the Darija AI models from Morocco and Mistral AI.