When I saw the reference video, my first reaction was: We should be able to copy this kind of video. The story is not important, just take one of us now, mainly depends on the effect. Now that the picture is there, the story is there, we can synthesize the speech. My feeling at that time was... everything is ready, let's make one first.
How does the story map work?
The story is very simple: When the child was a child, he always opened his hands to hug him, but his mother laughed at him while hugging him. Later, one day when crossing the road, the child said for the first time,"I'll walk by myself." His mother's hand stopped in mid-air and suddenly couldn't remember the last time she hugged him. That night, she picked up the sleeping child again.
We broke it down into 6 pictures:
-
The child opened his hands and ran to his mother.
-
His mother picked him up and said,"It weighs too much."
-
When crossing the road, the child said "I'll walk by myself" for the first time.
-
The mother's hand hangs in the air, and the child's back is getting farther and farther away.
-
At night, the mother picked up the sleeping child again.
-
The two backs of one big and one small walked away, using them to contain the sentence "never give advance notice the last time".
Each of the six pictures is useful. The first two pictures make people believe that "he always used to hug", the third one interrupts daily life, the fourth one makes my mother react, the fifth one shows her movements, and the last one is qualified to finish. If you miss any one in the middle, your emotions will become stubborn.
Then fix the characters. We first prepared a reference picture of our mother and a reference picture of our child: my mother has black shoulder-length short hair and a pink and purple sweater with bean paste; my child is five years old, with black hair, round face, missing a front tooth, and wearing a mustard yellow T-shirt. For all six pictures, these two character reference pictures were handed over to the picture generation tool together, and the clothes were not changed during the entire process.
Don't just guess the scene based on words. Prepare an environmental reference map for the living room, street corner and night bedroom. When making a living room picture, bring the living room reference, and when making a zebra crossing, bring the street reference. The background doesn't have to be exactly the same, but don't suddenly change the decoration in different pictures for the same home or the same street.
The pictures use Codex's built-in picture generation function. Make one picture separately for each scene, 3:4 vertical picture, and do not generate text in the picture. The common reminder is roughly like this:
同一位妈妈：黑色齐肩外翘短发，鹅蛋脸尖下巴，细长脖子，
豆沙粉紫圆领毛衣。
同一个五岁男孩：头大身小，黑色乱发，圆脸短脖，
缺一颗门牙，芥末黄 T 恤。全程不换衣服。
画风：1950 年代美式中古动画概念稿，传统水粉和彩铅，
画在白纸上，大面积留白，环境像舞台布景，不像照片。
构图：3:4 竖版，顶部至少 35% 留白，四个角保持白色，
不要日系动漫脸、3D、矢量插画、满构图、文字、水印、
额外人物和服装变化。Then each picture only fills in the actions and environment that are really needed for this game. For example, sheet 4:
同一街口。妈妈豆沙粉紫袖口和悬空的手占前景，
手指微微收拢；远处是儿子穿芥末黄 T 恤走远的小背影。
画面保留大片空白，色调比上一幕稍冷，情绪克制。You don't need to write too much literature in the prompt words, first fix the things that will change: the same person, the same piece of clothing, the same style of painting, and the same type of blank space. Only make one picture for a scene, and don't use one picture to dominate two plots.
If you do it from scratch, you must get at least 6 qualified illustrations. How many times it will actually be generated depends on whether the character has changed his face, whether the hands are broken, and whether the composition can leave a place for text. The production document at that time only left the six final drawings, and did not completely save each retry and deduction, so I did not compile a "How much money was spent in total".
After the six pictures were prepared, the original short film was also very simple: each picture was 4.25 seconds, first typed in Chinese word for word on the white paper, then let the gray version of the same picture appear from left to right, and finally let the color pictures cover it from left to right. Switch directly between scenes to prevent the characters from moving. The six pictures add up to 25.5 seconds, with background music and typing sound, and no narration.
At this point, the original short static story is finished.
Layout of videos
Let's first analyze the video: The layout of each game is basically unchanged: one sentence at the top, one picture in the middle, and Chinese and English at the bottom; the voice is only read in English. The picture first changes from blank to gray, and then scans from left to right to color.
It looks like the picture was generated on the spot, but in fact, it doesn't use a video model. Copy two layers of the same picture, remove the color from the lower layer, keep the color image from the upper layer, and mask the two layers from left to right once, that is, gradually release the visible area. Also explain the tools I use:
-
Chrome is used to view original X posts and reference videos.
-
macOS comes with say that reads English words into speech.
-
Remotion is a tool that uses React to layout and generate videos, responsible for pictures, text, masks and Timeline.
-
FFmpeg is a command-line tool for processing video and audio, responsible for audio format conversion and final mixing of background music; its own ffprobe is used to check resolution, frame rate, duration and audio track.
-
MLX Whisper is a local speech recognition model. After the film is ready, let it dictation out English again and check for any missing words or truncation.
The environment in which the scripts are running is Node.js 22.22.1, and Remotion 4.0.484 and FFmpeg 8.1.2 are used for the video section. There are many tools named, but in fact, the picture belongs to Remotion and the sound belongs to FFmpeg. Other tools are mainly responsible for inspection.
The animation parameters for each game are finally set: blank for 0.2 seconds first, scan the gray image in 1.45 seconds, and scan the color image in another 1.2 seconds. The edges of the mask are softened by 10% and the image is slowly zoomed in from 1x to 1.022x. The amplitude is very small, but if you don't move, 6 seconds really look like a PowerPoint.
typescript
const sketchProgress = interpolate(t, [0.2, 1.65], [0, 1]);
const colorProgress = interpolate(t, [1.65, 2.85], [0, 1]);
show(grayscale(image), leftToRightMask(sketchProgress));
show(image, leftToRightMask(colorProgress));The gray layer and the color layer must use the same picture, the same clipping position, and the same zoom value. Don't generate a "more like a line draft" and place it below. As long as the two pictures have a slightly different facial features, their faces will jump when the color pictures scan in, which is particularly obvious.
I made a separate list of scenes. Only record the picture name, Chinese title, Chinese text, English oral broadcast and the top and bottom offset position of the picture for each game:
json
{
"image": "03-i-walk.png",
"title": "后来，他说我自己走",
"chinese": "那天过马路，他第一次说：「我自己走。」",
"english": "Then one day, at the crosswalk, he said it for the first time. I can walk by myself.",
"imagePosition": "center 64%"
}
Finally, don't save this last location parameter. The characters in each picture stand in a different position, and they are all mechanically placed in the center, either cutting off their heads or cutting off their hands.
How to connect English sounds and subtitles
The English oral broadcast first uses the Samantha voice that comes with macOS, and the speed is 165 English words per minute, with a separate paragraph generated for each session. This process is called text-to-speech and is often abbreviated to TTS.
If the machine configuration is high, use CosyVoice The effect of synthesizing speech will be better
bash
say -v Samantha -r 165 -o scene.aiff "English narration here"
ffmpeg -i scene.aiff -ar 48000 -ac 1 scene.wavDon't read six games into one big document at a time. If there was a problem with the pronunciation of the third game, only the third game was changed, and there was no need to redo the first two games.
After the speech is generated, the real duration of each paragraph is read. Take the greater of "at least 6.4 seconds" and "voice-over duration plus 0.55 seconds" for each game. The short one stops for at least 6.4 seconds, and the long one stops for half a second more when waiting for the person to finish reading. Don't guess the duration by how many letters there are in English.
There is no additional word-by-word time code for English subtitles. I used "the time that has been played ÷ the total duration of this speech" to calculate how many words should be exposed now. Each paragraph only has 18-20 words, which is enough for subtitles of this kind of story. If you want to do serious English teaching and require your voice to accurately point to every word you read, you still need to time-stamp each word.
Chinese is only displayed and will not be read aloud. Read it in English and Chinese, and the 39.6 seconds will quickly drag on to one minute, and the audience will be tired of waiting.
The picture and Mr. Voice are in the same version, and the background music is mixed at the end. The music volume is set to 0.08 first. When the human voice appears, the music is automatically lowered. Finally, the maximum peak volume is limited to avoid sudden bursts. The original specific parameters are threshold 0.02, ratio 9, attack 18ms, release 300ms, and the peak value does not exceed 0.95. They are only suitable for this piece and this music. If you change the song, you have to listen to it again. Don't use it as an ancestral secret recipe.
There is another place where it is easy to feel good about yourself: Whisper can give a complete dictation, which only means that the machine understands it, but does not mean that the sound is natural. There is no problem with Samantha's version of the verification method. Before the real text, I still have to listen to it completely on my mobile phone. Whether you are tired or not, whether you look like a stop announcement is up to the human ear.
summary
First, the reference map of the characters should be separated. It's usually better to write "the same mother" a hundred times in the prompt, but it's usually better to bring the same person reference picture with it every time.
Second, one picture per game. Don't take a picture and say "The child refuses to hug" and "Mom holds him again at night" at the same time. The audience will not know which sentence to read.
Third, grayscale color maps are not real manuscripts. The background of the original picture is too complex, and it will still be dirty after removing the color. Changing to a cleaner original picture is more useful than frantically pulling contrast.
Fourth, don't let the video model redo the entire piece again. This form requires the stability of characters, composition and words, and there is no content that must be explained clearly by continuous actions. The video model changes one face, hands or clothes, and reworks more.
Fifth, background music is added last. Don't rush to match music until the screen and audio broadcast are decided, otherwise you won't know later whether the story is slow, the sound is slow, or the music is stealing the voice.
Sixth, only the final MP4 will be recognized. I handed the piece back to local speech recognition, and matched all six English paragraphs; I automatically checked the canvas, frame rate, coding and audio track again. The preview in the editor is normal, but that does not mean that the exported file must be normal.
I'm not going to turn it into a new assembly line this time. I only verified one thing by taking this piece into pieces: I already have a set of story illustrations with stable characters and clean composition in my hand. By changing the format and Timeline, I can create this bilingual hand-drawn effect.
If I really want to continue later, I will change to a more natural English voice (CosyVoice) first, send it out to look at the data, and then decide whether it is worth doing more.