Today, with the rapid development of AI video generation technology, more and more creators are beginning to explore a more warm form of content:Hand-painted animation story video。
Compared with AI videos that pursue highly realistic, the hand-drawn style is more like a moving picture book-simple lines, gradually emerging pictures, and stories with subtitles, which are more suitable for children's stories, knowledge popularization, personal creations, Short Video accounts and other scenes.
And open source projects story-to-handdrawn-video It is precisely in response to this need that a set of automated tools was born: it attempts to convert "a text story" or "a set of hand-drawn pictures" into a portrait hand-drawn animated video with a comic feel.
1. What is story-to-handdrawn-video?
story-to-handdrawn-video It is an AI video production project for creators, with the goal of lowering the threshold for hand-drawn animation production.
Traditionally, making a hand-painted story video usually requires:
- Writing story scripts;
- Split the story and mirror;
- Draw characters and scenes;
- Create frame-by-frame animations;
- Add subtitles;
- Synthetic video.
The entire process often requires animation design experience.
This project attempts to automate the process:
Story text → mirror split → hand-painted picture → dynamic display → video output
Users only need to provide story content, and the system can assist in generating hand-drawn comic animations suitable for the dissemination of Short Video.
II. Analysis of core functions
1. Automatic hand-drawn animation generation of Chinese stories
The project supports direct input of Chinese stories.
For example:
The little girl found an injured bird in the forest. She took care of it every day. Finally, the bird flew back into the sky.
The system can complete it around the story content:
- Text disassembly;
- Scenario planning;
- Picture generation;
- Animation rendering;
- Subtitle display.
The final output is similar to a "dynamic picture book" effect.
This approach is particularly suitable for:
- Children's story account;
- Bedtime Story Channel;
- Knowledge popularization video;
- AI content creator.
2. Convert static hand-drawn pictures to animation
In addition to text generation, the project also supports animation of existing picture materials.
For example, you already have:
01.jpg
02.jpg
03.jpg
04.jpg
Several consecutive comic pictures.
Tools can:
- Keep the order of the pictures;
- Add dynamic effects;
- Create a page-turning feeling;
- Generate video.
This is very valuable to comic writers and picture book designers.
3. Hand-painted reveals the effect
A major feature of the project is the simulation of the "painting process".
The video does not directly show the complete picture, but rather:
Phase 1:
文字出现
↓
黑白线稿逐渐绘制
↓
颜色慢慢填充
↓
完整插画呈现
This effect is similar to Whiteboard Animation and can enhance the sense of expectation during viewing.
3. Technical realization ideas
From a technical perspective, such systems are actually a multi-stage AI content pipeline.
The overall structure is roughly as follows:
用户故事
|
↓
文本理解
|
↓
故事分镜
|
↓
AI图片生成
|
↓
手绘风格处理
|
↓
动画合成
|
↓
MP4视频输出
It involves multiple technical modules:
1. Story understanding and lens generation
First, you need to understand the text:
For example:
Original story:
The cat waits for its owner to come home every day.
AI will break down into:
Lens 1:
- Scene: Room
- Main character: Kitten
- Action: Sit by the window and wait
Lens 2:
- Scene: Dusk Street
- protagonist: master
- Action: Open the door
Only in this way can a continuous picture be formed.
2. picture generation
Generate corresponding illustrations based on the split lenses.
It is usually necessary to maintain:
- Consistent roles;
- Unified scenes;
- Stable style.
Otherwise, the characters may change in every frame.
3. animation rendering
After the still pictures are processed:
- Add lens movement;
- Add drawing animations;
- Add transition;
- Add subtitles.
Eventually a video was formed.
A similar direction of "sketch-to-dynamic video" has also been a popular direction in AI video research in recent years.
4. Project operating environment
According to the project description, running this tool usually requires:
- Node.js;
- Python;
- FFmpeg;
- npm;
- Browser rendering environment.
The installation process is roughly:
git clone https://github.com/gnipbao/story-to-handdrawn-video.git
cd story-to-handdrawn-video
npm ci
Then perform project inspections:
npm run check
You can verify whether the environment is normal.
5. Who is suitable for use?
1. Short Video creator
For example:
- Xiaohongshu story account;
- Douyin knowledge account;
- Creator of YouTube Shorts.
In the past, a professional animation team was needed, but now basic content can be automatically generated through AI.
2. education workers
Teachers can put:
- Historical stories;
- Popular science knowledge;
- Classroom cases;
Convert it into a more understandable video.
For example:
"Three Minutes to Understand the Solar System"
Can become:
Hand-painted sun → planet appearance → subtitle explanation.
3. Picture book author
For picture book authors:
Existing works:
绘本图片
↓
自动动画化
↓
电子绘本视频
New communication channels can be expanded.
6. Advantages and disadvantages
advantages
Lower the threshold for animation production
Ordinary users do not need to learn:
- After Effects;
- Blender;
- Animate。
You can create simple animations.
Keep the hand-painted temperature
Compared to fully AI-generated videos:
Hand-painted styles include:
- Humanities;
- Sense of story;
- Affinity.
Open source extensible
Developers can, based on their needs:
- Modify animation effects;
- Connect to your own model;
- Add dubbing module.
insufficient
Role consistency remains a challenge
One of the biggest problems in AI image generation:
The same character may change in different shots.
Difficulties in making long stories
Short stories work better.
If you produce:
30-minute animation,
Still need to:
- Manual review;
- Lens adjustment;
- Post editing.
Limited audio capabilities
The current focus is on screen generation.
The full video also requires:
- AI dubbing;
- music;
- Sound effects.
7. Future development direction
Similar tools may be further developed in the future:
1. Generate a complete animation in one sentence
For example:
Enter:
"Make a short story about environmental protection."
Automatic generation:
- script;
- role;
- Splitting mirror;
- Dubbing;
- Video.
2. Personalized role
User uploads:
His own comic book character.
AI automatic hold:
- appearance;
- character;
- Painting style.
3. AI animation director
Future AI will not just generate pictures, but will be responsible for:
- Plot design;
- Camera language;
- rhythm control;
- Emotional expression.
summary
story-to-handdrawn-video Represents a new way of content production:
Let the story no longer stay in words, but automatically turn into a vibrant hand-drawn animation.
It is not a replacement for professional animators, but a lowering of the threshold for creation and allowing more ordinary people to transform their stories, knowledge and ideas into visual works.
As AI image and video generation technology continues to mature, the creation process of "text → animation → video" may become an important way to produce personal content in the future.
For those who want to enter the field of AI content creation, such open source projects are worthy of attention.