AI Video Generation Pack

What this pack is for

Generating AI video is no longer one model. It is a pipeline: pick a model, write a prompt, condition on a keyframe, control camera or motion, upscale, then cut. This pack assembles the ten picks that cover each stage — opinionated, not exhaustive — so you can move from a blank prompt to a watchable clip without stitching docs from ten different repos.

The stack is split deliberately: open-source models for local control and zero per-second cost, plus a commercial-API bridge for the moments when you genuinely need Sora-class quality and are willing to pay for it. Most working pipelines end up with both.

Install in this order

1. Choose a model

• CogVideo (#2458) — text-to-video and image-to-video. Mature open-source baseline from THUDM. Runs on a single high-VRAM GPU. Start here if you want a reproducible local pipeline.
• Open-Sora (#109) — the open-source effort to replicate Sora-style results. More capable than CogVideo on long shots and motion coherence, heavier on hardware.
• Diffusers (#111) — the Hugging Face hub that loads CogVideo, Stable Video Diffusion, HunyuanVideo, Wan, Mochi, and every new model the week it ships. If you want a single Python interface instead of one repo per model, install Diffusers first and treat the rest as weights.
• Together AI Video Generation Skill (#777) — the commercial-API bridge. Use this when you need Sora / Veo / Runway / Pika output quality and don't want to manage GPU infra. Pay per second, ship the same day.

2. Write the prompt

• Same rules as image generation, plus motion verbs: "dolly forward", "orbit left", "static lock-off". Models trained on cinematic data respond to film vocabulary.
• Keep the subject in the first 12 tokens. Most text-to-video models still front-load attention.

3. Condition on a keyframe

• ControlNet (#4664) — feed a pose / depth / canny image to lock down composition. Use it when you have a specific framing in mind and don't want the model to reinvent the shot.
• For image-to-video runs, the input image is the keyframe — no ControlNet needed.

4. Add motion

• AnimateDiff (#2463) — plug-and-play motion module for Stable Diffusion–family models. Animates an existing image-gen pipeline without retraining. Great for stylized or anime content.
• Motion Canvas (#4618) — when the motion you want is deterministic (UI demos, data viz, programmatic camera moves), don't fight a diffusion model — write the motion in code.

5. Upscale

• Real-ESRGAN (#2495) — practical 4× super-resolution that handles video. Most generation models output 512×512 or 720p; Real-ESRGAN is how you ship 4K. Run it as the last step before encode.

6. Assemble

• OpenCut (#4027) — open-source AI video editor. Trim, splice, color-match generated clips. Avoids the export round-trip to a closed NLE.
• Generative Media Skills (#3602) — the muapi + npx skill installer that unifies a dozen commercial generation APIs behind one CLI. Useful when an agent needs to call "generate a 5-second clip" without picking a vendor every time.

How they fit together

Prompt ─► CogVideo / Open-Sora / Diffusers / Together API
               │
               ▼
        Raw 720p clips
               │
   ControlNet ─┤ (optional: lock framing)
   AnimateDiff ┤ (optional: add motion to image)
   Motion Canvas ┤ (optional: deterministic moves)
               ▼
        Real-ESRGAN  ─►  4K upscaled clips
               │
               ▼
            OpenCut
               │
               ▼
         Final cut (mp4)

The split that matters: diffusion models hallucinate motion, code-based tools dictate motion. Use diffusion (CogVideo, Open-Sora, AnimateDiff) when you want surprises and atmosphere. Use Motion Canvas when the camera path is non-negotiable and the audience will notice drift.

Tradeoffs you'll hit

• Local vs API — local generation is free per second but costs you GPU time and tuning. API generation is fast and high-quality but priced per second and locked behind quotas. Run local for iteration, API for the hero shots.
• CogVideo vs Open-Sora — CogVideo is more stable to set up and runs on lower VRAM. Open-Sora produces longer, more coherent shots when it works. Start with CogVideo; graduate when the gap matters.
• AnimateDiff vs native video models — AnimateDiff bolts motion onto SD checkpoints (huge ecosystem of styles, mediocre coherence). Native video models train end-to-end on video (cleaner motion, fewer styles). Pick by content: stylized → AnimateDiff, realistic → CogVideo / Open-Sora.
• Real-ESRGAN vs paid upscalers — Real-ESRGAN is free and good enough for most web delivery. Topaz Video AI and similar paid tools are sharper on faces but cost real money. Ship Real-ESRGAN first; upgrade only if reviewers complain.

Common pitfalls

• VRAM math — CogVideo-5B needs ~24 GB for 720p generation. Open-Sora can demand 40 GB+. Read the model card before renting a GPU.
• Prompt drift across frames — long shots from any diffusion model drift in identity and lighting after ~3 seconds. Generate in 3-second chunks and stitch in OpenCut rather than fighting the model for a 10-second take.
• Audio is separate — none of these tools generate matching audio. Plan a separate TTS / SFX pass; the assembly happens in OpenCut.
• Commercial-API terms — every commercial generation provider has different rules on commercial reuse, training opt-out, and watermarking. Read the TOS before publishing client work.