Scaling AI Content Pipelines with ComfyUI: LoRA Training, Voice Cloning and Video Generation

Scaling AI content pipelines with ComfyUI means treating it as a GPU execution engine driven by API, not as a user interface. This guide covers the architecture behind generating 100+ consistent synthetic characters an hour — LoRA lifecycle management, voice cloning, video generation, perceptual-hash deduplication and channel-aware job control.

Generating one good AI image is a solved problem. Generating hundreds of distinct characters that each hold a persistent identity — the same face, the same voice, the same presence on video, across thousands of assets and without manual intervention — is an engineering problem, and almost none of it is about prompting.

Architecture Overview

Four layers, each with a clear responsibility: a Next.js dashboard for control and monitoring, a FastAPI backend holding orchestration logic, a set of pipeline services for the individual generation stages, and ComfyUI underneath as the GPU execution engine.

The important decision is the last one. ComfyUI is an engine, not a UI. Its workflows are JSON node graphs, and those graphs can be manipulated programmatically — load a template, substitute parameters, submit, poll for completion. Once you treat it that way, it becomes a scriptable renderer rather than something a human has to sit in front of.

Three Pipelines, Routed by Character Type

No single model handles every visual style well. Three independent pipelines run in parallel — Z-Image and FLUX2 for realistic output, SDXL for stylised — with routing decided by character archetype. Archetypes span K-Pop idols, fitness athletes, influencers, rappers, testimonial personas and mascots; group structures are modelled explicitly, down to roles like Leader, Main Vocalist, Main Rapper, Main Dancer, Visual and Maknae.

Running them independently rather than as a switchable configuration means one pipeline's failure or queue backlog never affects the others.

LoRA Training and Lifecycle

This is the identity anchor, and the part that makes consistency achievable at all. The Z-Image process runs in four stages: base generation, refinement, LoRA training on the accepted variations, then testing against held-out prompts.

Once trained, a character's LoRA is mounted and unmounted per job rather than kept loaded. That keeps VRAM available for concurrency while guaranteeing every asset for a given character is generated against the same identity weights. Without a per-character LoRA, drift is not a risk — it is a certainty.

Prompt Engineering at Scale

Hand-writing prompts does not survive contact with hundreds of characters. A prompt builder derives four prompt types automatically from character metadata — appearance, scene, style and negative — so the input to the system is structured character data, not prose.

Voice and Video

Audio uses three models for three jobs: VibeVoice for speaker-adaptive cloning (a 1.5B-parameter model at 5.4 GB in production), Qwen TTS for designing new voices, and SoulX Singer for voice conversion. Video runs on WAN Animate 2.2 with InfiniteTalk for lip-sync and LTX as an alternative path.

Splitting audio across specialised models rather than forcing one to do everything is the same principle as the three image pipelines — each is better at its narrow job than any general model is at all three.

Batch Processing and Deduplication

Two input modes cover real use: auto-generation from archetype templates, and CSV import for bulk character definitions. Missing fields are auto-filled from archetype defaults and name uniqueness is enforced on ingest, because duplicate character names corrupt the LoRA mapping downstream.

Output deduplication uses perceptual hashing. Comparing pHash values by Hamming distance catches near-duplicate images that a byte comparison would miss entirely, and triggers regeneration automatically. It is cheap to compute and prevents a subtle failure: a pipeline that appears productive while producing hundreds of near-identical frames.

Monitoring and Channel-Aware Control

Lessons Learned

The dashboard is driven by WebSockets rather than polling — with jobs running for minutes at a time across multiple channels, polling is both slower to update and heavier on the backend.

Stop control is channel-aware: cancelling work on one pipeline leaves concurrent jobs on the others untouched. On a system running long batches, a global stop button means losing an hour of unrelated work to cancel one bad run. A scheduler handles daily batch runs on top of all this.

Sustained throughput reached 100+ characters an hour with 48+ hours of continuous operation without failure. The remaining work is closing the last manual gap — CSV upload straight through to publication.

We build generative content infrastructure and the GPU orchestration behind it. Explore our generative AI development services or get in touch.