Multimodal AI and Vibe Coding: How AI Is Changing Software Development

Multimodal AI and vibe coding are changing how software gets built — but not evenly. This guide covers what AI-assisted development genuinely handles well, where it reliably fails, and what a four-modality production stack (image, video, speech-to-text, text-to-speech) actually costs to run on real GPU hardware.

Two shifts arrived at once. Models that handle images, audio and video alongside text became production-viable, and code generation became good enough that describing intent often beats typing implementation. Neither is a total replacement for engineering. Both change what engineers spend their time on.

What Vibe Coding Means in Practice

Vibe coding is directing intent rather than writing every line. Ask for a data visualisation dashboard and a capable coding agent will produce one with correct typing, sensible component structure and the API client patterns the codebase already uses — not scaffolding, but code that fits.

It works because most application code is a variation on a pattern the model has seen thousands of times. CRUD endpoints, form handling, list-and-detail views, data transformations, responsive grids with loading and error states — roughly 70% of typical application work falls into this category.

Where It Breaks Down

The other 30% is where engineering still lives, and the boundary is sharp:

The failure mode is confident plausibility. Generated concurrency code looks right, passes a casual review, and breaks under load. Review effort should be concentrated where the model is weakest, not spread evenly.

The Four-Modality Production Stack

Image generation, video generation, speech-to-text and text-to-speech each have different maturity levels, and treating them as one capability leads to disappointment.

Image Generation

Node-based workflows in ComfyUI make image pipelines reproducible — a workflow is a saved graph, not a prompt someone remembers. The hard problem is consistent character generation: producing the same recognisable subject across many images. The combination that solves it is IP-Adapter for identity, ControlNet for pose, and a LoRA for style, tuned together.

Working parameters: SDXL at 30 steps, CFG 7.5, Karras scheduler, 1024×1024; IP-Adapter weight 0.85 with 0.1 noise; ControlNet strength 0.7; LoRA strength 0.65.

VRAM Optimisation

This is the unglamorous constraint that determines throughput. On an A10G with 24 GB, SDXL base needs about 6.5 GB, but with IP-Adapter and ControlNet loaded it exceeds 12 GB — so two adapters halve your concurrency.

Four techniques recover it: FP16 inference, sequential model loading, tiled VAE decoding (which cuts VAE memory by 60%), and attention slicing. Together they allow three concurrent SDXL jobs per A10G at above 80% GPU utilisation.

Speech-to-Text

The most production-ready of the four. Whisper large-v3 achieves under 5% word error rate on clean English with word-level timestamps. Running it on a shared GPU cluster with queue-based time-sharing during off-peak hours keeps cost reasonable.

Video Generation

The Convergence

What This Means for Engineering Leaders

Still the frontier. Runway Gen-2, Stable Video Diffusion and Pika all produce impressive short clips and none is reliable enough for unsupervised production use. The genuinely useful application today is generating synthetic training data for computer vision, where variety matters more than perfection.

The interesting part is these capabilities composing. A working loop: speak a requirement, Whisper transcribes it, a coding agent with codebase context via MCP implements it, and asset generation runs in ComfyUI — with a human reviewing at each boundary.

Nothing in that chain is speculative. Each piece is deployable now; the composition is what is new.

We build multimodal AI systems and the GPU infrastructure they depend on. Explore our generative AI development services or get in touch.