Fine-Tuning LLMs for Domain-Specific Applications: A Practical Guide

Fine-tuning an LLM for a domain-specific application is mostly a data problem, not a training problem. This guide covers when fine-tuning beats prompting and RAG, how to build and deduplicate the dataset, QLoRA training on a single GPU, and the MLOps pipeline that keeps a fine-tuned model honest in production.

Fine-tuning has a reputation for being the expensive, expert-only option. In practice the training itself is the cheap part — six hours on a single A100. The work is in the dataset, and the payoff, when the use case fits, is substantial: in one deployment, replacing GPT-4 API calls with a fine-tuned open model took inference cost from $15,000 a month to roughly $400.

When Fine-Tuning Is the Right Call

Try prompt engineering first. Then try retrieval — our guide to building a production RAG pipeline covers that path in detail. Fine-tuning earns its cost in four situations:

If knowledge changes weekly, retrieval is still the right answer. Fine-tuning bakes knowledge in; updating it means training again.

Dataset Preparation Is 80% of the Work

The honest effort split on one project was three weeks on model training and eight weeks on building the dataset. From 50,000 collected documents, MinHash LSH deduplication at a 0.85 similarity threshold removed 4,832 near-duplicates — about 10% — leaving 28,400 training and 6,775 validation samples.

Deduplication matters more than it seems. Near-identical examples inflate the apparent dataset size while teaching the model nothing new, and they leak between training and validation splits, producing evaluation scores that flatter the model. Class balance deserves the same scrutiny: a dataset that mirrors production's natural skew will train a model that ignores the rare cases you most need it to catch.

Choosing the Base Model

Bigger is not automatically better for a narrow task. Evaluated against GPT-4, GPT-3.5 Turbo and several Llama 2 variants, a fine-tuned Mistral 7B won on the target task — 93.7% accuracy against GPT-4's 91.2%, with hallucination dropping from 8.4% to 0.2% and entity extraction reaching 91.1% F1.

That result is not a claim that Mistral beats GPT-4 generally. It is the ordinary outcome of specialisation: a small model shown thousands of examples of one task will outperform a large generalist on that task, and cost a fraction to run.

Training: QLoRA and VRAM Economics

QLoRA makes single-GPU fine-tuning of a 7B model routine. Loading the base model in 4-bit (BitsAndBytes nf4 with bfloat16 compute) and training low-rank adapters at r=64 means updating 83.9 million of 7.3 billion parameters — 1.15%.

Target the attention and MLP projections together (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) rather than attention alone; the quality difference is consistent. With gradient checkpointing, paged AdamW 8-bit and gradient accumulation, three epochs at a 2e-4 learning rate with a cosine scheduler completes in about six hours on one A100 40GB. Production inference then runs comfortably on a T4 at roughly $0.526 an hour.

Evaluation Beyond Accuracy

Accuracy alone will hide the failure mode that matters. Track hallucination rate as a first-class metric alongside F1 — a model that is 95% accurate and confidently invents the other 5% is more dangerous in production than one that is 90% accurate and abstains.

Deployment and MLOps

Lessons From the Trenches

Route low-confidence predictions to human review. A 0.85 confidence threshold worked well as a starting point, tuned afterwards against how much review capacity actually exists.

Serving runs on vLLM with AWQ quantisation. Around it sits a five-stage pipeline: ingest new labelled data, trigger retraining once 1,000+ new samples accumulate in a month, gate on evaluation results, shadow-deploy for 48 hours against live traffic, then promote only if accuracy improves by at least 0.5%. MLflow tracks experiments; Airflow orchestrates.

The shadow stage is what makes this safe. The candidate model answers every production request in parallel without those answers reaching users, so distribution shift shows up as measured divergence rather than as customer complaints.

We build and operate fine-tuned model pipelines end to end, from dataset construction through evaluation gates and serving. Explore our generative AI development services or get in touch.