Instruction Tuning from Scratch

By Michael Samuel Naeem · February 20, 2026 · 5 min read

Base Llama completes "The capital of France is" with encyclopedic continuation — not "Paris, as you asked in your travel planning question." Instruction tuning closes that gap by supervised fine-tuning on thousands of (instruction, response) pairs so the model learns that user messages demand direct, task-shaped answers in a chat template. Whether you are adapting Mistral-Instruct to healthcare triage or building a private assistant from Llama-Base, instruction tuning is the first alignment layer before DPO or RLHF — and the layer most sensitive to dataset garbage and template mismatches.

Base vs instruct starting point

Instruct models already know roles; you add domain. Base models need general instruction capability mixed in:

mix = {
    "domain": 0.7,
    "general_open_orca_sample": 0.2,
    "format_json_tool": 0.1,
}

Skipping general mix on base → brittle domain parrot.

Chat template consistency

Tokenizer chat template must match inference:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system", "content": "You are a concise legal summarizer."},
    {"role": "user", "content": "Summarize clause 4 in plain English."},
    {"role": "assistant", "content": "Clause 4 requires..."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False)
# train on `text` with labels masking non-assistant tokens

Mask loss on user/system tokens — train only assistant completion:

labels = input_ids.copy()
labels[:assistant_start_token] = -100  # ignore in loss

Wrong masking trains model to predict user prompts — leakage weirdness.

SFT with TRL

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

dataset = load_dataset("json", data_files="instr_train.jsonl")

training_args = SFTConfig(
    output_dir="sft-out",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    max_seq_length=4096,
    packing=True,  # efficient short examples
    dataset_text_field=None,  # use formatting_func
)

def formatting_func(example):
    return tokenizer.apply_chat_template(example["messages"], tokenize=False)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    processing_class=tokenizer,
    formatting_func=formatting_func,
)
trainer.train()

Use LoRA for 7B+ on consumer GPUs; full fine-tune only with budget and anti-forgetting eval.

Hyperparameter starting points (7B LoRA)

Early stop on validation loss and instruction eval — overfitting memorizes training phrasing.

Evaluation suite

• MT-Bench / domain golden set — multi-turn
• IFEval — verifiable instruction constraints (word count, JSON keys)
• Exact tool-call parse rate — if agents
• Toxicity/refusal — safety regression

Qualitative error buckets weekly: "too verbose", "ignored context", "hallucinated citation."

Instruction tuning vs DPO ordering

Standard pipeline:

1. SFT — teach format and domain knowledge
2. DPO/RLHF — refine preferences (tone, helpfulness trade-offs)

DPO without SFT on base for complex domains often underperforms — model lacks foundation to distinguish good/bad within format.

Common mistakes

• Training on prompts without system policy you deploy with
• Max length truncating tail of long context examples silently
• Duplicate near-identical instructions → overfit phrasing
• Evaluating with different temperature than production

Instruction tuning is dataset engineering first — hyperparameters second.

Dataset quality over quantity

LIMA showed 1,000 high-quality examples can outperform 52,000 mediocre ones. Quality criteria for instruction examples:

• Correct — verified answer, not hallucinated
• Diverse — cover task types, not 500 variations of "summarize this"
• Formatted — match production chat template exactly
• Scoped — include system prompt policy you deploy with
• Challenging — include edge cases, not just easy wins

{
  "messages": [
    {"role": "system", "content": "You are a helpful coding assistant. Be concise."},
    {"role": "user", "content": "Write a Python function to merge two sorted lists."},
    {"role": "assistant", "content": "def merge(a, b):\n    result = []\n    i = j = 0\n    ..."}
  ]
}

Every training example should use the exact system prompt and chat template deployed in production.

Multi-turn instruction datasets

Single-turn SFT teaches format; multi-turn teaches context maintenance:

{
  "messages": [
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "user", "content": "What's its population?"},
    {"role": "assistant", "content": "Paris has approximately 2.1 million people in the city proper."}
  ]
}

Include 30–50% multi-turn examples in domain datasets. Models trained only on single-turn fail at conversation context.

Production deployment checklist

After SFT, validate before deploy:

• Run IFEval or domain golden set — instruction following rate
• Compare base vs SFT on held-out eval set — ensure improvement, not regression
• Test with production system prompt and chat template
• Safety eval — refusal rate on harmful prompts unchanged or improved
• Latency benchmark — SFT shouldn't change inference speed (same model size)
• Rollback plan — keep previous model weights tagged in registry

Failure modes

• Training/eval template mismatch — train with Llama template, deploy with ChatML
• System prompt not in training data — model ignores system instructions at inference
• Overfitting to training phrasing — model parrots exact training responses
• Truncated long examples — tail of context window cut silently during training
• No held-out eval set — overfit discovered only in production

Production checklist

• Chat template matches production deployment exactly
• System prompt included in every training example
• 1k+ diverse, verified examples minimum for domain SFT
• Multi-turn examples included (30%+ of dataset)
• Held-out eval set with IFEval or domain golden set
• LoRA for 7B+ models; full fine-tune only with budget
• Previous model version tagged for rollback

Common production mistakes

Teams get fine tuning instruction tuning wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of fine tuning instruction tuning fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Resources

• TRL SFTTrainer documentation
• LIMA: Less Is More for Alignment
• Alpaca instruction tuning dataset
• Chat templates (Hugging Face)
• IFEval instruction following benchmark