As an application AI engineer, you don't need to train LLMs—but you must understand how they're built to make correct architectural decisions. Think of it like understanding database internals without writing a storage engine.
This guide consolidates the critical parameters you need to evaluate when selecting, deploying, or debugging LLMs in production.
| Aspect | Details |
|---|
| What it is | Next-token prediction (left-to-right) generator |
| Key questions | Is it decoder-only? Any special heads for classification/tooling? |
| Why it matters | Determines best-fit tasks (generation vs embedding) and streaming behavior |
| Common options | Decoder-only (GPT/LLaMA/Qwen), Encoder-decoder (T5), Encoder-only (BERT) |
| What you control | Prompting pattern, stop sequences, structured output constraints |
| Tradeoffs | Decoder-only can hallucinate; not ideal alone for retrieval/classification |
| Smell test | Code assistant or refactor agent → decoder-only is the right choice |
| Aspect | Details |
|---|
| What it is | Stored capacity vs per-token compute actually used |
| Key questions | Is it MoE? What's the activated compute per token? |
| Why it matters | Cost/latency can be close to a small model with large-model quality |
| Common options | Dense, MoE (top-k), Hybrid |
| What you control | Batch sizing, expert parallelism, routing config (if exposed) |
| Tradeoffs | MoE can have load imbalance; routing instability; harder to serve |
| Smell test | "30B total, ~3B active" → cheap inference, big VRAM footprint |
| Parameter | What it is | Why it matters | Tradeoffs |
|---|
| Layers (Depth) | Transformer block count (24, 32, 48, 80+) | Correlates with multi-step reasoning and abstraction | Deeper = higher latency; harder to quantize aggressively |
| Hidden size (Width) | Representation capacity per layer (2k–10k+) | Affects richness of representations + code reasoning | Wider increases VRAM + compute; may not help if data mix is weak |
| Aspect | Details |
|---|
| What it is | Attention parallelism mechanism; GQA/MQA reduces KV cache size |
| Key questions | Is it MHA, GQA, or MQA? What's the Q:KV head ratio? |
| Why it matters | Big driver of long-context feasibility and throughput |
| Common options | MHA (Multi-Head), GQA (Grouped Query), MQA (Single KV) |
| What you control | KV cache precision, batching, context window policy |
| Tradeoffs | Too aggressive KV sharing can slightly reduce quality |
| Smell test | GQA (32Q/4KV) → much smaller KV cache for long context |
Context and positioning
Context length
| Aspect | Details |
|---|
| What it is | Maximum tokens handled as trained |
| Key questions | Native or RoPE-scaled? What's quality past 32k/64k? |
| Why it matters | Long-context correctness is non-linear; "supports" ≠ "understands" |
| Common options | 8k / 16k / 32k / 128k / 256k+ |
| What you control | Chunking, sliding window, retrieval strategy, summarization |
| Tradeoffs | KV cache explosion; "lost in the middle" phenomenon; higher latency |
| Smell test | If it forgets key constraints in long prompts → long-context weakness |
| Aspect | Details |
|---|
| What it is | How the model represents token positions |
| Key questions | RoPE? ALiBi? Any long-context scaling (NTK/YaRN)? |
| Why it matters | Governs long-range generalization |
| Common options | RoPE, ALiBi, Absolute, RoPE scaling variants |
| What you control | Prompt structure (headings), chunking order, retrieval placement |
| Tradeoffs | Scaling hacks can break beyond the trained window |
| Smell test | Native 256k trained usually beats RoPE-hacked 256k |
Only applicable to MoE models (e.g., Mixtral, Qwen-MoE, DeepSeek-V2)
| Parameter | What it is | Key questions | Tradeoffs |
|---|
| # Experts | Count of expert FFNs (8–128+) | How many? Are they specialized (code/math)? | Bigger memory footprint; more complex routing |
| Activated experts (top-k) | How many experts run per token | k=2/4/8? Does it vary? | Higher k = better quality, more compute; tail latency spikes |
| Router / load balancing | Mechanism selecting experts | Token-level routing? Load balance loss? | "Hot" experts cause congestion; quality variance |
| Parameter | What it is | Key questions | Engineering impact |
|---|
| Tokenizer + vocab | How text/code becomes tokens | BPE/SentencePiece? Code-optimized? Multilingual? | Bad tokenizer → 20–40% extra tokens; worse code formatting |
| Activation function | Non-linearity in FFN layers | SwiGLU? GeGLU? ReLU? | Some activations quantize better than others |
| Aspect | Details |
|---|
| What it is | Pretrain gives capability; post-train adds instruction-following/alignment |
| Key questions | How much instruction data? RLHF? Tool-use training? Safety tuning? |
| Why it matters | "Usable in prod" depends more on post-train than size |
| Common options | SFT only, RLHF/DPO/IPO, Tool-use finetune, Safety filters |
| What you control | System prompt, policy layer, tool schema, eval harness |
| Tradeoffs | Over-alignment/refusals; shallow SFT can be prompt-fragile |
| Smell test | If it ignores instructions or breaks JSON easily → weak post-train |
| Aspect | Details |
|---|
| What it is | Ratio of code/text/math + freshness |
| Key questions | How much code? Which languages? License-clean? |
| Why it matters | Strong predictor of code quality and hallucination rate |
| Common options | Code-heavy, Balanced, Math-heavy |
| Tradeoffs | Data contamination/licensing risk; stale frameworks |
| Smell test | If it's bad at modern React/TypeScript patterns → stale data mix |
| Parameter | What it is | Key questions | Tradeoffs |
|---|
| Precision | Numerical format (FP32/BF16/FP16/FP8) | FP16 vs BF16 vs FP8? Mixed precision? | FP16 can overflow; FP8 needs calibration |
| Quantization | Lower-bit weights (INT8/INT4) | GPTQ/AWQ? KV quantization? | Biggest lever for $/token but can break math/JSON/tool calls |
| Parameter | What it is | Key questions | Why it matters |
|---|
| Attention kernel | GPU attention algorithm | FlashAttention2/3? Paged attention? | Can mean 2–4× throughput difference |
| KV cache strategy | How past tokens' K/V are stored | KV precision? Paged KV? Prefix caching? | Dominant factor for long-context serving economics |
| Parameter | Details |
|---|
| Batching strategy | Static vs continuous batching; drives throughput and cost |
| Parallelism (TP/PP/EP) | Tensor/Pipeline/Expert parallelism; determines scaling + infra cost |
| Aspect | Details |
|---|
| What it is | Sampling parameters for generation |
| Key questions | Temperature/top-p/top-k? Repetition penalty? |
| Common profiles | Code mode: temp 0–0.2, low top-p; Creative: temp 0.7–1.0 |
| Tradeoffs | Too creative → wrong code; too greedy → brittle phrasing |
| Aspect | Details |
|---|
| What it is | Whether model learned tool usage |
| Key questions | Native tool-calling? Or prompt-hacked? |
| Why it matters | Determines agent reliability |
| Common options | Trained function calling, XML tags, Prompt-only |
| What you control | Tool schema design, retries, self-heal loops |
| Smell test | If it keeps "thinking" instead of calling a tool → weak tool training |
| Aspect | Details |
|---|
| What it is | Ability to emit valid JSON/DSL |
| Key questions | Does it support JSON schema / constrained decoding? |
| Why it matters | Critical for agents and tool calls |
| Common options | Free-form, JSON mode, CFG/constrained decoding |
| What you control | Use constrained decoding, schema validation + retries |
| Smell test | If tool calls break 5% of the time → production pain |
| Metric | What to measure | Why it matters |
|---|
| Throughput | Tokens/sec, req/sec at target context length | Real production cost = throughput × latency |
| Latency | Time-to-first-token (TTFT) + per-token decode speed | UX + agent planning loops |
| Determinism | Same input → same output? | Needed for CI, debugging, compliance |
| Aspect | Details |
|---|
| Safety/refusal | Guardrails (inline or post-hoc classifier); can impact valid request completion |
| Context management | How you place info in prompt; RAG placement; "lost in middle" mitigation |
| Failure modes | Known weak spots: Where does it hallucinate? Over-refuse? Loop? |
| Parameter | What it is | Key questions | Tradeoffs |
|---|
| Fine-tuning compatibility | Whether you can adapt it | LoRA/QLoRA? Full finetune? SFT pipeline? | Drift, regressions, catastrophic forgetting |
| Instruction tuning depth | Breadth/quality of instruction-following | Multi-turn? Code-specific SFT? | Overfit to chatty style; refusals |
Before deploying any model, run these smell tests:
Remember: Predictability beats raw intelligence. A model that's slightly less capable but reliable is often better for production than a brilliant but brittle one.