AI Model Strategy
Large models, small models, and the system around them.
The useful question is not whether a large language model or a small language model is better. The useful question is what the product needs, where the model runs, what each request costs, and how much confidence the business needs before the answer reaches a customer.
By Herb Trevathan
Same foundation
Both model classes are built from transformer-style blocks that predict the next token from prior context.
Different constraints
Small models are shaped by memory, power, latency, and local deployment. Large models are shaped by throughput, quality, and serving cost.
Best used together
Production AI systems often combine small and large models instead of treating model size as a single permanent choice.
Foundation
Small and large models start from the same basic idea.
A language model reads tokens, compares the current token with earlier tokens, mixes that information through repeated layers, and produces a probability distribution for what should come next. The learned weights inside the model are called parameters. A small language model usually has far fewer parameters than a large one, which means it is easier to run but has less internal capacity.
The size is not the strategy. Size is a result of constraints. If the model must run on a device with limited memory, the design has to be lean. If the model serves difficult open-ended requests from a cloud system, the design can spend more compute to gain broader capability.
Deployment target
Where the model runs determines memory, latency, battery, privacy, and hardware limits. A phone, browser, edge device, and cloud server all push the design in different directions.
Inference economics
Training is paid up front. Serving is paid every time a customer uses the product. At scale, the recurring inference cost usually matters more than the original training cost.
Training budget
A smaller training budget pushes teams toward better data, distillation, compression, and specialization instead of raw size.
Architecture
The runtime cost shows up in the attention cache.
During generation, a model stores attention information for previous tokens so it can continue the response without recomputing everything. This stored state is often called the KV cache. As the conversation grows, the cache grows too.
Small models are designed to keep that cost under control. Grouped-query attention lets several query heads share key-value groups. Sliding-window attention lets some layers focus on recent context instead of the full conversation. Cache sharing can reuse stored state across layers. The purpose is practical: reduce memory and bandwidth while preserving enough quality for the job.
Training
Small models depend on training quality.
A small model cannot simply rely on massive parameter count. It has to learn efficiently. That makes the training recipe more important: the quality of the examples, the teacher signal, and the decision to spend more training work up front so production serving becomes cheaper later.
- Data curation: carefully selected and synthetic training data can teach a small model more efficiently than a huge pile of noisy text.
- Knowledge distillation: a small student model learns from a larger teacher model's outputs, not only from raw text.
- Overtraining: small models are often trained on more tokens than a compute-optimal formula would suggest because better quality can reduce serving cost later.
Deployment
The model has to fit the machine.
A model design is not complete until it runs under real conditions. Local models must respect memory, power, heat, and response time. Cloud models must respect throughput, queueing, rate limits, and cost per request.
- Quantization stores parameters with fewer bits so the model uses less memory and can run on smaller hardware.
- KV cache management reduces the stored attention state that grows as the conversation gets longer.
- Hardware mapping matches the model to the target processor, memory bandwidth, batching behavior, and power limits.
Tradeoffs
Small models are useful, but they have real ceilings.
A small model can be fast, private, inexpensive, and good at a narrow job. It can also be brittle when the request moves outside the training distribution. Large models usually carry broader knowledge and stronger reasoning, but they cost more and may require cloud execution.
Generalization
Small models can be excellent inside the tasks they were trained for and brittle outside that distribution.
Reasoning depth
Complex multi-step work still tends to favor larger models, especially when the problem is ambiguous or spans many concepts.
World knowledge
Parameters act like stored memory. Smaller models have less room for broad factual recall and often need retrieval from trusted data sources.
Production Pattern
The strongest answer is often a hybrid system.
In production, the model is only one part of the system. A practical AI workflow may use one model to classify the request, one model to handle simple work, one larger model for difficult work, retrieval for verified knowledge, and guardrails before and after the response.
That composition matters more than a benchmark chart. The goal is not to use the biggest model everywhere. The goal is to route the right work to the right capability with clear fallback behavior and measurable confidence.
Routing
A small classifier or small language model handles common requests and escalates uncertain or complex work to a larger model.
Guardrails
Small models classify, filter, redact, or score inputs and outputs around a larger model so the system stays safer and cheaper to operate.
Drafting
A small fast model drafts candidate tokens or responses while a larger model verifies the result before it reaches the user.
Decision Frame
Choose from constraints, not from model size.
HerbDev Perspective
Model choice is system design.
A reliable AI product is not built by picking a model name and wiring it to a text box. It is built by matching the model to the workflow, adding retrieval where facts matter, controlling cost, measuring confidence, protecting customer data, and giving humans clear places to approve high-impact decisions.