The fastest path to local AI is not building a server. It is installing a maintained runtime, downloading one model that fits comfortably, and testing a private task you already understand.
Use LM Studio when you want a desktop interface and model browser. Use Ollama when you want a simple command-line workflow and local service. Move to llama.cpp, MLX, or vLLM after you know why the simpler path is insufficient.
Install the runtime and choose a small first model
Download LM Studio from its official site or install Ollama using the vendor instructions for your operating system. Begin with a current 7B–14B instruct model at a 4-bit quantization. Confirm that the model license permits your intended use.
Do not download a 70B checkpoint because a forum post says it technically runs. A smaller model that produces a responsive answer and completes the task is a better first system.
Measure fit, speed, and quality
Run five prompts from the real workflow. Record time to first token, generation speed, peak memory, and whether the answer passes your quality check. Increase context gradually. If the system begins swapping memory or becomes unresponsive, the model does not fit operationally even if it eventually completes.
Compare two nearby model sizes or quantizations before buying hardware. A higher-bit small model can outperform an aggressively compressed larger one on the work you care about.
- Keep 15–25 percent memory free.
- Use the exact same prompts for each comparison.
- Measure prompt processing and output generation separately.
- Check that tools and structured output work in the chosen runtime.
Protect the local service
A local API often uses an OpenAI-compatible endpoint, which makes it easy to connect existing tools. Keep it bound to localhost unless another device truly needs access. For LAN use, add authentication and a firewall rule; do not forward the port directly through your router.
Know whether the chat interface stores history and where model files, logs, and document indexes live. Local inference is a privacy opportunity only when the surrounding application follows the same boundary.
Finish one offline workflow with Wi-Fi disabled. That verifies the entire chain is actually local.
Primary sources