5 min read
5 to 7 words per second. That is how fast Microsoft says its open-source BitNet framework can run a very large, specially compressed AI model on a single ordinary computer processor, roughly the speed a person reads. A few years ago that would have required a rack of specialized hardware. Today it is a free download.
You do not need BitNet itself to benefit from that shift. The bigger point is this: capable AI models now run entirely on a normal laptop, with nothing sent to the cloud. For a solo business holding client contracts, medical intake forms, financial records or unreleased work, that is a practical privacy option, not a hobbyist experiment. This guide shows you how to set one up in under an hour, and, just as importantly, when it is not worth the trouble.
Why run AI on your own machine at all
Cloud assistants like ChatGPT, Claude and Gemini are more capable than anything your laptop can run. So the case for local AI is narrow but real:
- Confidential documents. Some client agreements forbid uploading their material to third-party services. A local model never sends the text anywhere.
- Regulated information. If you handle health, legal or financial details, keeping a copy of that data off outside servers simplifies the conversation with clients and, often, with the rules covered in our plain-English guide to new state AI laws.
- Offline work. Planes, rural job sites and spotty cafe Wi-Fi stop mattering.
- No per-use cost. Once it is installed, a local model costs nothing to run beyond electricity.
The trade-off: local models are smaller, so they are weaker at complex reasoning and long documents. Think of one as a capable assistant for drafting, summarizing and sorting, not a strategist.
Step 1: Check your laptop
The single number that matters most is memory (RAM). A practical rule of thumb:
- 8 GB of RAM: small models (around 3 billion “parameters,” a rough measure of model size). Fine for short summaries and rewrites.
- 16 GB of RAM: mid-size models (around 7 to 8 billion parameters). This is the sweet spot for most business writing tasks.
- 32 GB or more: larger models that start to feel genuinely useful for longer documents.
Recent Apple Silicon Macs and Windows laptops with a modern processor both work well. You do not need a gaming graphics card to start.
Step 2: Choose a friendly app
Two free tools cover almost every solo owner.
LM Studio (easiest, point and click)
LM Studio is a desktop app for Mac, Windows and Linux that looks and feels like a chat assistant. You browse a catalog of open models, click download, and start chatting. If you have never opened a terminal window, start here.
Ollama (simple, slightly more technical)
Ollama installs like any app and runs models with one short typed command. It is popular because other tools plug into it easily, including free chat interfaces such as Open WebUI if you later want a nicer window.
Both are free. Pick LM Studio if you want zero typing, Ollama if you expect to connect other apps later.
Step 3: Download a small open model
Open models are AI models whose files are published so anyone can run them. Families you will see in both apps include Llama (Meta), Gemma (Google), Mistral and Qwen. For a first test:
- Search the app’s catalog for a model labeled with a size that matches your RAM from Step 1.
- Prefer versions marked “instruct” or “chat,” which are tuned to follow instructions.
- Download one. Expect a file of a few gigabytes.
In Ollama, the whole process is a single command such as ollama run llama3.2, which downloads a small Llama model and opens a chat in your terminal.
Step 4: Test it on a real (non-sensitive) task first
Before trusting it with client files, run three tests using material you would be comfortable sharing:
- Summarize a two-page document you know well. Did it miss anything important?
- Rewrite a rough email in a friendlier tone. Does it sound like you?
- Extract dates, names and amounts from a sample invoice. Are they correct?
If results are weak, try the next model size up, or a different family. Quality varies more between models than between apps.
Step 5: Build a simple “private lane” workflow
The smartest setup is not local only. It is a two-lane system:
- Private lane (local model): anything containing client names, health or financial details, or material under a confidentiality agreement. Summaries, first drafts, and pulling details out of documents.
- Public lane (cloud assistant): marketing copy, general research, strategy questions and anything already public.
A useful trick: let the local model strip out names and identifying details, then paste the anonymized version into your cloud assistant for heavier thinking. You get cloud-level quality without exposing the original.
When local AI is not worth it
Skip this entirely if your laptop has less than 8 GB of RAM, if none of your work is truly confidential, or if you mostly need deep reasoning and research. In those cases, the paid business tiers of cloud assistants, which typically promise not to train on your data, are the better buy. Our breakdown of myths about free AI tools explains why “free” is not always cheaper, and our look at Anthropic’s October price cuts shows how fast cloud costs are falling.
The bottom line
Local AI is no longer a science project. With a 16 GB laptop, a free app and a 30-minute setup, you can keep your most sensitive client work entirely on your own machine while still using cloud tools for everything else. Install one app, test one model, and give it one confidential task this week. You will quickly learn whether it earns a permanent place in your toolkit.
