Local or cloud AI: what matters beyond the model

• The app isn’t the model
• What is Local AI?
• What does it take to run AI locally?
• Starting with the hardware I already have
• Is local AI ready for everyone?

During a recent six-week sabbatical in England, staying with my brother and his family, I got to see AI in use outside my usual IT circles. My brother works in sales, and Claude is his AI assistant of choice. For much of his research, it has become his starting point instead of a web search. He uses it to summarize documents, develop written content, and support his work. In his world, AI is a research partner and documentation expert.
It wasn’t how or why he used AI that caught my attention. It was how little he needed to think about the technology behind it. His device, the app, the cloud service, and the hosted model worked together to help him get a result.
That got me thinking about local AI versus cloud AI. Is one better than the other? When would I choose to run a model myself? And how can local AI offer the same ease of access that makes cloud services useful to someone like my brother?
In my previous article, I wrote about moving beyond prompts and starting to work with agents. This is a related question, but a different one. Being agentic describes how a system works toward a goal. Local versus cloud describes where the model runs.

§The app isn’t the model

I find it helpful to separate the parts of the experience.
The model generates a response from the information supplied to it. The application provides the interface and may add conversation history, document handling, web search, or connections to other tools. An agent adds a workflow for choosing actions and checking what happened.
One way I picture this is as a person carrying out a task. The model is the brain, helping interpret information and work out what to do next. The tools are the hands, able to retrieve a document, search for information, or make an approved change. APIs are the connections that let those tools interact with other systems. The agent is the working system as a whole, bringing the brain and hands together to pursue a goal within the permissions I give it. It’s an analogy, not a suggestion that the AI thinks or understands like a person.
Those capabilities don’t all come from the model itself. A model doesn’t automatically have access to the live web, my files, or my business applications just because I can chat with it.
A desktop app can send requests to a cloud model. A browser can connect to a model running on a machine I control. The screen in front of me doesn’t tell me where the processing happens.
That distinction matters when comparing a complete cloud service with a downloaded model. I need to compare the experience around the model as well as the model’s answers.

§What is Local AI?

Ask around, and you might get the impression that local AI is for the tinfoil-hat crowd, geeky IT folks who enjoy configuring everything themselves, or people with deep pockets for expensive hardware. That’s a narrow way to look at it, although it hints at some of the reasons people become interested.
For this article, I mean running the AI model on my own computer rather than sending requests to a model operated by a cloud provider. Running it on a server within infrastructure I control is a related approach, usually described as self-hosting. In either case, I’m taking responsibility for operating the model instead of consuming someone else’s hosted model service.
Wanting to keep data within that boundary isn’t necessarily paranoia. It can be a business requirement, a contractual restriction, or simply a preference about where information is processed. But running the model locally doesn’t automatically keep the whole workflow local. Connected tools, external searches, and cloud backups still need consideration.
The deep-pockets argument needs some qualification too. Running a large, demanding model can require expensive hardware, but that doesn’t mean every useful local model needs a dedicated workstation. Smaller models may fit the computer I already own. Whether they’re good enough depends on the task and the performance I expect. The cost includes my time setting up and maintaining the software, not just the hardware I buy.

§What does it take to run AI locally?

After spending some time researching local AI and doing my own tinkering, one comparison keeps coming up: people want a model running on their own hardware to deliver the same results as a frontier model in the cloud.
Let’s put that expectation to bed, at least for a typical laptop or home workstation. Buying expensive hardware doesn’t automatically give me the model, capability, or supporting infrastructure behind a frontier AI service. That doesn’t mean a local model can’t produce comparable results on a particular task. It means I shouldn’t expect the same breadth of capability simply because I’ve invested in a powerful machine.
For a sense of scale, published examples of large models include Meta’s Llama 3.1 at 405 billion parameters and DeepSeek-V3 at 671 billion total parameters. Parameters are the learned numerical values within a model, not a count of facts it knows.
This is where memory becomes a practical consideration. The software that runs the model is called an inference engine. It loads the model’s learned weights and performs the calculations needed to generate a response. For efficient use, those weights generally need to be held in fast memory: system RAM, a graphics card’s dedicated memory (VRAM), or unified memory shared by the CPU and GPU.
Memory capacity is only part of the story. Memory bandwidth, how quickly the processor can read and write that data, matters too. GPUs were originally designed for demanding graphics work, but their ability to perform many calculations at once and read data quickly from high-bandwidth memory also makes them useful for AI. As a model generates a response, it repeatedly reads large amounts of model data. For a single conversation, that reading speed can be a major limit on how quickly the next words appear.
Problem is, GPUs that have the fast memory bandwidth and enough storage to host large models are very expensive, power intensive, and not portable.

§Starting with the hardware I already have

That doesn’t mean I need to build an AI workstation to get started. A modern laptop or desktop can be a useful place to experiment with smaller models. Instead of trying to squeeze in the largest model possible, I would start with one that leaves enough room for the computer to keep doing its everyday work.
For example, Qwen3 comes in smaller sizes including 4 billion and 8 billion parameters. Ollama lists compressed versions of those models at about 2.5 GB and 5.2 GB respectively.That compression is called quantization: storing the model’s learned values with less numerical precision to reduce its size, with a possible tradeoff in quality.
On a laptop with 16 GB of RAM, a quantized 4-billion-parameter model would be a candidate to try with short conversations and modest documents, rather than a guarantee of good performance. On a desktop with a supported graphics card and 8–12 GB of dedicated GPU memory, a quantized 8-billion-parameter model could be a reasonable next experiment, provided its weights and working space fit. These are starting points, not fixed hardware rules.
Apple Silicon devices also come up frequently in local AI discussions because of their unified memory. Rather than keeping system RAM and graphics memory in separate pools, the CPU and GPU share memory; Apple’s MLX framework is designed to use that arrangement without copying data between the two. That makes higher-memory Macs worth considering for local models.

§Is local AI ready for everyone?

My take is that local AI is viable, but it still isn’t the obvious choice for everyone. For people who want a turnkey experience where they open an app and get on with their work, I expect cloud services to remain the more practical option. The provider handles the infrastructure and packages the model, tools, and interface into a service. There are still limits and data policies to understand, but much less of the technical work falls to the person using it.
For organizations and individuals with stricter requirements about where data is processed, local or self-hosted AI is worth considering. That control comes with responsibility for securing the system, managing access, maintaining the software, and checking the results. Local doesn’t automatically mean secure or compliant, just as cloud doesn’t automatically mean unsuitable. The choice needs to meet the actual requirements.
Even with smaller models and friendlier applications, my experience so far is that running local AI remains a specialized area. It tends to appeal to enthusiasts, developers, and people willing to spend time matching a model to their hardware and needs. Getting it running is becoming easier; making it consistently useful still takes some understanding and patience.
Cost isn’t as simple as comparing a monthly cloud bill with a free model download, either. A cloud subscription or pay-as-you-go API can work out cheaper once I factor in buying or upgrading hardware, electricity, maintenance, and the time I spend keeping a local setup running. If I’m using hardware I already own, the additional cost of local AI may be much lower, and regular, heavy use can change the comparison. Equally, usage-based cloud bills can grow quickly. I need to compare the total cost of getting useful work done, at an acceptable level of quality, rather than assume local is cheaper because there isn’t a charge for each prompt.
For me, the decision comes back to the outcome. If an approved cloud service does the job well, I don’t need to recreate it locally just because I can. If keeping the processing under my control solves a specific problem, a smaller local model may be enough. I would choose based on the work I need done and the responsibility I’m prepared to take on.