Why Companies Are Bringing AI Back On-Premises — and What It Means for German SMBs

For three years, the conventional wisdom was clear: AI belongs in the cloud. Models are too large, GPUs too expensive, operations too complex. If you built with AI, you built on OpenAI, Anthropic, Azure, or Bedrock. In 2026, the wind has shifted. At Dell Technologies World in May, a trend that had been wandering through conference stages crystallized into something operational: companies are pulling their AI workloads back into their own data centers. Honeywell, Samsung, and a growing list of industrial heavyweights talk openly about why they’re doing it — and what they’re building to support it.

This article situates the trend without romanticizing it. What’s marketing, what’s substantive, and most importantly: what does this mean for mid-sized companies in Germany that don’t have Samsung’s GPU budget but must meet the same data protection requirements?

1. The hype has shifted

The first AI hype cycle (2023–2024) rewarded speed. Whoever shipped a ChatGPT wrapper first won attention. Architecture was secondary. The model was OpenAI. Data lived wherever it happened to live.

The second cycle (2025) rewarded reliability. RAG systems had to deliver reproducible answers, not just charming ones. Hallucinations grew more expensive than latency. The first compliance audits revealed that many pilots could never go to production — the data protection impact assessment failed.

The third cycle, just beginning, rewards control. In 2026, anyone building AI must be able to explain where data is processed, which model version is in production, who has access to inference logs, and how the whole thing is auditable under the EU AI Act. These questions are hard to answer in a shared public cloud — and comparatively easy in a sovereign stack.

“We’re not giving up on the cloud. We’re giving up the assumption that the cloud is automatically the right place for every AI workload.”

That was how a CIO at a German mechanical engineering company put it in a recent advisory conversation. The sentence captures the movement well. It’s not about cloud abstinence. It’s about deliberate segmentation.

2. What really happened at Dell Tech World 2026

Dell Technologies World has traditionally been a hardware conference with a storage, networking, and server focus. In 2026, the conference was almost entirely shaped by on-premise AI themes — not in the form of vendor pitches, but in the form of customer reports.

Three things stood out:

First: Customer sessions got concrete. Instead of generic “we’re investing in AI” slides, Honeywell, Samsung, a major European pharma group, and several US banks presented concrete architectures: which models run on which GPUs, what the data path looks like, where the audit loops close, what it costs per token.

Second: The open-weight world had matured. Llama 3.3, Mistral Large, Qwen3-Coder, and DeepSeek-V3 were named as models of choice — not as “cheap alternative,” but as the operational default. Closed-source models were still used for specific tasks, but as a special case, not the norm.

Third: Hardware became routine. What looked like a GPU procurement odyssey in 2023 is now a normal procurement process in 2026. H100/H200 clusters are orderable on plannable timelines, the newer Blackwell-based platforms (B100, B200, GB10/Grace Blackwell) are available, and supply markets have settled.

Most importantly, Dell Tech World marked something simple: on-premise AI is no longer a research project. Customer reports cover 12 to 18 months of production operation — with real customers, real SLAs, real audits. The learning curve is behind us.

3. Honeywell, Samsung & Co — what they’re bringing back

A closer look reveals: companies aren’t pulling everything back, but selectively — exactly the workloads where cloud inference has structural disadvantages.

Honeywell runs AI for industrial anomaly detection, predictive maintenance, and process optimization. These workloads run directly on production lines, often in plants with limited internet connectivity. Latency requirements are in the millisecond range. A cloud roundtrip to an API isn’t just expensive — it’s technically impossible. The models are smaller, often domain-fine-tuned, and run on edge compute right on site.

Samsung demonstrates the opposite extreme: massive central clusters for internal AI applications — from code generation to document processing to RAG systems over internal IP. This isn’t about latency, it’s about data control. Source code, roadmaps, patents — none of it leaves the corporate network.

Banks and insurers run hybrid setups: sensitive workloads (credit decisions, underwriting, fraud detection) on-premise; generic tasks (marketing copy, basic classification) still on public cloud APIs.

The pattern is clear: the strategically most important and most heavily regulated workloads are the first to come back. The non-critical ones stay where they are. Companies that try the reverse — pulling everything back without prioritization — quickly lose their economic footing.

4. The four drivers

Twelve months of project work surfaces four recurring drivers that move companies toward on-premise decisions:

Data protection and confidentiality. The obvious argument, but not the only one. Personal data, trade secrets, source code, contracts, patient records — anything covered by GDPR, professional confidentiality, arbitration clauses, or NDAs is hard to handle cleanly via a public cloud API. Data processing agreements with US hyperscalers exist, but they don’t shield against every regulatory risk.

Cost control. API pricing is linear with token usage. As usage grows — and in productive applications, it always grows — costs grow with it. Owned infrastructure has a high upfront investment, after which marginal cost per token approaches zero. Past a certain volume threshold (see above: roughly 1M tokens/day), the math flips. What was mostly theoretical in 2024 has become a concrete CFO question in 2026.

Latency and availability. Edge use cases (manufacturing, logistics, medical technology) need response times under 100ms. A cloud API doesn’t deliver that reliably — certainly not over transatlantic routes. Availability is a second issue: if AI fails because OpenAI has an incident, that affects your business, with nothing you can do about it.

Regulation. The EU AI Act requires demonstrability, reproducibility, and auditability for high-risk systems. GDPR remains. Industry guidelines (BaFin, MaRisk, ISO 27001, TISAX) demand control over data flows. A black-box API is not a good companion here. A sovereign stack with a documented model, verified model hash, and complete inference logs is auditable.

These four drivers often combine. Rarely is it just one. A typical 2026 project has data protection + regulation + cost control as the primary motivation, with latency as an occasional bonus argument.

5. What German SMBs should take away

Honeywell and Samsung can afford GPU clusters in the eight-figure range. The German Mittelstand can’t. Does that make the trend irrelevant to them? On the contrary.

The crucial difference in 2026: you don’t need a hyperscaler stack to be sovereign. The tools have matured. A setup with one or two H100 GPUs or modern Blackwell workstations is enough for most mid-sized use cases. A Llama 3.3 70B or Mistral Large runs on it productively. RAG systems with tens of thousands of documents are no longer a technical problem.

Three realities apply to the Mittelstand:

First: you don’t need to build from scratch. Sovereign AI stacks are documented. Open-source components are battle-tested. Implementation partners have experience. You’re not the pioneer — you’re the fast follower, and that’s the economically best position.

Second: you can build hybrid. On-premise doesn’t mean banning APIs. For sensitive internal data: your own RAG system with a local model. For non-critical tasks (marketing drafts, generic copy): API model, no problem. A simple classifier handles the routing — no architectural mega-project.

Third: you have to start with the right use case. Not the biggest, but the most valuable. We most often recommend an internal AI assistant with access to internal documents — it provides daily measurable benefit, isn’t public, and every compliance question has a clean answer.

6. The pragmatic entry point

Here’s how to start in 2026:

1. Use-case inventory. What do you actually want to do with AI? Three to five concrete use cases. Not “we want ChatGPT,” but “we want to pre-classify service tickets” or “we want technical documentation to be searchable.”
2. Data classification. Which data is sensitive, which isn’t? This one hour of work saves months of architecture debate later.
3. Architecture workshop. Which workloads run where? On-premise, EU cloud, public API. Who routes? What does the audit path look like?
4. 8-week pilot. One use case, one model, one defined data corpus. Production, with real users. Not a POC in a vacuum.
5. Scale afterwards. Only when the pilot lands do the next use cases follow. Each new one builds on the first’s infrastructure.

On the hardware side: a single-server setup with two Blackwell GPUs covers most mid-sized workloads. Investment: 50,000 to 150,000 euros for hardware plus setup, depending on requirements. Operationally: container orchestration with Kubernetes or a leaner alternative, inference via vLLM or TGI, monitoring with Prometheus/Grafana. Standard components, all proven.

For a deeper dive into the layers, see our field guide The sovereign AI stack 2026.

7. Conclusion

The trend Honeywell and Samsung are making public in 2026 has been quietly visible in the Mittelstand for months: data protection, regulation, and sheer AI workload volume create reasons to bring AI back into your own data center. What looked ideological or paranoid two years ago is now an economically sound decision.

The good news for the Mittelstand: you don’t need a hyperscaler stack. You need a clear use case, an operational open-weight model, a clean data path, and a partner who has built the stack before. In 2026, that’s an 8-to-12-week effort.

If you want to know which workload in your company should be the first to come back under your own control: let’s talk.