Running Gemma 4 locally on a Mac Mini M4 — from zero to your own domain

There’s a specific kind of satisfaction that comes from watching a large language model respond to your prompt and knowing — with complete certainty — that those tokens never left your building. No API key, no metered usage, no third-party server logging your queries. Just a model, a machine, and a local network.

I recently set this up on a Mac Mini M4 (2025, 16 GB RAM) running macOS 26 Tahoe. What follows is a faithful reconstruction of exactly what I did — including the wrong turns. If you have a similar setup, you should be able to follow this from start to finish.

This guide assumes you’re comfortable with Terminal and basic shell commands. No prior experience with Ollama or Docker is needed.

Why Gemma 4 — and which variant?

Google’s Gemma 4 family released in early 2026 brings something genuinely interesting to local deployment: Mixture-of-Experts (MoE) architecture in an approachable package. The headline models are large (26B and beyond), but the MoE variants activate only a fraction of parameters per inference — which translates to lower RAM pressure than raw parameter counts suggest.

Here’s how the lineup looks for local use:

Model	Download size	RAM needed	Context	Notes
gemma4:2b	~2 GB	4 GB+	128K	Fast, limited quality
gemma4:4b	~3 GB	6 GB+	128K	Good for light tasks
gemma4:e4b	9.6 GB	12 GB+	128K	MoE, multimodal, this guide
gemma4:26b	~18 GB	24 GB+	128K	Best quality, needs more RAM

On 16 GB RAM, gemma4:e4b is the right choice. It’s the “Efficient 4B” MoE variant — despite the name, it punches well above its weight, handles multimodal input, and fits comfortably within the machine’s memory headroom. The 26B model technically fits on 16 GB but leaves almost no breathing room for the OS and other processes.

Step 1 — Install Homebrew

If you’re starting fresh on macOS, chances are Homebrew isn’t there yet. Check:

brew --version

If you get command not found, install it:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

On Apple Silicon, you also need to add it to your PATH — the installer won’t always do this automatically:

echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile
eval "$(/opt/homebrew/bin/brew shellenv)"

If you skip the PATH step and open a new terminal window, Homebrew will appear to be gone. This tripped me up briefly.

Step 2 — Install Ollama and start the service

Ollama is the runtime layer that manages model downloads, storage, and inference. Installing it via Homebrew keeps updates clean:

brew install ollama

Then start it as a background service so it launches automatically at login:

brew services start ollama

==> Successfully started `ollama` (label: homebrew.mxcl.ollama)

Ollama is now running as a local server on port 11434. You can verify with curl http://localhost:11434 — it should respond with Ollama is running.

Step 3 — Pull the model

This is where a small gotcha hides. The model tag I initially tried (gemma4:27b) doesn’t exist in Ollama’s registry. The correct tag for our target model is:

ollama pull gemma4:e4b

This downloads approximately 9.6 GB. Depending on your connection, expect anywhere from 5 to 30 minutes. When it finishes, confirm the model is available:

ollama list

Step 4 — First test in terminal

Before adding any interface layer, it’s worth verifying the model actually works. The fastest sanity check:

ollama run gemma4:e4b "What is the capital of Poland?"

The capital of Poland is Warsaw.

If you see a coherent response, the model is running correctly. You can also drop into an interactive session by running ollama run gemma4:e4b without a prompt argument — type /bye to exit.

At this point you already have a fully functional local LLM. The remaining steps are about giving it a usable interface and making it accessible beyond your local machine.

Step 5 — Install Docker Desktop

Open WebUI — the browser-based chat interface — is distributed as a Docker container. Install Docker Desktop with Homebrew:

brew install --cask docker

After installation, launch Docker from Applications at least once. It needs to run in the background before you can start any containers. Wait until the whale icon in your menu bar stops animating — that means the Docker daemon is ready.

Step 6 — Launch Open WebUI

With Docker running, one command pulls and starts Open WebUI, pre-connected to your local Ollama instance:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Give it about 60 seconds on first launch — it’s pulling the container image and initializing. Then open:

http://localhost:3000

You’ll be prompted to create a local admin account (name, email, password — none of this leaves your machine). After that, select gemma4:e4b from the model dropdown and you’re ready to chat through a full browser UI.

Open WebUI persists conversation history in the Docker volume open-webui. If you’re deploying this for multiple users, everything in that volume is personal data — worth thinking about your retention policy before sharing access.

Inside Open WebUI — getting the configuration right

Once you’ve landed on http://localhost:3000, Open WebUI presents a clean chat interface. But the real value is in the settings panel — there’s considerably more surface area than the default view suggests.

The first account you create automatically becomes the admin. Subsequent accounts are regular users by default. If you’re deploying this for others, go to Admin Panel → Settings → General and set Default User Role to pending — this forces new registrations to wait for manual approval rather than getting immediate access.

This one change prevents the interface from being open to anyone who finds the URL.

Connecting to Ollama

Open WebUI auto-detects Ollama at http://host.docker.internal:11434 when launched with the --add-host flag from our Docker command. Verify this under Admin Panel → Settings → Connections. You should see Ollama listed with a green indicator.

If it shows as disconnected:

brew services restart ollama
# then refresh the browser

System prompts and model presets

Under Workspace → Models, you can create named presets — a model + system prompt combination saved as a distinct “assistant”. Each preset appears as a selectable option in the chat interface.

A few practical examples:

Name: Document Analyst
Model: gemma4:e4b
System prompt: You are a precise document analyst. When given text or a file,
extract key information, identify main arguments, and flag any ambiguities.
Respond in the same language as the input document.

Name: Code Reviewer
Model: gemma4:e4b
System prompt: You are an experienced software engineer performing code review.
Focus on correctness, readability, and potential edge cases. Suggest specific
improvements with brief explanations. Do not rewrite the entire code unless asked.

Once saved, these appear as distinct options in the model dropdown — users don’t need to think about system prompts at all.

RAG — attaching documents to conversations

Open WebUI has built-in Retrieval Augmented Generation. Under Workspace → Knowledge, upload documents (PDF, DOCX, TXT, Markdown) that the model can query during conversations.

Reference a document in chat with the # prefix — type # and the document name will autocomplete. The model retrieves relevant chunks before responding. With Gemma 4’s 128K context, it handles surprisingly long source documents without losing coherence.

All uploaded documents are stored in the Docker volume open-webui. Back this up if the documents matter. On macOS, Docker volumes live at ~/Library/Containers/com.docker.docker/Data/vms/0/.

User management at a glance

Feature	Where to find it
Approve pending users	Admin Panel → Users
Promote user to admin	Admin Panel → Users → edit role
Disable registration entirely	Admin Panel → Settings → General → toggle “New User Signup”
Export conversation history	User menu → Settings → Account → Export Data
Set default model for new users	Admin Panel → Settings → Interface

Making it available outside your local network

Running Open WebUI on localhost:3000 is fine for a single machine. But the moment you want to access it from another device — a phone on mobile data, a laptop at a café, a colleague connecting remotely — you need to expose it to the internet.

The naive approach is port forwarding on your router. Don’t do this. You’d be exposing a web service directly to the public internet with no additional protection layer.

The right approach: Cloudflare Tunnel.

Cloudflare Tunnel — how it works

A Cloudflare Tunnel creates an outbound-only encrypted connection from your Mac to Cloudflare’s edge network. Traffic flows like this:

User browser → ai.yourdomain.com (Cloudflare) → Tunnel → Mac Mini → Open WebUI

Your Mac never opens an inbound port. There’s no firewall rule to configure, no dynamic DNS to manage, no IP address to expose. Cloudflare terminates HTTPS at their edge, so your custom domain automatically gets a valid TLS certificate.

The tunnel daemon (cloudflared) runs locally, calls out to Cloudflare, and Cloudflare routes traffic back through that persistent connection.

Setting up Cloudflare Tunnel — step by step

Prerequisites:

A domain added to Cloudflare (it must use Cloudflare as its nameserver — this is the only hard requirement)
A free Cloudflare account

Install cloudflared

brew install cloudflared

Authenticate with Cloudflare

cloudflared tunnel login

This opens a browser window. Log in and select the domain you want to use. A certificate file is saved to ~/.cloudflared/cert.pem.

Create the tunnel

cloudflared tunnel create gemma-local

This registers a named tunnel and creates a credentials file at ~/.cloudflared/<TUNNEL-UUID>.json. Note the UUID printed in the output — you’ll need it next.

Create the configuration file

nano ~/.cloudflared/config.yml

Paste this, substituting your tunnel UUID and username:

tunnel: YOUR-TUNNEL-UUID
credentials-file: /Users/YOUR-USERNAME/.cloudflared/YOUR-TUNNEL-UUID.json

ingress:
  - hostname: ai.yourdomain.com
    service: http://localhost:3000
  - service: http_status:404

The ingress block tells Cloudflare: route requests to ai.yourdomain.com to localhost:3000. The final catch-all line is required — it handles any requests that don’t match a rule.

Create the DNS record

cloudflared tunnel route dns gemma-local ai.yourdomain.com

This adds a CNAME record in your Cloudflare DNS pointing ai.yourdomain.com to the tunnel. No need to touch the Cloudflare dashboard manually.

Test the tunnel

Start it manually first:

cloudflared tunnel run gemma-local

Open https://ai.yourdomain.com in a browser. If you see the Open WebUI login screen, it’s working — HTTPS with a valid certificate, handled automatically by Cloudflare. Press Ctrl+C to stop.

Run the tunnel as a persistent service

sudo cloudflared service install
sudo launchctl start com.cloudflare.cloudflared

The tunnel now starts automatically at boot, before any user logs in. Check the status:

sudo launchctl list | grep cloudflare

Locking down access — Cloudflare Access

Your Open WebUI is now reachable at https://ai.yourdomain.com from anywhere in the world. Open WebUI’s own authentication is one layer of protection — but for sensitive deployments, Cloudflare Access adds a second layer in front of the application entirely, before a request even reaches your Mac.

In the Cloudflare dashboard:

Go to Zero Trust → Access → Applications → Add an application
Choose Self-hosted
Set the domain to ai.yourdomain.com
Under Policies, create a rule: Emails → is → your@email.com

With this in place, any visitor to ai.yourdomain.com first sees a Cloudflare authentication page. Cloudflare sends a one-time code to their email. Only after that does the request reach Open WebUI.

This is free for up to 50 users on the Cloudflare Zero Trust free tier.

The full picture

At this point, the complete stack looks like this:

Layer	Component	Role
Model runtime	Ollama (Homebrew service)	Manages models, handles inference
Model	gemma4:e4b	The LLM itself, 9.6 GB on disk
Web interface	Open WebUI (Docker)	Browser UI, user management, RAG
Tunnel	cloudflared (Homebrew service)	Secure outbound connection to Cloudflare
Edge / TLS	Cloudflare	HTTPS termination, DNS, optional auth gate

Every component runs locally. The only outbound connections: cloudflared maintaining the tunnel to Cloudflare, and Ollama making no network calls at all after the initial model download.

Total monthly cost: zero — assuming you own a domain (typically €10–15/year) and use Cloudflare’s free tier.

What I noticed after extended use

Response latency on the M4 is genuinely good. The Neural Engine handles quantized inference well — typical responses start streaming within 2–3 seconds, with sustained throughput around 20–30 tokens/second for the e4b variant. That’s conversational speed.

The 128K context window is real and usable. I tested it with a long PDF pasted as text and the model held coherent references to content from the beginning throughout its response.

One practical note: brew services start ollama means Ollama runs in the background at all times and restarts after reboots. If you’d rather launch it manually, use brew services stop ollama and start it on demand with ollama serve.

Troubleshooting quick reference

Open WebUI shows “Ollama disconnected”

brew services restart ollama
# then refresh the browser

Cloudflare tunnel won’t start — credentials error

cloudflared tunnel login
# re-authenticate, then re-run the tunnel

ai.yourdomain.com returns a 502 error The tunnel is up but Open WebUI isn’t running. Check Docker:

docker ps | grep open-webui
docker start open-webui

Open WebUI container stopped after Mac restart The --restart always flag handles this — but Docker Desktop itself must be set to launch at login. Check Docker Desktop → Settings → General → Start Docker Desktop when you sign in.

Hardware: Mac Mini M4 2025, 16 GB RAM, macOS 26 Tahoe · Model: gemma4:e4b via Ollama · Interface: Open WebUI via Docker · Tunnel: Cloudflare