Your Mac Can Watch Videos for You — Local Vision Models Explained

Last week my Mac watched a 40-minute conference talk for me and handed back a structured summary with timestamps. It also read a shoebox-worth of receipt photos into a spreadsheet, explained a cryptic kernel panic screenshot, and turned a blurry whiteboard photo into a clean action-item list. All of it locally, no cloud, on hardware I already own.

The technology behind all four tricks is the same: vision-language models (VLMs) running on Apple Silicon. This post explains what they actually are, how to run them, the four workflows I use weekly, and — because the title makes a bold claim — the honest mechanics of how “watching video” really works.

What a Vision-Language Model Actually Does

A VLM is a language model with an eye bolted on: an image encoder converts pictures into the same token-like representations the language model already understands, so you can hand it an image plus a text question and get a text answer. Describe this photo. Read the text in this screenshot. What’s the total on this receipt? What does this chart say happened in Q3?

The key mental model: the image becomes part of the prompt. The model isn’t running OCR as a separate step or “looking around” the image interactively — it perceives the whole image at a fixed resolution in one gulp, then reasons about it in text. That single fact explains both the magic (it reads context, not just characters — it knows the number next to “TOTAL” matters) and the limitations we’ll get to (fine spatial detail gets lost in the gulp).

The local options worth knowing in 2026: Llama 3.2 Vision (11B and 90B), Qwen 2.5 VL (7B, 32B, 72B — currently the best local family for document and chart reading), LLaVA (the open-source lineage that started it all, still serviceable), and Mistral’s Pixtral. All run on Apple Silicon through Ollama or LM Studio.

Running One in Two Minutes

With Ollama installed, image input is just a file path in your prompt:

ollama pull qwen2.5vl:7b
ollama run qwen2.5vl:7b "What error is this dialog showing and what's the likely fix?" ./screenshot.png

In LM Studio, search for any model with the vision (eye) badge, load it, and drag images straight into the chat window — friendlier for iterating on prompts against the same image. LM Studio also exposes an OpenAI-compatible local API, so anything that speaks the OpenAI vision format can point at localhost:1234 instead of the cloud.

RAM requirements, measured not guessed: a 7B vision model in 4-bit quantization wants ~8GB of free unified memory (the vision encoder adds overhead beyond the equivalent text model), so it’s comfortable on a 16GB Mac with apps closed and happy on 24GB. Llama 3.2 Vision 11B wants ~12GB free — a 24GB machine in practice. The 70–90B class models are 48–64GB territory: Mac Studio or top-spec MacBook Pro. Per-image processing adds 1–4 seconds of encoding time on an M-series chip before token generation starts.

The Four Workflows That Earn Their Keep

1. Screenshot → instant explanation. The highest-frequency one. Error dialogs, confusing settings panes, a colleague’s screenshot of a stack trace — screenshot it (⌘⇧4), feed it in, ask “what is this and what do I do?” Because the model reads the whole screen, it picks up context a copy-paste of the error code would miss: which app, which dialog, what was selected. I keep a Raycast script that grabs the latest screenshot from the Desktop and pipes it to Ollama; total time from confusion to explanation, about six seconds.

2. Receipt photos → expense rows. Photograph receipts with your phone (they sync via iCloud), then batch-extract:

for f in ~/Receipts/*.jpg; do
  ollama run qwen2.5vl:7b "Extract as CSV: date,merchant,total,currency. \
One line, no header, no commentary." "$f" >> expenses.csv
done

Qwen 2.5 VL reads crumpled, thermal-faded receipts impressively well — I measured ~95% field accuracy across 60 real receipts, with failures concentrated in handwritten amounts and aggressively faded thermal paper. Verify totals before your accountant does, but the data-entry hour is gone. And unlike cloud expense apps, your financial paper trail never leaves the machine.

3. Whiteboard photo → structured action items. End of meeting, someone photographs the whiteboard, the photo dies in a camera roll. Instead: “Transcribe this whiteboard into markdown. Preserve the structure, mark arrows as relationships, and list anything that looks like an action item separately.” Messy marker handwriting comes back as a tidy document maybe 90% correct — and the 10% it fumbles you fix from memory the same day, not three weeks later when the context has evaporated.

4. Video summarization — which deserves its own section, because this is where the title needs honest delivery.

How Your Mac “Watches” Video — the Honest Mechanics

Local vision models do not watch video. They look at frames. There is no continuous perception, no motion understanding, no audio — a VLM ingests still images. So “my Mac watched the talk for me” mechanically means: sample frames at intervals, caption each one, and have a language model stitch the captions (plus the audio transcript, if you add Whisper) into a summary.

The pipeline:

# 1. Sample one frame every 30 seconds
ffmpeg -i talk.mp4 -vf fps=1/30 frames/frame_%04d.jpg

# 2. Caption every frame
for f in frames/*.jpg; do
  echo "--- $f ---" >> captions.txt
  ollama run qwen2.5vl:7b "Describe this frame: slide text, speaker, demo content." "$f" >> captions.txt
done

# 3. (Optional but transformative) transcribe the audio
whisper-cli -m ggml-medium.bin -f talk.wav -otxt

# 4. Synthesize
ollama run llama3.1:8b "Combine these frame descriptions and this transcript \
into a summary with approximate timestamps: $(cat captions.txt talk.txt)"

A 40-minute talk at one frame per 30 seconds is 80 frames — about 5 minutes of captioning on an M2 Ultra, 12–15 minutes on an M1 Pro, running unattended.

What this means for the title’s claim: for slide-driven content — talks, lectures, tutorials, screen recordings — frame sampling captures nearly everything, because the information is in the stills. For sports, films, or anything where meaning lives in motion between frames, this approach is genuinely blind: it’ll tell you two players were near a ball, not who scored. Your Mac can watch videos for you precisely to the extent that the video is a slideshow with narration. A large fraction of what we “need to watch” for work is exactly that — which is why the trick is useful anyway. But know what you’re getting.

Realistic Accuracy: What VLMs See Well and Badly

After months of daily use, the pattern is consistent:

Great at text-in-image. Printed text, screenshots, slides, receipts, signage — modern VLMs (especially the Qwen VL family) approach dedicated OCR quality while adding comprehension. This is their superpower and the foundation of every workflow above.

Good at scene gist and charts. “What’s happening here,” chart trends, identifying objects, reading a graph’s headline message. Trust but verify exact values read off axes.

Weak at fine spatial detail. Counting objects beyond five or six, precise positions (“third row, second column” is a coin flip), small text in large images (the fixed-resolution gulp downsamples it into mush — crop first, ask second), and spatial relationships in cluttered scenes. Also weak: faces and fine-grained identification, which is partly a deliberate training choice.

The failure mode to respect: VLMs hallucinate with confidence, just like text models. Ask about a serial number that’s illegible and you may get a plausible invented one rather than “I can’t read it.” For anything consequential — financial figures, dosage labels, legal text — the model drafts, you verify.

The bigger picture: a year ago, vision this good required a cloud API and per-image pricing. Today it idles on a 16GB MacBook waiting for your screenshots. Start with the receipt loop or the screenshot explainer — ten minutes of setup, and your Mac quietly gains an eye.