All technical writing

06 / UNDER THE HOOD

Searching WhatsApp Voice Notes on a CPU

Wawa turns the WhatsApp conversations you choose into tasks, actions, questions, reminders and things you’re waiting for. Each item links back to the messages it came from.

A lot of what matters in a WhatsApp chat isn’t text. It’s a photo, a screenshot, a voice note, a short video. Searching the text is easy. Searching the rest is not, because there’s no text to match.

So Wawa has Spotlight: press ⌘/Ctrl K and search across saved items and conversation messages, including what’s inside image, audio and video attachments. It runs on Ollama with EmbeddingGemma 2, and it works on a CPU.

Exact first, model second

Not every search needs a model. If you type a word that is literally in a message or a task, a lowercase substring match finds it. That works with no model installed at all.

const exact = docs.filter((doc) =>
  doc.text.toLocaleLowerCase().includes(query.toLocaleLowerCase()),
);

Exact matches come first, up to 50. Below them come up to 20 “related” results from embeddings, skipping anything that already matched exactly. If Ollama isn’t there, or the model call fails, you still get the exact results and a status that says related search is unavailable.

Two models

There are two EmbeddingGemma 2 models:

  • embeddinggemma-2:270m-q8_0-text for text: task titles, notes, steps, message bodies.
  • embeddinggemma-2:740m-q8_0 for media: images, audio, and video frames.

The full model is about 1.3 GB and shares its text weights with the smaller one. Queries are always embedded with the small text model, and that one query vector is compared against both the text index and the media index.

Both are called through Ollama’s /api/embed, asking for 256 dimensions. Vectors are normalised and stored as raw float32 blobs, so similarity is just a dot product in a loop.

body: JSON.stringify({ model, input, dimensions: 256, truncate: false, keep_alive: '10m' }),

Text is cut into 1,000-character chunks and sent with a title: ... | text: ... prompt. The query goes in as task: search result | query: .... A chunk only counts as related at a similarity of 0.7 or more.

Turning media into model input

The media model takes images and audio. Video isn’t an input type, so video becomes both.

An image goes in as-is, base64. Audio and video go through FFmpeg first. ffprobe checks which streams exist, then:

  • For video, ffmpeg samples one frame every 5 seconds, scaled down to at most 768 px wide.
  • For any audio track, ffmpeg converts to 16 kHz mono WAV and splits it into 30-second segments.
// Video frames
['-i', source, '-vf', "fps=1/5:start_time=0,scale='min(768,iw)':-2", '-q:v', '4', 'frame-%06d.jpg']

// Audio, 30-second segments
['-i', source, '-map', '0:a:0', '-vn', '-ac', '1', '-ar', '16000',
 '-f', 'segment', '-segment_time', '30', '-reset_timestamps', '1', 'audio-%06d.wav']

Each frame and each segment gets its own vector. A two-minute voice note is four vectors. A message’s media score is the best score of any of its parts, so one matching segment is enough to surface the whole message.

Audio and video are prepared in a temporary directory; the source file is written with mode 0600. The directory and its contents are deleted when the attachment is done. FFmpeg and FFprobe ship in the Docker image. Anywhere else, they need to be on the workspace machine.

Attachments come from Wawa’s local cache, or from the WhatsApp gateway if it still has them. There are limits: 20 MB per image, 100 MB per audio or video attachment. Anything bigger isn’t indexed.

How a media attachment enters the search indexA missing attachment is loaded from cache or the gateway. Images go directly to the media embedding model. Audio and video are probed first; video is sampled into frames every five seconds and audio is split into thirty-second segments. The 740 million parameter media model embeds the parts. Failures can be retried from the UI, and a changed digest causes reindexing. Queries first match exact substrings, then a 270 million parameter text embedding is compared with text and media indexes at different thresholds.ATTACHMENT INDEXINGNot indexedLoad attachmentlocal cache → gateway fallbackImageas-isAudio / videoffprobe streamsVideo: frame / 5sAudio: 30s partsvideo may have bothEmbed each part740m media modelIndexed in SQLiteChanged digest → remove vectors and reindexLoad, prepare or embed failsFailed → retry from UI → load againQUERY PATHExact substring results first270m text query embeddingText index: similarity ≥ 0.70Media index: similarity ≥ 0.65Related results follow exact matches
Images bypass FFmpeg; audio and video are prepared as separate searchable parts.

Indexing in the background

Nothing is indexed up front. The first search compares what should be in the index against what is, and starts a background job for the difference. Text goes to the model in batches of 8, media in batches of 4. Text chunks use a digest of the model, dimensions, server URL and text. Media uses the model, dimensions, server URL and message metadata, including image_digest; it does not hash the attachment bytes again on every search. When the relevant digest changes, the old vectors are dropped and the item is indexed again.

While a job is running, the search response says indexing with a count of indexed parts, and the UI asks again every second. While text indexing is running, you get exact results. Once text indexing finishes, related results can also use media parts that have already been indexed.

If an attachment can’t be loaded or processed, it’s marked failed rather than retried forever. A retry from the UI clears the failed marks and tries them again.

The index lives in the workspace’s SQLite database, in four tables. A workspace transfer drops them, along with the search server choice, so the restored copy builds its own index against its new host instead of trusting vectors from somewhere else.

Keeping text search fast while media indexes

The first version used one Ollama for everything. That works, but while one server is busy indexing media, text searches sent to it can be slower.

The fix was two Ollama workers in compose.search.yml, one for text and one for media, pointed at the same model volume so the weights are only downloaded once:

services:
  chat2tasks:
    environment:
      LOCAL_SEARCH_OLLAMA_URL: http://search-model:11434
      LOCAL_SEARCH_MEDIA_OLLAMA_URL: http://search-media-model:11434
  search-model:
    image: ollama/ollama:0.40.13
    volumes:
      - search-model-data:/root/.ollama
  search-media-model:
    image: ollama/ollama:0.40.13
    cpus: 4.0
    environment:
      OLLAMA_NUM_PARALLEL: '1'
    volumes:
      - search-model-data:/root/.ollama

The media worker is capped at 4 CPUs and one request at a time, so it can’t take the whole machine. Neither publishes a host port, and both set OLLAMA_NO_CLOUD. The text worker stays free to answer queries.

If you point Spotlight at a single Ollama server instead, it still works, but text searches can be slower while that server indexes media.

A lower cutoff for media

A text query against a media vector can be a relevant match and still score under 0.7, the cutoff used for text-against-text.

So media gets its own cutoff, 0.65. The test for it builds an image whose similarity to the query is 0.68 and checks it comes back as a related result. 0.65 is a hand-picked number, not the result of a measured evaluation.

Running it on CPU

On a Mac or a directly hosted workspace, install Ollama and pull both models once:

ollama pull embeddinggemma-2:270m-q8_0-text
ollama pull embeddinggemma-2:740m-q8_0

Everything here runs on a CPU. First-time media indexing is the slow part, and it’s slower than it would be on a GPU. I haven’t measured it, so I won’t put a number on it.

Docker Desktop on macOS can’t give a container the Mac’s GPU. If you have a machine with a GPU, Spotlight has an Another Ollama server option. Search queries, indexed text and media are then sent to that server, so it should be one you trust, over an encrypted connection if the traffic leaves the workspace host.

Search never uses the task-generation or decision models Wawa is configured with. Those settings still decide where task processing sends evidence, but Spotlight only talks to Ollama.

What it doesn’t do

  • No transcripts. An embedding of a voice note is a vector, not words. Spotlight can tell you a voice note is probably related to your query, but it can’t show you what was said, or which part matched.
  • Results are suggestions. Related results can miss something that’s there, and include something that isn’t relevant. Exact matches are the only ones you can fully trust.
  • No numbers. I haven’t measured recall, precision or indexing time. The cutoffs come from trying it, not from a benchmark.
  • Whole workspace. Search covers everything, regardless of the item filters in the workspace or the Chrome companion.
  • Size limits. Attachments over 20 MB (images) or 100 MB (audio and video) are skipped, and so is anything neither the cache nor the gateway still has.

The lesson

Search over a chat history doesn’t need a hosted vector database. One small embedding model, a substring match before it, vectors in the same SQLite file as everything else, and FFmpeg to turn audio and video into something the model can read. Keep the slow part, media, off the path the user is typing into.

The code is at github.com/liongkj/wawa.