Nova
Local AI assistant for macOS · Native SwiftUI · 2026 to present · In active development
The question
I wanted an assistant that could actually do things on my Mac, not just chat. And I wanted it to run on my own machine by default, instead of sending everything to the cloud.
What it does today
- Talks and listens. Push to talk records my voice, whisper.cpp transcribes it, and Nova answers out loud through a Piper neural voice. Speech starts as each sentence finishes, so she does not wait for the whole reply.
- Controls the Mac. About 37 actions: open and quit apps, check battery and system stats, set volume, read and write files, search the web, check the weather, play music on Spotify, and more. These run through AppleScript and NSWorkspace.
- Reads the screen. An overlay uses Apple's Vision OCR to read what is on screen, so I can ask about an error or a document. It can also grab selected text from any app.
- Remembers things. Nova saves facts I tell her and uses them in later chats. I can view and edit them in Settings.
- Knows her limits. For hard requests, an optional Cloud Assist mode hands the task to Claude and brings the answer back.
Nova is local by default. Cloud Assist is optional, and only it sends requests off the machine.
How it is built
- Native SwiftUI app for macOS, about 7,200 lines of Swift
- qwen3:14b through Ollama, with streaming replies
- whisper.cpp, built by hand as an XCFramework
- Piper for text to speech
- A custom HUD with animated amber reactor rings
- Runs on a MacBook Pro M4 with 24GB of RAM
What I learned
I tried rebuilding how Nova calls actions, with a typed tool registry, JSON tags, and then native tool calling. Each version was cleaner code, and each one made the model less likely to use its actions at all, until opening an app stopped working. I rolled back to the simpler design that worked. The lesson: with a local model, the format the model is good at matters more than how clean the code looks.
Next
A hands-free "Hey Nova" wake word, dictation into any app, an agent mode that runs multi-step tasks on its own, and moving to an in-process MLX model fine-tuned on Nova's own action format.