Project 03
JARVIS
It runs my laptop by voice, and gets faster the more I use it.
Three layers, and the model is only allowed in the middle one. Median time from my sentence to the machine moving: under a millisecond.
Tony Stark is the reason I started building things. Not the suit — the workshop. A man who could point himself at any problem and have something working by morning, because he had JARVIS holding the whole thing in its head while he did the thinking. I have wanted that since I was a kid. So in September I stopped admiring it and built one.
The first version was bad in an instructive way. I said "open youtube and play stand up video" and it tried to launch an application literally called youtube and play stand up video. I said "movie scene by karan aujla" and it typed that into Spotify's search box instead of finding the song already sitting in my library. Both failures had the same cause: the thing had captured my words instead of looking them up.
And underneath that was a worse problem. Every assistant puts a model in the hot path. I measured it on my own laptop: one cold model turn takes twenty-three seconds. Nothing you say after a twenty-three second pause sounds intelligent, no matter how clever the answer is. Speed is not a nice-to-have here — speed is the intelligence.
So I rebuilt it in three layers, each only handing upward what it genuinely cannot do. The bottom layer is pure code with no model anywhere near it: it takes your sentence, splits it into clauses, and looks each target up against a live catalogue of every app, folder, setting and site actually on the machine. Because it looks things up, a target can never swallow the rest of the sentence. Worst case across five thousand test sentences: twelve and a half milliseconds. Median: under one.
The middle layer is where it learns. Ask for something it cannot do and a model writes a recipe — but only out of forty-three verified building blocks, never raw shell commands. The recipe runs, the machine checks the thing actually happened, and only a recipe that verified gets saved. Ask again tomorrow and it replays as plain code, no model, no network. It genuinely gets faster the more you use it.
Two details I am quietly proud of. It speaks Hinglish as a first language — "youtube kholo aur stand up comedy chalao" resolves exactly like the English, because verbs are matched anywhere in a clause rather than assumed to sit at the front. And it can drive apps nobody wrote support for, by reading the window's own accessibility tree and clicking controls by name. Word, Discord, VS Code, Settings — no code per app.
The hardest engineering was the ears. A laptop that listens all the time hears its own speakers, so I feed what the machine is playing back in as a reference and subtract it from the microphone before anything downstream hears a thing — twelve decibels of my own audio removed, for two milliseconds of CPU per tenth of a second. And it learns my voice from how I already use it, switching on only once six clips agree, and built so it can never lock me out of my own computer.
Why it was hard
Voice assistants feel stupid for two reasons, and neither is the size of the model. First, they put a model in the hot path — one cold model turn on this laptop takes 23 seconds, and nothing said after a 23-second gap sounds intelligent. Second, they match what you said with rigid patterns, so "open youtube and play stand up video" becomes a request to launch an application literally named "youtube and play stand up video". Both are solvable, but only by refusing to let the model anywhere near the things the machine already knows how to do.
How it actually works
Three layers, and each one only hands upward what it genuinely cannot do itself. The bottom layer has no AI in it at all, which is exactly why it is instant.
It works in three layers and each one only passes upward what it genuinely cannot handle. The reflex arc is pure code and answers in twelve and a half milliseconds at its very worst, under one at its median. Above it, a model writes a new skill exactly once. Above that, it just talks to you.
1 receipt
- measured · source private84 commits. The architecture, the decision log and the audit scripts are all in it.
It does not try to understand the name of the thing you asked for — it looks it up in a live list of everything actually on the machine. That is why “open youtube and play stand up video” stops trying to launch an app called “youtube and play stand up video”.
The reflex layer never captures what you said — it looks it up. Your words are matched against a live catalogue of every installed app, settings page, folder and site on the machine, so a target can never swallow the rest of the sentence. Later clauses inherit the earlier target, which is what turns "open youtube and play stand up video" into two correct orders instead of one impossible one.
1 receipt
- measured · source privateCatalogue lookup rather than regex capture — the specific fix for the defect class that broke the previous version.
“youtube kholo aur stand up comedy chalao” works exactly as well as the English, because the verb is allowed to sit anywhere in the sentence instead of being assumed to come first.
It speaks Hinglish natively, not as a translation layer. "youtube kholo aur stand up comedy chalao" resolves identically to the English, because verbs are matched anywhere in the clause rather than assumed to be at the front.
1 receipt
- measured · source privateHindi word order handled in the resolver itself.
Ask for something it has never done and it writes itself a recipe — out of 43 pre-approved building blocks, never raw commands. It runs it, checks the thing really happened, and only keeps recipes that worked. Tomorrow the same request is plain code with no AI in it at all.
Ask for something it cannot do and a model writes a recipe from a fixed vocabulary of 43 verified building blocks. The recipe runs, the machine checks the result actually happened, and only a recipe that verified gets saved. Ask again and it replays as plain code — no model, no network. It literally gets faster the more you use it.
1 receipt
- measured · source private43 primitives in a registry; a plan may only name registry entries, never a shell command.
It can drive apps nobody wrote any support for, by reading the window the way a screen reader does and clicking things by their name. Word, Discord, VS Code, Settings — not one line of code written per app.
It is not a list of apps someone remembered to support. Anything the catalogue does not know gets driven through the accessibility tree — it reads the window's named controls and clicks them by name. That works in Word, Discord, VS Code, Settings, anything. No code per application.
1 receipt
- measured · source privateAccessibility-tree driving, with the window verified to be in front before any input is sent.
A laptop that listens all the time hears its own speakers. So it takes whatever the machine is playing, feeds it back in as a reference, and subtracts it before anything downstream hears a thing.
The hardest part was the ears. A laptop listening all the time hears its own speakers, so the system subtracts whatever the machine is playing out of the microphone before anything downstream hears it. And it learns your voice from how you already use it — holding the talk key, clapping it awake — switching on only once six clips agree, and built so it can never lock you out.
1 receipt
- measured · source privateLoopback echo cancellation plus a speaker-embedding gate that self-enrols and degrades to off with a visible badge.
The numbers
receipts
- measured · source privateMeasured across 5,052 sentences generated from this machine's own catalogue.
receipts
- measured · source privateAn audit script generates them from the real catalogue; a test is the gate.
receipts
- measured · source privateA plan may only name registry primitives, never a shell. A learned skill cannot become a learned mistake.
receipts
- measured · source privateWASAPI loopback as the far-end reference into echo cancellation.
receipts
- measured · source privateMeasured on the same laptop — the reason a model is never in the hot path.
receipts
- measured · source privateSingle user, runs on the machine, works with no key at all.
The part most people would have skipped
The first version of the resolver was fixed the way everyone fixes these — say a sentence, watch it break, patch that sentence. So I wrote a sweep that generates every phrasing the machine could possibly be asked, out of its own catalogue, and ran all 5,052 at once. The first run found six entire classes of bug in one go: a question's own words ending up inside the argument, a two-word Hindi command cut in half, a bracket in an app's name killing the lookup, a longer name resolving to a shorter one hiding inside it, a verb hiding at the start of a name, and — my favourite — "my desktop kholo" opening the gesture that minimises every window, because the possessive had been stripped as politeness. You cannot find those one sentence at a time.
What it can't do
- Windows only, and built for one person — mine. It enrols my voice, learns my catalogue, and is not multi-user.
- The source is private for now.
- Anything irreversible is refused rather than confirmed, so it will not delete or send things for you. That is deliberate.
- Both the echo cancellation and the voice gate degrade to off with a visible badge rather than silently — neither can stop the app starting.
- Python
- Windows accessibility APIs
- WASAPI loopback
- WebRTC echo cancellation
- speaker embeddings
- local wake word
- no cloud
