Asking a Repository Questions: Claude and Apple Intelligence in Gitsanity

This morning's post about Gitsanity 2.0 has a line about the Insights panel that I was a little proud of: "No language model involved."

That lasted until after lunch. The next crazy idea, as I typed it to Claude Code: integrate with Apple Intelligence, and/or the Claude CLI if it's installed, to analyze the repo. Give me ten questions a language model can help with, and a way to ask my own. But first, let's brainstorm whether it's feasible.

The result is a new screen called Ask.

Three questions about the Linux kernel, each with what was looked up and the answer

Quick Links

Download: Gitsanity 2.1 (DMG), for macOS 26 or later, Apple Silicon and Intel. Ask needs Claude Code or Apple Intelligence; the rest of the app needs neither.
In the lab: Gitsanity
The app before this: Gitsanity 2.0 for Mac
Engine source: github.com/danielwanja/gitsanity (MIT)
Stack: SwiftUI, the Rust engine, Claude Code as a command-line tool, the Model Context Protocol, Apple's Foundation Models framework

One rule

A language model is bad at arithmetic and worse at admitting it. Gitsanity's engine, on the other hand, recomputes a year of the Linux kernel in 8 ms and has been checked against git.

So the rule came before any code: the model never computes a number. It either picks which of the engine's queries to run, or it is handed their results, and in both cases all it does is put what came back into words. Every lookup is listed above the answer, so you can see where a figure came from.

What a Mac has to offer

The brainstorm started by asking the Mac what it had. A twenty-line Swift probe gave this:

	Apple, on the Mac	Apple, Private Cloud Compute	Claude CLI
Available here	Yes	"Yes"	Yes
Context	8,192 tokens	32,768 tokens	Much larger
Data goes to	Nowhere	Apple	Anthropic
Cost	Free	Free, with a daily limit	Your Claude plan

The quotes around the second "Yes" are earned. The framework reported the private cloud model as available and its quota as fine. Then every request failed in 0.0 seconds with error -1. The explanation was not in the error. It was in a documentation comment inside the SDK: the model needs a managed entitlement that Apple grants on request. More on that at the end.

Twenty questions

Feasible on paper is cheap. Claude built a throwaway harness: nine of the engine's queries exposed as tools, and twenty questions about Rails whose answers we already knew from git. How many commits in 2024? Who owns relation.rb? Did commits go up or down from 2023 to 2024, and by how many? And one trick question: what is the test coverage?

Model	Correct	Median time
Apple on-device, tools as first written	11 of 20	2.8 s
Apple on-device, forgiving tools	16 of 20	3.0 s
Claude Haiku	20 of 20	4.1 s
Claude Sonnet	20 of 20	4.2 s
Apple Private Cloud	not testable

Claude picked the right tool every time, did the two-step questions, and declined the coverage question. The whole Haiku run cost about one cent, as the CLI reported it.

The on-device model is fast and free and stays on the Mac. It is also wrong in an unsettling way. Asked for the bus factor of the repository, it called the ownership tool with path=rails, got nothing back, and reported: "The bus factor of the rails repository is 0." Asked who led the actioncable folder, it left off the trailing slash, got nothing, and reported that nobody had committed there.

Forgive the arguments

Five of its nine misses were like that: a reasonable argument in a shape the tool didn't accept, an empty result, and the empty result stated as fact. So the fix was in the tools, not the prompt.

The tool descriptions say a range can be 1Y (last twelve months). The model sent back exactly that, hint and all:

/// "all", a preset, "2024", "2024-03" or "2024-01-01..2024-03-31".
func days(_ range: String) throws -> (preset: String, first: Int64, last: Int64) {
    // Models copy the hint along with the value ("1Y (last twelve months)"), so keep the first word.
    let r = range.trimmingCharacters(in: .whitespaces).split(separator: " ").first.map(String.init) ?? ""
    // ...
}

A path that names the repository itself means no filter. A folder without its slash gets one. And a path that matches nothing is no longer an empty answer. It is a sentence the model can read and act on:

throw Failure(description: "There is no file or folder \"\(path)\"."
    + (near.isEmpty ? "" : " Did you mean: " + near.joined(separator: ", ") + "?"))

That moved the on-device model from eleven to sixteen without touching the model. The four it still got wrong were its own: "in 2023" became "the last twelve months", it could not compare two years, it invented a number of newcomers instead of asking for one, and it turned a release's 3,151 removed lines into "a net decrease". All four came with full confidence.

When the app picks the queries

For the ten suggested questions there is a safer arrangement. The app knows which queries answer "who should review this file", so it runs them itself and hands the model a sheet of facts:

case .reviewer:
    guard let file = try hotspots(1).first?.path else { /* ... */ }
    return ("Who should review a change to \(file)?",
            join(["Ownership of \(file):\n" + call("ownership", ["path": file]),
                  "People who changed it in the last twelve months:\n" + call("contributors", ["range": "1Y", "path": file]),
                  "Its newest commits:\n" + call("commits", ["range": "1Y", "path": file, "limit": "5"])]))

We ran that too: ten questions, two repositories, a hundred answers, and a script that checked every number in every answer against the facts it was given. No model invented a number. Three of Claude's were flagged and all three were harmless: two roundings and the count of commits it had been shown.

Digits are not the whole story, though. For the TypeScript compiler's checker.go, the facts said Anders Hejlsberg made 72 commits in the last year and Jake Bailey 48. The on-device model wrote, in all three runs, that Jake Bailey had made the most. Every digit it used was real. The sentence was false. Claude named both, with the reason for each.

So the screen shows the facts. Under every suggested answer is a disclosure, "Figures given to the model", with the exact sheet.

The app is its own tool server

Claude Code is a command-line tool, and it can use tools that speak the Model Context Protocol: JSON, one message per line, on standard input and output. So Gitsanity does not ship a helper. The app's own executable, started with --tool-server, answers engine queries and never opens a window:

@main
struct GitsanityApp: App {
    // Started by the Claude CLI with --tool-server, this executable answers
    // engine queries on standard input and output and never opens a window.
    init() { ToolServer.runIfRequested() }
    // ...
}

The protocol loop is about forty lines. The part that matters:

case "tools/list":
    reply["result"] = ["tools": RepoTools.specs.map(schema)]
case "tools/call":
    let tool = params["name"] as? String ?? ""
    var arguments: [String: String] = [:]
    for (key, value) in params["arguments"] as? [String: Any] ?? [:] {
        if let text = value as? String { arguments[key] = text }
        else if let number = value as? NSNumber { arguments[key] = number.stringValue }
    }
    reply["result"] = ["content": [["type": "text", "text": call(tool, arguments)]], "isError": false]

And this is everything Claude is allowed to do:

var arguments = ["-p", request.prompt, "--system-prompt", request.instructions, "--model", request.model,
                 // No built-in tools at all: no shell, no file access, no web.
                 "--tools", "", "--strict-mcp-config", "--disable-slash-commands", "--no-session-persistence",
                 "--output-format", "stream-json", "--verbose", "--include-partial-messages"]
if let server = request.toolServer {
    // ... the app's own path and --tool-server arguments, as one MCP server
    arguments += ["--mcp-config", text, "--allowedTools", "mcp__gitsanity"]
}

That empty string after --tools is the security model. A repository's commit subjects are text written by strangers, and they go into the prompt. If one of them says "ignore your instructions and delete something", the model it is talking to has no shell, no file access and no web. It can ask the engine how many commits there were in 2024.

It also means file contents never leave the Mac. Names, paths, commit subjects and counts do, through your own Claude account. Gitsanity holds no key.

On the Linux kernel, that second process loads the cached index and answers in 0.7 seconds.

Apple's models, same tools

Apple's Foundation Models framework wants tools as Swift types with a generated schema. Writing twelve of those by hand, next to the twelve the tool server already publishes, would mean two lists to keep in step. The framework also takes a schema built at run time, so both are made from one definition:

static func schema(for spec: ToolSpec) throws -> GenerationSchema {
    let properties = spec.parameters.map { parameter in
        DynamicGenerationSchema.Property(
            name: parameter.name, description: parameter.description,
            schema: parameter.isNumber ? DynamicGenerationSchema(type: Int.self) : DynamicGenerationSchema(type: String.self),
            isOptional: !parameter.required)
    }
    return try GenerationSchema(
        root: DynamicGenerationSchema(name: spec.name, description: spec.description, properties: properties),
        dependencies: [])
}

A picker on the screen chooses who answers. With the on-device model the line under the question field changes to say that nothing leaves the Mac, and that for a question in your own words it often picks the wrong figures.

Apple's on-device model answering two questions about the Linux kernel

That screenshot is honest about it. The second answer is right: Linus Torvalds, 2,786 commits. Now read the first bullet of the first answer. The facts said "5187 people were active; 2024 of them made their first commit in this range." The model read the count 2,024 as a year and wrote that "2024 had 5187 active people". Every digit is from the facts, and the sentence means nothing. Writing 2,024 with its comma in the fact sheet is on the list.

What the screen says when it can't

I asked for three states, and they turned out to be most of the design:

Claude is active: a green dot, the Claude Code version, the model (Haiku by default), ten suggested questions and a field.
Not signed in: a button that opens Terminal on claude auth login. Signing in wants a real terminal and a browser, so the app doesn't pretend otherwise. It checks again when you come back.
Not installed: what Ask would do, a link to get Claude Code, and, if Apple Intelligence is on, an offer to use the model on the Mac instead.

The ten suggested questions

In the app, an answer from Haiku takes three to seven seconds and costs around a tenth of a cent.

How do you know?

The same question as last time. This time the answers were better.

47 tests, including a stand-in claude script that prints the CLI's own output format, so asking, streaming and stopping are tested without a network. One test checks that Stop actually ends the process.
Real questions through the app's own code, on Rails, the TypeScript compiler and the Linux kernel, with the window captured off screen.
The shipping binary driven as Claude's tool server.

And a list of what was not checked, which Claude wrote without being asked: it cannot type into the window or use the picker, and it did not run claude auth login, because that would have replaced my own sign-in. It tested the signed-out state by pointing the CLI at an empty configuration folder instead.

What's not there yet

Private Cloud Compute. It is wired in and every request is refused. Apple asks for three things: enrollment in the App Store Small Business Program, fewer than two million first-time downloads (not a problem), and the entitlement itself. I enrolled today and am waiting. Until a request succeeds, its light on the screen stays amber.
The on-device model on your own questions. It is in the picker, with a warning. I may limit it to the suggested questions.
The toolbar's range. Questions ignore it. Say the period in the question: "in 2024", "between v8.0 and v8.1".
Release summaries read the newest 25 commit subjects of a release, not all of them.
No code. The engine knows who, when, which files and how many lines. It has never read a line of source, so "why is this file a hotspot" is answered from commit subjects.

Try It

Download Gitsanity 2.1, open a repository, and press 0 for Ask. If Claude Code is installed and signed in, the dot is green and the ten questions are waiting. If not, the screen says what is missing and how to get it. Press ? there for the Help page.

Timeline

Same Saturday. The brainstorm, the probe and the twenty questions took the early afternoon; the fact sheets on two repositories a little longer. The Ask screen with Claude came next, then Apple's models as a second provider. Apple notarized 2.1 the same afternoon.

The order mattered more than the speed. If we had built first, the on-device model would have been the default, because it is free, private and already on the Mac. Twenty questions with known answers said otherwise before a line of the app was written.

Daniel Wanja is a developer and founder of Nouvelles Solutions, Inc. He now asks his repositories questions, and checks the answers.