Marv turns an Obsidian vault into a hand-controlled AR workspace

A different way to navigate a growing vault

An Obsidian vault can become difficult to scan once notes, folders, and unfinished tasks accumulate. File trees and search are effective for deliberate retrieval, but they do not naturally provide a spatial overview of related work. Marv, an AR-style web application command center, appears designed to make that overview more immediate for someone working from a local vault.

The project places a 2D or 3D knowledge graph over a webcam feed. In that graph, folders, files, and tasks from an Obsidian vault become an interactive network around the user. Rather than treating the vault only as a set of documents to browse, Marv presents it as a visual workspace that can be explored with a hand and voice.

Its stated audience is an AI Engineer, though the underlying workflow could be relevant to builders who keep project context and tasks in a local Obsidian vault. The repository describes a setup that remains local in an important respect: it reads from a vault path supplied by the user. At the same time, its conversational AI feature requires a Google Gemini API key, which is a practical boundary for anyone assessing where their workflow and data connect to external services.

From webcam gesture to vault context

Marv's interaction model combines visual targeting, gesture recognition, and spoken questions. After opening the web app, the user starts the system to activate the webcam and microphone. A hand in the camera view produces a tracking cursor.

Pointing is the lightweight inspection gesture. According to the README, hovering an index finger over a folder cluster for about a second causes Marv to read out and analyze that cluster's contents. This gives the graph a behavior closer to a context-sensitive interface than a static visualization: the place a user indicates becomes the subject of the assistant's response.

Pinching supports a grab-and-drag action for moving graph nodes through the physical-space view. That distinction matters. Pointing is for asking about existing structure, while grabbing changes the arrangement of the visual representation. The documentation describes nodes as movable, but it does not say that these movements alter the underlying vault, so they should be understood as graph interactions rather than file-management operations.

Voice completes the loop. The project connects Marv to the Gemini Live API and describes multimodal streaming with vision and audio. A user can ask what they are pointing at, request a focus recommendation, or ask for a summary of pending tasks in a particular area of the vault. The README says Marv can see what the user is pointing at and hear the user's voice. It also says the assistant can offer recommendations, read pending tasks, and summarize context.

A small web stack with distinct jobs

The repository is primarily TypeScript, and its manifest confirms a Vite client alongside an Express backend. Development runs both processes concurrently: Vite serves the client application, while tsx watch runs server/index.ts. The README identifies the default client address as http://localhost:5173/ and the API as http://localhost:3001/.

On the visualization side, the documented stack uses D3.js for a force-directed graph. A force-directed graph lays out connected nodes by simulating forces such as attraction and separation. That is a sensible fit for the project's goal of showing folders, files, and tasks as a network rather than a rigid hierarchy. The README does not document the graph's data model or link rules, so it would be premature to assume exactly how relationships are derived.

Hand input is provided by MediaPipe Tasks Vision, confirmed by the @mediapipe/tasks-vision dependency. In Marv, this supports the hand tracking that enables pointing, hover behavior, and pinch-based dragging.

The AI layer uses @google/genai, which the README identifies as the Gemini Live API integration. Its documented role is multimodal vision and audio streaming. Express and CORS are listed as backend dependencies, while dotenv supports the local environment-variable setup described in the README.

The project expects two values in a root .env file: GEMINI_API_KEY and OBSIDIAN_VAULT_PATH. The latter is an absolute path to the local vault. Node.js 18 or later is recommended, and installation is the standard npm install followed by npm run dev.

Practical lessons in the interface

Marv illustrates a useful interaction-design idea: a graph becomes more useful when it is paired with a clear way to ask for context. Network visualizations can communicate breadth, but dense node layouts can also make direct reading difficult. Here, pointing selects the area that needs explanation, and voice supplies the question. The user does not have to formulate a search query before seeing the broader landscape.

There is also an operational tradeoff in the project's ambition. The app combines a local filesystem source, a live camera, a microphone, hand-tracking inference, graph rendering, a Node backend, and a Gemini API integration. That can create a compelling hands-on workflow, but it also means setup requires more than starting a typical note viewer. A usable local vault path, a Gemini API key, camera and microphone access, and a browser environment that supports the app's media needs are all part of the documented path.

For builders, the project is a concrete example of treating a personal knowledge base as an interactive input to an assistant, not just a text corpus. The evidence supplied does not establish how note content is selected, transmitted, or retained by connected services, so prospective users should inspect the implementation and consider their own privacy requirements before connecting a vault containing sensitive material.

Marv is a focused experiment in navigating notes and tasks through space, gesture, and conversation. Inspect the verified repository at https://github.com/MuLIAICHI/marv-ar-command-center to evaluate its local setup and implementation details.