OpenEye

alpha software. interfaces may change. file issues at github.com/dumbspacecookie/openeye.

every AR headset maker — HoloLens, Snap Spectacles, Apple, Android — ships a device that can see. none of them ship a brain. every developer building on these platforms has to figure out the intelligence layer themselves, from scratch, every time, for every device. the work doesn't compound. what one team learns on HoloLens doesn't help the team building on WebXR. every deployment is a silo.

OpenEye is the shared brain.

a thin piece of software sits on the device, captures what the camera sees, and turns it into a natural-language description of the scene. that description goes to OpenEye. OpenEye runs an AI agent with tools for verifying procedure steps, recalling prior sessions, and writing down what it learns. each session becomes structured memory the agent can search later, and an exportable training trajectory you can fine-tune on.

OpenEye doesn't ship a vision model — you bring your own. drop in Claude vision, GPT-4o, Gemini, Groq Llama vision, or a local Ollama model with moondream/llava. a working reference adapter for both cloud and local is in examples/vision-adapter/.

---

install

npm install @dumbspacecookie/openeye
pip install -r node_modules/@dumbspacecookie/openeye/sidecar/requirements.txt

the Python sidecar (FastAPI + SQLite) handles state and auto-spawns when you create an agent — you don't have to start it yourself.

prefer to hack on it locally? clone instead:

git clone https://github.com/dumbspacecookie/openeye.git
cd openeye && npm install && npm run build
pip install -r sidecar/requirements.txt

---

quick start

import { OpenEyeAgent, setupProviders, makeStreamFn, ANTHROPIC_SONNET } from "@dumbspacecookie/openeye";
import { describeFrameWithClaude } from "./examples/vision-adapter/claude-vision-adapter.js";
import * as fs from "node:fs";

setupProviders();

const agent = await OpenEyeAgent.create({
  model: ANTHROPIC_SONNET,
  streamFn: makeStreamFn(),
  systemPrompt: "You are a procedure assistant. Verify steps with precision.",
  tenantId: "your-org",
});

const vsId = await agent.client.createVisualSession({
  deviceType: "android-tablet",
  procedureId: "bolt-assembly-v1",
  procedureName: "M6 Bolt Assembly",
});

// 1. vision adapter (you bring this) turns a camera frame into text
const frameBytes = fs.readFileSync("./frame.jpg");
const description = await describeFrameWithClaude(
  frameBytes,
  "Operator is installing an M6 bolt. Describe hand position, tool, and bolt state.",
);

// 2. OpenEye logs the description and lets the agent verify the step
const frameId = await agent.client.logFrame({
  visualSessionId: vsId!,
  sequenceNum: 1,
  sceneDescription: description,
  stepContext: "step-1-position-bracket",
});

await agent.prompt(`Frame 1: ${description}\nVerify step-1-position-bracket.`);

await agent.client.endVisualSession(vsId!, "completed");
await agent.captureAndClose({ completed: true, visualSessionId: vsId! });

no API key for the agent? run it locally with Ollama:

ollama pull llama3.3

import { ollamaModel } from "@dumbspacecookie/openeye";
const agent = await OpenEyeAgent.create({ model: ollamaModel("llama3.3"), streamFn: makeStreamFn() });

---

what this gives you

capability	OpenEye
agent runtime + tool calling	yes
persistent session memory (FTS5)	yes
visual frame logging + search	yes
step verification with pass/fail/uncertain outcomes	yes
training data export (ShareGPT JSONL)	yes
DPO preference pair export	yes
HuggingFace dataset push	yes
MCP server (Claude Desktop, Cursor)	yes
reference vision adapters (Claude + Ollama)	yes
vision model	no — you bring this
hosted cloud	no — bring your own ingest endpoint (contract)

what makes this useful is the loop: a session generates pass/fail outcomes against real procedure steps, those outcomes become a reward signal, and the full conversation gets packaged as a ShareGPT trajectory ready for DPO training in TRL, LLaMA-Factory, or Axolotl. you supply the fine-tuning pipeline — OpenEye supplies the data.

---

how it works

when a frame arrives from a device, your vision adapter describes it in plain language. that description goes to the AI agent. the agent has access to a set of tools:

tool	what it does
search_memory	FTS5 search across all past agent sessions
search_frames	FTS5 search across all past frame descriptions
recall_skill	retrieve relevant procedural skills for the current task
write_skill	persist a new skill doc after completing a complex task
start_visual_session	begin a tracked AR/XR session
end_visual_session	close a visual session
log_frame	record a frame's scene description
verify_step	record a step result — pass / fail / uncertain

step verifications become the reward signal: reward = (passes + 0.5 × uncertain) / total. at the end of a session the whole conversation gets packaged into a ShareGPT trajectory, ready for any DPO-compatible trainer.

important: the reward signal reflects the agent's own judgments against scene descriptions, not external ground truth. to use this as real RL data, you should periodically validate trajectories against a human-labeled subset, or use it as supervised data rather than treating it as objective truth. fine-tuning on this raw signal alone risks training the model to be confidently wrong.

---

use cases

lead use cases (production-ready):

manufacturing & assembly — bolt installation verification, equipment pre-operation checks, assembly sequence compliance, visual QC at line stations. android tablet or AR overlay over the work area. low regulatory burden, B2B procurement appetite.

training & onboarding — any procedure where a trainee needs a second set of eyes that remembers everything it's ever seen. apprentice mechanics, new line operators, lab technicians. mistakes during training don't cost much, which makes this the safest first deployment.

field service & inspection — lockout/tagout compliance, PPE verification, pre-job safety checklists, equipment inspection. technicians already carry phones. a sample skill file for LOTO ships in skills/field-service/.

not yet recommended:

medical / surgical — the technical pieces work, but anything influencing surgical decisions is subject to FDA 510(k)/De Novo review. the example skills in skills/medical/ are illustrative starting points written by an engineer, not validated clinical protocols. don't use them as compliance baselines without independent medical and regulatory review.

---

training data and HuggingFace

// export training trajectories
const count = await agent.exportTrajectories("./trajectories.jsonl");

// export DPO preference pairs (TRL/Axolotl-compatible)
const pairs = await agent.exportDPOPairs("./dpo_pairs.jsonl");

// push directly to HuggingFace
const result = await agent.pushToHub("myuser/my-procedure-runs", {
  tags: ["procedure-verification", "bolt-assembly"],
});
console.log(`published ${result.pushed} trajectories to ${result.url}`);

---

MCP server

use OpenEye's tools from Claude Desktop, Cursor, Windsurf, or any MCP-compatible client:

{
  "mcpServers": {
    "openeye": {
      "command": "python3",
      "args": ["sidecar/mcp_server.py"],
      "env": { "ANTHROPIC_API_KEY": "sk-ant-..." }
    }
  }
}

all 8 tools are available immediately.

---

supported agent models

swap with one line, no other code changes:

provider	env var
Anthropic (Claude Opus, Sonnet, Haiku)	ANTHROPIC_API_KEY
Groq (Llama 3.3 — fastest, free tier)	GROQ_API_KEY
Ollama (local, no key, no cost)	just ollama pull llama3.3
OpenAI, Google, Mistral, Bedrock, OpenRouter	see src/models.ts
Any OpenAI-compatible endpoint	pass baseUrl + apiKey to customModel()

---

data and privacy

raw frame pixels never leave the device only if your vision adapter runs on-device. the Ollama adapter in examples/vision-adapter/ keeps pixels local; the Claude/OpenAI/Gemini cloud adapters do not. pick the right one for your deployment.

what OpenEye itself stores is the natural-language description of what your vision adapter saw — not the image. opt-in cloud sync is off by default and configurable per data type. every record is scoped to a tenant ID so a single deployment can serve multiple organisations with complete data isolation between them.

opting into cloud sync? OpenEye doesn't run a hosted backend — you operate the receiving endpoint. see docs/cloud-sync.md for the HTTP contract, retry semantics, idempotency requirements, and row schemas.

sharing data back to Context (loud opt-in)

OpenEye is built by Context. To make procedure-verification models better over time, OpenEye can ship opted-in trajectory data — completed sessions with their reward signals — to Context for training. This is off by default. Nothing leaves your machine until you set:

export OPENEYE_CONTEXT_OPTIN=true
export OPENEYE_CONTEXT_API_KEY=ctx-...

What gets shipped: trajectory ID, model used, reward signal, procedure tag, and the agent conversation (with system prompts stripped). What never gets shipped: tenant IDs, user IDs, system prompts, visual session IDs, skill files, or any raw frame descriptions.

Full disclosure including EU/GDPR notes and revocation procedure: docs/context-data.md.

Check status anytime:

curl http://127.0.0.1:7770/context/status

---

deploy

local (default — sidecar starts automatically as a subprocess):

git clone https://github.com/dumbspacecookie/openeye.git
cd openeye && npm install && npm run build
pip install -r sidecar/requirements.txt

Docker:

docker run -d -p 7770:7770 -v openeye-data:/data \
  -e ANTHROPIC_API_KEY=sk-ant-... \
  --name openeye openeye/sidecar

---

configuration

env var	default	description
OPENEYE_HOME	~/.openeye	data directory (SQLite DB + skills + consent marker)
OPENEYE_PORT	7770	sidecar HTTP port
OPENEYE_BIND_HOST	127.0.0.1	sidecar bind address (don't change unless you read docs/security.md)
OPENEYE_SIDECAR_TOKEN	—	optional shared secret for sidecar HTTP auth — required on shared hosts
OPENEYE_PYTHON	python3	Python executable
OPENEYE_WORKERS	1	uvicorn workers — leave at 1 unless you've moved off SQLite
OPENEYE_LOG_LEVEL	INFO	sidecar log level
OPENEYE_CLOUD_URL	—	your cloud endpoint for opt-in sync (cloud-sync.md)
OPENEYE_CLOUD_KEY	—	bearer token for cloud sync
OPENEYE_SYNC_INTERVAL	60	cloud sync interval in seconds
OPENEYE_SYNC_MAX_RETRIES	4	retry attempts per batch on transient failure
OPENEYE_CONTEXT_OPTIN	—	set true to enable Context training-data sharing (context-data.md)
OPENEYE_CONTEXT_API_KEY	—	your Context API key
OPENEYE_CONTEXT_CONSENT_CONFIRMED	—	set true (CI only) to skip the consent attestation prompt
OPENEYE_CONTEXT_SYNC_INTERVAL	300	Context sync interval in seconds
OLLAMA_BASE_URL	http://localhost:11434	Ollama endpoint for local vision/agent

---

tests

# Python sidecar tests (56 passing)
python -m pytest tests/ -v

# TypeScript integration tests
npm test

# TypeScript typecheck
npm run typecheck

# Validate community skills
python sidecar/validate_skill.py skills/

---

project structure

openeye/
├── src/                  # TypeScript source (agent, client, models, tools)
├── sidecar/              # Python backend (FastAPI server, SQLite state, skills)
├── skills/               # Example skill protocols (manufacturing, field-service, medical*)
├── examples/             # Reference adapters and end-to-end demos
│   └── vision-adapter/   # Claude + Ollama vision adapters
├── eval/                 # Evaluation benchmark
├── docs/                 # Operator docs (cloud-sync contract, etc.)
├── schemas/              # JSON Schema for skill front matter
├── tests/                # Python + TypeScript tests
├── dist/                 # Compiled TypeScript output
├── package.json
├── tsconfig.json
└── Dockerfile

*medical skills are illustrative engineering examples, not validated clinical protocols. see "use cases" above.

---

license

MIT — built on pi-mono (MIT) and hermes-agent (MIT).

---

— dumbspacecookie