Designing for Visual Intelligence

Status. The sample capability described on this page, sarja/visual-intelligence-sample, is not listed in the registry at the moment (the registry lists only the Example .adcap; see getting started). This page documents the design and what the sample was built to show. It does not include install commands or outputs, because there is nothing to run today.

Visual Intelligence is intended to become a major Arcotheon capability: thousands of videos and images, searchable embeddings, extracted techniques and their relationships, reusable creative knowledge. Nothing like that can live in a package. The format is designed so it does not have to.

This project does not build or train Visual Intelligence. It ships a small, honest sample that exercises every mechanism the real one will need.

The shape#

package (small, signed)                         remote store (R2 / any HTTPS)
  capability.json ──resources──▶ shard map (hash-pinned) ──▶ shard 0001 … shard N   (hash-pinned each)
  skills/  tools/  workflows/                    media indexes, embeddings, technique graphs
  knowledge/ (core, hand-curated)                video/image files (referenced, fetched only if a tool needs one)

• The manifest pins one small file: a shard map. The map lists every shard's path, SHA-256 and size, plus whatever routing data helps a client choose shards.
• A tool asks the host for the shards a query needs. The host fetches them from any mirror, verifies each against the map, and caches them. A user who asks about camera motion downloads the motion shards and nothing else.
• Big media (the videos themselves) are further resources referenced from shard records by URL and hash and fetched only when a tool asks for a specific clip or frame.
• Everything is verified through the chain signed manifest → map → shard → media, so the remote store needs no trust.
• Tools run with no network: the host does the fetching. The same tools can serve any agent through MCP.
• Entitled or Pro content uses access: "entitled" on the resource; the host's authoriser decides.
• Updating the dataset means publishing a new shard map and a new package version that pins it. Old versions keep working because their shards are content-addressed and immutable.

The sample: sarja/visual-intelligence-sample (not currently listed)#

What it is not#

• The embeddings are lexical feature hashes, not a neural model. Queries must use the vocabulary of photography and film. A real deployment would declare a neural embedding model as an adapter and a model resource; the manifest already supports that (see section 10 of the specification), but no model is shipped here.
• The image analysis covers 8-bit non-interlaced PNGs and its hints are heuristics with their evidence shown.
• The video and audio assets are synthetic test media: nothing here analyses video yet.
• The 32 techniques are a small curated set, not extracted from a corpus.

Trying it#

There is nothing to install while the sample is unlisted. When it is listed again it installs and runs like any other capability: see getting started.

Open design work for the real thing#

• Large media access (range reads, thumbnails, keyframe indexes) beyond whole-file resources.
• A neural embedding adapter and ANN index format, and how a host chooses a local or hosted inference backend.
• Provenance and licensing metadata per media record, enforced at retrieval.
• Entitlement service integration for Pro shards.
• Corpus extraction, curation and evaluation: out of scope here.