Runtime Discovery

Runtime Discovery is a service that watches container runtimes (Docker, Kubernetes) for workloads and provides a gRPC API for querying them. In addition, it also resolves workloads using various resolvers (A2A, OASF) to extract and provide details about the workload's capabilities.

Architecture

The system is split into two independent components:

flowchart LR
 subgraph Discovery["Discovery (discovery/)"]
        RA["Runtime Adapters<br>• Docker<br>• Kubernetes<br>• Process"]
        RES["Resolvers<br>• A2A<br>• OASF"]
        SW["Store Writer"]
  end
 subgraph Server["Server (server/)"]
        SPACE[" "]
        API["gRPC API<br>/ListWorkloads"]
  end
    RA --> RES
    RES --> SW
    SPACE ~~~ API
    Discovery ~~~ Server
    SW -- write/manage --> SB["Storage Backend<br>• etcd (distributed)<br>• Kubernetes CRDs (native)"]
    SB -- read/watch --> API

    style SPACE fill:none,stroke:none

Components

Discovery (`discovery/`)

The discovery component is responsible for:

Watching runtimes for workloads with the org.agntcy/discover=true label. Supported runtimes:
- Docker: Watches Docker daemon for labeled containers.
- Kubernetes: Watches Kubernetes API for labeled pods/services.
- Process: Watches a directory of workload descriptor files written by processes running directly on the host.
- Extensible architecture allows adding more runtimes in the future.
Resolving workload metadata using configurable resolvers:
- A2A resolver: Extracts A2A agent card from workloads with org.agntcy/agent-type=a2a label.
- OASF resolver: Resolves OASF records from Directory for workloads with org.agntcy/agent-record=<fqdn> label.
- Extensible architecture allows adding more resolvers in the future.
Writing workloads to the storage backend (etcd or CRDs).

The storage backend can be used to expose discovered workloads to other components (e.g., clients/servers) without coupling them directly and to reduce attack surface.

Choose the storage backend based on where the components run:

etcd is recommended in non-Kubernetes environments where discovery and server run as separate services (e.g. Docker Compose or across hosts), since both only need network access to etcd.
CRDs can be used in Kubernetes environments for a more native experience, so clients can query workloads via both gRPC and the Kubernetes API.
SQLite (sqlite) suits a single host where discovery and server run as local binaries, e.g. a developer machine. They share a database file instead of requiring an etcd instance, so both must be configured with the same path. The file is opened in WAL mode, so several processes can use it at once: readers don't block the writer, and writes are serialized (one writer at a time, others wait up to 5s). This only works for processes on the same host using a local filesystem; don't put the file on a network filesystem (e.g. NFS) or share it between containers through Docker Desktop bind mounts, where file locking is unreliable. Use etcd in those cases.

flowchart LR
    n1["Runtime Discovery"] --> n3["Runtime Server (for querying workloads via gRPC)"] & n4["CRD (for querying workloads via Kubernetes API)"]

    n1@{ shape: rect}

Server (`server/`)

The server component provides a gRPC API for querying discovered workloads.

Example Setup

Build Container Images

IMAGE_TAG=latest task build

Docker Compose

# Deploy the stack
docker compose -f install/docker/docker-compose.yml up -d

# Deploy example workloads
docker compose -f install/examples/docker-compose.yml up -d

# Add discovery to all networks (required for resolvers to work)
docker network connect examples_team-a runtime-discovery
docker network connect examples_team-b runtime-discovery
docker restart runtime-discovery

# Query the API
grpcurl -plaintext localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads

# Cleanup
docker compose -f install/docker/docker-compose.yml down
docker compose -f install/examples/docker-compose.yml down

Kubernetes

The Helm chart supports both CRD and etcd storage backends.

Setup KIND Cluster

# Create cluster
kind create cluster --name runtime

# Load images into KIND
kind load docker-image ghcr.io/agntcy/dir-runtime-discovery:latest --name runtime
kind load docker-image ghcr.io/agntcy/dir-runtime-server:latest --name runtime

Deploy Example Workloads

kubectl apply -f install/examples/k8s.workloads.yaml

Deploy with CRD Storage

# Install the chart with CRD storage (default)
helm install runtime install/chart/

# Wait for pods
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=discovery --timeout=60s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=server --timeout=60s

# Query the gRPC API
kubectl port-forward svc/runtime-server 8080:8080 &
grpcurl -plaintext localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads

# Query the Kubernetes API to see discovered workloads
kubectl get dw

Deploy with etcd Storage

# Install the chart with etcd storage
helm install runtime install/chart/ \
  --set etcd.enabled=true \
  --set discovery.config.store.type=etcd \
  --set server.config.store.type=etcd

# Wait for pods
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=etcd --timeout=60s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=discovery --timeout=60s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=server --timeout=60s

# Query the gRPC API
kubectl port-forward svc/runtime-server 8080:8080 &
grpcurl -plaintext localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads

Cleanup

kind delete cluster --name runtime

Process (Host)

The process runtime discovers agents running directly on a host. Each process announces itself by writing a JSON descriptor file to a watched directory, and is removed when the file is deleted or the process exits. Discovery runs as a host binary, since it must see host PIDs (this does not work from a container on Docker Desktop).

Descriptor Format

One file per process at <dir>/<id>.json; the file name without .json is the workload ID. Descriptors must be regular files of at most 64 KiB; symlinks and other file types are ignored. See install/examples/process/hello-agent.json.

Field	Required	Description
`name`	yes	Workload name
`pid`	yes	Process ID; the workload is listed only while this process is alive
`locators`	no	Endpoints the process can be reached at, each `{"protocol": ..., "url": ...}` with a full URL, e.g. `{"protocol": "slim", "url": "slim://org/namespace/agent"}`
`ports`	no	Ports the process listens on (strings or numbers); turned into `tcp://<address>:<port>` locators
`addresses`	no	Addresses for `ports`; defaults to `["127.0.0.1"]`
`labels`	no	Must include `org.agntcy/discover=true`; resolver labels work as for containers
`annotations`	no	E.g. `org.agntcy/agent-record`

Discovery sets runtime and type to process, hostname to the host name, and isolationGroups to ["host"].

Locators

Every workload lists the endpoints it can be reached at in locators, each with a protocol and a full URL. For a process, they come from the descriptor's locators, plus one tcp locator per address and port. An agent that is not reachable over an address and port (for example one served over SLIM) only lists its locator. The A2A resolver probes tcp, http and https locators for an agent card and skips other protocols, while the OASF resolver still resolves org.agntcy/agent-record.

Trust

A descriptor is an unauthenticated claim: discovery checks that the PID is alive, not that the process is the agent its descriptor or record names. Anyone who can write to the directory can register a workload, so keep it private to the user (discovery creates it with mode 0700). Consumers that need to know an instance is genuine must verify it themselves, e.g. by probing the agent and checking a reply signed with the key its record names (see #94).

Write descriptors atomically (write a dotfile or <id>.json.tmp, then rename it to <id>.json) and delete them on clean shutdown. Discovery never modifies or removes descriptor files.

Run Locally

# Start a toy A2A agent serving an agent card on port 9999
mkdir -p /tmp/hello-agent/.well-known
echo '{"name": "Hello Agent", "description": "Example agent"}' > /tmp/hello-agent/.well-known/agent-card.json
python3 -m http.server 9999 --directory /tmp/hello-agent &
AGENT_PID=$!

# Create the descriptor directory, private to the user (discovery also creates it on start)
mkdir -p -m 700 ~/.agntcy/dir-runtime/workloads.d

# Start discovery and server, sharing a SQLite store
DISCOVERY_RUNTIME_TYPE=process DISCOVERY_STORE_TYPE=sqlite DISCOVERY_RESOLVER_OASF_ENABLED=false \
  go -C discovery run ./cmd &
SERVER_HOST=127.0.0.1 SERVER_STORE_TYPE=sqlite go -C server run ./cmd &

# Announce the agent
cat > ~/.agntcy/dir-runtime/workloads.d/hello-agent.json <<EOF
{
  "name": "hello-agent",
  "pid": $AGENT_PID,
  "labels": {"org.agntcy/discover": "true", "org.agntcy/agent-type": "a2a"},
  "ports": ["9999"]
}
EOF

# Query the API (services.a2a holds the agent card)
grpcurl -plaintext localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads

# Stop the agent; the workload disappears within the poll interval
kill $AGENT_PID

To resolve OASF records from a local Directory (dirctl daemon start), drop DISCOVERY_RESOLVER_OASF_ENABLED=false, set DIRECTORY_CLIENT_SERVER_ADDRESS=localhost:8888 and DIRECTORY_CLIENT_AUTH_MODE=insecure, and add "org.agntcy/agent-record": "<cid or name:version>" to the descriptor's labels or annotations.

The server is bound to 127.0.0.1 here because it has no authentication: with the default SERVER_HOST=0.0.0.0 anyone who can reach the machine can list its workloads. To share workloads with other machines, put transport security and access control in front of the server, and list locators (or addresses and ports) in the descriptors that those machines can reach (the 127.0.0.1 default is only reachable locally).

The walkthrough uses SQLite so no extra service is needed; etcd works the same way by running etcd and setting DISCOVERY_STORE_TYPE=etcd and SERVER_STORE_TYPE=etcd.

Docker and process discovery on the same machine can share one store and one server: on startup each instance only removes stale workloads of its own runtime. See Sharing a Store for several instances of the same runtime.

Several discovery instances can write to the same store, and one server then lists all their workloads. On startup, discovery removes stored workloads its runtime no longer reports, but only those it owns: workloads of the same runtime type and the same instance ID.

Different runtimes (e.g. Docker and process discovery on one machine) need no configuration.
Several instances of the same runtime (e.g. Docker discovery on two hosts writing to one etcd) must each set a unique, stable DISCOVERY_INSTANCE_ID. Discovery records it on every workload it writes in the org.agntcy/discovery-instance annotation. The annotation is reserved: a value supplied by the workload itself (e.g. in a process descriptor or pod annotations) is replaced, or removed when no instance ID is set. Do not derive it from a container or pod hostname, since those change when the container is recreated and the new instance would no longer clean up the old one's workloads.

Workload IDs must be unique across all instances sharing a store; an instance that reports an existing ID overwrites that workload. Container IDs and pod UIDs are unique in practice; for the process runtime, use descriptor file names that are unique across hosts (e.g. a UUID or <host>-<agent>).

Workload Labels

Workloads are discovered based on labels. The discovery component watches for workloads with specific labels and processes their metadata.

Discovery Labels

Label	Runtime	Description
`org.agntcy/discover=true`	Kubernetes	Marks a pod/service for discovery
`org.agntcy/discover=true`	Docker	Marks a container for discovery
`org.agntcy/discover=true`	Process	Marks a process descriptor for discovery

Resolver Labels

Resolvers extract metadata from discovered workloads based on their labels. They provide additional information about the workload's capabilities.

Label/Annotation	Description
`org.agntcy/agent-type=a2a`	Enables A2A resolver - fetches A2A agent card from workload
`org.agntcy/agent-record=<fqdn>`	Enables OASF resolver - resolves record from Directory (e.g., `my-agent:v1.0.0`)

To configure OASF resolver, the Directory client must be set up using environment variables (e.g., DIRECTORY_CLIENT_SERVER_ADDRESS, DIRECTORY_CLIENT_AUTH_MODE).

Workload Services

Discovered workloads have a services field that holds metadata extracted by resolvers:

{
  "id": "4467371c-84fd-4683-ab30-93895d78bab7",
  "name": "service-a2a",
  "hostname": "service-a2a",
  "runtime": "kubernetes",
  "type": "pod",
  "labels": {
    "app": "service-a2a",
    "org.agntcy/discover": "true",
    "org.agntcy/agent-type": "a2a",
    "org.agntcy/agent-record": "my-agent:1.0.0"
  },
  "locators": [
    {"protocol": "tcp", "url": "tcp://10-244-0-9.team-a.pod:8080"},
    {"protocol": "tcp", "url": "tcp://10-244-0-9.team-a.pod:9999"}
  ],
  "isolationGroups": [
    "team-a"
  ],
  "services": {
    "a2a": {
      "name": "My Agent",
      "description": "...",
      "capabilities": [...]
    },
    "oasf": {
      "cid": "baf123",
      "name": "my-agent:1.0.0",
      "record": {
        "name": "my-agent",
        "version": "1.0.0",
        "skills": [...]
      }
    }
  }
}

Configuration

Discovery Component

Environment Variable	Description	Default
`DISCOVERY_WORKERS`	Number of resolver workers	`16`
`DISCOVERY_INSTANCE_ID`	Instance ID that scopes workload cleanup when several instances of the same runtime share a store (see Sharing a Store)	``
`DISCOVERY_STORE_TYPE`	Storage type (`etcd`, `crd`, `sqlite`)	`etcd`
`DISCOVERY_STORE_ETCD_HOST`	etcd server hostname	`localhost`
`DISCOVERY_STORE_ETCD_PORT`	etcd server port	`2379`
`DISCOVERY_STORE_ETCD_USERNAME`	etcd username for authentication	``
`DISCOVERY_STORE_ETCD_PASSWORD`	etcd password for authentication	``
`DISCOVERY_STORE_ETCD_DIAL_TIMEOUT`	Timeout for connecting to etcd	`5s`
`DISCOVERY_STORE_ETCD_WORKLOADS_PREFIX`	etcd key prefix for workloads	`/discovery/workloads/`
`DISCOVERY_STORE_CRD_NAMESPACE`	Namespace to store workloads in	`default`
`DISCOVERY_STORE_CRD_KUBECONFIG`	Path to kubeconfig file (empty for in-cluster)	``
`DISCOVERY_STORE_CRD_RESYNC_PERIOD`	How often to resync the cache from the API server	`30s`
`DISCOVERY_STORE_SQLITE_PATH`	SQLite database file shared with the server (`~` expands to home)	`~/.agntcy/dir-runtime/workloads.db`
`DISCOVERY_RUNTIME_TYPE`	Runtime type (`docker`, `kubernetes`, `process`)	`docker`
`DISCOVERY_RUNTIME_DOCKER_HOST`	Docker daemon socket path	`unix:///var/run/docker.sock`
`DISCOVERY_RUNTIME_DOCKER_LABEL_KEY`	Label key to filter containers	`org.agntcy/discover`
`DISCOVERY_RUNTIME_DOCKER_LABEL_VALUE`	Label value to filter containers	`true`
`DISCOVERY_RUNTIME_KUBERNETES_KUBECONFIG`	Path to kubeconfig file (empty for in-cluster)	``
`DISCOVERY_RUNTIME_KUBERNETES_NAMESPACE`	Namespace to watch (empty for all namespaces)	``
`DISCOVERY_RUNTIME_KUBERNETES_LABEL_KEY`	Label key to filter pods	`org.agntcy/discover`
`DISCOVERY_RUNTIME_KUBERNETES_LABEL_VALUE`	Label value to filter pods	`true`
`DISCOVERY_RUNTIME_PROCESS_DIR`	Directory containing workload descriptor files (`~` expands to home)	`~/.agntcy/dir-runtime/workloads.d`
`DISCOVERY_RUNTIME_PROCESS_POLL_INTERVAL`	How often to rescan the descriptor directory	`2s`
`DISCOVERY_RUNTIME_PROCESS_LABEL_KEY`	Label key to filter process descriptors	`org.agntcy/discover`
`DISCOVERY_RUNTIME_PROCESS_LABEL_VALUE`	Label value to filter process descriptors	`true`
`DISCOVERY_RESOLVER_A2A_ENABLED`	Enable A2A resolver	`true`
`DISCOVERY_RESOLVER_A2A_TIMEOUT`	Timeout for A2A discovery	`5s`
`DISCOVERY_RESOLVER_A2A_PATHS`	Comma-separated list of paths to probe for A2A discovery	`/.well-known/agent-card.json,/.well-known/card.json`
`DISCOVERY_RESOLVER_A2A_LABEL_KEY`	Label key to identify A2A workloads	`org.agntcy/agent-type`
`DISCOVERY_RESOLVER_A2A_LABEL_VALUE`	Label value to identify A2A workloads	`a2a`
`DISCOVERY_RESOLVER_OASF_ENABLED`	Enable OASF resolver	`true`
`DISCOVERY_RESOLVER_OASF_TIMEOUT`	Timeout for OASF resolution	`5s`
`DISCOVERY_RESOLVER_OASF_LABEL_KEY`	Label key to identify OASF workloads	`org.agntcy/agent-record`

Discovery OASF Resolver

The OASF resolver requires Directory client configuration via environment variables. These are the same as those used by the Directory client library, e.g. DIRECTORY_CLIENT_SERVER_ADDRESS. Refer to the Directory Client documentation for all available options.

When a workload has the configured OASF resolver label, the resolver attempts to fetch the corresponding record from Directory and validate its signature before attaching it to the workload.

Server Component

Environment Variable	Description	Default
`SERVER_HOST`	Server bind address	`0.0.0.0`
`SERVER_PORT`	Server listen port	`8080`
`SERVER_STORE_TYPE`	Storage type (`etcd`, `crd`, `sqlite`)	`etcd`
`SERVER_STORE_ETCD_HOST`	etcd server hostname	`localhost`
`SERVER_STORE_ETCD_PORT`	etcd server port	`2379`
`SERVER_STORE_ETCD_USERNAME`	etcd username for authentication	``
`SERVER_STORE_ETCD_PASSWORD`	etcd password for authentication	``
`SERVER_STORE_ETCD_DIAL_TIMEOUT`	Timeout for connecting to etcd	`5s`
`SERVER_STORE_ETCD_WORKLOADS_PREFIX`	etcd key prefix for workloads	`/discovery/workloads/`
`SERVER_STORE_CRD_NAMESPACE`	Namespace to read workloads from	`default`
`SERVER_STORE_CRD_KUBECONFIG`	Path to kubeconfig file (empty for in-cluster)	``
`SERVER_STORE_CRD_RESYNC_PERIOD`	How often to resync the cache from the API server	`30s`
`SERVER_STORE_SQLITE_PATH`	SQLite database file shared with discovery (`~` expands to home)	`~/.agntcy/dir-runtime/workloads.db`
`SERVER_OASF_SCHEMA_URLS`	Comma-separated OASF schema servers whose A2A card extensions are read when filtering by agent; `https`, or plain `http` on a loopback host (e.g. a local schema server)	`https://schema.oasf.outshift.com`

gRPC API

The server exposes a gRPC API defined in proto/agntcy/dir/runtime/v1/discovery_service.proto.

GetWorkload

Get a specific workload by ID, name, or hostname.

grpcurl -plaintext -d '{"id": "my-service"}' \
  localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/GetWorkload

ListWorkloads

Stream workloads, optionally filtered. Filters of different kinds combine with AND; a filter with several values matches any of them.

Field	Matches
`labels`	Workload labels; values are regular expressions matched anywhere in the label value
`runtimeTypes`	`runtime` (`docker`, `kubernetes`, `process`)
`reachableFromWorkloadId`	Workloads sharing an isolation group with that workload, within the same runtime and discovery instance, excluding it
`locatorProtocols`	A locator's `protocol` (`slim`, `grpc`, `http`, `tcp`, …)
`names`	The agent's name: the A2A card's `name`, else the OASF record's
`skills`, `domains`	The agent's OASF skills and domains
`authors`	The DID of the human vouching for the agent (A2A card), else the OASF record's authors
`schemaVersions`	The OASF schema version of the skills and domains

Names, skills, domains, authors and schema versions match as in Directory record search: exactly (case-insensitive), or with * and ? wildcards, e.g. natural_language_processing/*.

The agent filters read what the workload says about the agent it runs:

A2A card (services.a2a): the discovery extension (skills, domains, schema_version) and the identity extension (human_did) in capabilities.extensions, recognised by a uri of the form <schema URL>[/<version>]/objects/discovery (or /objects/identity) on a schema server listed in SERVER_OASF_SCHEMA_URLS. Extensions from other servers are ignored: dir-runtime doesn't validate the payload, so the list keeps workloads on one vocabulary.
OASF record (services.oasf), resolved from org.agntcy/agent-record, for whatever the card doesn't provide. A record the Directory doesn't know (for example one kept on a private node) is not an error: the workload is listed without it, and the resolver doesn't retry.

When both are present, the card wins. Values are claims made by whoever deployed the workload; nothing is verified (see #94).

# List all workloads
grpcurl -plaintext -d '{}' \
  localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads

# Filter by labels (supports regex)
grpcurl -plaintext -d '{"labels": {"org.agntcy/agent-type": "a2a"}}' \
  localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads

# Agents with a debugging skill reachable over SLIM
grpcurl -plaintext -d '{"skills": ["software_engineering/code_debugging/*"], "locatorProtocols": ["slim"]}' \
  localhost:8080 agntcy.dir.runtime.v1.DiscoveryService/ListWorkloads