Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 documentation-specialist
|
1496da1abf | ||
|
documentation-specialist
|
534d48005a | ||
|
documentation-specialist
|
cf20fcdfe7 |
@@ -7,7 +7,6 @@ on:
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
|
||||
@@ -42,7 +42,7 @@ Common runtime environment variables:
|
||||
```bash
|
||||
WORKSPACE_ID=ws-123
|
||||
WORKSPACE_CONFIG_PATH=/configs
|
||||
PLATFORM_URL=http://platform:8080
|
||||
PLATFORM_URL=http://host.docker.internal:8080
|
||||
PARENT_ID=
|
||||
AWARENESS_URL=http://awareness:37800
|
||||
AWARENESS_NAMESPACE=workspace:ws-123
|
||||
|
||||
@@ -8,6 +8,23 @@ Entries are published daily at 23:50 UTC.
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-13
|
||||
|
||||
### ✨ New features
|
||||
|
||||
- **Graceful shutdown support for remote agents**: `run_heartbeat_loop()` and `run_agent_loop()` in `molecule-sdk-python` now accept a `stop_event: threading.Event` parameter. Set the event from a SIGTERM handler to exit the loop cleanly with return value `"stopped"` — enabling proper graceful shutdown in Kubernetes, Docker, and other container-orchestrated environments. (`molecule-sdk-python` [#8](https://git.moleculesai.app/molecule-ai/molecule-sdk-python/pulls/8))
|
||||
|
||||
### 🔧 Fixes
|
||||
|
||||
- **PLATFORM_URL defaults aligned across all runtime modules**: all workspace runtime modules (`a2a_cli.py`, `a2a_client.py`, `a2a_mcp_server.py`, and 10 others) now consistently default `PLATFORM_URL` to `http://host.docker.internal:8080`. Previously some modules defaulted to `http://platform:8080`, causing connection failures in containerized deployments where the Docker host is not named `platform`. (`molecule-ai-workspace-runtime` [#12](https://git.moleculesai.app/molecule-ai/molecule-ai-workspace-runtime/pulls/12))
|
||||
|
||||
### 🧹 Internal
|
||||
|
||||
- **Canvas CI hardening**: publish workflow updated to pipefail-safe shell probes; Gitea cache export no longer masks errors; canvas image published to ECR. (`molecule-core` [#773](https://git.moleculesai.app/molecule-ai/molecule-core/pulls/773), [#776](https://git.moleculesai.app/molecule-ai/molecule-core/pulls/776), [#777](https://git.moleculesai.app/molecule-ai/molecule-core/pulls/777))
|
||||
- **Go lint CI hardening**: `golangci-lint run` no longer masked with `|| true`, so lint failures now fail the build loudly instead of being silently swallowed. (`molecule-core` [#781](https://git.moleculesai.app/molecule-ai/molecule-core/pulls/781))
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-12
|
||||
|
||||
### 🔒 Security
|
||||
|
||||
@@ -119,12 +119,20 @@ secrets = client.pull_secrets() # Phase 30.2 — decrypt API keys
|
||||
print("Secrets:", list(secrets.keys()))
|
||||
|
||||
# Keep alive + respond to platform commands
|
||||
import threading, signal, sys
|
||||
|
||||
stop_event = threading.Event()
|
||||
signal.signal(signal.SIGTERM, lambda *_: stop_event.set())
|
||||
|
||||
client.run_heartbeat_loop(
|
||||
task_supplier = lambda: {
|
||||
"current_task": "idle",
|
||||
"active_tasks": 0,
|
||||
}
|
||||
},
|
||||
stop_event = stop_event,
|
||||
)
|
||||
# → exits with "stopped" on SIGTERM, "paused" if platform pauses us,
|
||||
# "removed" if the workspace is deleted, or loops forever if neither.
|
||||
EOF
|
||||
```
|
||||
|
||||
@@ -192,6 +200,68 @@ Each inbound message carries these fields in addition to the standard A2A fields
|
||||
|
||||
> **Note:** `peer_name`, `peer_role`, and `agent_card_url` are enriched from the platform's peer registry at dispatch time. They are `None` if the sending peer has not registered an agent card.
|
||||
|
||||
### run_heartbeat_loop(stop_event=, max_iterations=, task_supplier=)
|
||||
|
||||
Drives heartbeat + state-poll on a timer. Returns the terminal status when the loop exits.
|
||||
|
||||
```python
|
||||
import threading, signal
|
||||
|
||||
stop_event = threading.Event()
|
||||
signal.signal(signal.SIGTERM, lambda *_: stop_event.set())
|
||||
|
||||
status = client.run_heartbeat_loop(
|
||||
max_iterations = None, # None = run until paused/deleted; int = stop after N ticks
|
||||
task_supplier = lambda: { # optional — report current task to the canvas
|
||||
"current_task": "idle",
|
||||
"active_tasks": 0,
|
||||
},
|
||||
stop_event = stop_event, # set() to exit cleanly with return value "stopped"
|
||||
)
|
||||
# status is one of: "stopped" | "paused" | "removed" | "max_iterations"
|
||||
```
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `stop_event` | `threading.Event \| None` | When set, the loop exits cleanly with `"stopped"`. Use in a SIGTERM handler for graceful Kubernetes/Docker shutdown. Ignored when `None`. |
|
||||
| `max_iterations` | `int \| None` | Stop after N loop iterations. `None` (default) = run until the workspace is paused or deleted. |
|
||||
| `task_supplier` | `callable \| None` | Zero-arg callable returning `{"current_task": str, "active_tasks": int}`. Reports activity to the canvas on each tick. |
|
||||
|
||||
Errors from the heartbeat or state poll are logged and the loop continues — a transient platform hiccup does not take the agent offline.
|
||||
|
||||
### run_agent_loop(handler, delivery=, stop_event=, max_iterations=, task_supplier=)
|
||||
|
||||
Combined heartbeat + state-poll + inbound-delivery loop. The recommended entry point for external agent authors: registers, heartbeats, state-polls, and dispatches inbound A2A messages in one synchronous call.
|
||||
|
||||
```python
|
||||
from molecule_agent import RemoteAgentClient, PollDelivery
|
||||
import threading, signal
|
||||
|
||||
stop_event = threading.Event()
|
||||
signal.signal(signal.SIGTERM, lambda *_: stop_event.set())
|
||||
|
||||
async def handle(msg):
|
||||
print(f"Got message: {msg.method}")
|
||||
return "Acknowledged"
|
||||
|
||||
status = client.run_agent_loop(
|
||||
handler = handle,
|
||||
delivery = None, # defaults to PollDelivery — correct for agents without a public URL
|
||||
stop_event = stop_event, # set() to exit cleanly
|
||||
max_iterations = None,
|
||||
task_supplier = lambda: {"current_task": "idle", "active_tasks": 0},
|
||||
)
|
||||
# status is one of: "stopped" | "paused" | "removed" | "max_iterations"
|
||||
```
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `handler` | `Callable[[InboundMessage], str \| None]` | Called once per inbound A2A message. Return a non-empty string to auto-reply; `None` to skip the reply. |
|
||||
| `delivery` | `InboundDelivery \| None` | Delivery mechanism. Defaults to `PollDelivery` (polling, no inbound URL needed). Pass `PushDelivery` wrapped around an `A2AServer` for push-mode agents. |
|
||||
| `stop_event` | `threading.Event \| None` | When set, the loop exits cleanly with `"stopped"`. Ignored when `None`. |
|
||||
| `max_iterations` | `int \| None` | Stop after N loop iterations. `None` = run until paused/deleted. |
|
||||
| `task_supplier` | `callable \| None` | Zero-arg callable returning `{"current_task": str, "active_tasks": int}`. |
|
||||
|
||||
### Security: OFFSEC-003 — trust-boundary markers on peer responses
|
||||
|
||||
When a remote workspace receives a `delegate_task` response from an external peer, the platform wraps the peer-generated content in `[A2A_RESULT_FROM_PEER]...[/A2A_RESULT_FROM_PEER]` trust-boundary markers. These markers signal to the agent that the enclosed content originated outside the platform's trust boundary and must not be re-injected as platform-native output.
|
||||
|
||||
@@ -1,198 +0,0 @@
|
||||
---
|
||||
title: Self-Hosted Workspace Deployment with Docker
|
||||
---
|
||||
|
||||
# Self-Hosted Workspace Deployment with Docker
|
||||
|
||||
This guide covers running a Molecule AI workspace agent as a Docker container on a self-hosted server or VM. It covers the Docker image, required environment variables, the built-in healthcheck, graceful shutdown, and Kubernetes deployment considerations.
|
||||
|
||||
> **Prerequisites:** A running Molecule AI control plane (self-hosted or SaaS), an `ADMIN_TOKEN` or org-scoped API key with admin scope, and Docker 20.10+ on the host.
|
||||
|
||||
## How the workspace container works
|
||||
|
||||
The Molecule AI workspace Dockerfile includes:
|
||||
|
||||
- A uvicorn server on port 8000 (configurable via `PORT`)
|
||||
- A healthcheck endpoint at `/.well-known/agent-card.json` (used by Docker and Kubernetes probes)
|
||||
- Graceful SIGTERM handling via uvicorn — the heartbeat loop and adapter tasks shut down cleanly
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Docker host (your VM / bare metal) │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────┐ │
|
||||
│ │ workspace container │ │
|
||||
│ │ │ │
|
||||
│ │ uvicorn (port 8000) │ │
|
||||
│ │ └─ /.well-known/agent-card.json ← HEALTHCHECK │ │
|
||||
│ │ │ │
|
||||
│ │ heartbeat loop + A2A agent │ │
|
||||
│ └──────────────┬──────────────────────┘ │
|
||||
│ │ │
|
||||
│ host.docker.internal:8080 │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────┐ │
|
||||
│ │ Molecule AI control plane │ │
|
||||
│ │ (platform on port 8080) │ │
|
||||
│ └─────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Step 1: Create an external workspace
|
||||
|
||||
First register the workspace as an external (self-managed) agent on the platform.
|
||||
|
||||
```bash
|
||||
ADMIN_TOKEN="your-admin-token"
|
||||
PLATFORM_URL="https://platform.moleculesai.app" # or http://localhost:8080 for local dev
|
||||
WORKSPACE=$(curl -s -X POST "${PLATFORM_URL}/workspaces" \
|
||||
-H "Authorization: Bearer ${ADMIN_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "self-hosted-agent", "runtime": "external"}')
|
||||
|
||||
WORKSPACE_ID=$(echo "$WORKSPACE" | python3 -c "import json,sys; print(json.load(sys.stdin)['id'])")
|
||||
echo "Workspace ID: $WORKSPACE_ID"
|
||||
```
|
||||
|
||||
Save the returned `WORKSPACE_ID`. The workspace agent obtains its bearer token automatically during its first registration with the platform.
|
||||
|
||||
## Step 2: Pull the workspace image
|
||||
|
||||
The workspace image is published to the Molecule AI ECR registry. Contact your platform administrator for the registry prefix and credentials, then log in:
|
||||
|
||||
```bash
|
||||
aws ecr get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin "${REGISTRY_PREFIX}.dkr.ecr.us-east-1.amazonaws.com"
|
||||
|
||||
docker pull "${REGISTRY_PREFIX}.dkr.ecr.us-east-1.amazonaws.com/molecule-workspace:latest"
|
||||
```
|
||||
|
||||
## Step 3: Configure environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `PLATFORM_URL` | `http://localhost:8080` | Platform API URL. Inside a Docker container, use `http://host.docker.internal:8080` to reach the platform on the host machine. |
|
||||
| `WORKSPACE_ID` | — | Workspace ID from Step 1 (required; no default) |
|
||||
| `PORT` | `8000` | Agent server port. Must match `containerPort` in Kubernetes and the port mapped with `-p` in Docker. |
|
||||
|
||||
## Step 4: Run the container
|
||||
|
||||
### Docker (standalone)
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name molecule-workspace \
|
||||
-p 8000:8000 \
|
||||
-e PLATFORM_URL="http://host.docker.internal:8080" \
|
||||
-e WORKSPACE_ID="your-workspace-id" \
|
||||
-e PORT=8000 \
|
||||
"${REGISTRY_PREFIX}.dkr.ecr.us-east-1.amazonaws.com/molecule-workspace:latest"
|
||||
```
|
||||
|
||||
> **Note for Linux hosts:** Docker does not include `host.docker.internal` by default. On Linux, either add `--add-host=host.docker.internal:host-gateway` to the `docker run` command, or use the host machine's IP address directly (e.g. `http://192.168.1.100:8080`).
|
||||
|
||||
### Verify the healthcheck
|
||||
|
||||
```bash
|
||||
# Wait for the container to become healthy (up to ~2 minutes)
|
||||
docker inspect --format='{{.State.Health.Status}}' molecule-workspace
|
||||
|
||||
# Expected output: healthy
|
||||
# Once healthy, the agent card is reachable:
|
||||
curl -s http://localhost:8000/.well-known/agent-card.json | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
molecule-workspace:
|
||||
image: "${REGISTRY_PREFIX}.dkr.ecr.us-east-1.amazonaws.com/molecule-workspace:latest"
|
||||
ports:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
PLATFORM_URL: "http://host.docker.internal:8080"
|
||||
WORKSPACE_ID: "your-workspace-id"
|
||||
PORT: "8000"
|
||||
# Linux hosts: add host.docker.internal resolution
|
||||
# extra_hosts:
|
||||
# - "host.docker.internal:host-gateway"
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/.well-known/agent-card.json"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
```
|
||||
|
||||
## Step 5: Graceful shutdown
|
||||
|
||||
When the container receives SIGTERM (e.g. from `docker stop` or Kubernetes pod deletion), the workspace's uvicorn server initiates graceful shutdown: the heartbeat loop stops, active A2A tasks are given a grace period to complete, and any snapshotable state is persisted before the process exits.
|
||||
|
||||
To integrate the heartbeat loop into custom agent code:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os, signal
|
||||
from heartbeat import HeartbeatLoop
|
||||
|
||||
# SIGTERM is handled by the Docker runtime, which sends the signal to the
|
||||
# workspace process. The workspace (via uvicorn) initiates graceful shutdown:
|
||||
# the heartbeat loop is stopped, any active adapter tasks are cancelled, and
|
||||
# in-flight A2A requests are given a grace period to complete.
|
||||
#
|
||||
# For custom integration with the heartbeat loop directly:
|
||||
async def main():
|
||||
heartbeat = HeartbeatLoop(
|
||||
platform_url=os.environ["PLATFORM_URL"],
|
||||
workspace_id=os.environ["WORKSPACE_ID"],
|
||||
)
|
||||
heartbeat.start()
|
||||
try:
|
||||
await asyncio.Event().wait() # keep running
|
||||
finally:
|
||||
await heartbeat.stop()
|
||||
print("Heartbeat loop stopped.")
|
||||
```
|
||||
|
||||
The Docker `stop` command sends SIGTERM and waits up to 10 seconds by default before sending SIGKILL. The healthcheck ensures orchestrators detect an unhealthy container before the SIGTERM timeout.
|
||||
|
||||
## Kubernetes deployment
|
||||
|
||||
For Kubernetes deployments, use the native liveness/readiness probe configuration instead of the Docker HEALTHCHECK:
|
||||
|
||||
```yaml
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8000
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /.well-known/agent-card.json
|
||||
port: http
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /.well-known/agent-card.json
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
terminationGracePeriodSeconds: 120
|
||||
```
|
||||
|
||||
> **Note:** The Kubernetes `terminationGracePeriodSeconds` should exceed the liveness probe failure threshold so that the probe can register a failure before the pod is killed. With `periodSeconds: 30` and `failureThreshold: 3`, the probe does not register a failure until approximately 120–150s after the container becomes unhealthy. Set `terminationGracePeriodSeconds: 120` or higher.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| Container shows `unhealthy` after startup | Platform unreachable from container | Verify `PLATFORM_URL` uses `host.docker.internal` (Docker) or the correct host IP |
|
||||
| `curl: (7) Failed to connect` on healthcheck | Container not fully started | Wait up to 30s; increase `start_period` |
|
||||
| Agent not appearing on canvas | Wrong `WORKSPACE_ID` or expired token | Re-run registration; check platform logs |
|
||||
| `host.docker.internal` not resolved | Linux host without the Docker flag | Use `--add-host=host.docker.internal:host-gateway` or the host's LAN IP |
|
||||
Reference in New Issue
Block a user