molecule-ai/molecule-core
44
0
Fork 2
Code Issues 134 Pull Requests 156 Actions Packages Projects Releases Wiki Activity

[handover] Add amazon-ecr-credential-helper to tenant AMI — fix 12h-stale-ECR-auth failure mode #658

New Issue
Open
opened 2026-05-12 04:20:51 +00:00 by hongming · 1 comment
hongming commented 2026-05-12 04:20:51 +00:00
Owner
Copy Link
Copy Source

Discovered tonight (2026-05-12)

During manual promote validation (CP#135 RFC#279 dry-run), hongming tenant's redeploy 403'd on the first attempt:

Error response from daemon: unknown: failed to resolve reference "153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/platform-tenant:latest": 403 Forbidden

Root cause: /root/.docker/config.json on the tenant EC2 stores a static ECR auth token (12h TTL) baked at last docker login time. After expiry, all docker pull calls 403. amazon-ecr-credential-helper is NOT installed.

Worked around tonight by manually aws ecr get-login-password | docker login via SSM, then retried redeploy successfully.

Note: chloe-dong redeploy worked on first try because earlier in the session I'd already SSM'd a docker login refresh while debugging — coincidental, not by design.

Fix

Bake amazon-ecr-credential-helper into the tenant AMI + flip /root/.docker/config.json to use credHelpers:

{
  "credHelpers": {
    "153263036946.dkr.ecr.us-east-2.amazonaws.com": "ecr-login",
    "004947743811.dkr.ecr.us-east-2.amazonaws.com": "ecr-login"
  }
}

This auto-refreshes the ECR auth from the EC2 IAM role on every pull, eliminating the 12h-stale failure mode.

Scope

  • Update packer build for tenant AMI (workspace-server / platform-tenant)
  • Verify fresh AMI bake passes a docker pull :latest smoke test 13+h after bake
  • Roll new AMI into CP provisioner default
  • Existing live tenants: SSM rolling update to install helper + flip config.json (or accept attrition: next AMI rebake recovers)

Related

  • CP#135 (RFC#279 auto-update chain) — this is the EC2-side piece complementing the ECR-side auto-tag promotion
  • Memory: feedback_no_secrets_in_docker_cmd_args (cred-helper avoids leaking secrets in docker config)
  • Tonight's manual workaround SSM command:
    aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <acct>.dkr.ecr.us-east-2.amazonaws.com
## Discovered tonight (2026-05-12) During manual promote validation (CP#135 RFC#279 dry-run), hongming tenant's redeploy 403'd on the first attempt: ``` Error response from daemon: unknown: failed to resolve reference "153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/platform-tenant:latest": 403 Forbidden ``` Root cause: `/root/.docker/config.json` on the tenant EC2 stores a static ECR auth token (12h TTL) baked at last `docker login` time. After expiry, all `docker pull` calls 403. `amazon-ecr-credential-helper` is NOT installed. Worked around tonight by manually `aws ecr get-login-password | docker login` via SSM, then retried redeploy successfully. Note: chloe-dong redeploy worked on first try because earlier in the session I'd already SSM'd a `docker login` refresh while debugging — coincidental, not by design. ## Fix Bake `amazon-ecr-credential-helper` into the tenant AMI + flip `/root/.docker/config.json` to use credHelpers: ```json { "credHelpers": { "153263036946.dkr.ecr.us-east-2.amazonaws.com": "ecr-login", "004947743811.dkr.ecr.us-east-2.amazonaws.com": "ecr-login" } } ``` This auto-refreshes the ECR auth from the EC2 IAM role on every pull, eliminating the 12h-stale failure mode. ## Scope - Update packer build for tenant AMI (workspace-server / platform-tenant) - Verify fresh AMI bake passes a `docker pull :latest` smoke test 13+h after bake - Roll new AMI into CP provisioner default - Existing live tenants: SSM rolling update to install helper + flip config.json (or accept attrition: next AMI rebake recovers) ## Related - CP#135 (RFC#279 auto-update chain) — this is the EC2-side piece complementing the ECR-side auto-tag promotion - Memory: `feedback_no_secrets_in_docker_cmd_args` (cred-helper avoids leaking secrets in docker config) - Tonight's manual workaround SSM command: `aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <acct>.dkr.ecr.us-east-2.amazonaws.com`
triage-operator added the tier:medium label 2026-05-12 04:21:35 +00:00
hongming added tier:high and removed tier:medium labels 2026-05-12 04:25:00 +00:00
core-devops was assigned by hongming 2026-05-12 04:25:22 +00:00
hongming commented 2026-05-12 05:09:47 +00:00
Author
Owner
Copy Link
Copy Source

Cross-link to molecule-controlplane#135 (RFC#279 auto-update chain) — this issue is the EC2-side credential-helper half. CP#135 is the platform-orchestration half. Both must land for the publish→replicate→pin→redeploy chain to be fully automated; until then scripts/promote-tenant-image.sh (core#672, closes #660) is the manual fallback that ships the SSM-refresh workaround. Posted by orchestrator handover sweep 2026-05-12.

Cross-link to molecule-controlplane#135 (RFC#279 auto-update chain) — this issue is the EC2-side credential-helper half. CP#135 is the platform-orchestration half. Both must land for the publish→replicate→pin→redeploy chain to be fully automated; until then `scripts/promote-tenant-image.sh` (core#672, closes #660) is the manual fallback that ships the SSM-refresh workaround. Posted by orchestrator handover sweep 2026-05-12.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
No Label tier:high
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees core-devops
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#658
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?