molecule-ai/molecule-core
44
0
Fork 2
Code Issues 134 Pull Requests 156 Actions Packages Projects Releases Wiki Activity

[handover] Codify manual ECR promote operation as scripts/promote-tenant-image.sh #660

New Issue
Closed
opened 2026-05-12 04:20:52 +00:00 by hongming · 0 comments
hongming commented 2026-05-12 04:20:52 +00:00
Owner
Copy Link
Copy Source

Context

Tonight (2026-05-12 03:30Z) we manually validated the prod-tenant image promotion end-to-end with Hongming's GO:

  1. Saved current :latest manifest as :latest-prev-20260512 (rollback tag)
  2. Re-tagged :staging-latest manifest as :latest via aws ecr put-image
  3. Triggered CP redeploy on chloe-dong + hongming via /cp/admin/tenants/<slug>/redeploy
  4. Verified /buildinfo shows new git_sha (210da3b1 vs prior 0276b29) + /health=ok

All done as ad-hoc bash. Not codified.

Asks

  1. Write scripts/promote-tenant-image.sh <src-tag> <dst-tag> in operator-config repo with:
    • Preflight: confirm both src+dst exist in ECR; refuse if src digest == dst digest (no-op)
    • Save current dst manifest as <dst>-prev-<YYYYMMDD> for rollback
    • put-image with src manifest tagged as dst
    • Print before/after digests for audit
  2. Write companion scripts/redeploy-tenant.sh <slug> [target_tag] with:
    • Optional pre-flight docker login refresh via SSM (avoids 12h-stale-auth failure — see sibling issue on amazon-ecr-credential-helper)
    • Call CP /cp/admin/tenants/<slug>/redeploy
    • Poll /buildinfo until git_sha changes (or timeout 60s)
    • Verify /health=ok post-deploy
  3. Document retention policy for :<tag>-prev-<date> rollback tags (e.g., keep last 3, purge older)

Rollback tag still in ECR

Tonight's specific rollback tag: molecule-ai/platform-tenant:latest-prev-20260512 (sha256:e08c68090b9a...) — preserved for rollback if needed. NOT YET on retention schedule.

Related

  • CP#135 RFC#279 (auto-update chain) — when that lands fully, manual promote becomes the rollback path only
## Context Tonight (2026-05-12 03:30Z) we manually validated the prod-tenant image promotion end-to-end with Hongming's GO: 1. Saved current `:latest` manifest as `:latest-prev-20260512` (rollback tag) 2. Re-tagged `:staging-latest` manifest as `:latest` via `aws ecr put-image` 3. Triggered CP redeploy on chloe-dong + hongming via `/cp/admin/tenants/<slug>/redeploy` 4. Verified `/buildinfo` shows new git_sha (210da3b1 vs prior 0276b29) + `/health=ok` All done as ad-hoc bash. Not codified. ## Asks 1. Write `scripts/promote-tenant-image.sh <src-tag> <dst-tag>` in operator-config repo with: - Preflight: confirm both src+dst exist in ECR; refuse if src digest == dst digest (no-op) - Save current dst manifest as `<dst>-prev-<YYYYMMDD>` for rollback - put-image with src manifest tagged as dst - Print before/after digests for audit 2. Write companion `scripts/redeploy-tenant.sh <slug> [target_tag]` with: - Optional pre-flight `docker login` refresh via SSM (avoids 12h-stale-auth failure — see sibling issue on amazon-ecr-credential-helper) - Call CP `/cp/admin/tenants/<slug>/redeploy` - Poll `/buildinfo` until git_sha changes (or timeout 60s) - Verify `/health=ok` post-deploy 3. Document retention policy for `:<tag>-prev-<date>` rollback tags (e.g., keep last 3, purge older) ## Rollback tag still in ECR Tonight's specific rollback tag: `molecule-ai/platform-tenant:latest-prev-20260512` (sha256:e08c68090b9a...) — preserved for rollback if needed. NOT YET on retention schedule. ## Related - CP#135 RFC#279 (auto-update chain) — when that lands fully, manual promote becomes the rollback path only
triage-operator added the tier:medium label 2026-05-12 04:21:35 +00:00
core-devops was assigned by hongming 2026-05-12 04:25:23 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
No Label tier:medium
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees core-devops
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#660
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?