fix(deploy): migrate 跨 pod 串行化 + rescale 迁移幂等守卫(测试环境×10⁶事故复盘)

事故:连推 3 commit 触发 3 轮 rolling 部署,多个新 pod 并发跑 migrate 且崩溃重跑,
积分 ×10 rescale 被交错重放——ai.0025 执行 6 次(单价/任务计价 ×10⁶),accounts.0008
执行 2 次(限额 ×100),billing.0003/0004 从未完成(django_migrations 漏记录)。
测试库数据已按精确倍率手工修复并补记迁移记录(备份于本机)。

两层防复发:
1. docker-entrypoint 用 MySQL GET_LOCK('airshelf_migrate') 串行化 migrate,
   后到 pod 等锁,拿到时迁移已被记录 → 自然 no-op;
2. 三个 rescale 迁移加 airshelf_rescale_marker 幂等标记(与数据变更同事务提交):
   记录丢失/崩溃重跑时,标记在 → 跳过,不会重复 ×10。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
zyc
2026-07-07 11:25:40 +08:00
co-authored by Claude Fable 5
parent d466bd60c6
commit 2fcd6d996c
4 changed files with 133 additions and 11 deletions
+31 -2
View File
@@ -3,10 +3,39 @@ set -e
# Only the web (gunicorn) container should run migrations / collectstatic.
# The celery worker shares this image but skips DB schema mutation to avoid races.
#
# ⚠️ migrate 必须跨 pod 串行化(MySQL GET_LOCK):2026-07-07 事故——连推 3 个 commit 触发
# 3 轮 rolling 部署,多个新 pod 并发跑 migrate,数据迁移(积分 ×10 rescale)被交错重放 6 次,
# 测试库单价被乘成 ×10⁶(¥2 → 2,000,000)。锁把并发压成串行;后到者拿到锁时迁移已被
# 先到者记录,migrate 自然 no-op。锁超时 600s 拿不到 → 快速失败重启,绝不裸跑。
case "$1" in
gunicorn)
echo "[entrypoint] running migrations..."
python manage.py migrate --noinput
echo "[entrypoint] running migrations (serialized via DB advisory lock)..."
python - <<'PYEOF'
import os
import django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "airshelf.settings.production")
django.setup()
from django.core.management import call_command
from django.db import connection
if connection.vendor == "mysql":
with connection.cursor() as cursor:
cursor.execute("SELECT GET_LOCK('airshelf_migrate', 600)")
acquired = cursor.fetchone()[0]
if not acquired:
raise SystemExit("[entrypoint] FATAL: migrate advisory lock timeout (another pod stuck?)")
try:
call_command("migrate", interactive=False)
finally:
with connection.cursor() as cursor:
cursor.execute("SELECT RELEASE_LOCK('airshelf_migrate')")
else:
call_command("migrate", interactive=False)
PYEOF
echo "[entrypoint] collecting static..."
python manage.py collectstatic --noinput
;;