zycandClaude Fable 5
2fcd6d996c
事故:连推 3 commit 触发 3 轮 rolling 部署,多个新 pod 并发跑 migrate 且崩溃重跑,
积分 ×10 rescale 被交错重放——ai.0025 执行 6 次(单价/任务计价 ×10⁶),accounts.0008
执行 2 次(限额 ×100),billing.0003/0004 从未完成(django_migrations 漏记录)。
测试库数据已按精确倍率手工修复并补记迁移记录(备份于本机)。
两层防复发:
1. docker-entrypoint 用 MySQL GET_LOCK('airshelf_migrate') 串行化 migrate,
后到 pod 等锁,拿到时迁移已被记录 → 自然 no-op;
2. 三个 rescale 迁移加 airshelf_rescale_marker 幂等标记(与数据变更同事务提交):
记录丢失/崩溃重跑时,标记在 → 跳过,不会重复 ×10。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
45 lines
1.6 KiB
Bash
45 lines
1.6 KiB
Bash
#!/bin/sh
|
||
set -e
|
||
|
||
# Only the web (gunicorn) container should run migrations / collectstatic.
|
||
# The celery worker shares this image but skips DB schema mutation to avoid races.
|
||
#
|
||
# ⚠️ migrate 必须跨 pod 串行化(MySQL GET_LOCK):2026-07-07 事故——连推 3 个 commit 触发
|
||
# 3 轮 rolling 部署,多个新 pod 并发跑 migrate,数据迁移(积分 ×10 rescale)被交错重放 6 次,
|
||
# 测试库单价被乘成 ×10⁶(¥2 → 2,000,000)。锁把并发压成串行;后到者拿到锁时迁移已被
|
||
# 先到者记录,migrate 自然 no-op。锁超时 600s 拿不到 → 快速失败重启,绝不裸跑。
|
||
case "$1" in
|
||
gunicorn)
|
||
echo "[entrypoint] running migrations (serialized via DB advisory lock)..."
|
||
python - <<'PYEOF'
|
||
import os
|
||
|
||
import django
|
||
|
||
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "airshelf.settings.production")
|
||
django.setup()
|
||
|
||
from django.core.management import call_command
|
||
from django.db import connection
|
||
|
||
if connection.vendor == "mysql":
|
||
with connection.cursor() as cursor:
|
||
cursor.execute("SELECT GET_LOCK('airshelf_migrate', 600)")
|
||
acquired = cursor.fetchone()[0]
|
||
if not acquired:
|
||
raise SystemExit("[entrypoint] FATAL: migrate advisory lock timeout (another pod stuck?)")
|
||
try:
|
||
call_command("migrate", interactive=False)
|
||
finally:
|
||
with connection.cursor() as cursor:
|
||
cursor.execute("SELECT RELEASE_LOCK('airshelf_migrate')")
|
||
else:
|
||
call_command("migrate", interactive=False)
|
||
PYEOF
|
||
echo "[entrypoint] collecting static..."
|
||
python manage.py collectstatic --noinput
|
||
;;
|
||
esac
|
||
|
||
exec "$@"
|