在 Rancher 中升级容器时，出现数据库迁移重复执行，进而报错的排查分析

1、在 Rancher 中第一次升级容器时，出现数据库迁移重复执行，进而报错升级失败。如图1

图1

*** applying m210106_122427_add_columns_to_task_table

> add column article_type_id integer NOT NULL DEFAULT 0 COMMENT '文章类型ID' AFTER `type` to table {{%task}} ...Exception: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'article_type_id'

The SQL being executed was: ALTER TABLE `cpa_task` ADD `article_type_id` int(11) NOT NULL DEFAULT 0 COMMENT '文章类型ID' AFTER `type` (/mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Schema.php:678)

#0 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1304): yii/db/Schema->convertException(Object(PDOException), 'ALTER TABLE `cp...')

#1 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1099): yii/db/Command->internalExecute('ALTER TABLE `cp...')

#2 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(377): yii/db/Command->execute()

#3 /mcloud/www/ccp_api/console/migrations/m210106_122427_add_columns_to_task_table.php(15): yii/db/Migration->addColumn('{{%task}}', 'article_type_id', Object(yii/db/mysql/ColumnSchemaBuilder))

#4 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(114): m210106_122427_add_columns_to_task_table->safeUp()

#5 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(744): yii/db/Migration->up()

#6 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(202): yii/console/controllers/BaseMigrateController->migrateUp('m210106_122427_...')

#7 [internal function]: yii/console/controllers/BaseMigrateController->actionUp(0)

#8 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)

#9 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Controller.php(181): yii/base/InlineAction->runWithParams(Array)

#10 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Controller.php(184): yii/base/Controller->runAction('', Array)

#11 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Module.php(534): yii/console/Controller->runAction('', Array)

#12 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(181): yii/base/Module->runAction('migrate', Array)

#13 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(148): yii/console/Application->runAction('migrate', Array)

#14 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Application.php(392): yii/console/Application->handleRequest(Object(yii/console/Request))

#15 /mcloud/www/ccp_api/yii(23): yii/base/Application->run()

#16 {main}

*** failed to apply m210106_122427_add_columns_to_task_table (time: 0.006s)

2、已经确定数据库结构并未被手动修改过。分析容器的健康检查。周期性向容器发出请求，以检测其健康状态。默认设置下，readiness和liveness使用相同的配置参数。对于应用初始化较长的容器，需要增加就绪检测时间。初步分析结论：由于数据库迁移执行时间过长，导致容器初始化时间过长，导致健康检查失败，再导致容器不断启动运行，数据库迁移重复执行。决定增加就绪检测时间。运行多少时间后开始检测：10秒 -> 100秒。如图2

图2

3、在初始部署时仍然报错：ReplicaSet “api-77dfcc7cfb” has timed out progressing.; Deployment does not have minimum availability.。如图3

图3

4、查看日志，仍然是数据库迁移重复执行的问题。查看 MySQL 中的表结构与迁移表中的记录，确定迁移记录：m210402_052030_add_columns_to_channel_app_source_table 在迁移表中不存在，但是 MySQL 中的表结构已经是此迁移记录执行后的结果。如图4

图4

m210402_052030_add_columns_to_channel_app_source_table

*** applying m201105_020453_add_article_read_count_column_to_channel_app_task_table

> add column article_read_count integer NOT NULL DEFAULT 0 COMMENT '文章的阅读数' AFTER `have_pub_number` to table {{%channel_app_task}} ...Exception: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'article_read_count'

The SQL being executed was: ALTER TABLE `cpa_channel_app_task` ADD `article_read_count` int(11) NOT NULL DEFAULT 0 COMMENT '文章的阅读数' AFTER `have_pub_number` (/mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Schema.php:678)

#0 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1304): yii/db/Schema->convertException(Object(PDOException), 'ALTER TABLE `cp...')

#1 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1099): yii/db/Command->internalExecute('ALTER TABLE `cp...')

#2 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(377): yii/db/Command->execute()

#3 /mcloud/www/ccp_api/console/migrations/m201105_020453_add_article_read_count_column_to_channel_app_task_table.php(15): yii/db/Migration->addColumn('{{%channel_app_...', 'article_read_co...', Object(yii/db/mysql/ColumnSchemaBuilder))

#4 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(114): m201105_020453_add_article_read_count_column_to_channel_app_task_table->safeUp()

#5 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(744): yii/db/Migration->up()

#6 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(202): yii/console/controllers/BaseMigrateController->migrateUp('m201105_020453_...')

#7 [internal function]: yii/console/controllers/BaseMigrateController->actionUp(0)

#8 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)

#9 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Controller.php(181): yii/base/InlineAction->runWithParams(Array)

#10 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Controller.php(184): yii/base/Controller->runAction('', Array)

#11 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Module.php(534): yii/console/Controller->runAction('', Array)

#12 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(181): yii/base/Module->runAction('migrate', Array)

#13 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(148): yii/console/Application->runAction('migrate', Array)

#14 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Application.php(392): yii/console/Application->handleRequest(Object(yii/console/Request))

#15 /mcloud/www/ccp_api/yii(23): yii/base/Application->run()

#16 {main}

*** failed to apply m201105_020453_add_article_read_count_column_to_channel_app_task_table (time: 0.006s)

0 from 27 migrations were applied.

Migration failed. The rest of the migrations are canceled.

5、运行多少时间后开始检测。：10秒 -> 100秒。如图5

图5