前几天收到监控告警,服务器磁盘容量不足,进一步查看,发现是一数据库实例错误日志暴增导致。

具体报错如下:

1
2
3
4
5
6
7
8
9
10
2023-04-03T10:52:09.465937+08:00 11220648 [ERROR] InnoDB: Column table_name in table `mysql`.`innodb_table_stats` is VARCHAR(597) NOT NULL but should be VARCHAR(192) NOT NULL (length mismatch).
2023-04-03T10:52:09.465947+08:00 11220648 [ERROR] InnoDB: Fetch of persistent statistics requested for table `db_name_xxx`.`abs_loan_def` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2023-04-03T10:52:09.466215+08:00 11220648 [ERROR] InnoDB: Column table_name in table `mysql`.`innodb_table_stats` is VARCHAR(597) NOT NULL but should be VARCHAR(192) NOT NULL (length mismatch).
2023-04-03T10:52:09.466226+08:00 11220648 [ERROR] InnoDB: Fetch of persistent statistics requested for table `db_name_xxx`.`abs_loan_feature` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2023-04-03T10:52:09.466468+08:00 11220648 [ERROR] InnoDB: Column table_name in table `mysql`.`innodb_table_stats` is VARCHAR(597) NOT NULL but should be VARCHAR(192) NOT NULL (length mismatch).
2023-04-03T10:52:09.466478+08:00 11220648 [ERROR] InnoDB: Fetch of persistent statistics requested for table `db_name_xxx`.`abs_payment` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2023-04-03T10:52:09.467033+08:00 11220648 [ERROR] InnoDB: Column table_name in table `mysql`.`innodb_table_stats` is VARCHAR(597) NOT NULL but should be VARCHAR(192) NOT NULL (length mismatch).
2023-04-03T10:52:09.467054+08:00 11220648 [ERROR] InnoDB: Fetch of persistent statistics requested for table `db_name_xxx`.`abs_pi_rec_pool` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2023-04-03T10:52:09.467317+08:00 11220648 [ERROR] InnoDB: Column table_name in table `mysql`.`innodb_table_stats` is VARCHAR(597) NOT NULL but should be VARCHAR(192) NOT NULL (length mismatch).
2023-04-03T10:52:09.467329+08:00 11220648 [ERROR] InnoDB: Fetch of persistent statistics requested for table `db_name_xxx`.`abs_pool_feature` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

 从内容看,感觉是迁移过来的元数据和实际版本不匹配,数据是通过xtrabackup备份,传输过来用IDC服务器本地mysql启动的。本地版本是5.7.21。

因为迁移过来后云上实例已经销毁了,难以拿到具体云上的小版本号,通过云上其他实例查询发现都是5.7.18。本地下载了5.7.18,启动发现还是报错。那就是这个实例后来升级过。询问云上工程师,他们也查不到具体的初始小版本和升级后的小版本,但是给了另一个用的比较多的小版本,5.7.32。节前最后一天下班后告知的,今天上午来了马上试验,确实是这个版本,启动后数小时,到写文章此刻,再未观察到报错。如果是小版本不匹配,启动直接会报错的。

那原因就是21到32某个小版本官方升级了 `mysql`.`innodb_table_stats` 表结构导致的。具体找时间refman排查下。