-
Bug
-
Resolution: Done
-
Normal
-
None
-
2
-
False
-
None
-
False
-
Testable
-
No
-
-
-
-
-
-
-
1.27.0-24
-
No
-
No
-
N/A
-
None
-
-
-
ML Ops Sprint 1.26, ML Ops Sprint 1.27
During sandbox performance tests, we noticed that there is an idler task that scale down all Deployments to save resources. For the MariaDB deployment, it keeps trying to create a new pod, putting in an endless loop with the idler terminating the pod until we get the InnoDB files corrupted. In another scenario, the frequent MariaDB pod crashes end in a corruption in the data files.
We need to find out how to avoid that scenario, not only for sandbox clusters but for production cluster.
Error we encounter:
=> sourcing 20-validate-variables.sh ... => sourcing 25-validate-replication-variables.sh ... => sourcing 30-base-config.sh ... ---> 19:35:18 Processing basic MySQL configuration files ... => sourcing 60-replication-config.sh ... => sourcing 70-s2i-config.sh ... ---> 19:35:18 Processing additional arbitrary MySQL configuration provided by s2i ... => sourcing 40-paas.cnf ... => sourcing 50-my-tuning.cnf ... ---> 19:35:18 Starting MySQL server with disabled networking ... ---> 19:35:18 Waiting for MySQL to start ... 2023-03-28 19:35:18 0 [Note] /usr/libexec/mysqld (mysqld 10.3.35-MariaDB) starting as process 25 ... 2023-03-28 19:35:18 0 [Note] InnoDB: Using Linux native AIO 2023-03-28 19:35:18 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2023-03-28 19:35:18 0 [Note] InnoDB: Uses event mutexes 2023-03-28 19:35:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2023-03-28 19:35:18 0 [Note] InnoDB: Number of pools: 1 2023-03-28 19:35:18 0 [Note] InnoDB: Using SSE2 crc32 instructions 2023-03-28 19:35:18 0 [Note] InnoDB: Initializing buffer pool, total size = 512M, instances = 1, chunk size = 128M 2023-03-28 19:35:18 0 [Note] InnoDB: Completed initialization of buffer pool 2023-03-28 19:35:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2023-03-28 19:35:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1625470 2023-03-28 19:35:18 0 [ERROR] InnoDB: Missing MLOG_CHECKPOINT at 1625470 between the checkpoint 1625470 and the end 1625488. 2023-03-28 19:35:18 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error 2023-03-28 19:35:19 0 [Note] InnoDB: Starting shutdown... 2023-03-28 19:35:19 0 [ERROR] Plugin 'InnoDB' init function returned error. 2023-03-28 19:35:19 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2023-03-28 19:35:19 0 [Note] Plugin 'FEEDBACK' is disabled. 2023-03-28 19:35:19 0 [ERROR] Unknown/unsupported storage engine: InnoDB 2023-03-28 19:35:19 0 [ERROR] Aborting
