Introduction
This document describes the possible cause for the Redis service failure issue in CyberVision Center.
Problem
In some cases, the redis.service on a CyberVision Center can be in a failed state. The failure of this service can also cause other services like marmotd and the backend to go down.
One possible cause for redis.service not starting gracefully is the Redis Append Only File (AOF) being in a corrupted state.
Redis AOF is a persistence mechanism that allows the Redis server to keep track of and log every command executed on the server. Using AOF, Redis appends each command sequentially executed on the server. This prevents any data loss due to incorrect command orders.
Solution
In order to confirm if the service has failed due to the context explained:
- Using
systemctl check the status of the Redis service via CLI or in systemctl file in the diagnostics bundle:
root@Center-4:~# systemctl --failed
* redis.service loaded failed failed Redis data structure
- Look into the
redis.service logs for specific errors using this command or journal_redis log from the diagnostic bundle:
root@Center-4:~# journalctl -u redis.service
2023-08-08T09:07:59+0000 center redis[973]: Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
2023-08-08T09:07:59+0000 center systemd[1]: redis.service: Main process exited, code=exited, status=1/FAILURE
2023-08-08T09:07:59+0000 center systemd[1]: redis.service: Failed with result 'exit-code'.
2023-08-08T09:07:59+0000 center systemd[1]: Failed to start Redis data structure server.
journal_error log:
2023-08-08T09:07:59+0000 center systemd[1]: Failed to start Redis data structure server.
2023-08-08T09:07:59+0000 center systemd[1]: Failed to start Redis data structure server.
- In order to fix the issue, the procedure is to take an AOF file backup, perform a fix on the corrupted file, and reboot the Center:
root@Center-4:~#sudo cp /data/redis/appendonly.aof /data/redis/appendonly.aof.bak
root@Center-4:~#sudo redis-check-aof --fix /data/redis/appendonly.aof
root@Center-4:~#reboot
- Post reboot, validate that the
redis.service is in an active state.
root@Center-4:~# systemctl status redis.service
● redis.service - Redis data structure server
Loaded: loaded (/lib/systemd/system/redis.service; enabled; vendor preset: enabled)
Active: active (running)
Feedback