Contents

Introduction

Problem

Solution

Introduction

This document describes the possible cause for the Redis service failure issue in CyberVision Center.

Problem

In some cases, the redis.service on a CyberVision Center can be in a failed state. The failure of this service can also cause other services like marmotd and the backend to go down.

One possible cause for redis.service not starting gracefully is the Redis Append Only File (AOF) being in a corrupted state.

Redis AOF is a persistence mechanism that allows the Redis server to keep track of and log every command executed on the server. Using AOF, Redis appends each command sequentially executed on the server. This prevents any data loss due to incorrect command orders.

Solution

In order to confirm if the service has failed due to the context explained:

- Using systemctl check the status of the Redis service via CLI or in systemctl file in the diagnostics bundle:

root@Center-4:~# systemctl --failed

* redis.service                      loaded failed failed    Redis data structure

- Look into the redis.service logs for specific errors using this command or journal_redis log from the diagnostic bundle:

root@Center-4:~# journalctl -u redis.service

2023-08-08T09:07:59+0000 center redis[973]: Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>

2023-08-08T09:07:59+0000 center systemd[1]: redis.service: Main process exited, code=exited, status=1/FAILURE

2023-08-08T09:07:59+0000 center systemd[1]: redis.service: Failed with result 'exit-code'.

2023-08-08T09:07:59+0000 center systemd[1]: Failed to start Redis data structure server.

journal_error log:

2023-08-08T09:07:59+0000 center systemd[1]: Failed to start Redis data structure server.

2023-08-08T09:07:59+0000 center systemd[1]: Failed to start Redis data structure server.

- In order to fix the issue, the procedure is to take an AOF file backup, perform a fix on the corrupted file, and reboot the Center:

root@Center-4:~#sudo cp /data/redis/appendonly.aof /data/redis/appendonly.aof.bak

root@Center-4:~#sudo redis-check-aof --fix /data/redis/appendonly.aof

root@Center-4:~#reboot

- Post reboot, validate that the redis.service is in an active state.

root@Center-4:~# systemctl status redis.service

● redis.service - Redis data structure server

Loaded: loaded (/lib/systemd/system/redis.service; enabled; vendor preset: enabled)

Active: active (running)