ACI Switch Node SSD Lifetime Explained

Introduction

This document describes the SSD lifespan exhaustion behavior on ACI switch platforms.

If you have landed on this page as a result of an SSD fault raised against APIC (not a switch node), refer to FN64329. 

This TechNote documents the APIC SSD replacement procedure.

Background Information

ACI Leaf and Spine nodes leverage solid-state drives (SSD) for storage and event logging. All SSD, irrespective of the vendor type, have a fixed lifespan based on specific attributes defined by the manufacturer. Some examples include the number of writes and erases as well as the volume of data transferred to the drives over a period of time. The SSD lifespan on Leaf and Spine nodes can become exhausted due to regular wear and tear. If the user does not take proper action against a switch that has SSD approaching exhaustion, the switch could crash due to the SSD going into read-only mode. For this reason, it is imperative to prioritize SSD replacement in the event F3073 fault is raised against a given node. In such a scenario, the whole switch is replaced as the SSD is not a field replaceable unit.

F3073 was implemented in 2.1(4), 2.2(4), 2.3(1o), and 3.1(2m). Any subsequent software versions have the functionality to raise F3073.

Note: If you run an ACI software version that does not have SSD monitoring functionality, it is strongly advised to upgrade to a version where such monitoring is available.

Typical Symptoms if SSD is not Replaced

1. When the switch fails to boot, on the console you can see these errors:

/dev/hd-cfg0: ********** WARNING: Filesystem still has errors **********
e2fsck 1.42.1 (17-Feb-2012)
/dev/hd-cfg1: recovering journal
/sbin/e2fsck: unable to set superblock flags on /dev/hd-cfg1

2. The system is able to proactively raise faults related to SSD (F3073 or F3074 based on the SSD lifespan usage) on later code releases (covered in later section)

• F3074: fltEqptFlashFlash-minor-alarm (80% lifetime) 

• F3073: fltEqptFlashFlash-worn-out    (90% lifetime) 

3. You may also see fault F3525 raised for high SSD usage. This fault is often confused with F3073/F3074.

# fault.Inst
code : F3525
ack : no
annotation :
cause : equipment-flash-warning
changeSet : deltape (New: 21), peCycles (New: 1678), tbw (New: 32.465179), warning (New: yes)
childAction :
created : 2019-08-05T18:22:01.455-07:00
delegated : no
descr : High SSD usage observed. Please check switch activity and contact Cisco Technical Support about high SSD usage.
dn : topology/pod-1/node-206/sys/ch/supslot-1/sup/flash/fault-F3525
domain : infra
extMngdBy : undefined
highestSeverity : warning
lastTransition : 2019-08-05T18:24:02.029-07:00
lc : raised
modTs : never
occur : 1
origSeverity : warning
prevSeverity : warning
rn : fault-F3525
rule : eqpt-flash-flash-warning-alarm
severity : warning
status :
subject : flash-warning-alarm
type : operational

Fault F3525 is raised if P/E (program-erase) cycles increment by more than 21 in 7 days. It does not mean that the SSD is worn out, simply that there is a lot of churn which may eventually cause the SSDs to be worn out.
You need to work with TAC to understand what is causing this churn and address it.
There is also a policy to change the thresholds that cause F3525 to be raised for a specific type of SSDs (Access Policies > Policies > Switch > Equipment Flash Config Policies)

How to Verify Current Health of an SSD?

There are two ways to check the SSD lifespan status:

• Proactive System Alerting
• Manual Data Analysis

Proactive System Alerting

SSD lifetime monitoring was added as part of CSCve88634 in these versions of code:

• 2.1(4) and above for 2.1 train
• 2.2(4) and above for 2.2 train
• 2.3(1o) and above for 2.3 train
• 3.1(2m) and all future releases

A minor fault is raised when the drive approaches 80% of its lifetime and a major fault is raised when it exceeds 90% of its lifetime.  

This allows network operators the capability to monitor and proactively replace any switch before it fails due to SSD lifetime exceeded.

• F3074: fltEqptFlashFlash-minor-alarm (80% lifetime)
• F3073: fltEqptFlashFlash-worn-out    (90% lifetime)

Previous versions of code did not have SSD lifetime check for switch SSDs and therefore, even if  the switch was approaching the SSD expiration there would not be a fault raised. 

Manual Data Analysis

SSD usage logging was introduced starting in these releases 3.2(5d) 3.2(6i) 3.2(7f) 4.1(1i).

a-leaf101# cat /mnt/pss/ssd_log_amp.log   (Tested on 4.2.3l)
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Mon Dec 16 19:00:01 EST 2019   16531370        0       29398      12350    100             0               122396965772    3825894828           60841125365               16         0.00 %              0               0
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Tue Dec 17 19:00:01 EST 2019   16531725        0       29422      12355    100             0               122447755358    3827482016           60858831240               16         0.00 %          12355      61606629993
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Wed Dec 18 19:00:01 EST 2019   16532222        0       29446      12359    100             0               122495197374    3828964594           60875667807               16         0.00 %          12359      61636366134
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Thu Dec 19 19:00:01 EST 2019   16533056        0       29470      12364    100             0               122542269672    3830435617           60892533342               16         0.00 %          12364      61666601865
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Fri Dec 20 19:00:01 EST 2019   16534010        0       29494      12367    100             0               122587126224    3831837398           60908759296               16         0.00 %          12367      61694592882
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Sat Dec 21 19:00:01 EST 2019   16535311        0       29518      12372    100             0               122631804318    3833233605           60925205662               16         0.00 %          12372      61723045022
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Sun Dec 22 19:00:01 EST 2019   16536727        0       29542      12376    100             0               122676829388    3834640654           60941741722               16         0.00 %          12376      61751623726
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Mon Dec 23 19:00:01 EST 2019   16538321        0       29566      12380    100             0               122722137414    3836056544           60958504969               16         0.00 %          12380      61780395692
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Tue Dec 24 19:00:01 EST 2019   16539858        0       29591      12384    100             0               122767087802    3837461258           60975050343               16         0.00 %          12384      61808582833
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Wed Dec 25 19:00:01 EST 2019   16541121        0       29615      12388    100             0               122812238580    3838872234           60991578411               16         0.00 %          12388      61837188237
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Thu Dec 26 19:00:01 EST 2019   16542653        0       29639      12393    100             0               122857007868    3840271286           61008101155               16         0.00 %          12393      61865429457
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Fri Dec 27 19:00:01 EST 2019   16544352        0       29663      12397    100             0               122901807026    3841671272           61024757511               16         0.00 %          12397      61893705871
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Sat Dec 28 19:00:01 EST 2019   16546213        0       29687      12401    100             0               122946927200    3843081289           61041459287               16         0.00 %          12401      61922169689
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Sun Dec 29 19:00:01 EST 2019   16549522        0       29711      12405    100             0               122991584018    3844476827           61058889999               16         0.00 %          12405      61950124731
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Mon Dec 30 19:00:01 EST 2019   16551885        0       29735      12410    100             0               123036252528    3845872732           61075846564               16         0.00 %          12410      61977362529
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Tue Dec 31 19:00:01 EST 2019   16553858        0       29759      12414    100             0               123098133082    3847806568           61093085451               16         0.00 %          12414      62022286518
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Wed Jan  1 19:00:01 EST 2020   16555127        0       29783      12418    100             0               123142018610    3849178024           61109758713               16         0.00 %          12418      62048933159
Model                     SerialNo        Software-Version               Date                           RawReadError(1) GBB(5)  UpTime(9)  PE(173)  LifeTime(202)   Attribute(210)  TBW(246)        HostPageCount(247)   BackgroundPageCount(248)  WAF        Delta_WAF    Delta_PE   Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1                                    Thu Jan  2 19:00:01 EST 2020   16556532        0       29808      12422    100             0               123186637556    3850572390           61126687626               16         0.00 %          12422      62076878843

Model                          SerialNo        SW-Version      Date         R.Error(1) GBB(5)     UpTime(9)  PE(173)  LifeTime(202)   Attr(210)  TBW(246)     WAF    DWAF       DPE      DTBW   
Micron_M600_MTFDDAT064MBF      MSA195207V1     14.2(3l)        2020-01-03   16557977   0          29832      12428    100             0          123278298184 16     0.00 %      0        0      
Micron_M600_MTFDDAT064MBF      MSA195207V1     14.2(3l)        2020-01-04   16558440   0          29856      12433    100             0          123323818606 16     0.00 %      5        45520422
Micron_M600_MTFDDAT064MBF      MSA195207V1     14.2(3l)        2020-01-05   16559031   0          29880      12438    100             0          123369348610 16     0.00 %      5        45530004
Micron_M600_MTFDDAT064MBF      MSA195207V1     14.2(3l)        2020-01-06   16561567   0          29904      12443    100             0          123415316270 16     0.00 %      5        45967660
Micron_M600_MTFDDAT064MBF      MSA195207V1     14.2(3l)        2020-01-07   16564701   0          29928      12448    100             0          123460483898 16     0.00 %      5        45167628
Micron_M600_MTFDDAT064MBF      MSA195207V1     14.2(3l)        2020-01-08   16565955   0          29952      12453    100             0          123505550532 16     0.00 %      5        45066634

ssd_log_amp.log file is in the 1of3 switch tech supports

For any additional manufacturer-related commands outputs would need privileges of a root user which would require a TAC service request.

Triggers Leading to the Switch SSD Lifespan Exhaustion

• In an effort to ensure serviceability and the ability to provide root cause for events long after that event has occurred, ACI APICs and switches have verbose logging for each component that is persistent to SSD.
  
  
• The above logging capabilities require a high amount of disk writes and overwrite. The capability of the disks to perform this is not infinite hence, there's a lifetime associated with the drive.  Instability in the network (huge number of moves or policy updates etc.), for a long-continued period, can lead to premature exhaustion of a switch SSD.
  
  
• There have been many enhancements to optimize the writes and maintain serviceability while decreasing the amount of data written to disk. These changes were introduced across different versions. Refer to the Code Optimization section.
  
  
• For a version with the below enhancements integrated, if you still experience fast SSD consumption like 2/3 P/E cycle each day, it is very likely caused by DB write instead of logging as most of them are written to RAMFS, logs are only moved to SSD after ZIP when rotation is required.

Code Optimization

There have been many enhancements to optimize the writes to maintain serviceability while decreasing the amount of data written to disk. These changes were introduced across different versions.

CSCve88634 Raise a fault when the SSD exceeds its lifetime.
                            • Added to 2.1(4), 2.2(4), 2.3(1o), 3.1(2m)
CSCvi32353 F3073 Revision to SSD lifetime exceeded detection logic
CSCvh73803 create ramfs partition for tmp_logs to decrease logging impact on SSD
CSCvm97108 Bug to track SSD overprovisioning changes

CSCvt36458  Removed SQL DB persistent Layer for switches.

                             • Added to 4.2(6d) and 5.1(1h)

3.2.4d and later, 4.x and later has all the fault monitoring and optimization enhancements required to prolong the SSD lifespan. This of course does not mean that upgrading to these releases absolutely prevents the SSD lifespan issue.

Reload Type Behavior Change

CSCvt36458 changed the expected behavior of an ACI Switch node ungraceful reload:

Reload Type	Before CSCvt36458	After CSCvt36458   [4.2(6d)+ and 5.1(1h)+]
Upgrades	Stateless	Stateless
Graceful Reloads (Manual reloads, hap-resets)	Stateful	Stateful
Ungraceful Reloads (Kernel panic, power cycle)	Stateful	Stateless

In summary, Graceful switch reloads continue to use their DB on bringup. Ungraceful switch reloads now require that all policies are pulled from the APIC, similar to an upgrade.

Miscellaneous FAQ

Can the SSD be replaced in the field?
No. The SSD is not a field-replaceable unit. The entire Chassis is RMAed when the failure is on the leaf. On modular Spines, you have to RMA the supervisor.

Should you proactively replace switch hardware?

TAC engineers can check smartctl values and evaluate which switches have more lifetime, so that replacements can be made for the most urgent switches immediately, and then gradually move to replace those switches that are not as urgent.

Are there specific hardware PIDs that are more susceptible to SSD failures?

No. The SSD lifespan usage can vary across different devices in the same environment. Usage of the SSD disks depends on how often the device needs to write the logs for event keeping. A device that registers a large number of events contributed by network instability has the potential of reaching its SSD lifespan sooner than others. Hence Cisco advocates the need to use the later releases to ensure that code optimization helps with controlling the SSD writes and prolongs the SSD lifespan.