The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the SSD lifespan exhaustion behavior on ACI switch platforms.
If you have landed on this page as a result of an SSD fault raised against APIC (not a switch node), refer to FN64329.
This TechNote documents the APIC SSD replacement procedure.
ACI Leaf and Spine nodes leverage solid-state drives (SSD) for storage and event logging. All SSD, irrespective of the vendor type, have a fixed lifespan based on specific attributes defined by the manufacturer. Some examples include the number of writes and erases as well as the volume of data transferred to the drives over a period of time. The SSD lifespan on Leaf and Spine nodes can become exhausted due to regular wear and tear. If the user does not take proper action against a switch that has SSD approaching exhaustion, the switch could crash due to the SSD going into read-only mode. For this reason, it is imperative to prioritize SSD replacement in the event F3073 fault is raised against a given node. In such a scenario, the whole switch is replaced as the SSD is not a field replaceable unit.
F3073 was implemented in 2.1(4), 2.2(4), 2.3(1o), and 3.1(2m). Any subsequent software versions have the functionality to raise F3073.
Note: If you run an ACI software version that does not have SSD monitoring functionality, it is strongly advised to upgrade to a version where such monitoring is available.
1. When the switch fails to boot, on the console you can see these errors:
/dev/hd-cfg0: ********** WARNING: Filesystem still has errors **********
e2fsck 1.42.1 (17-Feb-2012)
/dev/hd-cfg1: recovering journal
/sbin/e2fsck: unable to set superblock flags on /dev/hd-cfg1
2. The system is able to proactively raise faults related to SSD (F3073 or F3074 based on the SSD lifespan usage) on later code releases (covered in later section)
F3074: fltEqptFlashFlash-minor-alarm (80% lifetime)
F3073: fltEqptFlashFlash-worn-out (90% lifetime)
3. You may also see fault F3525 raised for high SSD usage. This fault is often confused with F3073/F3074.
# fault.Inst
code : F3525
ack : no
annotation :
cause : equipment-flash-warning
changeSet : deltape (New: 21), peCycles (New: 1678), tbw (New: 32.465179), warning (New: yes)
childAction :
created : 2019-08-05T18:22:01.455-07:00
delegated : no
descr : High SSD usage observed. Please check switch activity and contact Cisco Technical Support about high SSD usage.
dn : topology/pod-1/node-206/sys/ch/supslot-1/sup/flash/fault-F3525
domain : infra
extMngdBy : undefined
highestSeverity : warning
lastTransition : 2019-08-05T18:24:02.029-07:00
lc : raised
modTs : never
occur : 1
origSeverity : warning
prevSeverity : warning
rn : fault-F3525
rule : eqpt-flash-flash-warning-alarm
severity : warning
status :
subject : flash-warning-alarm
type : operational
Fault F3525 is raised if P/E (program-erase) cycles increment by more than 21 in 7 days. It does not mean that the SSD is worn out, simply that there is a lot of churn which may eventually cause the SSDs to be worn out.
You need to work with TAC to understand what is causing this churn and address it.
There is also a policy to change the thresholds that cause F3525 to be raised for a specific type of SSDs (Access Policies > Policies > Switch > Equipment Flash Config Policies)
There are two ways to check the SSD lifespan status:
SSD lifetime monitoring was added as part of CSCve88634 in these versions of code:
A minor fault is raised when the drive approaches 80% of its lifetime and a major fault is raised when it exceeds 90% of its lifetime.
This allows network operators the capability to monitor and proactively replace any switch before it fails due to SSD lifetime exceeded.
Previous versions of code did not have SSD lifetime check for switch SSDs and therefore, even if the switch was approaching the SSD expiration there would not be a fault raised.
SSD usage logging was introduced starting in these releases 3.2(5d) 3.2(6i) 3.2(7f) 4.1(1i).
a-leaf101# cat /mnt/pss/ssd_log_amp.log (Tested on 4.2.3l)
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Mon Dec 16 19:00:01 EST 2019 16531370 0 29398 12350 100 0 122396965772 3825894828 60841125365 16 0.00 % 0 0
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Tue Dec 17 19:00:01 EST 2019 16531725 0 29422 12355 100 0 122447755358 3827482016 60858831240 16 0.00 % 12355 61606629993
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Wed Dec 18 19:00:01 EST 2019 16532222 0 29446 12359 100 0 122495197374 3828964594 60875667807 16 0.00 % 12359 61636366134
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Thu Dec 19 19:00:01 EST 2019 16533056 0 29470 12364 100 0 122542269672 3830435617 60892533342 16 0.00 % 12364 61666601865
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Fri Dec 20 19:00:01 EST 2019 16534010 0 29494 12367 100 0 122587126224 3831837398 60908759296 16 0.00 % 12367 61694592882
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sat Dec 21 19:00:01 EST 2019 16535311 0 29518 12372 100 0 122631804318 3833233605 60925205662 16 0.00 % 12372 61723045022
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sun Dec 22 19:00:01 EST 2019 16536727 0 29542 12376 100 0 122676829388 3834640654 60941741722 16 0.00 % 12376 61751623726
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Mon Dec 23 19:00:01 EST 2019 16538321 0 29566 12380 100 0 122722137414 3836056544 60958504969 16 0.00 % 12380 61780395692
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Tue Dec 24 19:00:01 EST 2019 16539858 0 29591 12384 100 0 122767087802 3837461258 60975050343 16 0.00 % 12384 61808582833
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Wed Dec 25 19:00:01 EST 2019 16541121 0 29615 12388 100 0 122812238580 3838872234 60991578411 16 0.00 % 12388 61837188237
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Thu Dec 26 19:00:01 EST 2019 16542653 0 29639 12393 100 0 122857007868 3840271286 61008101155 16 0.00 % 12393 61865429457
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Fri Dec 27 19:00:01 EST 2019 16544352 0 29663 12397 100 0 122901807026 3841671272 61024757511 16 0.00 % 12397 61893705871
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sat Dec 28 19:00:01 EST 2019 16546213 0 29687 12401 100 0 122946927200 3843081289 61041459287 16 0.00 % 12401 61922169689
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sun Dec 29 19:00:01 EST 2019 16549522 0 29711 12405 100 0 122991584018 3844476827 61058889999 16 0.00 % 12405 61950124731
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Mon Dec 30 19:00:01 EST 2019 16551885 0 29735 12410 100 0 123036252528 3845872732 61075846564 16 0.00 % 12410 61977362529
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Tue Dec 31 19:00:01 EST 2019 16553858 0 29759 12414 100 0 123098133082 3847806568 61093085451 16 0.00 % 12414 62022286518
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Wed Jan 1 19:00:01 EST 2020 16555127 0 29783 12418 100 0 123142018610 3849178024 61109758713 16 0.00 % 12418 62048933159
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Thu Jan 2 19:00:01 EST 2020 16556532 0 29808 12422 100 0 123186637556 3850572390 61126687626 16 0.00 % 12422 62076878843
Model SerialNo SW-Version Date R.Error(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attr(210) TBW(246) WAF DWAF DPE DTBW
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-03 16557977 0 29832 12428 100 0 123278298184 16 0.00 % 0 0
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-04 16558440 0 29856 12433 100 0 123323818606 16 0.00 % 5 45520422
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-05 16559031 0 29880 12438 100 0 123369348610 16 0.00 % 5 45530004
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-06 16561567 0 29904 12443 100 0 123415316270 16 0.00 % 5 45967660
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-07 16564701 0 29928 12448 100 0 123460483898 16 0.00 % 5 45167628
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-08 16565955 0 29952 12453 100 0 123505550532 16 0.00 % 5 45066634
ssd_log_amp.log file is in the 1of3 switch tech supports
For any additional manufacturer-related commands outputs would need privileges of a root user which would require a TAC service request.
There have been many enhancements to optimize the writes to maintain serviceability while decreasing the amount of data written to disk. These changes were introduced across different versions.
CSCve88634 Raise a fault when the SSD exceeds its lifetime.
• Added to 2.1(4), 2.2(4), 2.3(1o), 3.1(2m)
CSCvi32353 F3073 Revision to SSD lifetime exceeded detection logic
CSCvh73803 create ramfs partition for tmp_logs to decrease logging impact on SSD
CSCvm97108 Bug to track SSD overprovisioning changes
CSCvt36458 Removed SQL DB persistent Layer for switches.
• Added to 4.2(6d) and 5.1(1h)
3.2.4d and later, 4.x and later has all the fault monitoring and optimization enhancements required to prolong the SSD lifespan. This of course does not mean that upgrading to these releases absolutely prevents the SSD lifespan issue.
CSCvt36458 changed the expected behavior of an ACI Switch node ungraceful reload:
Reload Type | Before CSCvt36458 ![]() |
After CSCvt36458 ![]() |
Upgrades | Stateless | Stateless |
Graceful Reloads (Manual reloads, hap-resets) | Stateful | Stateful |
Ungraceful Reloads (Kernel panic, power cycle) | Stateful | Stateless |
In summary, Graceful switch reloads continue to use their DB on bringup. Ungraceful switch reloads now require that all policies are pulled from the APIC, similar to an upgrade.
Can the SSD be replaced in the field?
No. The SSD is not a field-replaceable unit. The entire Chassis is RMAed when the failure is on the leaf. On modular Spines, you have to RMA the supervisor.
Should you proactively replace switch hardware?
TAC engineers can check smartctl values and evaluate which switches have more lifetime, so that replacements can be made for the most urgent switches immediately, and then gradually move to replace those switches that are not as urgent.
Are there specific hardware PIDs that are more susceptible to SSD failures?
No. The SSD lifespan usage can vary across different devices in the same environment. Usage of the SSD disks depends on how often the device needs to write the logs for event keeping. A device that registers a large number of events contributed by network instability has the potential of reaching its SSD lifespan sooner than others. Hence Cisco advocates the need to use the later releases to ensure that code optimization helps with controlling the SSD writes and prolongs the SSD lifespan.
Revision | Publish Date | Comments |
---|---|---|
1.0 |
09-Jan-2020 |
Initial Release |