Cisco HyperFlex Software Advisory for HX Release 5.0(2b)

Cisco Software Advisory for HX Release 5.0(2b)

Cisco engineering has identified issues with HXDP Release 5.0(2b) that may affect your use of this software. Please review the Software Advisory notice here to determine if any of the issues apply to your environment and apply any steps required to address the issue.

 

Contents:

 

 

Software Advisory for CSCwf98678: Potential All Paths Down after a Drive Replacement

 

 

December 12, 2023 (Update)

 

Dear Cisco Customer,

 

Cisco engineering has identified the following software issue with Cisco HyperFlex Release 5.0(2b), 5.0(2c), and 5.0(2d). Please review the Software Advisory notice here to determine if the issue applies to your environment and apply any steps required to address the issue.

 

For more comprehensive information about what is included in this software, refer to the Cisco software Release Notes, available from the Product Selector tool. From this page, select the product you are interested in. Release Notes are under "General Information" on the product page.

 

 

 

Reason for Advisory:

Potential cluster-wide All Paths Down (APD) at some point after a drive is replaced due to stale data.

 

New clusters should be installed with HXDP Release 5.0(2e) or later.

 

Affected Software:

Cisco HyperFlex Release 5.0(2b), 5.0(2c), and 5.0(2d).

 

Symptom:

The request is to ensure users are not impacted during the upgrade process.

Potential issues while using the affected releases include:

·         All Paths Down (APD)

·         Node Panic/outage

·         Slow or no healing during upgrade after SCVM restarts or if storfs process in the SCVM restarts.

 

Conditions:

A HyperFlex cluster running one of the affected software releases incurs a drive failure at some point in time.

Workaround:

If your HyperFlex cluster HXDP version is currently: 5.0(2b), 5.0(2c), or 5.0(2d):

·         Review https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwf98678

·         Review the Script Read Me Files:

https://software.cisco.com/download/home/286305544/type/286305994/release/5.0(2e)

https://software.cisco.com/download/home/286305544/type/286305994/release/5.5(1a)

 

All other HyperFlex clusters can upgrade without engaging TAC; unless Hypercheck results directs you to.

 

Software Advisory for CSCwf34019: A race condition during handling of flusher restart may result into discarding data to be flushed

 

 

December 12, 2023 (Update)

 

Dear Cisco Customer,

 

Cisco engineering has identified the following software issue with the release that you have selected that may affect your use of this software. Please review the Software Advisory notice here to determine if the issue applies to your environment. You may proceed to download this software if you have no concerns with the issue described.

 

For more comprehensive information about what is included in this software, refer to the Cisco software Release Notes, available from the Product Selector tool. From this page, select the product you are interested in. Release Notes are under "General Information" on the product page.

 

 

Reason for Advisory:

A race condition during handling of flusher restart may result into discarding yet-to-be-flushed data

 

Affected Software:

Cisco HyperFlex Release 5.0(2a), 5.0(2b) and 5.0(2c).

 

Symptoms:

·         Multiple disks on multiple nodes randomly get retired.

·         Disks becomes available for sometimes and then they again get retired.

·         A cluster enters in CRITICAL state.

 

Conditions:

·         During de-staging activity, user data from the caching tier is written to the persistent tier.

·         During the actual write operation, state of the one of the mirror on a remote node may change due to disk status change.

·         When this happens, the de-staging activity notices the change of the mirror status and retries the operation.

·         Due to an extremely rare race condition, the de-staging activity incorrectly ignored the mirror/disk status change and declared that de-staging activity has been successful (instead of retry). Due to this, it missed persisting of the data, however it updated other metadata structures with success. At this stage the actual contents of the data block on the disk are incorrect.

 

·         During the subsequent read access, when data is read form the block and when its correctness is verified, it did not match with the expected result. This caused the read operation to declare that disk block is affected resulting into disk retire. This sequence of access-retire continued whenever that particular data block has been accessed.

 

Workaround:

If your HyperFlex cluster HXDP version is currently: 5.0(2b), 5.0(2c), or 5.0(2d):

·         Review https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwf98678

·         Review the Script Read Me Files:

https://software.cisco.com/download/home/286305544/type/286305994/release/5.0(2e)

https://software.cisco.com/download/home/286305544/type/286305994/release/5.5(1a)

 

All other HyperFlex clusters can upgrade without engaging TAC; unless Hypercheck results directs you to.

 

 

 

 

Software Advisory for CSCwe41201: Stretched cluster may result in All Paths Down (APD) due to incorrect mirror distribution post upgrade to 5.0(2b)

 

 

February 27, 2023

 

Dear Cisco Customer,

 

Cisco engineering has identified the following software issue with the release that you have selected that may affect your use of this software. Please review the Software Advisory notice here to determine if the issue applies to your environment. You may proceed to download this software if you have no concerns with the issue described.

 

For more comprehensive information about what is included in this software, refer to the Release Notes for Cisco HX Data Platform, Release 5.0, available from the Product Selector tool. From this page, select the product you are interested in. Release Notes are under "General Information" on the product page.

 

Stretched clusters may experience All Paths Down (APD) due to incorrect mirror distribution post upgrade to 5.0(2b). Cisco recommends not upgrading a HyperFlex stretched cluster to HXDP Release 5.0(2b).

 

Affected Software:

Cisco HyperFlex Data Platform (HXDP) Release 5.0(2b)

 

Symptom:

After upgrading to HXDP 5.0(2b), a HyperFlex stretched cluster may experience all paths down (APD) condition due to a software issue where:

·         The Cluster resource manager's view and individual node's view about certain resources does not converge.

·         Due to this no new IOs are accepted by the cluster resulting in APD.

 

Note: Data Center, DC-no-FI and Edge cluster types are not impacted by this advisory.

 

Conditions:

HyperFlex stretched clusters initially deployed using HXDP Release 4.0(2f) or earlier and upgraded to HXDP Release 5.0(2b) may experience an all paths down (APD) condition with the first cluster rebalance event after the upgrade.

 

Workaround:

If a stretched cluster is upgraded to HXDP Release 5.0(2b) and there is an outage after the upgrade, please contact Cisco TAC for cluster recovery.

 

Cisco is working to release a new HXDP Release that will fix this defect.