Introduction to Persistent Memory

Persistent Memory Modules

Cisco IMC and Cisco UCS Manager Release 4.0(4) introduce support for the Intel® Optane™ Data Center persistent memory modules on the UCS M5 servers that are based on the Second Generation Intel® Xeon® Scalable processors. Starting with Cisco UCS Manager Release 4.2, the support for the Intel® Optane™ Data Center persistent memory modules on the UCS M6 servers that are based on the Second Generation Intel® Xeon® Scalable are also provided. Intel® Optane™ DC persistent memory modules can be used only with the Second Generation Intel® Xeon® Scalable processors.

This release provides the ability to configure Intel® Optane™ DC persistent memory modules through Cisco IMC and Cisco UCS Manager. Persistent memory modules are non-volatile memory modules that bring together the low latency of memory and the persistence of storage. Persistent memory modules provide faster access to data and retain data across power cycles, based on the mode.

Persistent Memory Module Population Guidelines

To ensure the best server performance, it is important that you are familiar with memory performance guidelines and population rules before you install or replace persistent memory modules.

The population guidelines can be divided into the following, based on the number of CPU sockets:


Note
For UCS M5 and M6 B-Series and C-Series servers, the Near Memory (NM): Far Memory ratio (FM) ratio (DRAM + PMEM) is supported between 1:4 and 1:16 in 100% memory mode.

Example, 8 + 4 DRAM (16G) in 8 slots [populated in slot-1 of each-channel] + PMEM (128G) in 4 slots [A2, C2, E2, G2] =128G [8*16G] : 512G [4*128G] that is 1:4.

Cisco UCS-Managed and Host-Managed Modes

You can manage persistent memory module configuration using UCS-managed mode (Cisco IMC or UCS Manager) or the host-managed mode. In the UCS-managed mode, you can use Cisco UCS Manager or Cisco IMC to configure and manage persistent memory modules. In the host-managed mode, you can use the host tools to configure and manage persistent memory modules. When using the UCS-managed, you can perform configuration tasks using the Cisco UCS management interfaces or the host tools.

Cisco recommends that you use Cisco UCS management interfaces for all security operations and region management, and use the host tools only for namespace configurations if required.

Goal

A goal is used to configure how persistent memory modules connected to a CPU socket are used. You can configure a persistent memory module to be used in Memory Mode, App Direct Mode, or Mixed Mode. When a persistent memory module is configured as 100% Memory Mode, it can be used completely as volatile memory. Conversely, when it is configured as 0% Memory Mode, it becomes App Direct Mode and can be used completely as persistent memory. When you configure a persistent memory module as x% Memory Mode, x% is used as memory and the remaining is used as persistent memory. For example, when you configure 20% Memory Mode, 20 percent of the persistent memory module is used as memory and the remaining 80 percent is used as persistent memory. This mode is called Mixed Mode.

In mixed mode, the percentage may not linearly translate into the actual memory available. The actual memory size obtained may not accurately correspond to the specified percentage. Also, if the percentage is changed, the resultant memory obtained may not change in the same proportion.


Note

In memory mode, DDR4 memory is used as a cache layer to the persistent memory module, and, is therefore not visible to the OS. For example, if you have 1.5 Tb of persistent memory in memory mode and 256 Gb DDR4, the OS/Hypervisor would still only see 1.5 Tb of total memory.

For completely persistent memory or mixed mode, you can configure the persistent memory type as App Direct or App Direct Non Interleaved. The App Direct type configures all the memory modules connected to a socket into one interleaved set, and creates one region for it. The App Direct Non Interleaved type configures one region for each memory module.

You can create a goal only at the server level for all sockets together, and not for each socket separately. After a goal is created and applied on a server, the regions that are created are visible in the server inventory. A region is a grouping of one or more persistent memory modules that can be divided up into one or more namespaces. When a host application uses namespaces, it stores application data in them.


Note

For UCS M5 S-Series servers:

  • The only supported goal configuration is 0% Memory Mode and App Direct Non Interleaved persistent memory type.

    The persistent memory modules for S-Series servers are shipped with 100% Memory Mode. To use the persistent memory modules for S-Series servers do one of the following:

    • Perform a persistent memory scrub (Cisco UCS Manager) or reset persistent memory module to factory defaults (Cisco IMC).

    • Create a goal with 0% Memory Mode.

  • The system does not restrict you from configuring any other combination of Memory Mode % and persistent memory type. However, unsupported goal configurations cannot be used.

  • After a persistent memory scrub (Cisco UCS Manager) or reset persistent memory module to factory defaults (Cisco IMC), the default goal is 0% Memory Mode and App Direct Non Interleaved persistent memory type.

Goal modification is a destructive operation. When a goal is modified, new regions are created based on the modified goal configuration. This results in the deletion of all existing regions and namespaces on the associated servers, which leads to the loss of data currently stored in the namespaces.

Before modifying the Persistent Memory Type in a goal, delete the existing namespaces. This is because, in the App Direct persistent memory type you do not specify a DIMM number for each namespace. In the App Direct Non Interleaved persistent memory type, each namespace has a DIMM number specified.

For UCS M5 and M6 B-Series and C-Series servers, deleting a goal deletes all related regions and namespaces on the associated servers, and disables security. For UCS M5 S-Series servers, deleting a goal deletes all namespaces on the associated servers, and disables security. Goal deletion also returns the persistent memory module to its default state. The default state of a persistent memory module is:

  • UCS M5 and M6 B-Series and C-Series servers—100% Memory Mode

  • UCS M5 S-Series servers—0% Memory Mode and App Direct Non Interleaved persistent memory type

Region

A region is a grouping of one or more persistent memory modules that can be divided up into one or more namespaces. A region is created based on the persistent memory type selected during goal creation.

When you create a goal with the App Direct persistent memory type, it creates one region for all the memory modules connected to a socket. When you create a goal with the App Direct Non Interleaved persistent memory type, it creates one region for each memory module.

Namespace

A namespace is a partition of a region. When using the App Direct persistent memory type, you can create namespaces on the region mapped to the socket. When using the App Direct Non Interleaved persistent memory type, you can create namespaces on the region mapped to a specific memory module on the socket.

A namespace can be created in Raw or Block mode. A namespace created in Raw mode is seen as a raw mode namespace in the host OS. A namespace created in Block mode is seen as a sector mode namespace in the host OS.

Deleting a namespace is a destructive operation, and results in the loss of data stored in the namespace.

Security

You can enable security on a persistent memory module and lock it by using a secure passphrase. In Release 4.0(4), the secure passphrase for persistent memory modules is stored and managed locally.

Local Security

You can configure local security for a persistent memory module. This contains the secure passphrase to be applied on the servers. All the persistent memory modules on a server are secured with a single secure passphrase. Until you configure a secure passphrase, the persistent memory modules are not locked or secured.

Configuring a secure passphrase has the following constraints:

  • The minimum length of the secure passphrase must be 8 characters, and the maximum length must be 32 characters.

  • The allowed characters are letters (A-Z, a-z), numbers (0-9), special characters (!, @, #, $, %, ^, &, *, -, _, +, =), or a combination of all of them.

A deployed secure passphrase is the passphrase that is currently deployed on a server. You can modify a configured secure passphrase after you correctly enter the currently deployed secure passphrase for verification, and the new secure passphrase to be set. Before disabling security, ensure that all persistent memory modules are unlocked.

Security States

The following table describes each possible security state of a persistent memory module:

Security State

Description

Disabled

This means that the security of persistent memory modules is disabled.

Enabled

This means that the security of persistent memory modules is enabled.

Locked

This means that security is enabled for persistent memory modules, and they are locked with a secure passphrase. The secure passphrase is required to unlock them.

Unlocked

This means that the security of persistent memory modules is enabled, and they are currently unlocked.

Frozen

This means that the host OS can configure the persistent memory modules and use them, but cannot configure the security of these persistent memory modules.

Not Frozen

This means that the host OS can configure the persistent memory modules, use them, and configure the security of these persistent memory modules. This state is typically seen in the host-managed mode.

Count Expired

This means that the Max Retry Count, which is maximum number of unlock attempts allowed, has expired. It is no longer possible to unlock the persistent memory modules until the next reset or reboot.

The maximum number of incorrect unlock attempts allowed:

  • Host-managed mode—3 incorrect unlock attempts

  • UCS-managed mode with security disabled—3 incorrect unlock attempts

  • UCS-managed mode with security enabled—2 incorrect unlock attempts

Count Not Expired

This means that the Max Retry Count, which is maximum number of unlock attempts allowed, has not yet expired. It is still possible to unlock the persistent memory modules with the secure passphrase.

The maximum number of incorrect unlock attempts allowed:

  • Host-managed mode—3 incorrect unlock attempts

  • UCS-managed mode with security disabled—3 incorrect unlock attempts

  • UCS-managed mode with security enabled—2 incorrect unlock attempts

These are the possible security statuses that are displayed for each persistent memory module :

Status

Description

Disabled, Unlocked, Frozen, Count Not Expired

Security is disabled, secure passphrase is not configured, the host OS can configure the persistent memory modules and use them, but cannot configure the security of these persistent memory modules, retry count has not expired

Disabled, Unlocked, Not Frozen, Count Not Expired

Security is disabled, secure passphrase is not configured, the host OS can configure security of persistent memory modules, retry count has not expired

Enabled, Unlocked, Frozen, Count Not Expired

Security is enabled, persistent memory modules are unlocked, the host OS can configure the persistent memory modules and use them, but cannot configure the security of these persistent memory modules, retry count has not expired

Enabled, Locked, Not Frozen, Count Not Expired

Security is enabled, persistent memory modules are locked by using the secure passphrase, the host OS can configure security of persistent memory modules, retry count has not expired

Enabled, Locked, Not Frozen, Count Expired

Security is enabled, persistent memory modules are locked by using the secure passphrase, the host OS can configure security of persistent memory modules, retry count has expired

Unknown

The host is powered down.

These are the overall security states that are displayed for each server.

Overall Security State

Description

Disabled-Frozen

Persistent memory modules are in UCS Managed mode and security is disabled on all persistent memory modules.

Disabled

Persistent memory modules are in Host Managed mode and security is disabled on all persistent memory modules.

Unlocked-Frozen

 Persistent memory modules are in UCS Managed mode and security is enabled on all persistent memory modules.

Enabled, Locked

Persistent memory modules are in Host Managed mode and security is enabled on all persistent memory modules.

Mixed-State

Some persistent memory modules have security enabled and the rest have security disabled.

BIOS Support for Persistent Memory Module Security

The following outline the BIOS support for persistent memory module security:

  • The BIOS supports one secure passphrase for all persistent memory modules in a server.

  • When the BIOS is provided with a secure passphrase to lock all the persistent memory modules, it does the following for each persistent memory module:

    • Enable security for the persistent memory module

    • Lock the persistent memory module with the secure passphrase provided

    After all the persistent memory modules are locked, the server is rebooted.

  • For the host OS to use the persistent memory modules, after the server reboots, the BIOS unlocks the persistent memory module and puts it in a Frozen state. In this state, the host OS can configure the persistent memory modules and use them, but cannot change the security passphrase of the persistent memory modules. The state of each persistent memory module, then, appears as Unlocked and Frozen.

  • The BIOS does not support goal modification and secure passphrase modification operations at the same time. These operations, however, can be performed one after the other. Performing these operations simultaneously will result in failure.

    Cisco UCS Manager prevents you from trying to perform goal modification and secure passphrase modification operations at the same time.

Persistent Memory Server Operations

Secure Erase

The secure erase functionality allows you to erase data in a region, namespaces and disable security in a specific persistent memory module. You can perform secure erase on a specific set of persistent memory modules, or all the persistent memory modules on a server. The secure erase functionality is also supported when security is disabled, in which case, no passphrase is required.

  • A set of persistent memory modules—You can use this option to perform secure erase on a specific set of one or more persistent memory modules. If the server is configured with a secure passphrase, you must provide the secure passphrase for verification. When this operation is complete, data in the regions for the selected persistent memory modules is erased, all namespaces on these persistent memory modules are deleted, and security is disabled on these persistent memory modules.

  • All the persistent memory modules on a server—You can use this option to perform secure erase on the persistent memory configuration of the server. If the server is configured with a secure passphrase, you must provide the secure passphrase for verification. When this operation is complete, data in all the regions on the server is erased, all namespaces on the server are deleted, and security is disabled on all persistent memory modules on the server.

Unlock Foreign Persistent Memory Modules

When a persistent memory module that is locked with a secure passphrase is moved to a different server that has security enabled with a different secure passphrase, it remains locked on the new server. You must unlock this persistent memory module to be able to use it on the new server. After you unlock this persistent memory module by using its deployed secure passphrase, it is secured with the single secure passphrase of the new server. For example, if persistent memory module DIMM_A2 from server 1 is locked with secure passphrase "A", and is then moved to server 2, it is identified as a locked persistent memory module on server 2. The secure passphrase of the persistent memory modules on server 2 is "B". To manage persistent memory module DIMM_A2 on server 2, you must unlock the module by using secure passphrase "A". After persistent memory module DIMM_A2 is successfully unlocked, it is secured with the secure passphrase of server 2, which is "B".

Persistent Memory Scrub

Persistent memory scrub allows you to remove the persistent memory configuration and data from the persistent memory modules on a server.

In Cisco IMC, you can scrub persistent memory by resetting the persistent memory modules to factory defaults.

In Cisco UCS Manager, you can scrub persistent memory by using one of the following methods:

  • Disassociating the service profile and the scrub policy, which has the persistent memory scrub option set to yes

  • Performing a Reset to Factory Default operation on the server with the persistent memory scrub option set to yes

  • Deleting a goal

After persistent memory scrub is complete, the following happen:

  • All persistent memory data is erased

  • Persistent memory configuration is reset to factory default settings.

    For B-Series and C-Series servers, 100% Memory Mode is applied. For S-Series servers, 0% Memory Mode and App Direct Non Interleaved type are applied.

  • Persistent memory module security is disabled

Persistent Memory Firmware Update

Persistent memory modules have firmware running on it. This firmware is packaged in the blade and rack server (B and C) bundles. Ensure that the blade and rack package versions are set to Release 4.0(4) or later releases.

In Cisco UCS Manager, you can use these firmware packages in service profiles to upgrade persistent memory firmware by defining a host firmware policy and including it in the service profile associated with a server. For instructions about defining a host firmware policy in Cisco UCS Manager, see the Cisco UCS Manager Firmware Management Guide.

You can upgrade or downgrade persistent memory firmware on the standalone Cisco UCS C-Series and Cisco UCS S-Series servers using the Cisco UCS Host Update Utility (HUU). For instructions to update the firmware, see Cisco Host Upgrade Utility User Guide

Cisco recommends that all the persistent memory modules on a server run the same and the latest firmware version.

Persistent Memory Policy and Its Components in Cisco UCS Manager

This section describes the persistent memory policy, its components, and guidelines for configuring them in Cisco UCS Manager.

Persistent Memory Policy for Cisco UCS Manager

In Cisco UCS Manager, a persistent memory policy allows you to configure how persistent memory modules are used. It contains goals and namespaces.

You must include this policy in a service profile and that service profile must be associated with a server for this policy to take effect. While each service profile can have one persistent memory policy, one persistent memory policy can be mapped to several service profiles.

The behavior of the persistent memory policy and its components will be based on whether the policy is referred to by a server or not. If the policy is not referred by any server, you can perform all operations—create, modify, delete—without any restrictions. If the policy is referred to by a server, specific restrictions apply. For example, when a persistent memory policy is referred to by a server, namespaces configured by it cannot be modified.

Some of the operations that you can perform on a persistent memory policy and its components are destructive. Such an operation results in the loss of created structures and data. The operations that you can perform on a persistent memory policy and its components, which could lead to loss of data are:

  • Modification of a goal

  • Deletion of a goal

  • Deletion of a namespace

  • Replacement of a persistent memory policy in a service profile associated with a server

To perform a destructive operation, you must explicitly apply the new configuration on the server. You can do this by using the Force Configuration option in the persistent memory policy. You must select this option everytime you perform a destructive operation.

Guidelines for Configuring Persistent Memory Policy Components

Here are the guidelines for configuring the persistent memory policy components:

Goal

Goal creation, modification, and deletion can all be done without any endpoint restriction when the service profile that it is included in is not associated to a server. Data loss is not applicable in these cases because the policy is not applied to a server.

Goal creation on a server with a pre-existing persistent memory configuration is a destructive operation. However, goal creation without any preexisting config is not destructive.

Goal modification is a destructive operation. When a goal is modified, new regions and namespaces are created based on the modified goal configuration. This results in the deletion of all existing regions and namespaces on the associated servers, which leads to the loss of data currently stored in the namespaces.

When you delete a goal, it deletes all related regions and namespaces on the associated servers. It also returns the persistent memory module to its default state. The default state of a persistent memory module is:

  • UCS M5 and M6 B-Series and C-Series servers—100% Memory Mode.

  • UCS M5 S-Series servers—0% Memory Mode and App Direct Non Interleaved persistent memory type.

Namespace

Namespace creation, modification, and deletion can all be done without any endpoint restriction when the persistent memory policy that contains the namespace is not referred to by a server. Data loss is not applicable in these cases because the policy is not applied to a server.

You can modify a namespace only if the persistent memory policy that contains the namespace is not referred to by a server. Modifying a namespace is not an allowed operation if the persistent memory policy that contains the namespace is referred to by a server.

Deleting a namespace is a destructive operation, and results in the loss of the namespace, and the data stored in it.

Local Security

In Cisco UCS Manager, a local security policy for persistent memory modules allows you to configure the secure passphrase for the server. It contains the secure passphrase for the persistent memory policy, which is then applied on servers. Initially, the security state for all the persistent memory modules on a server is set to Disabled. When a persistent memory policy with a secure passphrase is applied on the server, the security state for all the persistent memory modules on the server is set to Enabled, and the modules are locked with the specified, single secure passphrase.

You can modify local security configuration after you correctly enter the currently deployed secure passphrase for verification, and the new secure passphrase to be set.

Local security configuration can be deleted. Deleting the local security configuration unlocks the persistent memory module and disables security for the persistent memory policy.

Unlock Foreign Persistent Memory Modules

To unlock the foreign persistent memory modules in Cisco UCS Manager, you can use the following workflow:

  1. Decommission the server.

  2. Change the persistent memory modules.

  3. Recommission the server.

  4. Associate the server to a service-profile without a persistent memory policy.

  5. Ensure that the server is in the powered-on state, and BIOS POST is completed.

  6. In the persistent memory inventory, select the persistent memory modules to be unlocked, and perform the unlock foreign DIMMs operation by providing the secure passphrase of the persistent memory modules.

  7. Check whether the persistent memory modules get unlocked after the ExecuteActions FSM completes. Now, the persistent memory modules are ready to be used.

  8. Attach a persistent memory policy.

  9. Check whether the associate FSM completes.

Unlocking Foreign Persistent Memory Modules Based on Security

The following workflows apply to unlocking foreign persistent memory modules based on server security configuration:

When security is disabled:

  • Unassigning the persistent memory policy will change management to host-managed mode.

  • After the unlock operation, assigning a new persistent policy will override the existing configuration in the system. If the server has any existing regions or namespace, they will be deleted.

When security is enabled:

  • Server-level security is enabled.

  • Unassigning the security-enabled persistent memory policy will change management to host-managed mode.

  • You can unlock foreign persistent memory modules, after which you must apply server-level security to the unlocked persistent memory modules.

  • After the unlock operation, assigning a new persistent policy will override the existing configuration in the system. If the server has any existing regions or namespace, they will be deleted.