Customize Fleet Upgrade

This document covers the following topics:

Use Parallel Upgrades With Acceptable Failures

Fleet Upgrade provides the Parallel upgrades and Acceptable failures fields to help you both speed up and control long upgrade runs with many devices.

As explained in Run a Fleet Upgrade, whenever you create a Fleet Upgrade job, you can select up to 50 devices to be upgraded. Even though the upgrade process is entirely automated, depending on the number and size of the upgrades being performed on each of these devices, this can result in upgrade runs lasting many hours. This is especially true if you must perform each upgrade one at a time, in series, and if you can't cancel the run if you start experiencing upgrade failures.

To help with these issues, you can use the Parallel upgrades field to specify how many upgrades you want performed at the same time, in parallel. If you enter a Parallel upgrades value equal to the total number of devices to be upgraded (up to the maximum of 50), all the upgrades will take place at the same time. If you leave Parallel upgrades set to the default value of 1, Fleet Upgrade performs each of them one at a time.

As a practical matter, many users specify a lower Parallel upgrades value, such as 5 or 10. Doing this helps conserve processing resources and ensures that only a few of the network devices in a 50-device group will be offline at any one time.

With this type of Parallel upgrades value, Fleet Upgrade performs the upgrades in batches. For a 50-device upgrade group with a Parallel upgrades value of 5, this means 10 batches of five upgrades each. In this case, Fleet Upgrade performs all five of the upgrades in batch #1 at the same time, in parallel, and doesn't initiate any the upgrades in batch #2 until all of the upgrades in batch #1 are done.

How can you cancel a run that's failing too often? Fleet Upgrade will automatically cancel the remaining upgrades in a run depending on the number of Acceptable failures you set. The value you specify in this field acts as a failure "budget" that, when exceeded, triggers automatic cancellation of all of the remaining upgrades in the run. If you want to avoid automatic cancellation entirely, specify an Acceptable failures value equal to the total number of devices to be upgraded (up to the maximum of 50). Set it to the default value of 1 if you want the system to cancel remaining upgrades after the very first failure.

Bear in mind that, when executing parallel upgrades in batch mode, Fleet Upgrade will continue to execute each new batch until the failure budget set by Acceptable failures is actually exceeded. This can mean that the total number of failures will sometimes exceed the budget you set. It can also mean that it sometimes takes longer for cancellation to kick in than you might expect.

For example: Let's assume that our set of devices to upgrade is 50, our Parallel upgrades setting is 5 and our Acceptable failures setting is 5. That means we have 10 batches of 5 devices for Fleet Upgrade to perform. Let's further suppose that, during execution of batch #1, we encounter 4 failures. The 5-failures budget is not yet exceeded, so Fleet Upgrade will begin to execute all the upgrades in batch #2 in parallel. We then encounter 4 more failures in batch #2. The 5-failure budget is now exceeded, so Fleet Upgrade will automatically cancel execution of batch #3 and the remaining 7 other batches. However, we've actually encountered 8 failures, not 5. Similarly, we might encounter only 1 failure each in batches #1, #2, #3, and #4, then encounter 5 failures in batch #5, triggering cancellation of the run. In this case, we've actually encountered 9 failures, almost twice the number we specified. Also, cancellation wasn't triggered until batch #6 and device #30, 60 percent of the way through the entire run.

Download software upgrades using a proxy server

You can download Cisco SMUs to your local image repository using a proxy server, but you must configure the proxy first, using the following steps.

Procedure


Step 1

From the main menu, choose Device Management > Fleet Upgrade > Image Repository.

Step 2

With the contents of the Image Repository displayed, click the Cisco.com tab to display the catalog of all SMUs available from the Cisco.com software image download site, as shown below.

Step 3

Click Configure proxy server to display the popup window shown below.

Step 4

Complete the proxy server fields as required. The Server URL or IP address and port are always required. A valid Username and Password are also required if you specify a server using a secure protocol, such as HTTPS. When you are finished, click Save.


Troubleshoot Fleet Upgrade Failures

The following table summarizes common Fleet Upgrade issues and remedies.

Table 1. Common Fleet Upgrade Issues and Remedies

Issue

Description

Remedy

Upgrade fails; unable to transition state

Upgrade fails due to inability to transition state during image distribution

Increase timeouts to at least 3600 seconds for Action, Event, State, and Workflow. System Activity timeout should be at least 10 seconds.

FPD causes activation failure

The failure message will include a recommendation that you "Please reconfigure you setting and create a new job." Upgrade fails during "Activate" stage, with the following message: "Preforming FPDs upgrade on the device. FPDs upgrade success except for the following FPDs: PO-PrimMCU, PO-PrimMCU". This occurs on VXR devices.

Add the following code to the configuration input for the Activate Action:

"fpd": 
"false","isISSUUpgrade": "false" }

Fleet Upgrade GUI drop downs are not working

The Vendor, Product series and other GUI drop down selection lists don't appear when clicked on.

The issue occurs when NSO packages are not updated. Please install the correct packages when installing Fleet Upgrade and CWM Solutions.

Software Image Policy has no Vendor or Device drop down

The Vendor and Product series drop down selection lists don't appear when clicked on.

Check that at least one software image has been downloaded to the local image repository. Fleet Upgrade is designed to display Vendor and Product series selections only when software images are available in the local repository.

Upgrade fails due to failure to find FTP services

Upgrade fails with a message indicating that a Distribute action failed due to the FTP server not being found.

From the main menu, select Administration > File servers and ensure that Enable FTP and Enable SFTP server upload are both checked. Also ensure that your organization's FTP and SFTP servers are active on TCP Ports 30621 and 30622, respectively. This support is required, as explained in Meeting installation prerequisites.

Fleet Upgrade job fails to start

Inventory reports "access denied" issues.

Be sure that you have set NACM permissions on NSO correctly, as explained in NSO package pre-installation tasks.

Use the default MOPs

Fleet Upgrade comes installed with three default (or "pre-built") MOPs. These MOPs are intended for use in performing upgrades for specific supported vendors and device families:

  • Default Juniper Upgrade: For upgrading Juniper MX960 series devices running JunOS

  • Default XE Upgrade: For upgrading Cisco Systems ASR 900 series devices running Cisco IOS-XE

  • Default XR Upgrade: For upgrading Cisco Systems 8000 series devices running Cisco IOS-XR

In most cases, you will want to select one of these default MOPs when a conformance report indicates that it is time to upgrade a suppported device in the series. They are usually your safest choice. Each of the default MOPs is customized for the supported vendor and product series, and the customizations are extensive. Each MOP varies significantly in the number of workflow Action types it makes available, the selected Actions it performs during an upgrade, and the Stages it goes through as it performs these actions. For example:

  • Default Juniper Upgrade: Offers 17 Action Types. The majority of these Actions are Juniper-specific. It performs 19 Actions during an upgrade: nine during the Pre check stage, three during Distribute stage, one during the Activate stage, and six during the Post stage.

  • Default XE Upgrade: Offers 15 Action Types. It performs 17 Actions during an upgrade: seven during the Pre check stage, two during Distribute stage, one during the Activate stage, and seven during the Post stage.

  • Default XR Upgrade: Offers 22 Action Types. It performs 25 Actions during an upgrade: 11 during the Pre check stage, three during Distribute stage, three during the Activate stage, one during its unique Commit stage, and seven during the Post stage.

However, running a default MOP may not always be your best choice for an efficient, targeted upgrade. You may find it useful to view each default MOP as a comprehensive library of all the Fleet Upgrade Actions relevant to a particular vendor and product series, arranged as a useful workflow. You don't have to stick with the default. You can copy a default MOP, and then manipulate and customize the copy to suit your needs. For example, you may decide that a MOP step that checks existing disk space is not necessary when you are upgrading software on a set of devices that are all factory-fresh. Similarly, you may find running sanity checks against devices you know are up and running are a waste of time and don't belong in the MOP you want to run.

Before trying to create a custom MOP, you will want to become familiar with the default MOP and its Actions. To see what any default MOP is doing during each step of its execution, select from the main menu Device Management > Fleet Upgrade > MOPs to display the list of MOPs. If necessary, use the Search field to find the MOP, then click the MOP name to see a list of the Available actions for that MOP.

The Selected actions list at the right shows you which stages of the MOP use each of these action types. In the Available actions list, next to each listed Action type is an "i" icon. Hover your mouse cursor over the "i" icon to see a detailed text description of the action, as shown in the figure for the "command-capture" sanity-check Action being executed during the Pre stage.

Once you know more about the default MOP, you can decide on the kinds of changes you want to make. The easiest way to get started creating a custom MOP from the default MOP is to then follow the steps in Clone and Edit MOPs, removing and adding Actions as you require.


Note


When working with default MOPs and custom, user-defined MOPs, please bear in mind:

  • You cannot edit or delete the default MOPs.

  • You can edit or delete clones of the default MOPs, or custom MOPs you defined from scratch.

  • You will find the Edit and Delete options for your user-defined MOPs at the far right, in the same row as the custom MOP you want to work on, under the More icon ().


Import Devices in Bulk

Complete the steps below to create a CSV file that specifies multiple devices and then import it. This allows you to add multiple devices at once instead of adding each device individually.

For a list of the fields that you can use when preparing the CSV file, see

Procedure


Step 1

From the main menu, choose Device Management > Network Devices.

Step 2

Click the Import icon icon to display the Import CSV File dialog box.

Step 3

If you have not already created a device CSV file to import:

  1. Click the Download sample 'Device Management template (*.csv)' file link and save the CSV file template to a local storage resource.

  2. Open the template using your preferred tool. Begin adding rows to the file, one row for each device.

    For a list of the fields that you can use when preparing the CSV file, see the table in Onboard Devices.

    Use a semicolon to separate multiple entries in the same field. Use two semicolons with no space between them to indicate that you are leaving the field blank. When you separate multiple entries with semicolons, remember that the order in which you enter values in each field is important.

  3. Delete the sample data rows before saving the file, or they will be imported along with your data. The column header row can stay, as it is ignored during import.

  4. Save the new CSV file.

Step 4

Click Browse to navigate to the CSV file you created in the previous steps and then click Open to select it.

Step 5

With the CSV file selected, click Import and wait for the import to complete.


Export Devices

Exporting the device list is a handy way to keep a record of all devices in the system at any one time. You can also edit the CSV file as needed, and re-import it to overwrite existing device data.

Procedure


Step 1

From the main menu, choose Device Management > Network Devices.

Step 2

(Optional) Filter the device list as needed.

Step 3

Check the check boxes for the devices you want to export.

Step 4

Click the icon. Your browser will prompt you to select a path and the file name to use when saving the CSV file, or to open it immediately.


Clone and Edit MOPs

The easiest and quickest way to create a MOP is to clone one of the default (or "pre-built") MOPs supplied with CWM Solutions and then modify it as needed. You'll be required to give the cloned MOP a new unique name, but you can then add and remove MOP Actions as you wish.

You can create Fleet Upgrade MOPs using other means, such as:

It's also useful to remember when you're creating a new MOP that:

  • You can always use the "i" icon shown next to each Action type in the CWM Solutions Fleet Upgrade Available actions list to research what each Action type does. Hover your mouse cursor over the "i" icon to see a detailed text description of that Action type and what it does.

  • You can use the JMESPath query language to filter, sort, or transform the JSON data for any of Action type that permits configuration in the new MOP.

  • Whenever new Action types have been added to CWM that do not appear in the Available actions list, click the Sync actions button.

Procedure


Step 1

From the main menu, choose CWM Solutions > Fleet Upgrade > MOPs. CWM Solutions displays the MOPs window, listing all existing Fleet Upgrade MOPs.

Step 2

At the far right, in the same row as the MOP you want to clone, click the More icon (). CWM displays a drop down menu like the one at right in the following figure.

Step 3

From the drop-down menu, choose Clone. CWM displays an edit window for the MOP you are cloning, like the one shown in the following illustration.

As you can see in the figure, the MOP name field at the top of the window is empty, but all the other fields are filled with the information contained in the original MOP that you are cloning. The Available actions list on the left shows all the Action Types that are appropriate for the selected Vendor and Product series. The Selected actions list on the right shows each Action Type selected for use in this MOP, along with the custom Name that each Action Type was assigned when it was selected for use in the MOP, as well as the execution Stage the action will be performed in. For instance, in this example, the Action Type cisco-disk-space-cwm-sol was seleccted for use in the MOP. It was assigned the custom Name check disk space, and will be performed during the distribute Stage.

Step 4

Edit the five identification fields at the top of the window as shown in the table below:

In this field...

Enter or select...

MOP name

A new name for the cloned MOP. You must enter a unique name.

Vendor

Choose the name of the supported vendor.

Product series

An appropriate product series for the Vendor you selected.

Application group

Fleet Upgrade is automatically selected.

Tags

A comma-separated list of tags to be applied to the job whenever a user runs this MOP.

Step 5

To add one of the Available actions on the left to the Selected actions on the right:

  1. In the Action type list on the left, click the + icon next to the Action type you want to add. An Add Action window for the new Action Type appears, like the one shown below.

  2. Enter a Custom name for the newly selected Action, and select the Stage at which you want it to be performed.

  3. Click Add. The newly added Action appears in the Selected actions list, at the end of the Actions in the Stage you selected for it.

Step 6

To remove one of the Selected actions on the right:

  1. Click the icon shown to the right of the Action you want to delete.

  2. You will be prompted to verify that you want to delete the Action. Click Delete to remove it.

Step 7

To edit the Custom name, Stage, Position, or Configuration details for one of the Selected actions on the right:

  1. Click the icon shown to the right of the Action you want to edit. A Configuration window for the Action appears, with Details and Configuration tabs, like the one shown below.

  2. To change the selected Action's Custom name, Stage, or Position, edit the respective fields on the Details tab.

  3. To change the Action's Configuration (for Actions that permit configuration), click the Configuration tab to display the Input Schema tools as shown below. Then use the Text/View toggle to shift between JSON and Javascript views, and format the text in the JSON. You can also use the + icons to format JMEScript queries and sorts.

Step 8

When you are finished, click Save changes.


Create Custom MOPs

You can use the CWM Solutions Fleet Upgrade GUI enables to create MOPs from scratch in free-form fashion, using Action types pre-selected for the Vendor and Product series that you choose. The steps below show how to do this.

You can also create Fleet Upgrade MOPs using other means, such as:

It's also useful to remember when you're creating a new MOP that:

  • You can always use the "i" icon shown next to each Action type in the CWM Solutions Fleet Upgrade Available actions list to research what each Action type does. Hover your mouse cursor over the "i" icon to see a detailed text description of that Action type and what it does.

  • You can use the JMESPath query language to filter, sort, or transform the JSON data for any Action type that permits configuration in the new MOP.

  • Whenever new Action types have been added to CWM that do not appear in the Available actions list, click the Sync actions button.

Procedure


Step 1

From the main menu, choose CWM Solutions > Fleet Upgrade > MOPs. CWM Solutions displays the MOPs window, listing all existing Fleet Upgrade MOPs.

Step 2

Click + Create MOP. CWM displays the Create new MOP window, with the first five required identification fields blank .

Step 3

Edit the first three identification fields at the top of the window as shown in the table below:

In this field...

Enter or select...

MOP name

A new name for the cloned MOP. You must enter a unique name.

Vendor

Choose the name of the supported vendor.

Product series

An appropriate product series for the Vendor you selected.

Step 4

As soon as you select the Product series, the Create new MOP window populates the Available actions list with Action Types appropriate for the Vendor and Product series you selected, as shown in the following figure.

Step 5

You can now edit the remaining two identification fields:

In this field...

Enter or select...

Application group

Fleet Upgrade is automatically selected.

Tags

A comma-separated list of tags to be applied to the job whenever a user runs this MOP.

Step 6

To add one of the Available actions on the left to the Selected actions list on the right:

  1. In the Action type list on the left, click the + icon next to the Action type you want to add. An Add Action window for the new Action Type appears, like the one shown below.

  2. Enter a Custom name for the newly selected Action, and select the Stage at which you want it to be performed.

  3. Click Add. The newly added Action appears in the Selected actions list, at the end of the Actions in the Stage you selected for it.

Step 7

To remove one of the Selected actions on the right:

  1. Click the icon shown to the right of the Action you want to delete.

  2. You will be prompted to verify that you want to delete the Action. Click Delete to remove it.

Step 8

To edit the Custom name, Stage, Position, or Configuration details for one of the Selected actions on the right:

  1. Click the icon shown to the right of the Action you want to edit. A Configuration window for the Action appears, with Details and Configuration tabs, like the one shown below.

  2. To change the selected Action's Custom name, Stage, or Position, edit the respective fields on the Details tab.

  3. To change the Action's Configuration (for Actions that permit configuration), click the Configuration tab to display the Input Schema tools as shown below.

    Use the Text/View icons to toggle between JSON and Javascript views, and format JSON text. You can also use the + icons to format JMEScript queries and sorts.

Step 9

When you are finished, click Save changes.


Export, Edit and Import MOPs

Fleet Upgrade codes MOPs using JavaScript Object Notation. It can export and import MOPs written in JSON. If you're new to JSON, you may want to start here. But if you are comfortable working with it, you may find it convenient to create MOPs by exporting a copy of a default MOP, modifying it as needed, then importing it. Exporting MOPs as JSON files is also a handy way to keep backups of MOPs you create, and importing these files allows you to share custom MOPs with air-gapped locations in your organization.

In the following steps, we will create a very simplified custom MOP that does just two things:

  1. Check that the devices we run it against are accessible, using the check-nodes-cwm-sol Action Type in the Pre stage. If any of the devices are inaccessible, the MOP will return a failure for that device, so we'll know that's why the second Action also failed.

  2. Collect a snapshot of the configuration of each reachable, using the command-capture-cwm-sol Action Type.

We'll create it by:

  1. Exporting to a JSON file the pre-built Default XR Upgrade MOP supplied with every copy of Fleet Upgrade. This MOP file contain both of the Action Types we want to use.

  2. Modify the JSON file to rename the MOP, delete all the Actions we don't want to use, reorder the remaining selected actions, and otherwise make it fit to re-import.

  3. Re-import the modified custom MOP.

You can create Fleet Upgrade MOPs using other means, such as:

It's also useful to remember when you're creating a new MOP that:

  • You can always use the "i" icon shown next to each Action type in the CWM Solutions Fleet Upgrade Available actions list to research what each Action type does. Hover your mouse cursor over the "i" icon to see a detailed text description of that Action type and what it does.

  • You can use the JMESPath query language to filter, sort, or transform the JSON data for any of Action type that permits configuration in the new MOP.

  • Whenever new Action types have been added to CWM that do not appear in the Available actions list, click the Sync actions button.

Procedure


Step 1

From the main menu, choose CWM Solutions > Fleet Upgrade > MOPs. CWM Solutions displays the MOPs window, listing all existing Fleet Upgrade MOPs.

Step 2

In the search field under the MOPs title, enter XR. The Default XR Upgrade MOP name moves to the top of the list.

Step 3

At the far right, in the same row as the MOP you want to clone, click the More icon (). CWM displays a drop down menu like the one at right in the following figure.

Step 4

From the drop-down menu, choose Export (JSON) . CWM displays a message indicating that the MOP has been exported to a file. Check your local default download folder for a file with a filename the same as the title of the exported MOP and a JSON filename extension.

Step 5

Change the name of the JSON file to MyMOP.json.

Step 6

Edit MyMOP.json file to:

  1. Since we are not going to import any action types as part of this MOP: Delete all of the actions: arrays with blank values, and other actions: arrats except the first "name:" "node health check" at "position": 1 and the first "name": "device snapshot", at "position": 9.

  2. Change the "name": "device snapshot" "position": value to "position": 2.

  3. Change the value of the "name": "Default XR Upgrade" object to read "name": "MyMOP"

  4. Change the value of the "tags":[...] array to read "name": ["custom"]

  5. As we are not going to import any workflows as part of this MOP: Delete all of the "Workflows" arrays with blank values.

When you're done, the MyMOP.json file should look like this:

{
       ],
    "actions": [
        {
            "name": "node health check",
            "stage": "pre",
            "wfName": "check-nodes-cwm-sol",
            "wfVersion": "1.0.1",
            "position": 1,
            "ignoreFailure": false
        },
        {
            "data": {
                "commandCapture": [
                    "show version",
                    "show ip interface brief"
                ]
            },
            "name": "device snapshot",
            "stage": "pre",
            "wfName": "command-capture-cwm-sol",
            "wfVersion": "1.0.1",
            "position": 2,
            "ignoreFailure": false
        }
    ],
    "name": "MyMOP",
    "vendor": "Cisco Systems",
    "series": "Cisco 8000 Series Routers",
    "app": "Fleet Upgrade",
    "tags": [
        "custom"
    ]
}

Step 7

Validate the modified MyMOP.json file, then save it.

Step 8

From the main menu, choose CWM Solutions > Fleet Upgrade > MOPs.

Step 9

Click Import MOP (JSON). Browse to and select your modified MyMOP.json file, then click Import MOP. When the import is finished, the file appears in the MOPs window list.