Release Notes for Cisco IOS XE SD-WAN Release 16.12.x

These release notes accompany the Cisco IOS XE SD-WAN Software Release 16.12, which provides Cisco SD-WAN capabilities for Cisco IOS XE SD-WAN routers.

Supported Devices

The Cisco IOS XE SD-WAN software runs on the following devices.

Table 1. Supported Devices and Versions

Device Family

Device Name

Cisco ASR 1000 Series Aggregation Services Routers

  • ASR 1001-HX and ASR 1001-X

  • ASR 1002-HX and ASR 1002-X

Cisco ISR 1000 Series Integrated Services Routers

  • C1101-4PLTEP

  • C1109-4PLTE2P

  • C1111-8P, C1111-8P LTE EA, and C1111-8P LTE LA

  • C1117-4P LTE EA, C1117-4P LTE LA

  • C1111-4P LTE EA, C1111-4P LTE LA, C1116-4P LTE EA, C1117-4P MLTE EA

  • C1111-4P, C1116-4P, C1117-4P, C1117-4PM, C1101-4P, C1111X-8P (8GB RAM)

  • C1111-8PLTEEAWA

  • C1113-8PMLTEEA

  • C1121X-8P, C1121X-8PLTEP

  • C1121-8PLTEPWE

  • C1121-8PLTEPWB

  • C1121-8PLTEPWZ

  • C1121-8PLTEPWQ

  • C1126X-8PLTEP

  • C1127X-8PLTEP, C1127X-8PMLTEP

  • C1161X-8P

  • C1161X-8PLTEP

  • P-LTEAP18-GL

  • Pluggable modules: CAT6 LTE Advanced P-LTEA-EA(=), CAT6 LTE Advanced P-LTEA-LA(=)

Cisco ISR 1000 Series Integrated Services Routers with wireless services (WLanGigabitEthernet configuration required from vManage)

  • C1111-8PWY (WiFi domain WY; Y = A, B, E, F, H, N, Q, R, Z)

  • C1111-8PLTEEAWX^*^ (WiFi domain WX; X = A, B, E, R)

  • C1111-8PLTELAWY* (WiFi domain WY; Y = D, F, H, N, Q, Z S, E and A)

  • C1101-4PLTEPWX* (WiFi domain WX; X = A, B, D, E, Z)

  • C1109-4PLTE2PWZ* (WiFi domain WZ; Z = A, B, D, E, Q, R, Z)

  • C1121-8PLTEPWX* (WiFi domain WX; X = B, E, Z, Q)

Cisco ISR 4000 Series Integrated Services Routers

ISR 4221, ISR 4221-X, ISR 4321, ISR 4331, ISR 4351, ISR 4431, ISR 4451, ISR 4461

Cisco CSR 1000v Series Cloud Services Routers

CSR 1000v

Cisco 5000 Series Enterprise Network Compute System

  • ENCS 5104, ENCS 5406, ENCS 5408

  • ENCS 5412 with T1/E1 and 4G NIM modules

Cisco UCS E Series M2 servers

  • UCS-EN120S-M2/K9

  • UCS-EN140N-M2/K9

  • UCSE-140S-M2/K9

  • UCSE-160D-M2/K9

  • UCSE-180D-M2/K9

Cisco UCS E Series M3 servers

  • UCSE-160S-M3/K9

  • UCSE-180D-M3/K9

  • UCSE-1120D-M3/K9

Cisco 1101 Series Integrated Services Routers

Cisco SD-WAN capability can now be enabled on Cisco 1101 Series Integrated Services Routers.

Table 2. Supported Modules on Cisco 4000 Series Integrated Services Routers

Interfaces

Type

L3–Routed Ports

NIM-1GE-CU-SFP

NIM-2GE-CU-SFP

SM-X-6X1G

SM-X-4X1G-1X10

VDSL/ADS

NIM-VAB-A

NIM-VAB-M

3G/4G Modules

NIM-LTEA-EA

NIM-LTEA-LA

LAN–NIM & SM-X Modules

NIM-ES2-4

NIM-ES2-8

NIM-ES2-8-P

T1, E1, and G.703 Multiflex Trunk Voice and WAN Interface Cards

NIM-1MFT-T1/E1 (Data)

NIM-8MFT-T1/E1 (Data)

NIM-4MFT-T1/E1 (Data)

NIM-2MFT-T1/E1 (Data)

Cisco 1-Port Serial WAN Network Interface Card (NIM-1T)

What's New for Cisco IOS XE SD-WAN Release 16.12.1b, 16.12.1d, and 16.12.2r

This section applies to Cisco XE SD-WAN devices.

Cisco is constantly enhancing the SD-WAN solution with every release and we try and keep the content in line with the latest enhancements. The following table lists new and modified features we documented in the Configuration, Command Reference, and Hardware Installation guides. For information on additional features and fixes that were committed to the SD-WAN solution, see the Resolved and Open Bugs section in the Release Notes.

Table 3. What's New for Cisco XE SD-WAN Device

Feature

Description

Getting Started

Multitenancy support in Cisco XE SD-WAN Devices

Starting release Cisco IOS XE SD-WAN 16.12.2r, multitenancy is supported on the following platforms:

  • Cisco ASR 1000 Series Aggregation Services Routers, Cisco ASR 1001X

  • Cisco ISR 4000 Series Integrated Services Routers, Cisco ISR 4321, Cisco ISR 4461

  • Cisco ISR 1000 Series Integrated Services Routers. Cisco ISR 1111-4P

  • Cisco CSR 1000 Series Cloud Services Routers, Cisco 1000v

Multitenancy allows service providers to manage multiple customers or tenants.

Tenant data backup solution in multitenant mode

Starting from Cisco IOS XE SD-WAN 16.12.2r, when databases are shared by multiple tenants in2 a multitenant mode, you can back up data for a specific tenant and restore it.

Systems and Interfaces

IPv6 Support for NAT64 Devices

This release supports NAT64 to facilitate communication between IPv4 and IPv6 on Cisco IOS XE SD-WAN routers. For related information, see Configure NAT64 CLI Equivalent on Cisco XE SD-WAN Routers.

Secure Shell Authentication Using RSA Keys

You can now configure RSA keys to secure communication between a client and a Cisco SD-WAN server. For related information, see SSH Authentication using vManage on Cisco XE SD-WAN Devices.

DHCP option support

You can now use DHCP server options 43 and 191 to configure vendor-specific information in client-server exchanges. For related information, see Configure DHCP.

Communication with an UCS-E Server

This feature provides an interface in the interface feature template list to configure an UCS-E interface to connect to an UCS-E server. For related information, see Create a UCS-E Template.

Bridging, Routing, Segmentation, and QoS

Subinterface QoS

A physical interface may be treated as multiple interfaces by configuring one or more logical interfaces called subinterfaces. This feature enables Quality of Service (QoS) policies to be applied to individual subinterfaces. For related information, see QoS on Subinterface.

Policies

Packet Duplication for Noisy Channels

This feature helps mitigate packet loss over noisy channels, thereby maintaining high application QoE for voice and video. This feature is supported on Cisco XE SD-WAN devices as well as on Cisco vEdge devices. For related information, see Configure and Monitor Packet Duplication.

Integration with Cisco ACI

The SD-WAN and Cisco ACI integration functionality now supports predefined SLA cloud beds. It also supports dynamically generated mappings from a data prefix list and includes a VPN list to an SLA class that is provided by Cisco ACI. For related information, see Integration with Cisco ACI.

Encryption of Lawful Intercept Messages

Lawful intercept messages between a Cisco XE SD-WAN router and a Media Device can now be encrypted using static tunnel information. For related information, see Lawful Intercept.

Security

High-Speed Logging for Zone-Based Firewalls

High-Speed Logging (HSL) allows a firewall to log records with minimum impact to packet processing. For related information, see Firewall High-Speed Logging.

Self zone policy for Zone-Based Firewalls

Self-zone is an default zone in the firewall that is associated with the VPN for punt and inject interface. You can define policies to impose rules on the incoming and outgoing traffic. For related information, see Configure Firewall Policies Using vManage.

Secure Communication Using Pairwise IPsec Keys

This feature enables support to create and install private pairwise IPSec session keys to secure communication between IPSec devices and its peers. For related information, see IPSec Pairwise Keys Overview.

Network Optimization and High Availability

TCP Optimization

TCP optimization fine tunes the processing of TCP data traffic to decrease round-trip latency and improve throughput. For related information, see TCP Optimization: Cisco XE SD-WAN Routers. This feature support was added in Cisco IOS XE SD-WAN Release 16.12.1d

Commands

Loopback interface support for WAN (IPsec)

You can now configure a loopback transport interface on a Cisco IOS XE SD-WAN router to help in troubleshooting and diagnostics. For related information, see the bind command.

New and Enhanced Hardware Features

New Features

  • Support for UCS-E module—This feature adds a UCS-E template in Cisco vManage for configuring Cisco Unified Computing System (UCS) E-Series servers. For related information, see Getting Started Guide for Cisco UCS E-Series Servers and the Cisco UCS E-Series Network Compute Engine and Configuring Devices using vManage.

  • Support for Cisco IR1101 Integrated Services Router Rugged—Cisco SD-WAN capability can now be enabled onCisco IR1101 Integrated Services Router Rugged. The following notes apply to this support:

    • Controller devices (Cisco vBond orchestrators, Cisco vManage NMSs, and Cisco vSmart controllers) must run Cisco SD-WAN Release 19.2 or later.

    • The default topology is full mesh, but the hub and spoke topology is often used for IoT applications.

    • Cisco SD-WAN support on the Cisco IR1101 Integrated Services Router Rugged requires Cisco IOS-XE Release 16.12.

    • The Cisco IR1101 Integrated Services Router Rugged has four fixed switch-ports. Make sure to select the correct template.

    • The CLI template is not currently supported.

    • Starting from Cisco IOS-XE Release 16.12.1, Cisco IR1101 Integrated Services Router Rugged has dual LTE support with LTE extension module.

    • We recommend using up to 50 BFD sessions for scaling.

Important Notes, Known Behavior, and Workaround

  • Cisco IOS XE SD-WAN devices with the SFP-10G-SR module do not support online insertion and removal (OIR) of this module.

  • When you complete a Cisco SD-WAN software downgrade procedure on a device, the device goes into the configuration mode that it was in when you last upgraded the Cisco SD-WAN software on the device. If the device is in a different configuration mode when you start the downgrade than it was when you last upgraded, the device and Cisco vManage show different configuration modes after the downgrade completes. To put the configuration modes back in sync, reattach the device to a device template. After you reattach the device, both the device and Cisco vManage show that the device is in the vManage configuration mode.

Resolved and Open Bugs

About the Cisco Bug Search Tool

Use the Cisco Bug Search Tool to access open and resolved bugs for a release.

The tool allows you to search for a specific bug ID, or for all bugs specific to a product and a release.

You can filter the search results by last modified date, bug status (open, resolved), severity, rating, and support cases.

Resolved and Open Bugs

This section details all fixed and open bugs for this release. These bugs are available in the Cisco Bug Search Tool

Resolved Bugs for Cisco SD-WAN Release 16.12.2r

Table 4. Resolved Bugs for Cisco SD-WAN Release 16.12.2r

Bug ID

Description

CSCvp38857

unable to modify interface speed for CSRv XE SDWAN

CSCvr45260

The config on VBond rolls back when the configs are pushed through VManage CLI template

CSCvr51104

vManage cluster GUI SSO fails during the 2nd login attempt using old cookies

CSCvk32783

Standard IPSec support in IOS-XE SDWAN software

CSCvp11416

XE SD-WAN device- Template attach fails for a cedge device if theres a central policy with cflowd activated

CSCvp36883

SD-WAN QoS not work as expected after no class under policy-map

CSCvp37056

flow-visibility get broken and doesn't working properly on ASR1001HX platform with IPSec encap

CSCvp73389

OSPF is not setting the downbit for the default route.

CSCvp96887

Failed to attach template to Cisco XE SDWAN Rtr if qos-map name changed after policy-map is attached

CSCvq01813

Pending object for "SDWAN Overlay Cfg" and sessions are not downloaded with scale of sdwan session

CSCvq27599

Delete bandwidth queue with random-detect from template getting rejected on device side

CSCvq31153

SDWAN BFD session stuck and packet drops due to IN_CD_SW_IPSEC_ANTI_REPLAY_FAIL drops

CSCvq47444

CLI "config-exchange request" for any ikev2 profile has inconsistent behavior between IOS and confd

CSCvq49150

LAN ACL dropping packets with default-action accept

CSCvq64513

Differentiate sdwan control packets priority based on device_type for Inject path

CSCvq65906

admin/admin credentials are lost after reload

CSCvq66518

Data traffic classified into qos-group 0 improperly without qos policy enabled

CSCvq68449

QFP ucode reloads unexpectedly while processing large packet with NBAR enabled

CSCvq75871

IPSec SA receives anti-replay error for all packets for NAT session flap sometimes

CSCvq76075

HMAC failure due to incorrect stale nat fixup entry for the ipsec session after symnat session flap

CSCvq97694

Local internet breakout (DIA) doesn't work on subinterfaces in IOS-XE SD-WAN 16.11.1a, 16.12.1b

CSCvr12264

fman-fp crashed with "set vpn + tloc" in data-policy with tloc pointed to local

CSCvr23424

XE SD-WAN device rebooting continuously when upgraded to 16.12b

CSCvr23454

NBAR not turned off on datapath when unconfigure policy with app-visibility

CSCvr28506

ftpmd process core when two app-ids with invalid name used in centralized app-route-policy

CSCvr46085

QoS dscp rewrite doesn't work properly with one single rewrite-rule entry update

CSCvr47688

local data policy classification issue with prefix less specific than /24 on ISR1100 platform

CSCvr52767

loops because of redistribution OMP<>OSPF external with DN-bit are happening on IOS-XE SD-WAN

CSCvr55738

spanning-tree mode rapid-pvst is not part of the default config on 16.12.1 cEdge software anymore

CSCvs34879

Tracebacks seen when pushing ACL policy on C1111-8P

CSCvs46366

DNS configurations are not pushed to the XE-SDWAN device properly

CSCvp86463

key field of yang-model "snmp-server/host" incorrect

CSCvq45411

IOSd is crashing after configuration from vmanage is pushed

CSCvq69544

Improve datapath drop cause with proper code for OCT_UNSUPPORTED_CIPHER from Octeon based platform

CSCvr18395

policy seq with app-family network-service is not downloaded to datapath

CSCvr27773

Multiple times add and delete sym nat with cEdge cause BFD down with vEdge devices

CSCvr27819

Add/remove of symmetric nat on WAN link multiple times makes the link BFDs down forever

CSCvr48167

SD-WAN BFD session failure due to IPSec SA is down and stuck with non IPSec SA

Open Bugs for Cisco SD-WAN Release 16.12.2r

Table 5. Open Bugs for Cisco SD-WAN Release 16.12.2r

Bug ID

Description

CSCvm86435

confd_cli process is not terminated and hogging CPU

CSCvq28313

MTCVM:templates are changing to out of sync for CSR1000v devices after sometime

CSCvr22877

BFD staying down between a XE SD-WAN device and a Cisco vEdge device after a failure condition is triggered on the ISP side.

CSCvs27051

idle-timeout is improperly mapped on XE SD-WAN devive

CSCvs39216

IOS-XE SD-WAN CSR in Azure does not remove start up config.

CSCvs54333

c1100-4P/6P-LTE : Low Bandwidth over cellular is not working

CSCvs56121

sysmgrd core seen on CSR on reboot cases

CSCvp86463

key field of yang-model "snmp-server/host" incorrect

Resolved Bugs for Cisco SD-WAN Release 16.12.1e

Table 6. Resolved Bugs for Cisco SD-WAN Release 16.12.1e

Caveat ID Number

Description

CSCvp96887

Failed to attach template to Cisco XE SDWAN Rtr if qos-map name changed after policy-map is attached

CSCvq10160

Cellular IP is getting reset when primary transport interface Gi0/0/0 is shutdown.

CSCvq11615

Route is not getting removed from the routing table even if the BFD is down.

CSCvq61835

interface cant be moved from vrf 0 to service vrf when it has ip address

CSCvq61992

XE SDWAN router stuck in boot loop after power-cycle due to replaystore file corruption

CSCvq70071

flow data is not populated into /tmp/xml/fnf

CSCvq97954

Cellular interface doesn't get an IP address when brought up through the pnp workflow

CSCvr13244

19.2.0 regression: Can not configure NTP on SD-WAN and specify source interface in VPN

CSCvr15012

fman-fp keeps on crashing after attach app-route policy with app-family

CSCvr18082

xe-sdwan omp aggregate-only does not suppress component routes sometimes

CSCvr35568

CPP crash with Packet Duplication enabled on path failover with XE SDWAN router

CSCvr52767

microloops because of redistribution OMP<>OSPF external with DN-bit are happening on IOS-XE SD-WAN

CSCvq11615

Route is not getting removed from the routing table even if the BFD is down.

CSCvq61992

XE SDWAN router stuck in boot loop after power-cycle due to replaystore file corruption

CSCvq97694

Local internet breakout (DIA) doesn't work on subinterfaces in IOS-XE SD-WAN 16.11.1a, 16.12.1b

CSCvr55738

spanning-tree mode rapid-pvst is not part of the default config on 16.12.1 XE SDWAN software anymore

CSCvr71786

Pairwise-keying configuration not enabled when configured through a vManage template

Resolved Bugs for Cisco SD-WAN Release 16.12.1d

Table 7. Resolved Bugs for Cisco SD-WAN Release 16.12.1d

Caveat ID Number

Description

CSCvq67094

zbf drops hierarchical overlay traffic between spoke sites that go through hub ASR1001-X

CSCvq71921

ucode crash observed with ZBFW due to stuck thread processing data traffic

CSCvr27714

CSR+SDWAN on AWS will install default route in startup config which conflicts with some topologies

Resolved Bugs for Cisco SD-WAN Release 16.12.1b

Table 8. Resolved Bugs for Cisco SD-WAN Release 16.12.1b

Caveat ID Number

Description

CSCvj84204

XE SDWAN: Control connections fail if DNS server is not reachable thru one TLOC interface in ECMP

CSCvk48972

Admin-tech failure via vManage for multiple Cisco XE SD-WAN Router platforms

CSCvm47984

ISR4331: 16.9.1: snmpwalk error - OID not increasing

CSCvm55520

C9407R - C9400-PWR-3200AC Power Supply goes into faulty state randomly ( "n.a." )

CSCvn54741

Traffic not getting matched when using vsmart data policy

CSCvn55971

Cisco XE SD-WAN Router: Locally sourced packets using wrong interface with ECMP

CSCvn63395

ASR-1002-HX crash at headend running 16.9.3

CSCvn71472

'snmp-server user' config shown as part of sdwan running config

CSCvn95901

High memory utilization on ISR1K C1111-8P platform

CSCvo00790

Cisco XE SD-WAN Router cli_template: Unable to move interface from global vpn

CSCvo31413

fman_fp crash after upgrading to build 201

CSCvo60765

SD-WAN router experiences an IOSd crash when connected to a controller

CSCvo69625

Increase IPSec tunnel limit to 200 by default without HSECk9 on ISR1k

CSCvo83361

XE SDWAN: add the error code support on XE SDWAN asr1k

CSCvo90556

XE SDWAN: NTP should try all available interfaces with ECMP

CSCvp08310

Not enough disk space to carry on configuration DB error when trying to install third image on ISR

Open Bugs for Cisco SD-WAN Release 16.12.1b

Table 9. Open Bugs for Cisco SD-WAN Release 16.12.1b

Caveat ID Number

Description

CSCvj26197

Update statistics from Oecteon viptela code to platform

CSCvk72903

XE SDWAN-vDaemon: Sub-interface's control-local-properties shows state=UP even though it is admin-down

CSCvp15917

ciscosdwan.cfg located on the bootflash is ignored when bootstrapping a new router

CSCvp77035

vManage is pushing "negotiation auto" config to TenGigabitEthernet interface with optical SFPs

CSCvp79646

Unable to connect to vManage over the LTE interface when fail over executed from other transport.

CSCvq10160

Cellular IP is getting reset when primary transport interface Gi0/0/0 is shutdown.

CSCvq13727

CSR 1000v XE SDWAN instance keeps rebooting in AWS

CSCvq34185

Umbrella redirects not respecting local domain bypass list, it is not programmed to DP

CSCvq61835

interface cant be moved from vrf 0 to service vrf when it has ip address

CSCvq62993

Secondary Supervisor can't boot up after "redundancy force-switchover" command

CSCvq67094

zbf drops hierarchical overlay traffic between spoke sites that go through hub ASR1001-X

CSCvq70071

flow data is not populated into /tmp/xml/fnf

CSCvq79547

Bootflash space exhaustion causing watchdog to trigger on ISR4351

CSCvq83612

Polaris 16.9 QFP crash due to a stuck thread

CSCvr38887

NAT DIA with matching app-list is not supported officially

Compatibility Matrix

Table 10. Compatibility Matrix

Controllers

ENCS/ISR/ASR

ISRv

18.3.5

16.9.4

16.9.4 with NFVIS 3.9.1FC1 or NFVIS 3.9.2-FC4

18.4.0

16.10.1

16.10.1 with NFVIS 3.9.1FC1 or NFVIS 3.9.2-FC4

19.1.0

16.11.1a

16.11.1a with NFVIS 3.9.1FC1 or NFVIS 3.9.2-FC4

19.2.0

16.12.1b

16.11.1a with NFVIS 3.9.1FC1 or NFVIS 3.9.2-FC4

ROMmon Requirements Matrix

The following table lists the minimum ROMmon versions supported on the corresponding devices and releases:

Table 11. ROMmon Versions

Device

ROMmon Version for 16.10 Devices

ROMmon Version for 16.11 Devices

ROMmon Version for 16.12 Devices

ASR1000-X/HX

16.3(2r)

16.3(2r)

ASR1001-HX, ASR1002-HX, ASR1001-X: 16.9(4r

ASR1002-X: 16.7(1r)

ISR 4000

16.7(4r)

16.7(4r)

16.12(1r)

ISR 1000

16.9(1r)

16.9(1r)

16.12(1r)

Table 12. Recommended Rommon Release for SD-WAN for Cisco ISR 4000 series Integrated Services Routers (Cisco ISR 4000)

Cisco IOS XE Release

Cisco 4321 ISR

Cisco 4321 ISR

Cisco 4331 ISR

Cisco 4351 ISR

Cisco 4431 ISR

Cisco 4451 ISR

Cisco 4461 ISR

Cisco IOS XE 16.9.x

16.7(5r)

16.7(5r)

16.7(5r)

16.7(5r)

Cisco IOS XE 16.10.x

16.7(5r)

16.7(5r)

16.7(5r)

16.7(5r)

16.7(5r)

16.7(5r)

Cisco IOS XE 16.11.x

16.7(5r)

16.7(5r)

16.7(5r)

16.7(5r)

16.12(2r

16.12(2r)

Cisco IOS XE 16.12.x

16.12(2r)

16.12(2r)

16.12(2r)

16.12(2r)

16.12(2r)

16.12(2r)

16.12(2r)


Note

ROMmon auto-upgrade is supported on the ISR 4000 series routers, beginning with 16.9.1 and all subsequent releases/throttles.



Note

ROMmon auto-upgrade is supported on the ISR 1000 series routers, beginning with 16.10.3 and 16.12.1b.



Note

For the ISR 1000 series routers, ROMmon version 16.8(1r) is not compatible with 16.10 releases and ROMmon version 16.9(1r) is not compatible with 16.9 releases. If an ISR 1000 series router is upgraded to a 16.10 release without auto-upgrade support, it is required that ROMmon be upgraded to 16.9(1r) or later by the user.


ISRv routers must run the minimum required version of the CIMC and NFVIS software, as shown in the following table.

Table 13. Minimum CIMC and NFVIS Software Versions for ISRv Routers

Hardware Platform

CIMC

NFVIS

ISRv

3.2.6

3.9.2