Rakuten finds success through disruptive thinking and actions
The current, typical model for building a mobile network is outdated. Operators are constrained by legacy vendor architectures that have remained essentially unchanged for more than 25 years of mobile networks. Although these architectures were useful in prior generations, they aren’t suited for today’s more dynamic, application-driven environment. Operators urgently need a new model to ensure they remain competitive delivering new services faster, while decreasing both capital and operating expenses.
A software-defined architecture that includes cloud virtualization and automation will help operators meet these new application and operational demands. They will reap the benefits of true multivendor networks that are harmonized with a common feature set across all target markets. With the onset of a new software-defined architecture, the supply chain for mobile network infrastructure deployment changes at a fundamental level. It will support an unprecedented level of versatility, allowing operators to combine best-in-class functions from multiple vendors. Operators also can evolve services as needed to address the demands of a competitive environment.
Because the new architectures embrace a software-centric approach, they promote more automation and service versatility. A prime example would be 5G use cases that target enterprise and industry vertical markets. These services should be supported using APIs that are designed to manage resources in an end-to-end network and won’t function properly without network automation. Additionally, many of the software-defined functions will occur in virtualized environments at or near the edge of networks, which enables support for a newer breed of low-latency services defined in edge computing or multiaccess edge computing (MEC).1
This white paper addresses the limitations of current models for building 5G-ready mobile networks. It then highlights new strategies and their technical foundations. As a real-world illustration, we present the approach followed by Rakuten Mobile Network Inc. (RMN). RMN is creating a radically new business model which is enabled by applying cloud and automation techniques in innovative ways.
Any new end-to-end mobile network deployed today must adopt basic 5G system architecture principles even if it’s using LTE access. Operators should embrace the key principles that underpin the foundation of the 5G system architecture.
A monolithic functional implementation such as the base transceiver station (BTS) is an example of why the current mobile supply chain is outdated. With this implementation, operators must pick one vendor per market and harmonize the macro vendor markets to a “lowest common denominator” set of features. The result is a limited set of applications that operators can provide to their customers. Sometimes, the vendor dependencies and lock-in can propagate into other domains when proprietary features are implemented. For example, mobility management entities (MMEs) are often implemented with the RAN vendor, and small cells are often coupled to the macro RAN using features such as HetNet.
The new cloud RAN architecture addresses, among other things, the challenges of building multivendor networks and harmonizing to a common feature set. One fundamental characteristic is the decomposition of the radio signal processing stack using standardized splits (see Figure 1).2,3
The radio signal processing stack in both 5GNR and LTE is a “service chain” of functions which are processed sequentially. These functions are controlled using signaling derived from the packet core, specifically the MME in the evolved packet core (EPC), and the access and mobility management function (AMF) in 5GC. (If you’re not familiar with the functions and their sequence in this stack, please refer to “Radio signal stack decomposition” below.)
Figure 1. Radio signal processing stack.
The decomposition maps function in the stack as follows:
1. Service delivery adaptation protocol or SDAP (in NR only), packet data convergence protocol (PDCP), and radio resource control (the control functions) to a centralized unit (CU). These stack functions are packet-level manipulations (header compression, over-the-air ciphering) that aren’t timing sensitive and can be easily implemented in a virtualized environment. The midhaul8 links connects the CU to the DU defined below. The CU location is excellent for deploying user plane function (UPF) in the decomposed packet core architectures. The decomposed packet core is known as CUPS in 3GPP R14 and 5GC in 3GPP R15.10,11
2. The radio link control (RLC), medium access control (MAC); and physical (PHY) layers map to a distributed unit (DU). The DU performs significant preparation for the RF layer including rate adaptation, channel coding, modulation, and scheduling. For the MAC layer, the functions of the DU are time sensitive because a transport block of duration of the Transmit Time Interval (1 ms in LTE) is produced for consumption by the PHY layer. The DU link to the downstream entity is called front and transports digitized RF samples in either the time domain or frequency domain.
3. The baseband radio functions, up and down conversion, and amplification along with any analog beamforming map into a remote unit (RU) deployed at the cell-site also known as a remote radio head (RRH). In some cases, a LO-PHY can also be integrated into the RRH. The interface to the DU to the RU is typically implemented using well known interface standards such as CPRI or eCPRI.9
This decomposition and isolation of functions, along with the well-defined interfaces between them, allow operators to disaggregate software from hardware. Operators can procure software capabilities that will execute in hardware supplied by a different vendor. This disaggregation is natural and relatively simple for the centralized unit (CU) because those functions can be easily virtualized. Disaggregation of the Distributed Unit (DU), is more challenging because of the real-time processing requirements of the MAC and PHY layers. Many in the industry questioned whether it was even possible to achieve the benefits of virtualization for the DU. Cisco and Altiostar have accomplished a complete virtualization of the DU in situations where transport capacity is available. Where a virtualized solution isn’t possible, white box style DU designs provide the required functionality.
The decomposition and disaggregation of 5G enables a major architectural shift to an edge infrastructure that combines decomposed subscriber management with access functionality. This shift will also apply to wireline networks with increasing adoption of the 5GC wireline networks (see Figure 2).
Figure 2. Edge infrastructure.
As the RAN and core become decomposed, it’s logical to create an intermediate location. This location would consolidate the CU workloads and a user plane function (UPF) from the mobile core (see the Edge Cloud in Figure 2). These distributed UPFs facilitate offload, enabling local virtualized services or more efficient peering at a metro level. This unified platform approach supports infrastructure workloads and a variety of service-oriented workloads. These workloads can support business-to-business (B2B) services such as tenancies offered to other businesses, and business-to-consumer (B2C) services in support of the operator consumer business.
Splitting the architecture as described above creates two additional transport domains known as fronthaul and midhaul. These domains are implemented when remote radio head (RRH)/DU split or CU/DU splits are exposed. In some cases, it’s possible to implement fronthaul over dark fiber, which is basically, transport of time domain or frequency domain baseband samples between the DU and the RU/RRH. In other cases, transport techniques that rely on Ethernet, IP, or WDM must be employed. Midhaul is basically the transport of GTP-u packets and associated control plane between the CU and the DU. This type of transport is easily implemented over IP.
“As the newest MNO in Japan, Rakuten Mobile Network Inc. has partnered with Cisco to design and deliver an innovative 5G system architecture from the ground up. This is clearly the world’s first cloud-native network that is fully virtualized from RAN to core, with an unprecedented level of automation for both network and services.
This disruptive architecture will enable Rakuten to offer a broad suite of services including consumer mobile, NB-IOT, fixed wireless, connected everything, rich media, low latency services such as AR and VR and many more. And all these services will benefit from a unique mobile edge computing infrastructure that enables the best possible user experience.”Tareq Amin, CTO, Rakuten Mobile Network Inc.
The time has come for the industry to evolve past multiprotocol label switching (MPLS) and its derived models such as unified MPLS. While successful to date, MPLS techniques are characterized by a multiplicity of control planes which prove too complex as networks evolve. It’s far simpler to collapse to one proven control plane, the IP control plane, and to use the augmented capabilities of IPv6. We propose a transport infrastructure on a single control plane with traffic engineering supported by segment routing. One strength of segment routing is the consolidation of service chaining used in building virtualized services with the traffic engineering required in WAN transport into one simple seamless transport framework.
The transport approach we propose eliminates control plane protocols such as label distribution protocol (LDP) and resource reservation protocol – traffic engineering (RSVP-TE) where they are redundant with what IP can provide. It removes flow state from the network and still provides several options for control plane implementation. These options range from a fully distributed implementation to a hybrid approach in which the router and a software-defined networking (SDN) controller divide the functionality between themselves. The transport infrastructure based on segment routing can overlay a wide range of service technologies, with associated IP-based service-level agreements (SLAs), including BGP-based virtual private network (VPN) technologies, such as EVPNs and VPNv4/v6s, and emerging software-defined wide-area network (SD-WAN) VPN technologies.4
The automation architecture is relevant to the new approach described here. Model-driven automation has emerged as a major paradigm for configuration management of modern networks. In the model-driven approach, a definition language such as YANG provides models for network elements (NEs) and for services deployable over those NEs.5 An agile and programmable network configuration engine such as the Cisco Network Services Orchestrator parses the models and creates a framework for configuring the network using NETCONF or other interfaces into NEs. Configuration of the service is done directly using RESTful APIs or even using CLI commands.6,7
Model-driven configuration management should become pervasive in a 5G environment and apply across all 5G domains including RAN, transport, core, and services. Operating expenses (OpEx) are reduced through zero-touch automation. In addition to the OpEx benefit, a fully automated network is a prerequisite to API enablement, which is critical for easy consumption of 5G networks from other parties.
APIs are at the heart of any 5G strategy. A well-honed familiar set of APIs is a compelling vehicle for revenue generation in an economy that is increasingly dominated by B2B relationships. A set of APIs that enables an enterprise to consume 5G services such as Network-as-a-Service (NaaS) helps build a valuable channel into industry verticals. Network slicing is key area of API-enablement and APIs play a pivotal role. In network slicing, the operator’s infrastructure is divided into multiple tenancies, each with isolation and dedicated to a specific service.
The idea of disaggregation, virtualization, and decomposition of various components and functions in the network and in the overall network architecture has opened many new exciting possibilities for operators. In the new and evolved end-to-end mobile architecture, the idea of disaggregation and decomposition is being applied in multiple ways with many advantages.
Cloud and virtualization technologies have enabled operators to disaggregate software from the underlying hardware. This model offers benefits such as:
Functional decomposition allows even more flexibility beyond hardware and software disaggregation. The decomposition of the control and user plane for the evolved packet core is one example and the decomposition of radio functions into RRH/RU, DU, and CU outlined earlier is another. Decomposition makes it possible to:
To help an operator benefit from a multivendor approach, it’s necessary for the software implementation of the RAN functions to be decomposed from the hardware. The CU functions of a cloud RAN deployment can be instantiated on a mass-produced Intel x86 server. Functions are virtualized on a carrier-grade network functions virtualization (NFV) software framework. The DU functions, based on availability of type of transport, may be virtualized on a similar NFV platform or may be implemented as a network function on an Intel x86 server near the cell site.
A cluster of remote radio units (RRUs) can be aggregated into a DU. In turn, multiple DUs can be aggregated into a CU. The solution architecture allows the operator to scale their network as the number of cells, MIMO layers, frequencies, and user capacity increases. In the case where both the DU and CU functions are virtualized, scaling could entail instantiating more virtualized CU (vCU) or virtualized DU (vDU) functions as virtual network functions (VNFs) on the NFV platform or it could entail scaling up the processing capability of an existing VNF.
Figure 3. Cloud RAN network scalability
Disaggregating the baseband hardware from the software provides flexibility. Operators can choose to deploy best-of-breed solutions in terms of the RRU, baseband hardware, and software. Building a cloud RAN architecture has many benefits because it fundamentally changes the way networks are procured, built, and operated. Here are some of the benefits:
A distributed telco cloud platform that is fully automated can help today’s communication service providers evolve into tomorrow’s digital service providers. The distributed telco cloud essentially is the virtualized telco infrastructure built using data center, cloud, and virtualization technologies; spanning from centralized telco DCs all the way to 100s-1000s of edge locations that are closer to the consumers. They are interconnected using a programmable WAN with common policy and end-to-end automation that exposes rich APIs northbound. This approach enables delivery of a suite of services at cloud scale with agile operations and ease of consumption.
Figure 4 shows an architectural framework of a distributed telco cloud platform featuring a loosely coupled, modular architecture where the integration between layers is driven using open APIs and data models. The modularity of this architecture allows compliance to any disaggregation requirement. Loose coupling between layers guarantees isolation, decomposability, flexibility, and openness. Each layer of the architecture is capable of supporting a multivendor deployment model.
Figure 4. Architectural framework.
The highly scalable architecture covers deployments that include central and regional telco DCs, along with new emerging edge locations. The edge typically is highly distributed and is targeted for location types such as central offices, aggregation, or mobile switching offices and C-RAN hub or preaggregation.
A programmable transport network provides secured and optimally engineered connectivity between sites of the distributed cloud platform. The programmability of the transport network, enabled by Cisco’s IOS XR- and IOS XE-based routing portfolio ensures that the virtualization platform and upper layer automation and orchestration system can interact with it dynamically. It creates optimal control and forwarding plane constructs, performs resource management and enables the dynamic allocation of network resources such as traffic-engineered path and network slices.
The virtualization platform is fundamental to the framework, and in Figure 4 is depicted as NFVI or cloudlets. It’s a combination of hardware, such as compute, storage and network, plus virtualization software integrated together in a validated stack to create a common virtualized infrastructure that can host a wide variety of workloads. A workload could be a network function (NF), edge computing application, or eventually an IT application. The virtualization model could use virtual machines or containers with a microservices architecture.
The NFVI or cloudlets in this foundation can be enabled by Cisco Network Functions Virtualization Infrastructure (NFVI) solution that is powered with Cisco Virtual Infrastructure Manager (CVIM). CVIM provides rich cloud orchestration and lifecycle management features with extensive operational tooling to bring deep visibility and OAM capabilities to this framework. Cisco VIM enables a carrier-class NFVI solution that is built on a standards-based architecture with various open source components such as OpenStack, KVM, Linux, Docker, Kubernetes, OVS, DPDK, and fd.io. Cisco NFVI is engineered to enable various optimal footprints depending on the type of location, resource availability, physical constraints, and application requirements. It can support everything from large sites such as a central data center, all the way down to smaller sites like a central office or a preaggregation/cloud RAN hub. The number of servers can range from more than 100 in the largest footprint down to three servers in its minimal form. All footprints have the same availability, performance, resiliency, security, and operational capabilities.
These highly distributed NFVI or cloudlets could create a monumental management and operational challenge. While each NFVI or cloudlet is fully autonomous to drive its operations, certain capabilities like fault and performance management, along with lifecycle management need to be unified. The architecture provides a layer on top of distributed NFVI or cloudlets with a common set of management and operational capabilities across the platform and all location types. It includes common security and policy, comprehensive lifecycle management, application-aware resource management, DevOps and OAM tools, and infrastructure assurance through a multisite single pane of glass management capability.
The service orchestration shown at the top of Figure 4 provides intent-driven networking capabilities by performing multidomain and multivendor service orchestration. A user expresses intent and the orchestrator translates it into a language understood by the components underneath. Distributed resource management, multi-domain capacity management, service level analytics, assurance, interface with other elements like portal or operations support systems (OSS) will be delivered at this layer to provide an end-to-end architecture.
The service orchestration layer is realized with Cisco Elastic Services Controller (ESC) as a generic VNF Manager and Cisco Network Services Orchestrator (NSO) as the NFV orchestrator along with a service assurance solution. Cisco ESC, a multi-VIM and multicloud-capable VNFM, provides rich VNF lifecycle management for both Cisco and third-party VNFs. It can be deployed in a high-availability pair for a single larger site, or in a decomposed form supporting many smaller sites with VNF monitoring in each smaller site to increase reliability and scale. Cisco NSO is model driven, multivendor, and multidevice capable. It supports physical and virtual functions and provides rich orchestration capabilities for NFV, as well as other domains such as programmable transport/WAN and data centers. In a distributed telco cloud architecture, a multilayer NSO deployment is recommended to ensure modularity and operational demarcation. In this configuration, the lower layers perform individual domain orchestration (WAN, DC, cloudlets) while an upper layer NSO controls the various lower layer orchestrators to provide end-to-end cross-domain orchestration. Rich northbound APIs are exposed from the upper layer towards OSS, business support system (BSS), and to third-party partners to ensure easy consumption of the distributed telco cloud platform.
Rakuten Inc., an internet services company headquartered in Japan, is a global leader in e-commerce and financial technology. It also offers media services such as over-the-top video, live TV, music, ebooks, and social media messaging. Rakuten Mobile Network (RMN), a wholly owned subsidiary of Rakuten Inc., is the newest mobile network operator in Japan. They plan to launch their commercial services in October 2019 with an innovative architecture that is set to disrupt the telecom industry landscape globally.
Embracing a 5G systems architecture from inception, this will be the world’s first cloudified network that is fully virtualized from RAN to core with end-to-end automation for both network and services. Radio access will initially be 4G LTE (macro and small cells) and Wi-Fi; with plans for 5G radio technology to be added during early 2020 before the Summer Olympics. This disruptive architecture will enable RMN to deliver a broad suite of services including consumer mobile, narrowband IOT, rich media, and low-latency services including augmented and virtual reality. All services will benefit from an innovative MEC architecture enabling a differentiated user experience.
Figure 5 illustrates the high-level and simplified architecture established by RMN.
Figure 5. Rakuten network architecture.
Most current occurrences of telco clouds are siloed instantiations of network functions. In contrast, RMN is deploying a common, horizontal, and carrier-grade telco cloud for all virtualized applications from RAN to core. It uses a common NFV infrastructure management layer that will be deployed in a highly distributed manner across thousands of locations from the edge to centralized data centers. This converged infrastructure approach results in high efficiency, reduced cost, operational simplicity, service delivery speed, and optimal scale-out capability.
Cisco Virtual Infrastructure Manager (VIM) running on x86 Servers with Intel technology, combined with Cisco Nexus and ACI-based switching fabric, form the foundation of this distributed telco cloud. RMN plans to onboard multiple virtualized applications on this platform including: Altiostar vRAN, Cisco vEPC, Nokia and Mavenir vIMS, InnoEye OSS, Netcracker BSS, and many other VNFs related to security and sGi-LAN.
The packet core functions are provided by the industry-leading Cisco Ultra Services Platform. Control and user plane separation is a key enabler for this architecture to provide scalable multi-access edge computing (MEC) capabilities. Cisco Ultra enables CUPS from day one in the Rakuten Mobile Network, and has the flexibility to scale functions independently while laying the foundation for an easy migration to a full 5G systems architecture.
An open and virtualized RAN is at the epicenter of RMN’s technical strategy. It will be the first network to launch with a fully virtualized RAN solution from Altiostar Networks. The solution architecture enables a two-layer split employing fronthaul from the cell site towards pre-aggregation locations where the lower layer of the radio stack is hosted on Virtualized DU (vDU). The upper layer of the radio stack is hosted on virtualized CU that gets connected to the vDU using the midhaul interface.
Although virtualized CU solutions have existed in the industry for some time, this is the first commercial implementation of a fully virtualized DU function for a 4G LTE macro RAN. Given the stringent real-time performance and scale requirements necessary to process the digital RF signal, virtualizing the DU function for a production-grade deployment was not trivial.
This first fully virtualized DU came together for Rakuten with innovation and collaboration from multiple industry leaders in the Open vRAN partner ecosystem, including Altiostar Networks with its vRAN software solution, Cisco Systems with its VIM, ESC, NSO, Nexus switching, and NCS 5500 Routing solutions, Intel with its Xeon family of CPU, NICs, FPGA technology for acceleration and FlexRAN software framework, as well as Red Hat with RHEL and OSP solutions.
Figure 6 depicts a high-level schematic of the RAN architecture for RMN.
Figure 6. RAN architecture.
The architecture enables a lean cell site for RMN comprising the antenna and remote radio head. It supports coordinated radio processing to increase spectral efficiency, facilitates scale out, and exposes open APIs that enable multiple vendors to participate and contribute.
Figure 7. Simplified deployment.
The architecture enables RMN to generate a significant reduction in capital expenditures. It also reduced site construction time and costs, increased site acquisition success, and allowed better coordination and coverage, particularly in rural areas.
RMN’s innovative edge architecture uses vRAN, control and user plane separated (CUPS) packet core, and distributed telco cloud. It enables MEC for both infrastructure functions and a variety of low-latency and content-centric services. Examples of such services include optimized content delivery, live TV, connected car, augmented and virtual reality, on-line gaming, connected stadiums, and more. While others are looking at similar possibilities afforded by 5G, RMN will tap into these opportunities with both 4G and 5G to deliver the best possible user experience.
RMN is deploying a 5G system architecture today with the virtualization/NFV, vRAN, SDN, automation, CUPS packet core, edge computing, slicing, scale, and capacity that are inherent to the RMN architecture. This architecture creates a foundation that makes it easy to add 5G capabilities through software upgrades, which can reduce time to market.
Powered by Cisco NCS 5500 routers with an IOS XR-based solution, RMN’s mobile backhaul transport network is built with the capacity and scale of 5G in mind. The core of the network will have multiterabit capacity. Multiples of 100G of bandwidth will be made available to the cell site preaggregation network, in contrast to the rest of the world that has deployed either a 1G or 10G-based network. The network is based on IPv6 with a migration path to IPv6 segment routing, which will enable scalability and avoid address translation functions that are expensive and complex to operate.
RMN’s central and regional software defined data center fabrics are powered by Cisco ACI and will host a wide array of core and edge services. They are designed with 5G in mind and built as agile hubs of service delivery. They feature tens of terabits of capacity, horizontal scale, automation, and analytics.
A common problem across the industry is that operators usually maintain hundreds of SKUs for equipment in their network. RMN has standardized on fewer than ten SKUs. Limiting SKUs enables infrastructure standardization, easy scale-out, simplified operations, and easier spares management while maintaining a sound balance between cost and performance.
Automation is a fundamental objective of the RMN strategy. Cisco NSO along with the element management system (EMS) and OSS tools are enabling automation. All network infrastructure components, along with the services, are automated in RMN to perform initial deployment and ongoing lifecycle management. This reduces OpEx and minimizes human control to deploy and operate the network compared to a traditional operator. It also helps avoid issues caused by human error and facilitates faster response to problems, eventually paving the way to autoremediation.
OSS siloes are another significant problem for many traditional operators. RMN has adopted a unified OSS strategy where one single OSS/BSS layer, powered by InnoEye OSS and Netcracker BSS, will be used to enable true cross-domain service and function management.
The RMN open, automated, and software-defined architecture allows the creation of a new vendor ecosystem that can unleash higher innovation and new services at a fraction of the cost of traditional systems.
The Cisco CX team has been instrumental in the incubation of the Rakuten cloud-native architecture. The CX team has set-up a program and architecture management office (PAMO) that is responsible for the end-to-end design, solution validation, test automation, and deployment and management of workloads. The main focus is outcome-based program success. We’re deploying new tools and methodology using continuous integration and continuous deployment.
Integrated automation is a key requirement to manage this next-generation network. The CX team has developed full-fledged closed-loop automation with full lifecycle management using Network Service Orchestrator and Elastic Services Controller for over 100 VNF types (over 50,000 instances) in Rakuten’s network. The team also implemented the world’s first fully automated virtualized RAN deployment that brings down the time to deploy a radio site to a matter of minutes from weeks. It allows the management of more than 150,000 macro and small-cell radio sites without manual intervention.
CX is redefining the software delivery methodology of virtualized mobile network solutions to Rakuten with a software defined test framework. It integrates unified delivery of all VNFs from Cisco and our partners to Rakuten’s solution lab in an agile fashion. The KPIs captured in the portal include health, licenses, reservation of resources, and vendor score cards. We are deploying on-premise software platforms for accelerated development and evolution (SPADE) with the CICD tools used in CX with more than 300 customers as an extension to Rakuten. It gives Rakuten access to the Cisco Engineering teams to refine and get innovative features using an agile delivery model.
Security is designed based on the principle of “zero-trust, zero-touch,” which implies invisible security with full automation. The CX team has deployed security tools integrated into VNF so it is embedded into the overall solution. Security moves with workload mobility so that every transaction between all entities of the network is secured, while minimizing the operational burden by making these security constructs transparent to all end users.
There is finally an effort to define organization and models for cloud operations. The objective is to manage the lifecycle of the network while keeping a steady operational staff with a small footprint. We are using DevOps systems with site reliability engineering practices and distributed cloud operations to self-manage the entire infrastructure.
The mobile industry and its supply chain are on the verge of a major transformation. This transformation is software-driven and will result in a more agile and efficient mobile network architecture. The new mobile architecture is built on two ideas. First, the principle of disaggregation, which is the notion that software can be purchased separately from hardware. And second, the principle of decomposition, which separates previously monolithic systems into multiple functions that can be deployed in the best way to meet operator business requirements, while using standard interfaces.
The deconstructed mobile architecture can be deployed across a network and can be built using multiple suppliers. The net effect is a feature-harmonized network which is fundamentally and inherently multivendor. In this new mobile network, automation is built-in using a model-driven approach which also supports API enablement for B2B and NaaS capabilities. Edge sites play a prominent role as clouds are deployed to support better radio service. These same edge sites become the base locations for edge computing services.
The future starts now. This new mobile network architecture can be realized today for 5GNR or LTE, and operators can choose a variety of paths to get there. Prudent steps will enable more agile service delivery with reduced expenses, resulting in increased competitiveness and profitability while providing a superior quality of experience.
5. RFC6020, YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF), 2010.