Table Of Contents
Configuring IP Storage
Services Modules
Module Status Verification
IPS Module Upgrade
MPS-14/2 Module Upgrade
Supported Hardware
IPS Module Core Dumps
Configuring Gigabit Ethernet High Availability
VRRP for iSCSI and FCIP Services
Configuring VRRP for Gigabit Ethernet Interfaces
About Ethernet PortChannel Aggregation
Configuring Ethernet PortChannels
Configuring CDP
Displaying Statistics
Displaying Gigabit Ethernet Interface Statistics
Displaying Ethernet MAC Statistics
Displaying DMA-Bridge Statistics
Displaying TCP Statistics
Default Settings
Configuring IP Storage
Cisco MDS 9000 Family IP storage (IPS) services extend the reach of Fibre Channel SANs by using open-standard, IP-based technology. The switch connects separated SAN islands using Fibre Channel over IP (FCIP), and it allows IP hosts to access Fibre Channel storage using the iSCSI protocol.
Note FCIP and iSCSI features are specific to the IPS module and are available in Cisco MDS 9200 Switches or Cisco MDS 9500 Directors.
The Cisco MDS 9216I switch and the 14/2 Multiprotocol Services (MPS-14/2) module also allow you to use Fibre Channel, FCIP, and iSCSI features. The MPS-14/2 module is available for use in any switch in the Cisco MDS 9200 Series or Cisco MDS 9500 Series.
This chapter includes the following sections:
•Services Modules
•Supported Hardware
•IPS Module Core Dumps
•Configuring Gigabit Ethernet High Availability
•Configuring CDP
•Default Settings
Services Modules
The IP Storage services module (IPS module) and the MPS-14/2 module allow you to use FCIP and iSCSI features. Both modules integrate seamlessly into the Cisco MDS 9000 Family, and support the full range of features available on other switching modules, including VSANs, security, and traffic management. The following types of storage services modules are currently available for use in any switch in the Cisco MDS 9200 Series or in the Cisco MDS 9500 Series:
•The 4-port, hot-swappable IPS module (IPS-4) has four Gigabit Ethernet ports.
•The 8-port, hot-swappable IPS module (IPS-8) has eight Gigabit Ethernet ports.
•The MPS-14/2 module has 14 Fibre Channel ports (numbered 1 through 14) and two Gigabit Ethernet ports (numbered 1 and 2).
Gigabit Ethernet ports in these modules can be configured to support the FCIP protocol, the iSCSI protocol, or both protocols simultaneously.:
•FCIP—FCIP transports Fibre Channel frames transparently over an IP network between two Cisco MDS 9000 Family switches or other FCIP standards-compliant devices. Figure 45-1 shows how the IPS module is used in different FCIP scenarios.
Figure 45-1 FCIP Scenarios
•iSCSI—The IPS module provides IP hosts access to Fibre Channel storage devices. The IP host sends SCSI commands encapsulated in iSCSI protocol data units (PDUs) to a Cisco MDS 9000 Family switch IPS port over a TCP/IP connection. At this point, the commands are routed from an IP network into a Fibre Channel network and forwarded to the intended target. Figure 45-2 depicts the iSCSI scenarios in which the IPS module is used.
Figure 45-2 iSCSI Scenarios
Module Status Verification
After inserting the module, verify the status of the module using the show module command:
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 0 Caching Services Module DS-X9560-SMAP ok
2 8 IP Storage Services Module DS-X9308-SMIP ok <-------IPS-8 module
4 16 2x1GE IPS, 14x1/2Gbps FC Module DS-X9216i-K9-SUP ok <-------MPS-14/2 module
5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active *
6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby
9 4 IP Storage Services Module DS-X9304-SMIP ok <---------IPS-4 module
Mod Sw Hw World-Wide-Name(s) (WWN)
--- ----------- ------ --------------------------------------------------
1 2.0(1) 0.201 20:41:00:0b:fd:44:68:c0 to 20:48:00:0b:fd:44:68:c0
2 2.0(1) 0.201 20:41:00:0b:fd:44:68:c0 to 20:48:00:0b:fd:44:68:c0
4 2.0(1) 0.201 20:c1:00:05:30:00:07:1e to 20:d0:00:05:30:00:07:1e
9 2.0(1) 0.1 22:01:00:05:30:00:07:1e to 22:04:00:05:30:00:07:1e
Mod Application Image Description Application Image Version
-------- ----------------------------- -------------------------
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 00-05-30-01-49-c2 to 00-05-30-01-4a-46 JAB073907EP
2 00-05-30-00-9d-d2 to 00-05-30-00-9d-de JAB064605a2
4 00-05-30-01-7f-32 to 00-05-30-01-7f-38 JAB081405AM
5 00-05-30-00-2c-4e to 00-05-30-00-2c-52 JAB06350B1M
6 00-05-30-00-19-66 to 00-05-30-00-19-6a JAB073705GL
9 00-0d-bc-2f-d6-00 to 00-0d-bc-2f-d6-08 JAB080804TN
IPS Module Upgrade
Caution A software upgrade is only disruptive for the IPS module. The SAN-OS software continues to support nondisruptive software upgrades for Fibre Channel modules in the switch and for the switch itself.
IPS modules use a rolling upgrade install mechanism where each module in a given switch can only be upgraded in sequence. To guarantee a stable state, each IPS module in a switch requires a 5-minute delay before the next IPS module is upgraded.
MPS-14/2 Module Upgrade
Caution A software upgrade is only partially disruptive for the MPS-14/2 module. The SAN-OS software continues to support nondisruptive software upgrades for Fibre Channel modules in the switch and for the switch itself.
The MPS-14/2 modules have 14 Fibre Channel ports (nondisruptive upgrade) and 2 Gigabit Ethernet ports (disruptive upgrade). MPS-14/2 modules use a rolling upgrade install mechanism for the two Gigabit Ethernet ports where each module in a given switch can only be upgraded in sequence. To guarantee a stable state, each MPS-14/2 module in a switch requires a 5-minute delay before the next module is upgraded.
Supported Hardware
You can configure the FCIP and iSCSI features using one or more of the following hardware:
•IPS-4 and IPS-8 modules (refer to the Cisco MDS 9200 Series Hardware Installation Guide or the Cisco MDS 9500 Series Hardware Installation Guide for more information)
•MPS-14/2 module (refer to the Cisco MDS 9200 Series Hardware Installation Guide or the Cisco MDS 9500 Series Hardware Installation Guide for more information).
Note In both the MPS-14/2 module and the Cisco MDS 9216i integrated supervisor module, the port numbering differs for the Fibre Channel ports and the Gigabit Ethernet ports. The Fibre Channel ports are numbered from 1 through 14 and the Gigabit Ethernet ports are numbered 1 and 2.
•Cisco MDS 9216i Switch (refer to the Cisco MDS 9200 Series Hardware Installation Guide).
IPS Module Core Dumps
IPS core dumps are different from the system's kernel core dumps for other modules. When the IPS module's operating system (OS) unexpectedly resets, it is useful to obtain a copy of the memory image (called a IPS core dump) to identify the cause of the reset. Under that condition, the IPS module sends the core dump to the supervisor module for storage. Cisco MDS switches have two levels of IPS core dumps:
•Partial core dumps (default)—Each partial core dump consists of four parts (four files). All four files are saved in the active supervisor module.
Use the show cores command to list these files.
•Full core dumps—Each full core dump consists of 75 parts (75 files). The IPS core dumps for the MPS-14/2 module and the Cisco MDS 9216i Switch only contains 38 parts. This dump cannot be saved on the supervisor module because of its large space requirement. They are copied directly to an external TFTP server.
Use the system cores tftp: command to configure an external TFTP server to copy the IPS core dump (and other core dumps).
To configure IPS core dumps on the IPS module, follow these steps:
|
Command
|
Purpose
|
Step 1
|
switch# config terminal
switch(config)#
|
Enters configuration mode.
|
Step 2
|
switch(config)# ips core dump full
|
Configures a dump of the full core generation for all IPS modules in the switch.
|
switch(config)# no ips core dump full
|
Disables the configuration of dump of full core generation for all IPS modules in the switch.
|
Step 3
|
switch(config)# ips core dump partial
|
Configures a dump of the partial core (default) generation for the IPS modules in the switch.
|
switch(config)# no ips core dump partial
|
Disables the configuration of dump of partial core generation for all IPS modules in the switch.
|
Configuring Gigabit Ethernet High Availability
Virtual Router Redundancy Protocol (VRRP) and Ethernet PortChannels are two Gigabit Ethernet features that provide high availability for iSCSI and FCIP services.
VRRP for iSCSI and FCIP Services
VRRP provides a redundant alternate path to the Gigabit Ethernet port for iSCSI and FCIP services. VRRP provides IP address failover protection to an alternate Gigabit Ethernet interface so the IP address is always available (see Figure 45-3).
Figure 45-3 VRRP Scenario
In Figure 45-3, all members of the VRRP group must be IP storage Gigabit Ethernet ports. VRRP group members can be one or more of the following interfaces:
•One or more interfaces in the same IPS module or MPS-14/2 module
•Interfaces across IPS modules or MPS-14/2 modules in one switch
•Interfaces across IPS modules or MPS-14/2 modules in different switches
•Gigabit Ethernet subinterfaces
•Ethernet PortChannels and PortChannel subinterfaces
See the "Virtual Router Redundancy Protocol" section on page 44-16.
Note You can configure no more than seven VRRP groups, both IPv4 and IPv6, on a Gigabit Ethernet interface, including the main interface and all subinterfaces.
Configuring VRRP for Gigabit Ethernet Interfaces
To configure VRRP for Gigabit Ethernet interfaces using IPv4, follow these steps:
|
Command
|
Purpose
|
Step 1
|
switch1# config terminal
switch1(config)#
|
Enters configuration mode.
|
Step 2
|
switch(config)# interface
gigabitethernet 2/2
switch(config-if)#
|
Enters the interface configuration mode on the Gigabit Ethernet interface (slot 2, port 2).
|
Step 3
|
switch(config-if)# ip address
10.1.1.10 255.255.255.0
|
Assigns the IPv4 address (10.1.1.10) and subnet mask (255.255.255.0) for the Gigabit Ethernet interface.
|
Step 4
|
switch(config-if)# no shutdown
|
Enables the selected interface.
|
Step 5
|
switch(config-if)# vrrp 100
switch(config-if-vrrp)
|
Creates VR ID 100.
|
Step 6
|
switch(config-if-vrrp)# address
10.1.1.100
|
Configures the virtual IPv4 address (10.1.1.100) for the selected VRRP group (identified by the VR ID).
Note The virtual IPv4 address must be in the same subnet as the IPv4 address of the Gigabit Ethernet interface. All members of the VRRP group must configure the same virtual IPv4 address.
|
Step 7
|
switch(config-if-vrrp)# priority 10
|
Configures the priority for the selected interface within this VRRP group.
Note The interface with the highest priority is selected as the master.
|
Step 8
|
switch(config-if-vrrp)# no shutdown
|
Enables the VRRP protocol on the selected interface.
|
To configure VRRP for Gigabit Ethernet interfaces using IPv6, follow these steps:
|
Command
|
Purpose
|
Step 1
|
switch1# config terminal
switch1(config)#
|
Enters configuration mode.
|
Step 2
|
switch(config)# interface
gigabitethernet 2/2
switch(config-if)#
|
Enters the interface configuration mode on the Gigabit Ethernet interface (slot 2, port 2).
|
Step 3
|
switch(config-if)# ipv6 address
2001:0db8:800:200c::417a/64
|
Assigns the IPv6 address for the Gigabit Ethernet interface.
|
Step 4
|
switch(config-if)# no shutdown
|
Enables the selected interface.
|
Step 5
|
switch(config-if)# vrrp ipv6 100
switch(config-if-vrrp-ipv6)
|
Creates VR ID 100.
|
Step 6
|
switch(config-if-vrrp-ipv6)# address
2001:0db8:800:200c::417a
|
Assigns single primary link-local IPv6 address or one of the multiple secondary IPv6 addresses.
Note If this IPv6 address is the same as the physical IPv6 address, this switch is automatically the owner of this IPv6 address.
|
Step 7
|
switch(config-if-vrrp-ipv6)# priority
10
|
Configures the priority for the selected interface within this VRRP group.
Note The interface with the highest priority is selected as the master.
|
Step 8
|
switch(config-if-vrrp-ipv6)# no
shutdown
|
Enables the VRRP protocol on the selected interface.
|
Note If you configure secondary VRRP IPv6 addresses on an IPFC VSAN interface, before a downgrading to a release prior to Cisco Release 3.0(1), you must remove the secondary VRRP IPv6 addresses. This is required only when you configure IPv6 addresses.
Note The VRRP preempt option is not supported on IPS Gigabit Ethernet interfaces. However, if the virtual IP address is also the IP address for the interface, then preemption is implicitly applied.
About Ethernet PortChannel Aggregation
Ethernet PortChannels refer to the aggregation of multiple physical Gigabit Ethernet interfaces into one logical Ethernet interface to provide link redundancy and, in some cases, higher aggregated bandwidth and load balancing.
An Ethernet switch connecting to the MDS switch Gigabit Ethernet port can implement load balancing based on the IP address, IP address and UDP/TCP port number, or MAC address. Due to the load balancing scheme, the data traffic from one TCP connection is always sent out on the same physical Gigabit Ethernet port of an Ethernet PortChannel. For the traffic coming to the MDS, an ethernet switch can implement load balancing based on its IP address, its source-destination MAC address, or its IP address and port. The data traffic from one TCP connection always travels on the same physical links. To make use of both ports for the outgoing direction, multiple TCP connections are required.
All FCIP data traffic for one FCIP link is carried on one TCP connection. Consequently, the aggregated bandwidth is 1 Gbps for that FCIP link.
Note The Cisco Ethernet switch's PortChannel should be configured as a static PortChannel, and not the default 802.3ad protocol.
Ethernet PortChannels can only aggregate two physical interfaces that are adjacent to each other on a given IPS module (see Figure 45-4).
Note PortChannel members must be one of these combinations: ports 1-2, ports 3-4, ports 5-6, or ports 7-8.
Figure 45-4 Ethernet PortChannel Scenario
In Figure 45-4, Gigabit Ethernet ports 3 and 4 in slot 9 are aggregated into an Ethernet PortChannel. Ethernet PortChannels are not supported on MPS-14/2 modules and 9216i IPS modules.
Note PortChannel interfaces provide configuration options for both Gigabit Ethernet and Fibre Channel. However, based on the PortChannel membership, only Gigabit Ethernet parameters or Fibre Channel parameters are applicable.
Configuring Ethernet PortChannels
The PortChannel configuration specified in Chapter 17, "Configuring PortChannels," also applies to Ethernet PortChannel configurations.
To configure Ethernet PortChannels, follow these steps:
|
Command
|
Purpose
|
Step 1
|
switch1# config terminal
switch1(config)#
|
Enters configuration mode.
|
Step 2
|
switch(config)# interface port-channel 10
switch(config-if)#
|
Configures the specified PortChannel (10).
|
Step 3
|
switch(config-if)# ip address 10.1.1.1
255.255.255.0
|
Enters the IPv4 address (10.1.1.1) and subnet mask (255.255.255.0) for the PortChannel.
Note A PortChannel does not have any members when first created.
|
Step 4
|
switch(config-if)# no shutdown
|
Enables the interface.
|
Step 5
|
switch(config)# interface gigabitethernet 9/3
switch(config-if)#
|
Configures the specified Gigabit Ethernet interface (slot 9, port 3).
|
Step 6
|
switch(config-if)# channel-group 10
gigabitethernet 9/3 added to port-channel 10
and disabled
please do the same operation on the switch at
the other end of the port-channel, then do "no
shutdown" at both ends to bring them up
switch(config-if)#
|
Adds Gigabit Ethernet interfaces 9/3 to channel group 10. If channel group 10 does not exist, it is created. The port is shut down.
|
Step 7
|
switch(config-if)# no shutdown
|
Enables the selected interface.
|
Step 8
|
switch(config)# interface gigabitethernet 9/4
switch(config-if)#
|
Configures the specified Gigabit Ethernet interface (slot 9, port 4).
|
Step 9
|
switch(config-if)# channel-group 10
gigabitethernet 9/4 added to port-channel 10
and disabled
please do the same operation on the switch at
the other end of the port-channel, then do "no
shutdown" at both ends to bring them up
|
Adds Gigabit Ethernet interfaces 9/4 to channel group 10. The port is shut down.
|
Step 10
|
switch(config-if)# no shutdown
|
Enables the selected interface.
|
Note Gigabit Ethernet interfaces cannot be added to a PortChannel if one of the following cases apply:
- The interface already has an IP address assigned.
- The subinterfaces are configured on that interface.
- The interface already has an associated IPv4-ACL rule and the PortChannel does not.
Configuring CDP
The Cisco Discovery Protocol (CDP) is supported on the management Ethernet interface on the supervisor module and the Gigabit Ethernet interfaces on the IPS module or MPS-14/2 module.
See the "Configuring CDP" section on page 5-36.
Displaying Statistics
This section provides examples to verify Gigabit Ethernet and TCP/IP statistics on the IP storage ports.
Displaying Gigabit Ethernet Interface Statistics
Use the show interface gigabitethernet command on each switch to verify that the interfaces are up and functioning as desired. See Example 45-1 and Example 45-2.
Example 45-1 Displays the Gigabit Ethernet Interface
switch# show interface gigabitethernet 8/1
GigabitEthernet8/1 is up <-----------The interface is in the up state.
Hardware is GigabitEthernet, address is 0005.3000.a98e
Internet address is 10.1.3.1/24
MTU 1500 bytes, BW 1000000 Kbit
5 minutes input rate 744 bits/sec, 93 bytes/sec, 1 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
3343 packets input, 406582 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
8 packets output, 336 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
Example 45-2 Displays the Gigabit Ethernet Subinterface
switch# show interface gigabitethernet 4/2.100
GigabitEthernet4/2.100 is up
Hardware is GigabitEthernet, address is 0005.3000.abcb
Internet address is 10.1.2.100/24
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
1 packets output, 46 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
Displaying Ethernet MAC Statistics
The show ips stats mac interface gigabitethernet command takes the main Gigabit Ethernet interface as a parameter and returns Ethernet statistics for that interface. See Example 45-3.
Note Use the physical interface, not the subinterface, to display Ethernet MAC statistics.
Example 45-3 Displays Ethernet MAC Statistics
switch# show ips stats mac interface gigabitethernet 8/1
Ethernet MAC statistics for port GigabitEthernet8/1
Hardware Transmit Counters
0 collisions, 0 late collisions, 0 excess collisions
0 bad frames, 0 FCS error, 0 abort, 0 runt, 0 oversize
Hardware Receive Counters
427916 bytes, 3464 frames, 0 multicasts, 3275 broadcasts
0 bad, 0 runt, 0 CRC error, 0 length error
0 code error, 0 align error, 0 oversize error
3429 received frames, 237 transmit frames
0 frames soft queued, 0 current queue, 0 max queue
Displaying DMA-Bridge Statistics
You can display direct memory access (DMA) device statistics using the show ips stats dma-bridge interface gigabitethernet command. This command takes the main Gigabit Ethernet interface as a parameter and returns DMA bridge statistics for that interface. See Example 45-4.
Note Use the physical interface, not the subinterface, to display DMA-bridge statistics.
Example 45-4 Displays DMA-Bridge Statistics
switch# show ips stats dma-bridge interface gigabitethernet 7/1
Dma-bridge ASIC Statistics for port GigabitEthernet7/1
231117 Good, 0 bad protocol, 0 bad header cksum, 0 bad FC CRC
Hardware Ingress Counters
218255 Good, 0 protocol error, 0 header checksum error
0 FC CRC error, 0 iSCSI CRC error, 0 parity error
231117 good frames, 0 bad header cksum, 0 bad FIFO SOP
0 parity error, 0 FC CRC error, 0 timestamp expired error
0 unregistered port index, 0 unknown internal type
0 RDL ok, 0 RDL drop (too big), 0 RDL ttl_1
3656368645 idle poll count, 0 loopback, 0 FCC PQ, 0 FCC EQ
Flow Control: 0 [0], 0 [1], 0 [2], 0 [3]
Software Ingress Counters
218255 Good frames, 0 header cksum error, 0 FC CRC error
0 iSCSI CRC error, 0 descriptor SOP error, 0 parity error
0 frames soft queued, 0 current Q, 0 max Q, 0 low memory
0 out of memory drop, 0 queue full drop
0 RDL ok, 0 RDL drop (too big)
Flow Control: 0 [0], 0 [1], 0 [2], 0 [3]
This output shows all Fibre Channel frames that ingress or egress from the Gigabit Ethernet port.
Displaying TCP Statistics
Use the show ips stats tcp interface gigabitethernet to display and verify TCP statistics. This command takes the main Ethernet interface as a parameter, and shows TCP stats along with the connection list and TCP state. The detail option shows all information maintained by the interface. See Example 45-5 and Example 45-6.
Example 45-5 Displays TCP Statistics
switch# show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
0 active openings, 3 accepts
0 failed attempts, 12 reset received, 3 established
163 received, 355 sent, 0 retransmitted
0 bad segments received, 0 reset sent
Local Address Remote Address State Send-Q Recv-Q
0.0.0.0:3260 0.0.0.0:0 LISTEN 0 0
Example 45-6 Displays Detailed TCP Statistics
switch# show ips stats tcp interface gigabitethernet 4/1 detail
TCP Statistics for port GigabitEthernet4/1
355 segments, 37760 bytes
222 data, 130 ack only packets
3 control (SYN/FIN/RST), 0 probes, 0 window updates
0 segments retransmitted, 0 bytes
0 retransmitted while on ethernet send queue, 0 packets split
163 segments, 114 data packets in sequence, 6512 bytes in sequence
0 predicted ack, 10 predicted data
0 bad checksum, 0 multi/broadcast, 0 bad offset
0 no memory drops, 0 short segments
0 duplicate bytes, 0 duplicate packets
0 partial duplicate bytes, 0 partial duplicate packets
0 out-of-order bytes, 1 out-of-order packets
0 packet after window, 0 bytes after window
121 acks, 37764 ack bytes, 0 ack toomuch, 4 duplicate acks
0 ack packets left of snd_una, 0 non-4 byte aligned packets
8 window updates, 0 window probe
30 pcb hash miss, 0 no port, 0 bad SYN, 0 paws drops
0 attempts, 3 accepts, 3 established
3 closed, 2 drops, 0 conn drops
0 drop in retransmit timeout, 1 drop in keepalive timeout
0 drop in persist drops, 0 connections drained
115 segments timed, 121 rtt updated
0 retransmit timeout, 0 persist timeout
12 keepalive timeout, 11 keepalive probes
0 recovery episodes, 0 data packets, 0 data bytes
0 data packets retransmitted, 0 data bytes retransmitted
0 connections closed, 0 retransmit timeouts
15 entries, 3 connections completed, 0 entries timed out
0 dropped due to overflow, 12 dropped due to RST
0 dropped due to ICMP unreach, 0 dropped due to bucket overflow
0 abort due to no memory, 0 duplicate SYN, 0 no-route SYN drop
0 hash collisions, 0 retransmitted
Local Address Remote Address State Send-Q Recv-Q
0.0.0.0:3260 0.0.0.0:0 LISTEN 0 0
Use the show ips stats icmp interface gigabitethernet to display and verify IP statistics. This command takes the main Ethernet interface as a parameter and returns the ICMP statistics for that interface. See Example 45-7.
Example 45-7 Displays ICMP Statistics
switch# show ips stats icmp interface gigabitethernet 2/1
ICMP Statistics for port GigabitEthernet2/1
0 ICMP messages dropped due to errors
0 destination unreachable
0 destination unreachable
Default Settings
Table 45-1 lists the default settings for IP storage services parameters.
Table 45-1 Default Gigabit Ethernet Parameters
Parameters
|
Default
|
IPS core size
|
Partial
|