Randomness (via

Entries in Nexus (4)


Configuring Jumbo Frames on Cisco Data Center Switches

Ethernet jumbo frames are a common requirement in data center networks, especially ones that rely on Ethernet based storage protocols such as NFS and iSCSI. Jumbo frames are not enabled on Cisco switches by default - here are the commands to configure the larger MTUs on the most common switches seen within the data center.

Nexus 5000s and 5500s

Enabling jumbo frames on the Nexus 5000/5500 platform is done via a policy-map. When applied using class-default, jumbo frames are enabled across the whole switch.

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216
system qos
  service-policy type network-qos jumbo

Note: The MTU parameter in “show interface” will continute to show a value of 1500. The best way to verify that jumbo frames is enabled is to setup a ping across the switch to endpoints that are also configured with jumbo frames using a packet size of 9216 and the “do not fragment” bit set (e.g. another switch, a server, NAS controller, etc…).

ping packet-size 9216 df-bit count 10

Nexus 7000

Jumbo frame configuration on the Nexus 7000 is much simpler. Simply set the system jumbo MTU in global config mode:

switch-n7k(config)#system jumbomtu 9216

Then specify the MTU for specific interfaces

switch-n7k(config)#interface ethernet x/x
switch-n7k(config-if)#mtu 9216

MTU settings on individual interfaces can be verified via the “show interface” command:

 switch#sh interface vlan10
 Vlan100 is up, line protocol is up
  Hardware is EtherSVI, address is  4055.3927.2cc1
  Description: Sample
  Internet Address is
  MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,

Catalyst 6500 (IOS Native Mode)

Configuring jumbo frames on a 6500 also requires simply setting the MTU on the individual interfaces:

switch-6500(config)#int GigabitEthernet1/1
switch-6500(config-if)#mtu 9216

Verify using the “show interface” command.

Reference Configuration of Jumbo MTU on Nexus 5000 and 7000 Series Jumbo/Giant Frame Support on Catalyst Switches Configuration Example Enabling Jumbo Frames on the Cisco Nexus 1000v


Using Configuration Sync on the Nexus 5500

Cisco’s Virtual Port Channel (VPC) feature is a nice way to build multiple link redundancy between two switches without the limitations of spanning tree (notably having one link unused in “blocking” mode). VPC is an extension of traditional Ether Channel bundles, but across two separate switches, and is a feature on the Cisco Nexus 5000 and 7000 switches.

A challenge when using VPC is that it requires duplicate and identical configuration of ports on both switches that have member ports. Cisco introduced “configuration sync” in NX-OS 5.0(2)N1(1) for the Nexus 5000s to help with this issue.

Setting up Configuration Sync

The initial configuration to enable configuration sync is simple. You do have to have the mgmt0 interface configured on both Nexus switches so that they can communicate with each other over IP.

The initial configuration involves the following steps:

! Enable CFS distribution over IP
    configuration terminal
        cfs ipv4 distribute

! Configure a switch profile and the sync-peer
    config sync
        switch-profile [name]
        sync-peers destination [peer-IP-address]

The switch profile needs to be identically configured on both Nexus switches that are sync-peers. You will need to enter the switch-profile name everytime you want to add a synced configuration, so keep it short and simple - e.g. “switch-profile vpc”.

Once you have the initial configuration entered you can verify that things are working:

N5K01# sh switch-profile vpc status

switch-profile  : vpc

Start-time:  67539 usecs after Wed Apr 25 10:18:18 2012
End-time: 422813 usecs after Wed Apr 25 10:18:20 2012

Profile-Revision: 18
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Sync Success

Local information:
Status: Commit Success

Peer information:
Sync-status: In sync
Status: Commit Success

That’s pretty much it to get it setup.

Configuring Interfaces and Virtual PortChannels

Configuration sync is most useful when configuring Virtual PortChannels (VPC) since it requires an exact replica of the interface and port-channel configuration on two Nexus 5000 switches.

Entering configuration commands using config sync requires two additional steps beyond entering standard config mode.

  1. Enter normal configuration mode

    config t
  2. Enter Configuration Sync mode

    config sync
  3. Enter the switch profile name

    switch-profile [name]

Now you can enter configuration commands just as you would in normal configuration mode.

It’s About Commitment

There is one important additional difference over normal configuration - commands do not take effect, and are not synced, until you type the “commit” command. You can type any number of commands before committing them.

The “commit” process includes a mutual exclusion check (verify) step that validates that no existing commands are unintentionally overwritten. (You can also check this along the way with the “verify” command before you “commit” a large number of commands.) Commands that are entered in configuration sync mode cannot already be configured through regular command mode, and once configuration sync is used, subsequent changes to that configuration also need to be made in configuration sync mode (with the exception of some interface commands, shut/no-shuts, and system qos).

If commands in the commit buffer fail verification you can delete or edit them and then try the commit process again.

See which commands are sitting in the commit buffer:

    show switch-profile buffer

Commands can be moved in the commit buffer:

    buffer-move [current-seqID] [new-seqID]

Commands can also be deleted:

    buffer-delete [seqID]

To clear the entire commit buffer:

    buffer-delete all

Viewing Config Commands in the Switch Profile

Once a configuration is completed it can be hard to remember which interfaces and features were configured with config sync, and once you use config sync you need to modify those elements in the future with config sync.

To view the running or startup configuration commands that have been entered with config sync:

    show running-configuration switch-profile
    show startup-configuration switch-profile

Importing Existing Commands

If you already have a switch configuration that you want to migrate to a synced switch profile, there is an import process.

For details on importing and additional commands, features, and caveats check the Nexus 5000 NX-OS System Management Configuration Guide - Configuring Switch Profiles

Final Thoughts

The configuration sync is a useful option for setting up a large number of virtual port channels between two Nexus 5000 switches. It can also be used to configure physical interfaces for VPCs as long as identically numbered ports are used on both switches (which I consider a recommended practice anyway). Some may prefer to continue to copy-and-paste manually given the extra number of steps involved in the config-sync.

Update - Using CLI Alias to remember switch-profile

I find that remembering the switch-profile name is the item that trips me up the most when trying to use config sync. An easy way around this of course is to define an alias command for the switch profile:

    cli alias name swpr switch-profile PROFILE_NAME

This way, anytime I want to add config via the config sync feature I just type the following from Enable mode:

    nexus# config sync
    nexus(config-sync)# swpr
    nexus(config-sync-sp)# <enter commands>
    nexus(config-sync-sp)# commit



Quicktip: Nexus 7000 Minimum Recommended Releases


Cisco goes 40 and 100 Gigabit Ethernet

Last week Cisco announced new 40 and 100 Gigabit Ethernet modules for the Catalyst 6500 and Nexus 7000. I’m guessing that next up will be Nexus 5000s with 40GE uplink ports[1].

Nexus 7000

The Nexus 7000 now has a 6 port 40G M2 module (datasheet) and a 2 port 100G M2 module (datasheet) - both non-blocking, and both capable of layer 3 in addition to layer 2. Both modules are supported in all Cisco Nexus 7000 Series chassis and compatible with Fabric–1 or Fabric–2 fabric modules.

The only available optics for the 100G module are LR4 modules that run 10km over G.652 single-mode-fiber.

The 40G module can reach up to 150m (30m or 100m seem more likely unless the fiber runs are carefully engineered) over 50 micron multi-mode-fiber with available SR4 optics.

There are some interesting “Quad Small Form-Factor Pluggable Plus” (QSFP+) modules available for 40G, including 1, 3, and 5 meter copper cables (similar to the 10GE twinax cables), and a 40G QSFP+ to Four 10G SFP+ breakout cable. Other than the SR4 optics these other QSFP+ modules currently only seem to be compatible with the Nexus 3064 switch that has four fixed QSFP+ uplink ports.

Catalyst 6500

The 6500 is getting a 4 port (2:1 oversubscribed[2]) 40GE module (datasheet). Supported 40GE optics are an SR4 that works at 100m over OM3 multi-mode fiber, and an LR4 that will reach 10km over single-mode fiber. A 40GE port slot can also accept a FourX adapter that will then provide four 10GE SFP+ ports in a slot. A single module can have a mixture of 40GE and 10GE ports.

The 6500 40GE module (unsurprisingly) is only compatible with the new Supervisor 2Ts.

  1. I haven’t heard anything yet though.  ↩

  2. Two ports can be set into “performance” mode, disabling the other two ports, for a non-blocking configuration.  ↩