Beginner’s Guide to HPE 5000 Series Switches

I don’t closely track the popularity of my blog. If what I share helps people in their day to day job, it’s already good enough to me. But I do look at site statistics now and then just out of curiosity and it seems that network-related posts get a lot of popularity. A blog post I wrote a while ago on Dell N4000 switches has quickly got in top five over the last year.

So it seems that there is a demand for entry-level switch configuration guides. I’ve worked with a quite a few different switch brands over the years, so I thought I will build on the success of the Dell blog post and this time write about HPE FlexNetwork/FlexFabric 5000 switch series.

Operating Systems

HPE has several network switch product lines. I won’t even try to cover all of them in this post. But it’s important to know that there are a few different operating systems you can encounter, while working with HPE network switches. There is a familiar ProCurve product portfolio (now merged with Aruba), which is based on ProVision operating system.

HPE FlexNetwork/FlexFabric 5000 series, on the other hand, is based on Comware operating system. It has a different CLI command set and can be a complete surprise if you’ve worked only with ProCurve switches before. So this blog post will be particularly valuable for those who’re dealing with HPE 5000 for the first time.

The following guide has been tested on a pair of HPE FlexFabric 5700-series switches. Even though commands are mostly the same, on other switch series, like FlexNetwork 5800, there might be some minor differences.

Initial Configuration

When the switch is booted for the first time it will start automatic configuration by trying to obtain settings over DHCP, which you can interrupt by Ctrl+C to get straight to CLI.

You start in user view where you can run display commands to review switch settings. To start the configuration, change to system view:

> system-view

Let’s start by configuring remote access to the switch. There are two ways you can do that. You either use the out-of-band management port:

> interface M-GigabitEthernet 0/0/0
> ip address 10.10.10.10 255.255.255.0
> ip route-static 0.0.0.0 0.0.0.0 10.10.10.1

Or you can configure a VLAN interface IP address:

> interface vlan-interface 1
> ip address 10.10.10.10 255.255.255.0
> ip route-static 0.0.0.0 0.0.0.0 10.10.10.1

Then configure switch name, enable SSH, set passwords and you can start managing the switch over SSH:

> sysname switchname

> public-key local create rsa
> ssh server enable
> user-interface vty 0 15
> authentication-mode scheme
> protocol inbound ssh

> super password simple yourpassword
> local-user admin
> password simple yourpassword
> authorization-attribute user-role level-0
> service-type ssh

User “admin” will have an unprivileged role. You will need to run the following command and enter password once logged in, to elevate to network admin rights:

> super

Intelligent Resilient Framework

In small non-business-critical environments one standalone switch is usually sufficient. In larger environments switches are typically deployed in pairs for redundancy. To simplify management and to avoid network loops most switches support some sort of MLAG or stacking. IRF is HPE’s version of it.

Determine what ports you’re going to use for IRF. There are two QSFP+ ports on 5700-series dedicated for it. And then on on the first switch (master) run the following commands (it’s recommended to shut down the ports before you set them up as IRF):

> irf member 1 priority 32
> int range FortyGigE 1/0/41 to FortyGigE 1/0/42
> shutdown
> irf-port 1/1
> port group interface FortyGigE 1/0/41
> irf-port 1/2
> port group interface FortyGigE 1/0/42
> int range FortyGigE 1/0/41 to FortyGigE 1/0/42
> undo shut
> save
> irf-port-configuration active

On the second switch (slave) run the following commands to change the IRF ID to 2:

> irf member 1 renumber 2
> reboot

When the switch comes up, configure IRF ports:

> irf member 2 priority 30
> int range FortyGigE 2/0/41 to FortyGigE 2/0/42
> shutdown
> irf-port 2/1
> port group interface FortyGigE 2/0/41
> irf-port 2/2
> port group interface FortyGigE 2/0/42
> int range FortyGigE 2/0/41 to FortyGigE 2/0/42
> undo shut
> save
> irf-port-configuration active

Now you can connect the physical IRF ports. IRF is a ring topology, that means (in my case) port 1/0/41 should connect to 2/0/42 and port 1/0/42 should connect to 2/0/41.

Second switch will automatically reboot and if all is configured correctly, you should see both switches join the IRF fabric. Member switch 1 has the highest priority of 32 and becomes the master:

> display irf

Firmware Upgrade

Firmware upgrade is the next logical step after you set up IRF. The latest firmware revision for the switches can be download from HPE web-site. Keep in mind you will need a HPE passport account, with a valid service agreement (SAID) added to it.

You will also need a TFTP server to upgrade the firmware. There are a few of them out there, but the most commonly used is probably Tftpd64.

When you get the TFTP server up and running and copy the firmware file to it, perform an upgrade:

> tftp 10.10.10.20 get 5700-CMW710-R2432P03.ipe
> boot-loader file flash:/5700-CMW710-R2432P03.ipe slot 1 main
> boot-loader file flash:/5700-CMW710-R2432P03.ipe slot 2 main
> irf auto-update enable
> reboot

Confirm firmware has been updated:

> display version

VLANs, Aggregation Groups and Tagging

In Comware the term “aggregation group” is used to describe what is a “port channel” in Cisco world. Trunk/access ports are also called tagged/untagged ports throughout the documentation.

In this section we will discuss a few common port configuration scenarios:

• Untagged ports, which can be your iSCSI storage array ports
• Tagged ports, such as your VMware host uplinks
• Aggregation groups, typically used for LAGs to upstream switches

First of all create all VLANs and give them descriptions:

> vlan 10
> description iSCSI
> vlan 20
> description Server
> vlan 30
> description Dev and test

Then specify untagged ports:

> vlan 10
> port te 1/0/1
> port te 2/0/1

To configure tagged ports and allow certain VLANs (ports will be added to the VLANs automatically):

> int te 1/0/2
> description ESX01 vmnic0
> port link-type trunk
> port trunk permit vlan 20 30
> int te 2/0/2
> description ESX02 vmnic0
> port link-type trunk
> port trunk permit vlan 20 30

And to create an LACP aggregation group:

> interface bridge-aggregation 1
> description Trunk to upstream switch
> link-aggregation mode dynamic
> port link-type trunk
> port trunk permit vlan 20 30

> interface te 1/0/3
> port link-aggregation group 1
> interface te 2/0/3
> port link-aggregation group 1

Common Commands

Other useful commands that don’t fall under any specific category, but handy to know.

Display switch configuration:

> display current-configuration

Save switch configuration:

> save

Shut down a port:

> int te 1/0/27
> shutdown

Undo a command:

> undo shutdown

Conclusion

Whether you are a network engineer new to the Comware operating system or a VMware administrator looking for a quick cheat sheet for FlexNetwork/FlexFabric switches, I hope this guide has helped you get the job done.

If this blog post gets the same amount of popularity, maybe it will turn into another series. But for now – over and out.

Advertisement

Merging Brocade Fabrics

Recently I needed to merge two pairs of Brocade fibre channel fabrics for one of the customers. When I was doing a bit of my own research I realised that there is very scarce information on how to do that on the Interwebs. There were a few community posts on the Brocade forums, but there seemed to be some confusion around how zoning should be configured to let the switches merge successfully. I thought I would fill the gap with this post and share my own experience.

Prerequisites

First, make sure you have the right transceivers. Short wave 8Gb FC transceivers are limited to 190m when using OM4 fibre. If you need to connect switches over a longer distance, use long wave SFP+ modules, which have maximum distance of 10km.

Second, change the default switch Domain IDs. All switches within the same fabric must have unique IDs. By default Brocade switches come with the Domain ID set to 1. If you’re merging two redundant fabrics, make sure that the second pair of the switches have Domain IDs set to 2.

Third, verify that the switches you’re interconnecting have compatible zoning configuration. Brocade is very specific on how zoning should be configured for two fabrics to merge. There are at least nine different scenarios, but we’ll touch only on three most common ones. If you want to get more details, refer to the Brocade Fabric OS Administrator’s Guide and specifically the section called “Zone merging scenarios”.

Zone merging scenarios

Scenario 1: Switch A does not have a defined configuration. Switch B has a defined configuration.

This is the most straightforward scenario when you are adding a brand new Switch A to an existing fabric. As a result of the merge configuration from the Switch B propagates to the switch A.

Scenario 2: Switch A and Switch B have different defined configurations. Switch B has an effective configuration.

This is the scenario where you have two individual fabrics with their own set of aliases, zones and defined configurations. There is a catch here. If you want to merge such fabrics, you MUST have unique set of aliases, zones and configurations on each fabric. If this requirement is not met, fabrics won’t merge and you will end up with two segmented fabrics because of the zoning conflict. You also MUST disable effective zoning configuration on Switch A.

Outage is not required, because typically you have two redundant fabrics – fabric A and B in each location. And you can do one switch at a time. If you are still concerned, implement Scenario 3.

Scenario 3: Switch A and Switch B have the same defined and effective configuration.

This is the easiest path and is what Brocade calls a “clean merge”. Under this scenario you will have to recreate the same configs on both fabrics. That means you MUST have completely identical aliases, zones and configs on Switch A and Switch B.

This is the easiest and least disruptive path if you are worried that disabling effective configuration on the switches may cause issues.

Real world scenario

In my case I went with scenario 2 for two reasons: one – it was a DR site where I could temporarily bring down both fabrics and two – I didn’t need to manually add aliases/zones/configs to the switches as I would have to in scenario 3. Once fabrics are merged, zones from Switch B propagate to Switch A and you can simply combine them in one zone in the GUI, which is just a few mouse clicks.

Here is the step by step process. First step is to change Domain IDs on the second pair of switches. You can do that both from GUI and CLI. Bear in mind that even if you’ve picked scenario 3 as the least disruptive approach for merging zones, changing Domain IDs will still be disruptive. Because switch has to be disabled before making the change.

From the Web Tools go to Switch Administration, disable the switch in the Switch Status section, type in the new Domain ID and re-enable the switch:

If you want to take the CLI path, run the following. Switch will ask you a series of questions. You can accept all defaults, except for the Domain field:

> switchdisable
> configure
> switchenable
> fabricshow

Next disable the effective configuration on the Switch A either from GUI or CLI:

> cfgdisable
> cfgactvshow

At this point you can interconnect the switches and you should see the following log entry on Switch A:

The effective configuration has changed to SWITCHB_CONFIG

The fabrics are now merged an you should see both switches under the Web Tools. If you see the switch in the Segmented Switches section, it means that something went wrong:

Clean up steps

Once the fabrics are merged you will see all zones in the Zone Admin interface, however, the effective configuration will be configuration from the Switch B. You will need to create a new configuration which combines all zones to enable connectivity between the devices connected to the Switch A.

From the operational perspective you can now manage zoning on either of the switches and when you save or enable a configuration it will propagate to all switches in the fabric automatically.

If you have redundant fabrics, which you normally do, repeat the steps for the second pair of switches.

Conclusion

Steps described in this post are for a basic switch setup. If you have a non-standard switch configuration or using some of the advanced features, make sure to check “Zone Merging” section in the Fabric OS Administrator’s Guide for any additional considerations.

Let me know if this was helpful.

 

Force10 MXL Switch: Port Numbering

This is a quick cheat sheet fro MXL port numbering schema, which might seem a bit confusing if you see a MXL switch for the first time.

Above is the picture of the switches that I’ve worked with. On the right we have a 2-Port 40GbE built-in module. And then there’re two expansion slots – slot 0 in the middle and slot 1 on the left. Each module has 8 ports allocated to it. The reason being that you can have 2-Port 40-GbE QSFP+ modules in each of the slots, which can operate in 8x10GbE mode. You will need QSFP+ to 4xSFP+ breakout cables, but that’s not the most common scenario anyway.

As we have 8 ports per slot, it would look something like this:

This picture is more for switch stacking, but the rightmost section should give you a basic idea. One of the typical MXL configurations is when you have a built-in 40GbE module for stacking and one or two 4-Port SFP+ expansion modules in slots 0 and 1. In that case your port numbers will be: 33 and 37 for 40GbE ports, 41 to 44 in expansion slot 0 and 49 to 52 in expansion slot 1.

As you can see for QSFP+ module switch breaks 8 ports in two sets of 4 ports and picks the first number in each set for 40GbE ports. And for SFP+ modules it uses consecutive numbers within each slot and then has a 4 port gap.

Port numbering is described in more detail in MXL’s switch configuration guide, which you can use for your reference. But this short note might help someone to quickly knock that off instead of browsing through a 1000 page document.

Also, I’ve seen pictures of MXL switches with a slightly different port numbering: 41 to 48 in slot 0 and 33 to 40 in slot 1. Which seems like a mirrored version of the switch with a built-in module on the opposite side of it. I’m not sure if it’s just an older version of the same switch, but keep in mind that you might actually have the other variation of the MXL in your blade chassis.