Introduction
This article will show how to setup an F5 cluster and create a virtual server for the sole purpose of explaining a key limitation of the F5. This is manual setup, not through the device package.
*NOTE*: This article is not written with the intention of a tutorial on how to setup an F5, only to get it up and running enough to test failover of a virtual server and test the MAC Masquerade feature and GARPs.
Pre-requisites and assumptions
- Two F5’s in my case virtual
- licensed
- NTP must be setup on the F5 (will prevent successful clustering)
- This example will use AVS VXLAN with LS
In ACI i created a new AP and 3 new BDs with default settings, then 3 EPGs tied to those BDs. The EPGs and BDs are internal, external, and failover. The EPGs are tied to the AVS VMM domain.
F5 has a few accounts
admin account password admin
root account password default
I noticed that root is the only account that can be used to SSH to the F5 devices for CLI access.
Section 1: Individual F5 bringup
At this stage, each F5 must be configured individually so everything must basically be done twice. i will only go through the configuration once, make sure to replicate on the second F5 when it is specified.
Self-IPs
I configured a self IP for internal, external and failover. These are simple interfaces and only require an IP address, VLAN (which really doesn’t matter) and port lockdown set to allow default
Once all six addresses are configured (three on each F5) SSH to the command line and send a few pings to confirm ACI is forwarding appropriately.
Section 2: HA configuration
First step is to configure each device for syncing and failover and specify which interfaces will be used to build the failover network.
Device management > Devices
-click on self device then device connectivity and config-sync to set the local address (self IP of failover network)
then click failover
repeat for second F5
Now under Device management >device trust >peer list >add
enter the IP address and credentials of the other F5
do this part only once since the other F5 will automatically add the current F5 to its peer list
At this point, both F5s should sync bug are active/active. A Device Group is needed so go to
Device Management >Device Group
enter a name
Group type = sync-failover
move all members to include
Network Failover en
Auto sync en
Full sync en
Finish
now they should be active/standby and waiting for initial sync. To complete the initial sync, go to Device Management >Overview and click on the active F5 and then select “sync to group”
Note: I have had 100% success with “sync device to group” and following the advice the device gives me.
Section 3: Virtual Server
Local Traffic > Virtual Server
the bare minimum here is a name
destination = select host and enter an IP address
a pool is then needed so use a health monitor of gateway_icmp and for node use a VM in the server side EPG/Portgroup and use that IP address
must assign the virtual server to traffig-group-1 (default) since this is the floating traffic group
now the actual testing!
under device management > devices click on the self active device and click “force standby”
Problem
The Virtual Server address is now stuck on the original leaf and did not move to the new active F5/leaf
Virtual Server Failover is unsuccessful, ACI does not update its endpoint table and traffic is blacked holed when a failure event occurs.
This is the limitation on the F5 side. According to the F5 support engineer
** The F5 will not send a GARP for any floating IP that is in a different subnet than the self-ip addresses**
Solution
MAC Masquerade on the traffic group!
F5 is quite finicky with how it reads 0’s in a MAC address. Sometimes they are ommited other times they are truncated as shown above. The proper MAC i input was 0000:1313:0001
This feature allows a custom MAC to be assigned to all virtual server IP addresses using the traffic group.
With this feature configured, a GARP will be sent for the virtual server address, ACI inserts a bounce entry and all is right with the world. Traffic loss is minimal (testing shows 1-2 ping drops).
rtp-leaf4# tcpdump -i kpm_inb arp
11:27:34.946251 ARP, Reply 5.5.5.1 is-at 00:00:13:13:00:01 (oui Unknown), length 46
11:27:35.445158 ARP, Reply 4.4.4.1 is-at 00:00:13:13:00:01 (oui Unknown), length 46
11:27:35.445161 ARP, Reply 6.6.6.1 is-at 00:00:13:13:00:01 (oui Unknown), length 46
^^^THESE ARE THE GARPS^^^
rtp1-leaf4# vsh_lc -c "show system internal epmc endpoint mac 0000.1313.0001"
MAC : 0000.1313.0001 ::: Num IPs : 3
IP# 0 : 4.4.4.1 ::: IP# 0 flags :
IP# 1 : 5.5.5.1 ::: IP# 1 flags :
IP# 2 : 6.6.6.1 ::: IP# 2 flags :
Vlan id : 22 ::: Vlan vnid : 9273344 ::: BD vnid : 15826919
Encap vlan : VXLAN/9273344
VRF name : dpita-tenant:dpita-context ::: VRF vnid : 2719746
phy if : 0 ::: tunnel if : 0x1801000f ::: Interface : Tunnel15
Flags : 0x80004c04
Ref count : 7 ::: sclass : 32778
Timestamp : 01/13/1970 06:44:10.709249
last mv timestamp 01/01/1970 00:00:00.000000 ::: ep move count : 0
last loop_detection_ts 01/01/1970 00:00:00.000000
previous if : 0 ::: loop detection count : 0
EP Flags : local,IP,MAC,class-set,timer,
Aging:Timer-type : Host-tracker timeout ::: Timeout-left : 660 ::: Hit-bit : Yes ::: Timer-reset count : 0
PD handles:
Bcm l2 hit-bit : No
[L2]: Asic : NS ::: ADJ : 0x18 ::: LST SA : 0xa37 ::: LST DA : 0xa37 ::: GST ING : 0x30b ::: BCM : No
[L3-0]: Asic : NS ::: ADJ : 0x18 ::: LST SA : 0x86f ::: LST DA : 0x86f ::: GST ING : 0xdc0 ::: BCM : No
[L3-1]: Asic : NS ::: ADJ : 0x18 ::: LST SA : 0x197 ::: LST DA : 0x197 ::: GST ING : 0xb46 ::: BCM : No
[L3-2]: Asic : NS ::: ADJ : 0x18 ::: LST SA : 0xd1c ::: LST DA : 0xd1c ::: GST ING : 0xcc ::: BCM : No
<detail> SDB Data:
mod 4 ::: port 4
is_rmac_idx_valid : YES ::: rmac_idx 0x2
::::
Hope this helps
Dpita's blog of ACI and Networking Fun 