Introduction
As of 2.2.2 code, Transit routing is now allowed between L3 Outs in different VRFs. In other words, Inter-VRF Transit Routing / Route leaking between L3 outs/VRFs
Requirements
- ACI 2.2.2 (Danube MR1)
- Two VRFs
- One L3 out in each VRF
Configure
Network Diagram

Configurations
- Create two different L3 outs in two different VRFs (different tenant or same tenant, doesn’t matter)
- Ensure both L3 outs, individually, are working. Neighborships are up and routes are coming in and out.
- Now the fun part:
Traditionally, with transit routing, Subnets on the ingress L3 out are marked with “external subnet for external EPG” so that the source VRF and InstP can apply policy. The remote or egress L3 out will need an entry for the same subnet but marked as “export route control subnet”. That flag is the essential “transit routing” flag. The inverse must be done as well.
In our example topology above, dpita-tenant:dpita-context is learning 123.123.254.0/24 from the outside. This subnet is marked as “external subnet for external epg”. In the egress VRF dpita-tenant:2600, this subnet is marked as “export route control subnet” and the transit routing config is complete.
WIth Inter-VRF transit routing, 123.123.254.0/24 will be marked with:
- External subnet for external EPG = standard for subnets being learned from the outside. applies policy for anything in the same VRF
- Shared route control subnet = enables the subnet to be route-leaked
- Shared security import subnet = applies the correct pcTag of the InstP
In the remote VRF, nothing changes, 123.123.254.0/24 is marked for:
- Export route control subnet.
The config is now repeated for 55.55.254.0/24 in its source VRF dpita-tenant:2600:
- External subnet for external EPG
- Shared route control subnet
- Shared security import subnet
In the remote VRF, 55.55.254.0/24 is marked for “export route control subnet”
the InstPs should look something like this:


With any route-leaking configuration, the final piece of the puzzle will be the contract.
- Create a global scoped contract
- Have one InstP provide
- The other InstP consumes
Verify
rtp-f1-p1-leaf1# show ip route vrf dpita-tenant:dpita-context
IP Route Table for VRF "dpita-tenant:dpita-context"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
10.10.4.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.208.66%overlay-1, [1/0], 1d19h, static, tag 4294967295
10.10.4.1/32, ubest/mbest: 1/0, attached, pervasive
*via 10.10.4.1, vlan12, [1/0], 1d19h, local, local
10.10.13.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.208.66%overlay-1, [1/0], 1d19h, static, tag 4294967295
10.10.13.1/32, ubest/mbest: 1/0, attached, pervasive
*via 10.10.13.1, vlan18, [1/0], 1d19h, local, local
11.11.11.11/32, ubest/mbest: 2/0, attached, direct
*via 11.11.11.11, lo5, [1/0], 1d01h, local, local
*via 11.11.11.11, lo5, [1/0], 1d01h, direct
55.55.254.0/24, ubest/mbest: 1/0
*via 20.0.216.93%overlay-1, [200/401], 19:20:55, bgp-1, internal, tag 1
123.123.254.0/24, ubest/mbest: 1/0
*via 192.168.44.254, vlan22, [110/44], 21:21:49, ospf-default, intra
192.168.4.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.208.66%overlay-1, [1/0], 1d19h, static
192.168.4.1/32, ubest/mbest: 1/0, attached, pervasive
*via 192.168.4.1, vlan49, [1/0], 1d19h, local, local
192.168.13.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.208.66%overlay-1, [1/0], 1d19h, static, tag 4294967295
192.168.13.1/32, ubest/mbest: 1/0, attached, pervasive
*via 192.168.13.1, vlan9, [1/0], 1d19h, local, local
192.168.44.0/24, ubest/mbest: 1/0, attached, direct
*via 192.168.44.253, vlan22, [1/0], 1d01h, direct
192.168.44.253/32, ubest/mbest: 1/0, attached
*via 192.168.44.253, vlan22, [1/0], 1d01h, local, local
192.168.130.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.208.66%overlay-1, [1/0], 1d19h, static, tag 4294967295
192.168.130.1/32, ubest/mbest: 1/0, attached, pervasive
*via 192.168.130.1, vlan16, [1/0], 1d19h, local, local
192.168.131.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.208.66%overlay-1, [1/0], 1d19h, static, tag 4294967295
rtp-f1-p1-leaf1#
aci-n3k-1-bootcamp# show ip route vrf dpita-tenant:dpita-context
IP Route Table for VRF "dpita-tenant:dpita-context"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
11.11.11.11/32, ubest/mbest: 1/0
*via 192.168.44.253, Vlan2356, [110/41], 21:22:50, ospf-dpita-tenant, intra
55.55.254.0/24, ubest/mbest: 1/0
*via 192.168.44.253, Vlan2356, [110/1], 19:21:56, ospf-dpita-tenant, type-2,
tag 4294967295
123.123.254.0/24, ubest/mbest: 1/0, attached
*via 123.123.254.1, Vlan2357, [0/0], 1d00h, direct
123.123.254.1/32, ubest/mbest: 1/0, attached
*via 123.123.254.1, Vlan2357, [0/0], 1d00h, local
192.168.44.0/24, ubest/mbest: 1/0, attached
*via 192.168.44.254, Vlan2356, [0/0], 1d00h, direct
192.168.44.254/32, ubest/mbest: 1/0, attached
*via 192.168.44.254, Vlan2356, [0/0], 1d00h, local
aci-n3k-1-bootcamp#
rtp-f1-p2-leaf1# show ip route vrf dpita-tenant:2600
IP Route Table for VRF "dpita-tenant:2600"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
4.4.4.1/32, ubest/mbest: 1/0
*via 18.18.18.1, eth1/95, [110/401], 21:24:19, ospf-default, intra
18.18.18.0/30, ubest/mbest: 1/0, attached, direct
*via 18.18.18.2, eth1/95, [1/0], 1d00h, direct
18.18.18.2/32, ubest/mbest: 1/0, attached
*via 18.18.18.2, eth1/95, [1/0], 1d00h, local, local
19.19.19.0/30, ubest/mbest: 1/0
*via 18.18.18.1, eth1/95, [110/401], 21:24:19, ospf-default, intra
26.26.26.26/32, ubest/mbest: 2/0, attached, direct
*via 26.26.26.26, lo5, [1/0], 1d00h, local, local
*via 26.26.26.26, lo5, [1/0], 1d00h, direct
55.55.254.0/24, ubest/mbest: 1/0
*via 18.18.18.1, eth1/95, [110/401], 21:24:19, ospf-default, intra
123.123.254.0/24, ubest/mbest: 1/0
*via 10.0.32.95%overlay-1, [200/44], 19:20:32, bgp-1, internal, tag 1
rtp-f1-p2-leaf1#
2600#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
19.0.0.0/30 is subnetted, 1 subnets
C 19.19.19.0 is directly connected, FastEthernet0/1
18.0.0.0/30 is subnetted, 1 subnets
C 18.18.18.0 is directly connected, FastEthernet0/0
4.0.0.0/24 is subnetted, 1 subnets
C 4.4.4.0 is directly connected, Loopback4
55.0.0.0/24 is subnetted, 1 subnets
C 55.55.254.0 is directly connected, Loopback55
26.0.0.0/32 is subnetted, 1 subnets
O 26.26.26.26 [110/2] via 18.18.18.2, 21:26:19, FastEthernet0/0
123.0.0.0/24 is subnetted, 1 subnets
O E2 123.123.254.0 [110/1] via 18.18.18.2, 19:22:33, FastEthernet0/0
44.0.0.0/24 is subnetted, 1 subnets
C 44.44.44.0 is directly connected, Loopback0
2600#
GBL_C++: [INFO] sclass: 0011
module-1# show system internal aclqos prefix
3014660 0.0.0.0 ffffffff 37 15 FALSE FALSE
3014660 55.55.254.0 ff 37 17 FALSE FALSE
Shared Addr Mask Scope Class RefCnt
============== ======== ===== ====== ======
123.123.254.0 ff 0 10935 1
rtp-f1-p2-leaf1# show zoning-rule src-epg 17
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4442 17 10935 5 enabled 3014660 permit fully_qual(6)
rtp-f1-p2-leaf1#
Troubleshoot
Packet Flow
source – 55.55.254.1 (leaf 121)
dest – 123.123.254.1 (leaf 111)

High Level
- ping ingress on leaf 121
- elam trigger on NS
- elam RW goes to bcm for routing lookup
- elam trigger on NS with dst MAC 0c
- check encap ptr and ip for dest leaf.
<1 and 2>
GBL_C++: [MSG] - l2vec0 is complete
GBL_C++: [INFO] ce_da: 0022BDF819FF
GBL_C++: [INFO] ce_sa: 000C85B86860
GBL_C++: [MSG] - l3vec0 is complete
GBL_C++: [INFO] ip_da: 0000000000000000000000007B7BFE01
GBL_C++: [INFO] ip_sa: 0000000000000000000000003737FE01
GBL_C++: [MSG] - pktrw is complete
GBL_C++: [INFO] loopback: 1
GBL_C++: [INFO] uc_routed: 1
GBL_C++: [INFO] ol_encap_idx: 0002
GBL_C++: [INFO] ol_segid: 2E0004
GBL_C++: [INFO] ol_mark: 1
GBL_C++: [INFO] ol_lb: 1
GBL_C++: [INFO] ol_dl: 1
GBL_C++: [INFO] ol_e: 0
GBL_C++: [INFO] ol_sp: 1
GBL_C++: [INFO] ol_dp: 1
GBL_C++: [INFO] sclass: 0011
GBL_C++: [INFO] vpc_df: 1
GBL_C++: [INFO] brcm_proxy: 1
<3 and 4>
rtp-f1-p2-leaf1# bcm-shell-hw "l3 defip show" | grep 123.123
2160 3 123.123.254.0/24 00:00:00:00:00:00 100095 0 0 0 0 n
2161 37 123.123.254.0/24 00:00:00:00:00:00 100097 0 0 0 0 y
rtp-f1-p2-leaf1# bcm-shell-hw "l3 egress show" | grep 100097
Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC
100097 00:0c:0c:0c:0c:0c 4059 5014 17 4 -1 no no 1 no
module-1(NS-elam-insel3)# set outer ipv4 src_ip 55.55.254.1 dst_ip 123.123.254.1
module-1(NS-elam-insel3)# set outer l2 dst_mac 000c.0c0c.0c0c
GBL_C++: [MSG] - l2vec0 is complete
GBL_C++: [INFO] ce_da: 000C0C0C0C0C
GBL_C++: [INFO] ce_sa: 000C85B86860
GBL_C++: [MSG] - l3vec0 is complete
GBL_C++: [INFO] ip_da: 0000000000000000000000007B7BFE01
GBL_C++: [INFO] ip_sa: 0000000000000000000000003737FE01
GBL_C++: [MSG] - pktrw is complete
GBL_C++: [INFO] loopback: 0
GBL_C++: [INFO] uc_routed: 1
GBL_C++: [INFO] adj_vld: 1
GBL_C++: [INFO] adj_index: 0044
GBL_C++: [INFO] ol_encap_idx: 3004
GBL_C++: [INFO] ol_segid: 2E0004
GBL_C++: [INFO] ol_mark: 1
GBL_C++: [INFO] ol_lb: 1
GBL_C++: [INFO] ol_dl: 1
GBL_C++: [INFO] ol_e: 0
GBL_C++: [INFO] ol_sp: 1
GBL_C++: [INFO] ol_dp: 1
GBL_C++: [INFO] brcm_proxy: 0
module-1(NS-elam-insel3)# show platform internal ns forwarding encap 0x3004
======================================================================================================================================================
TABLE INSTANCE : 0
======================================================================================================================================================
Legend
MD: Mode (LUX & RWX) LB: Loopback
LE: Loopback ECMP LB-PT: Loopback Port
ML: MET Last TD: TTL Dec Disable
DV: Dst Valid DT-PT: Dest Port
DT-NP: Dest Port Not-PC ET: Encap Type
OP: Override PIF Pinning HR: Higig DstMod RW
HG-MD: Higig DstMode KV: Keep VNTAG
------------------------------------------------------------------------------------------------------------------------------------------------------
M PORT L L LB MET M T D DT DT E TST O H HG K M E
POS D FTAG B E PT PTR L D V PT NP T IDX P R MD V D T Dst MAC DIP
------------------------------------------------------------------------------------------------------------------------------------------------------
12292 0 800 0 1 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3 00:00:00:00:00:00 10.0.32.95
======================================================================================================================================================
TABLE INSTANCE : 1
======================================================================================================================================================
Legend
MD: Mode (LUX & RWX) LB: Loopback
LE: Loopback ECMP LB-PT: Loopback Port
ML: MET Last TD: TTL Dec Disable
DV: Dst Valid DT-PT: Dest Port
DT-NP: Dest Port Not-PC ET: Encap Type
OP: Override PIF Pinning HR: Higig DstMod RW
HG-MD: Higig DstMode KV: Keep VNTAG
------------------------------------------------------------------------------------------------------------------------------------------------------
M PORT L L LB MET M T D DT DT E TST O H HG K M E
POS D FTAG B E PT PTR L D V PT NP T IDX P R MD V D T Dst MAC DIP
------------------------------------------------------------------------------------------------------------------------------------------------------
12292 0 fff 0 1 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3 00:00:00:00:00:00 10.0.32.95
module-1(NS-elam-insel3)#
rtp-f1-p2-leaf1# acidiag fnvread | grep 10.0.32.95
111 1 rtp-f1-p1-leaf1 SAL1819SAN6 10.0.32.95/32 leaf active 0
rtp-f1-p2-leaf1#