(This blog posting was written with the understanding that the reader already understands how standard LISP operates from a control and data plane perspective.)
Overview
For those who have looked into LISP Extended Subnet Mode (ESM), we have all come to the realization that it is a great leap in technology but not quite ready for prime time. Well, not anymore: in fact, Cisco will be releasing LISP Multihop ESM during the next couple of releases of NX-OS code for the Nexus 7k product line. The difference between LISP ESM and LISP Multihop ESM is the ability for the xTR to be located a routed hop or multiple routed hops away from the ESM subnet’s default gateway. It is basically adding a level of indirection into the LISP ESM process where the xTR relies on the ESM subnet’s default gateway to propagate what EID is connected to its local segment. This allows for non-LISP aware devices to be utilized in a LISP ESM environment; security devices like firewalls being a big player here. RAD is a good example of LISP ESM and for those who have given a RAD demo or have participated in a RAD discussion, one of the questions that typically comes up is “where do firewalls fit into this architecture?” LISP Multihop ESM will allow for firewalls to be placed in between ESM subnet default gateways (I will refer to them as EID-notify eTR for the remainder of this post) and the Site xTR.
Control Plane
The LISP Multihop ESM control plane is a bit different from regular ESM in that the xTR is a routed hop or more away. There is an additional component that notifies the xTR when a /32 EID has moved from one site to another which is the EID-notify eTR (First Hop Router – FHR). The way the Site xTR (iTR/eTR for the site) operates is almost the same as the xTR in regular ESM; the difference is that the Site xTR no longer has the ESM subnet locally attached and relies on the EID-notify eTR router to update Site xTR for LISP convergence. For the rest of this blog post, I will be referring to the following diagram (which is the RAD setup):
Let’s now walk through the EID registration process.
Dynamic EID detection on the EID-notify eTR (FHR) in the East Datacenter
Multicast-map-notify sent to the EID-notify eTRs in both Data Centers
Note: This is sent over the OTV overlay. This also lets the West EID-notify eTR know to remove the host entry from the dynamic-EID table if one was present.
EID-notify message sent local Site xTR
Map-Register message sent to the Map Server (MS) from the Site xTR in the East Datacenter
Remote xTRs can now locate the dynamic EID
Note: There are some control plane steps here but it is a standard LISP control plane and thus outside the scope of this blog.
Now let’s take a look at the control plane after a dynamic EID has relocated to the other Datacenter.
Dynamic EID Move detected in West Datacenter by EID-notify eTR.
Note: This is sent over the OTV overlay. This also lets the losing EID-notify eTR know to remove the host entry from the dynamic-EID table.
EID-notify message sent local Site xTR
Map-Register message sent to the MS from the Site xTR in the West Datacenter
Map-Notify message sent from the MS to the East Datacenter Site xTR
Note: This portion of the control plane lets the “losing” Site xTR know to clear its dynamic-EID entry and install a NULL route in the routing table. This is a critically important step in the Multihop ESM control plane.
Remote xTRs will send continue to forward to the East Datacenter Site xTR which will trigger a Solicited Map-Request (SMR).
Remote xTRs can now locate the dynamic EID
Note: The SMR will cause the Remote xTR to clear the LISP Map-cache entry and go back and ask the MS for the location of the EID. This is also a standard LISP control plane.
As you can see from the preceding diagrams, LISP ESM Multihop adds a level of indirection/LISP control plane to the standard LISP ESM setup. This is needed in order for Firewalls or other devices that need to view the data stream non-encapsulated. This all may seem very straight forward and may appear to be a simple process, but we also need to examine how packet flow happens between Datacenters and non-LISP subnets in and between data centers. Another critical design consideration to examine is how the subnet routes from the EID-notify eTR (FHR) are propagated through the local Datacenter to the Site xTR.
The Site xTR must send the de-encapsulated packets a hop or multiple routed hops away. The Site xTR must have a local route to the ESM subnet in order to forward the packet; otherwise, it will hit the bit bucket. The answer is that we need to either inject a route for the entire ESM subnet(s) into the local IGP routing process in each Datacenter -or- place a static route at each Site xTR telling it where to forward the de-encapsulated packet. The Static route only works if you have a single ESM subnet and therefore will not be the answer in our setup. Another item to note is the NULL routes that the EID-Notify eTR insert into their routing tables along with the NULL route which the Site xTRs insert after a dynamic EID move to a different location.
Along with route propagation inside each local Datacenter, the Site xTRs can attract dynamic EID /32 routes for hosts that have moved to a different location. This is accomplished by redistributing LISP into the local datacenter IGP process. One thing to remember is that a LISP xTR in ESM will only have NULL routes in the routing table learned via the LISP process for dynamic EIDs that have moved to a different location; therefore the routes that we redistribute from LISP into the local datacenter IGP process are only the NULL routes.
This allows the Site xTR routers to pull the traffic to them from inside the data center for dynamic EIDs that have moved away. Another alternative to attracting /32 routes up to the Site xTRs is that you can establish a separate IGP routing process between the EID Notify (FHR) routers for the ESM subnets.
You then would redistribute the LISP into that IGP. This would allow the subnets behind a firewall or other stateful devices to route to each other. If this isn’t in place, traffic will get bit bucketed but the NULL route for the dynamic EID subnets.
The last thing to note is that the /32 propagated within the local Datacenter IGP process should not be propagated out to the WAN. This might not make sense now but it should become clearer after we walk through the next couple of traffic flow scenarios. We have seen some scenarios where propagating /32 makes sense in smaller environments as an alternative to host route injection (HRI).
The following is traffic flow between a non-LISP enable subnet in Datacenter East to a LISP Multihop ESM subnet stretched between datacenters. The first scenario is a LISP ESM subnet talking to a non-LISP subnet located in the same datacenter. This is a straightforward process and just local routes or IGP would be used for forwarding traffic between one subnet to another.
Now we are going to examine what happens when the dynamic EID moves to the other Datacenter. The traffic pattern after the move will look a bit different than you might think. The following is the control plane convergence (which is the same as the example above) when the dynamic EID moved from Datacenter East to Datacenter West.
The dynamic EID moves from Datacenter West to East and communicates on the LAN.
Multicast-map-notify sent to the EID-notify eTRs in both Data Centers
EID-notify message sent local Site xTR
Map-Register message sent to the MS from the Site xTR in the East Datacenter
Note: The West Site xTR will insert a NULL route from the LISP process into the routing table.
Note: This allows traffic to be attracted to the Site xTR for the dynamic EID that moved to the other datacenter. The IGP route will have a better administrative distance in the EID-notify eTR for the /32 EID than the NULL route added by the Multicast-map-notify sent by the EID-notify eTR in step 2.
(LISP convergence steps skipped; you can review them in the previous example)
Traffic can now flow from a non-LISP network(s) in Datacenter West to a LISP ESM network with the dynamic EID in the East Datacenter.
Another alternative to this would be to establish a separate IGP process between the EID Notify (FHR) routers and redistribute LISP dynamic EIDs into that IGP process. Here is how the traffic pattern would look like for that setup:
The physical path that the traffic would take looks like the following:
As with any LISP design, there are many ways to accomplish an architectural design because of the flexibility that the LISP protocol offers.
