In my role as a network architect, I often find myself validating a design based on documentation provided by the vendor. Whenever possible, I prefer to verify at least the core functionality in the lab.

Platforms such as Cisco Meraki on the one hand simplify configurations by introducing automation and abstraction mechanisms that have a great impact on large installations; on the other hand, abstractions and simplifications require careful analysis of requirements and deployment methods.

In this blog I present a scenario that I had the opportunity to test recently.

Table of Contents

Scenario

The current network includes many sites with Meraki MX firewalls .

The customer request is to route all the traffic from some of the VLANs of the remote sites to a central site. There is a 3rd party firewall in the central site, so we cannot take advantage of the AutoVPN feature, which is a proprietary technology of Meraki.

Solution

The solution is well documented, we can create an IPSec VPN to a central site and route traffic to the destination through the tunnel.

However, this solution raised some questions: how will the MX behave when the VPN tunnel announces remote network 0.0.0.0/0 but some local LANs do not have to go through the tunnel? And what happens to the traffic if the IPSec tunnel drops?

Let’s test in the lab!

The lab

The lab topology is simple: it includes an Internet-connected Meraki MX65, an IPSec-compatible remote firewall and a couple of laptops.

On the LAN side we created two VLANS:

  • VLAN 1 = LAN, 10.178.1.0/25, all the traffic of this network will be tunnelled to the remote site
  • VLAN 2 = GUEST, 10.178.1.128/25, the Internet access of this network goes through the local Internet access connected to WAN 1

Notice that VPN mode in enabled for the LAN network 10.178.1.0/25 only.

The next step is to create a site-to-site IPSec VPN connection to the remote firewall. Configurint it is trivial. The networks advertised in the IPSec are…

  • LOCAL network (Meraki side): 10.178.1.0/25
  • REMOTE network: 0.0.0.0/0, meaning all the destinations are reachable via tunnel

In the remote site LOCAL and REMOTE networks will be reversed.

Check the VPN status to confirm it works:

The results

We begin testing the LAN network, we expect that all traffic will be tunnelled to the remote site. We can verify this by performing a traceroute, a ping, and opening ifconfig.me page in a browser to confirm access to the Intenet via the public IP of the central site.

All the LAN tests were ok ✅

Then we test the GUEST. Remember we didn’t enabled VPN mode for the guest network, so we expect the traffic to be routed to the local internet access.

We repeat the same test, and that’s it! The GUEST network reaches the Internet via the WAN 1 interface of the Meraki MX. Opening ifconfig.me we get the local public IP.

All the GUEST tests were ok ✅

Failure scenarios

By anticipating potential failures, a network architect can design systems to withstand and recover from these events, ensuring that the network remains operational even during adverse conditions.

What happens if the VPN goes down?

We change the PSK to break the VPN:

Result:

  • the PC in LAN 1 cannot reach any outside network ❌
  • the PC in LAN 2 can reach any outside networks ✅

Since the LAN 1 network is in VPN Mode with all networks in the destination (0.0.0.0/0), the traffic is dropped.

Two VPNs?

To reduce the risk of a downtime, a possible solution would be to configure redundant VPNs to at least two remote sites.

Let’s try:

Now the REMOTE-SITE-1 VPN is down, but REMOTE-SITE-2 VPN is UP.

We can now access the Internet on the laptop connected to LAN 1, with VPN mode enabled.

I cannot test in my lab what happens when REMOTE-SITE-1 goes UP again.

According to the documentation , if both VPNs are UP:

when two subnets directly conflict, IP routing behavior will be undefined

Backup tunnel configured on the remote site is supported:

If the MX in question has an established VPN tunnel with a non-Meraki peer, the non-Meraki device will need to have the ability to designate a backup (failover) peer IP. By designating the public IP address of the MX’s secondary uplink as the back-up VPN IP on the non-Meraki VPN peer, you can ensure that the VPN tunnel will be re-established in the event of an uplink failure. Source

For redundancy the documentation shows some options to make it work across two datacenters.

I strongly suggest to test this failover scenarios to avoid issues and confirm the backup mechanism fits your design.

Final notes

When I work on a design, I make it a point to validate my work whenever feasible. Testing the entire scenario can be a challenge at times, requiring a network architect to adeptly analyze the entire landscape. The key is to extract the most crucial features and create validation tests that significantly boost the likelihood of project success. It’s about ensuring the design stands up to scrutiny and sets the stage for a successful outcome.

Sometimes the only option is to lean heavily on documentation, experience, and intuition. These elements are essentially a product of the lessons learned from both mistakes and successes in the past. They serve as our toolkit, forged through the trials and triumphs that pave the way for sound decision-making in the present.

There's always a certain level of risk lurking around – be it bugs, integrations, or the ever-elusive unknowns. What matters most is shouldering responsibility rather than leaving things to chance. Being consistently present alongside the delivery team is key, providing support and openly sharing uncertainties. This philosophy aligns with Jocko Willink’s perspective in his book Extreme Ownership , emphasizing the proactive stance of taking charge and owning up to challenges.

A technical note about my home network

For those interested in the details, I run the lab in my home network, placing the Meraki MX behing an Ubiquity USG . I’ve enabled port forwarding for UDP ports 500 and 4500, and added a firewall rule to permint inbound traffic for the public IP of the firewall in the remote site.

To see it there is some PSec traffic we can run tcpdump on the USG:

This command for the LAN interface (towards the Meraki MX):

sudo tcpdump -npi eth1 port 500 or port 4500 and host X.X.X.X

This command for the WAN interface (towards the Internet):

sudo tcpdump -npi pppoe0 port 500 or port 4500 and host X.X.X.X

UPDATES

Feb 22, 2024: this blog post was featured in the Human Infrastructure 339 newsletter.