Without a working site-to-site VPN, a business with two offices is effectively two separate islands – one office cannot access the other’s files, printers, or servers over the internet securely. This chapter explains how to build and fix those digital bridges, which is a core skill tested in the NSE7 Advanced Security exam.
Jump to a section
A simple way to picture Site-to-Site VPNs and IPSec Troubleshooting
A busy embassy in a foreign country. The ambassador needs to send a secret message to the government back home, but the local post office is run by a rival nation who might read or tamper with the letter.
The embassy’s solution is to use a diplomatic pouch. First, the message is written, sealed in a special tamper-proof envelope (encryption – IPsec Phase 2). That envelope is then placed inside a larger, heavy-duty diplomatic bag (the tunnel – IPsec Phase 1). The bag is locked with a unique, shared key that only the embassy and the home government possess (the Pre-Shared Key). The courier (the IPsec process) then delivers the bag through the public postal system (the internet).
Troubleshooting this system is like tracking a lost diplomatic pouch. Did the courier forget the key? (Check Phase 1 – authentication failed). Did the envelope inside the bag get opened by mistake? (Check Phase 2 – encryption mismatch). Did the courier take the wrong route because the map was wrong? (Check routing and firewall policies). Is the bag itself damaged, letting the message leak? (Check interface MTU and fragmentation). The embassy doesn’t fix the problem by guessing – it checks the courier’s logbook, inspects the lock, and tests the envelope. Similarly, a network engineer uses commands like 'diagnose vpn ike log' and 'diag debug app ike' to trace exactly where the tunnel broke, from the initial hello to the final data transfer.
A Site-to-Site VPN (Virtual Private Network) is a secure, encrypted tunnel between two or more networks (like two office buildings) over the public internet. Think of it as a private, armoured tunnel running through a public street. The technology that builds this tunnel is called IPsec (Internet Protocol Security).
IPsec is not a single protocol but a suite of protocols that work together to provide three critical security services: Confidentiality (encrypting data so no one can read it), Integrity (ensuring data has not been modified in transit), and Authentication (verifying that the data truly came from the expected source).
The process of building an IPsec VPN happens in two distinct phases:
Phase 1 – The Tunnel’s Handshake. This is where the two VPN gateways (the FortiGates at each office) say hello and agree on how to protect the tunnel itself. They use a protocol called IKE (Internet Key Exchange). During Phase 1, they negotiate a common encryption algorithm (like AES-256), a hashing algorithm (like SHA-256) for integrity, and a method to authenticate each other (usually a Pre-Shared Key or digital certificates). Once they agree, they establish an IKE SA (Security Association) – a secure control channel. This is like the two embassies first agreeing on a secret handshake and a secure courier route before any actual secret messages are exchanged.
Phase 2 – The Actual Data Tunnel. After the secure control channel is established, Phase 2 begins. This phase negotiates the specific parameters for encrypting the actual user traffic (like email, file shares, or database queries). This creates the IPsec SA – the actual data tunnel. The protocol used here is usually ESP (Encapsulating Security Payload), which encrypts the entire original IP packet. At the end of Phase 2, both FortiGates generate and share a pair of session keys (one for each direction) that will be used for the symmetrical encryption of the data.
Once both phases are successful, the VPN is up, and traffic can flow securely from one site to another. However, the tunnel alone is not enough. The FortiGates also need a firewall policy that explicitly allows the traffic to enter and exit the tunnel. If the policy is missing, the tunnel will be up but no traffic will pass.
Troubleshooting an IPsec VPN involves checking each of these layers in order:
First, verify basic connectivity between the two gateway IP addresses using ping. If the gateways cannot reach each other, the tunnel cannot form.
Second, check Phase 1 parameters: are the encryption algorithm, hashing algorithm, and authentication method identical on both sides? A common mistake is a mismatch in the DH (Diffie-Hellman) group, which is used for secure key exchange.
Third, check Phase 2 parameters: do the local and remote subnets (the networks behind each gateway) match exactly? Any mismatch will cause the tunnel to fail to establish.
Fourth, verify firewall policies: is there a policy allowing clear traffic from the local subnet into the tunnel interface? And is there a policy allowing decrypted traffic out of the tunnel to the remote subnet?
Finally, examine routing: does the FortiGate know how to send traffic to the remote subnet? It needs a static route pointing to the tunnel interface or a dynamic routing protocol (like OSPF) running over the VPN.
Check Physical Connectivity and Basic Reachability
Use the command 'execute ping <remote_gateway_ip>' from the FortiGate CLI. If the ping fails, the tunnel cannot form. This step isolates whether the problem is at the network layer (e.g., ISP issue, wrong public IP) or the VPN configuration itself.
Verify Phase 1 Parameters
Use 'diagnose vpn ike gateway list' to see the status of the IKE gateway. If it shows 'dead', check that the Pre-Shared Key, encryption algorithm (e.g. AES256), hash algorithm (e.g. SHA256), and Diffie-Hellman group (e.g. DH14) are identical on both FortiGates. A single character mismatch in the PSK will cause a failure.
Verify Phase 2 Parameters
Use 'diagnose vpn tunnel list' to check the IPsec SA. If the tunnel is established but the Phase 2 SA is not, use 'diagnose vpn ike config' to compare the 'local subnet' and 'remote subnet' (proxy IDs) on both sides. These subnets must match exactly – if one side uses 10.0.1.0/24 and the other uses 10.0.1.0/25, Phase 2 will fail.
Inspect Firewall Policies
Even if the tunnel is green, traffic will not flow without a valid firewall policy. Run 'diagnose firewall iprope list' to see packet counters. If the counter for the expected policy is zero, the traffic is not hitting it. Check that the policy's source and destination interfaces are correct (e.g., from 'internal' to 'vpn_tunnel').
Check Static Routes and Routing Table
Run 'get router info routing-table all' to confirm there is a route for the remote subnet pointing to the tunnel interface. If the route points to the wrong interface (e.g., the WAN interface instead of the tunnel), the traffic will be sent unencrypted or dropped. Add or correct the static route if needed.
Consider a real company, ‘GlobalTech’, with a headquarters in London and a branch office in Manchester. GlobalTech uses a central ERP (Enterprise Resource Planning) system hosted in London that the Manchester team must access daily. The IT manager decides to connect the two offices using a site-to-site IPsec VPN between two FortiGate firewalls.
Step-by-step, what does the IT professional actually do?
First, they configure the Phase 1 interface on both FortiGates. They set the same Pre-Shared Key (a long, complex password), the same encryption algorithm (AES256), and the same hash algorithm (SHA256). The London FortiGate has a public IP address of 203.0.113.1, and the Manchester FortiGate has 203.0.113.2.
Second, they configure Phase 2. For London, the local subnet is 10.10.1.0/24 (the London LAN), and the remote subnet is 10.20.1.0/24 (the Manchester LAN). For Manchester, the local and remote subnets are swapped. A common trap here is accidentally using the wrong subnet mask (e.g., /24 versus /16) – the tunnel will still come up, but some traffic may not route properly.
Third, they create firewall policies. On the London FortiGate, a policy is created from the ‘internal’ interface (where the London LAN lives) to the ‘VPN’ interface, allowing all traffic to the Manchester subnet. A second policy is created from the VPN interface to the internal interface, allowing return traffic. The same is done symmetrically on the Manchester FortiGate.
Fourth, they add static routes. The London FortiGate needs a route for 10.20.1.0/24 pointing to the VPN tunnel interface. The Manchester FortiGate needs a route for 10.10.1.0/24 pointing to its VPN tunnel interface.
Now, the tunnel is configured. The IT professional tests it by pinging from a computer in Manchester to the ERP server’s IP address (10.10.1.50). If the ping fails, they begin troubleshooting using the FortiGate CLI:
They run ‘diagnose vpn ike gateway list’ to see if Phase 1 is up. If the gateway status is ‘dead’ or ‘established->check_dpd’, Phase 1 is having issues.
They run ‘diagnose vpn ike config’ to verify Phase 1 and Phase 2 parameters match.
They check the logs using ‘diagnose debug application ike -1’ to see the detailed negotiation messages.
If Phases 1 and 2 are up but traffic still fails, they check the firewall policy counters using ‘diagnose firewall iprope list’ to see if packets are hitting the policy.
They also check the routing table with ‘get router info routing-table all’ to ensure the remote subnet is reachable via the tunnel interface.
In a real-world scenario, a common issue is that the ISP (Internet Service Provider) is blocking the UDP ports used by IKE (ports 500 and 4500). The IT professional would then need to contact the ISP or configure the VPN to use a different port or protocol (like NAT-T, which encapsulates IPsec inside UDP port 4500, avoiding many firewall blocks).
The NSE7 exam (specifically objective 3.1) is relentless about phase differentiation. The most frequent question pattern is:
‘Which phase is responsible for negotiating the encryption algorithm for the user data traffic?’
And the trap is: many candidates say Phase 1 because they remember ‘encryption starts there’. But Phase 1 negotiates the protection for the control channel, while Phase 2 negotiates the protection for the actual data. The correct answer is Phase 2.
Another classic trap: ‘During IPsec negotiation, which protocol provides encryption of the original packet?’
The options might include ESP, AH (Authentication Header), IKE, and ISAKMP. Beginners often pick IKE. But the correct answer is ESP, because ESP encrypts the entire payload. AH only provides authentication and integrity – no encryption. IKE is used for key exchange, not for encrypting user data.
Key concepts the exam loves to test:
IKEv1 versus IKEv2 differences: IKEv2 is more robust, supports MOBIKE (mobility), and has fewer messages for negotiation. The exam likes to ask about the number of messages in each mode (Main mode: 6 messages, Aggressive mode: 3 messages in IKEv1; IKEv2 uses 4 messages).
The role of Dead Peer Detection (DPD): it detects when the remote VPN gateway goes down, so the local gateway does not wait indefinitely to send traffic.
IPsec modes: Tunnel mode (encrypts the entire original IP packet and is used in site-to-site VPNs) versus Transport mode (encrypts only the payload and is used in host-to-host scenarios like remote access).
The ‘SPI’ (Security Parameter Index): a unique identifier for an IPsec SA. The exam may ask about what happens if two SAs have the same SPI.
The difference between ‘responder’ and ‘initiator’ in Phase 1: the initiator starts the negotiation, the responder replies.
The correct answer pattern when troubleshooting is always the same: verify connectivity first, then Phase 1, then Phase 2, then policies, then routing. The exam will present you with a scenario where the tunnel is ‘up’ but traffic is not passing. The correct answer is almost always ‘check the firewall policy’ or ‘check the static route’. They also love to present a configuration mismatch, where one side uses SHA-256 and the other uses SHA-1. The correct answer is to align the hash algorithms.
Definitions you must memorise verbatim for the exam:
IKE: Internet Key Exchange. Protocol for establishing a shared session key.
SA: Security Association. A set of security parameters for a given connection.
ESP: Encapsulating Security Payload. Provides confidentiality, data origin authentication, and integrity.
AH: Authentication Header. Provides data origin authentication and integrity but no confidentiality.
NAT-T: Network Address Translation Traversal. Allows IPsec to work when one or both gateways are behind a NAT device.
IPsec site-to-site VPNs use two distinct phases: Phase 1 secures the management channel, and Phase 2 secures the actual user data traffic.
The most common cause of a Phase 1 failure is a mismatch in encryption algorithms, hash algorithms, or Diffie-Hellman groups between the two VPN gateways.
The most common cause of a Phase 2 failure is a mismatch in the local and remote subnet definitions (the 'proxy IDs').
A tunnel can be 'up' yet still not pass traffic if the firewall policy on the FortiGate does not explicitly allow the traffic to and from the tunnel interface.
IPsec Encapsulating Security Payload (ESP) provides encryption, authentication, and integrity; Authentication Header (AH) provides only authentication and integrity without encryption.
When troubleshooting, always follow this order: physical connectivity, Phase 1, Phase 2, firewall policies, and then static routing.
These come up on the exam all the time. Here's how to tell them apart.
IKEv1 Main Mode
Uses 6 messages for Phase 1 negotiation
The Pre-Shared Key is never sent in plain text during negotiation
More secure and slower than Aggressive Mode
IKEv1 Aggressive Mode
Uses only 3 messages for Phase 1 negotiation
The Pre-Shared Key can be sent in plain text (hashed) allowing theft if intercepted
Less secure but faster – used primarily when the remote peer's identity is dynamic (e.g., DHCP)
ESP (Encapsulating Security Payload)
Provides encryption (confidentiality) of the entire original IP packet
Also provides data origin authentication and integrity
Used in 99% of modern IPsec VPNs; works well with NAT
AH (Authentication Header)
Does not provide any encryption – only authentication and integrity
Protects parts of the outer IP header, making it incompatible with NAT
Rarely used today; primarily relevant for legacy or compliance-only scenarios
Tunnel Mode
Encapsulates and encrypts the entire original IP packet
Adds a new outer IP header – packet size increases significantly
Used for site-to-site VPNs between two gateways
Transport Mode
Encrypts only the payload (data) of the original IP packet; the original IP header is left intact
Smaller packet overhead
Used for host-to-host VPNs (e.g., direct client-to-server protection)
Static Route over VPN
Manually configured route pointing to the remote subnet
Does not adapt to network changes – must be updated manually if remote subnets change
Simple and low overhead, but not suitable for large or redundant topologies
Dynamic Routing (e.g., OSPF) over VPN
Protocol like OSPF learns and advertises routes automatically
Adapts automatically to link failures or new subnets
Higher complexity and overhead, but necessary for multi-site or hub-and-spoke scenarios
Mistake
If the Phase 1 tunnel is up, the VPN is working perfectly and all traffic can pass through.
Correct
Phase 1 only establishes a control channel. The actual data tunnel (Phase 2) must also be established, and firewall policies and routing must be correctly configured for traffic to flow.
It feels logical that once the 'handshake' is done, everything should work, but Phase 1 and Phase 2 are separate negotiations with different parameters and purposes.
Mistake
IPsec encryption is only required for sensitive data like passwords; other traffic can go through the tunnel unencrypted.
Correct
IPsec encrypts all traffic that matches the Phase 2 selectors. If a VPN gateway receives a packet that matches the VPN policy, the entire payload is encrypted – not just parts of it.
People assume encryption is optional or selective because they are used to applications that only encrypt their own data (like HTTPS). IPsec operates at a lower layer and encrypts everything.
Mistake
If the tunnel is up, both sites can ping each other’s computers directly without any additional configuration.
Correct
The tunnel itself does not automatically create routes. Both FortiGates need explicit static routes to the remote subnet pointing to the tunnel interface, and the PC’s default gateway must be the FortiGate.
Non-network people often think a 'connection' between two devices means any device on either side can talk directly, not realising that each device needs a route telling it how to reach the other network.
Mistake
Using a longer Pre-Shared Key always makes the VPN more secure and faster.
Correct
While a longer key can be more secure, it does not directly affect VPN speed. The speed is determined by the encryption algorithm (e.g., AES-128 vs AES-256) and the hardware. Also, the PSK is only used during authentication, not for encrypting the data.
The term 'key' is ambiguous. People confuse 'Pre-Shared Key' (which is like a password for authentication) with 'encryption keys' (which are randomly generated for each session and are mathematically derived from the DH exchange).
Mistake
If you change any IPsec parameter on one FortiGate, the tunnel will automatically renegotiate.
Correct
Changing parameters like encryption algorithm does reset the tunnel, but only if the configuration is saved and the changes are applied. In some cases, you must explicitly clear the existing SAs using 'execute vpn tunnel reset' before the new parameters take effect.
People assume that modifying a configuration instantly disrupts the connection, but IPsec SAs are stateful. The old SA continues to exist until it times out or is manually cleared, which can lead to confusing behaviour where the tunnel works but uses old settings.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
IKEv1 uses more messages (6 in Main mode) and is more complex, while IKEv2 uses only 4 messages and is more efficient. IKEv2 also supports built-in NAT traversal and MOBIKE for mobile users.
The tunnel being 'up' only means the Phase 1 and Phase 2 negotiations succeeded. The problem is likely a missing firewall policy on either side, a missing static route, or an incorrect subnet definition in Phase 2.
DPD stands for Dead Peer Detection. It allows a VPN gateway to detect when the remote gateway has gone offline, so it can remove the stale Security Associations and rebuild the tunnel automatically when the peer comes back.
IPsec uses UDP port 500 for IKE and UDP port 4500 for NAT-T. If your ISP or an intermediate firewall is blocking these ports, the Phase 1 negotiation will fail. You can test by temporarily changing the VPN to use a different port or by using a tool like 'traceroute' to see where the packets are dropped.
IPsec adds extra headers (encapsulation) to each packet, increasing its size. If the total oversized packet is sent over a link with a smaller MTU, it gets fragmented or dropped. This often causes random connectivity failures. The solution is to reduce the TCP MSS (Maximum Segment Size) on the FortiGate interface.
Yes, technically you can, but it is a security risk. If one tunnel is compromised, all tunnels sharing that key are compromised. Best practice is to use a unique Pre-Shared Key for each tunnel or, better yet, use digital certificates for authentication.
You've finished Site-to-Site VPNs and IPSec Troubleshooting. Continue through the NSE7 study guide to build a complete picture of the exam.
Done with this chapter?