There can be many ways one can architect VPN Networks. The end objective is to tunnel private IP (typically LAN subnets) within public IP Networks (such as Internet). You can establish such a tunnel in various creative ways, such as Transport Mode, Tunnel Mode, IP-in-IP tunnelling, etc. Also you can choose any transport layer protocol of your choice such as send VPN traffic via TCP, or via UDP and so on.
Sometimes since our objective is to just create a tunnel on public Internet (i.e L3 Routed Network), we may never need a transport layer. In such cases, we can just tunnel via IP-in-IP alone as shown below:
Since the routers operate in Layer-3, such VPN packets in theory will perfectly work and will reach the intended destination. From there the VPN server(s) or an app can recreate original LAN packets.
So if you see that way we can compare the VPN packet encapsulation differences between just IP-in-IP vs via added Transport Layer (either TCP or UDP).
In some weird situation if this VPN data (in this case IP-in-IP) is encapsulated inside another VPN Network (again IP-in-IP), then you can get into a weird scenario like this as shown below. Although it is perfectly fine to do as per the IPv4 routed network standards, we should know the fact that each VPN tunnel/layer will reduce the MTU of the actual application payload due to all those extra added headers and as well performance due to encapsulation as well possibly VPN tunnel compression.
For VPNs which are built via Transport Layer network sockets (such as TCP/ UDP), one can choose either of the protocols. Since what we send the LAN data itself could be end-to-end TCP sessions which has reliability in place, we don’t need strictly VPNs with TCP as its Transport Layer protocol such as TCP. Instead we can use UDP itself. Which is why I say always UDP is nothing but Layer-2 on IPv4 Networks.
Refer:
PacketLife.net Packet Captures ↗
IP in IP tunneling ↗
Here is an extensive Youtube video of mine on the same with a deeper insight on the same with some live packet analysis:
I would also recommend this below Youtube video episode of mine shot long ago before I published my earlier video. In which I discussed overall VPN architecture, design possibilities and compared the architecture with live real-world examples: