It sounds like a simple task to configure a Windows client to connect to a L2TP/IPsec server hosted by Mac OS Server. However it’s not.

Requirements

To setup the connection you need the following informations:

• Username and password of a VPN enabled account
• the L2TP/IPsec shared secret
• official IP or hostname of the VPN server
• Administrator account on the Windows machine

Setup

The first step it the most important one. Open the registry editor (regedit) on the Windows PC as administrator and add the DWORD AssumeUDPEncapsulationContextOnSendRule with a hex value of 00000002 within HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PolicyAgent. To apply the changes you have to reboot.

Then add a new VPN connection using the Set up new connection or network assistant in the Network and Sharing Center. Be sure to enable the checkbox Don’t connect now, just set it up so I can connect later.

To finish the last step open the Properties dialog of your newly created VPN connection. Within the Security tap change the type to L2TP/IPsec and set the shared secret within the Advanced settings. Furthermore change the dropdown menu Data encryption to Optional encryption.

Now you are able to connect to the VPN service!