Random thoughts of a warped mind…

August 20, 2012

Link Amazon VPCs over a IPSEC site to site VPN

Filed under: All,Amazon EC2,AWS VPC,EC2,Linux,Virtualization — Srinivas @ 15:52

Typically, when you launch new hosts in AWS EC2, these come with a public IP address and a private IP address – and both of these can (and mostly will) change when these instances are stopped and re-launched. This may fit most needs but if you want to isolate your servers into a seperate subnet where IP addresses dont change on stop/start, then VPC is what you would launch your instances in… Launching instances within a VPC is akin to running them in a private subnet where you could have all ports open within the subnet without allowing other instances (that are not yours) into the segment.

AWS VPC provides an option to create a hardware based IPSEC tunnel between your VPC and your data center (or office VPN etc) and allow seamless traffic flow between both. This is handy if you want hosts in your VPC to be able to talk to hosts in your datacenter (by data center, I mean a typical rack and stack setup that you own/lease/have whatever…). See http://aws.amazon.com/vpc/ for details.

However lets say you have servers in both AWS US-EAST and US-WEST and want them linked into one private network, this is’nt something VPC does out of the box. This article deals with linking the two VPCs over a IPSEC VPN to form a site to site VPN(where each host can individually access any other host at the other end)

Why would you need this? Sample scenarios-

  • You want to load balance traffic across west and east sites and want a database(s) in each zone to handle traffic for that zone while also being able to replicate changes from one side to the other. This is not something you would want to do over the big bad open internet…
  • Federated qpid instances across AWS regions.
  • Internal mail hosts or numerous other possibilities…

AWS provides multiple options to create a VPC. Considering that this is a onetime setup, we will use the VPC console to do this (you could use the API tools as well but I dont see the need to do this often so console works just fine for me, anything I might have to do N times, I use the API).

The concept is simple -

  • Create a VPC (/16 network) in each region (US-EAST and US-WEST in this case) with a public and private subnet(each is a /24).
  • In each public subnet, create a Ubuntu instance with a fixed IP address we specify. (will refer to these as “NAT instances“)
  • Allot an elastic IP to the public subnet Ubuntu instances.
  • Confirm each of the public subnet instances can now access the internet.
  • Now in both west and east zones, create an Ubuntu instance in the private subnet.(Will refer to these as “Private/Inside instances“)
  • Create a custom routing table via VPC console for this private subnet and set the default route for that routing table to be the instance ID of the Ubuntu instance in the Public subnet (the one we alloted a Elastic IP address to)
  • Confirm that the private instances can access the internet.
  • Setup up Open Swan (https://www.openswan.org/projects/openswan/) on both public subnet instances to setup a IPSEC VPN with the other (public) instance and route traffic for their specific subnets(full /16 subnets).
  • Confirm that a private instance in East can access the private instance in West (and vice versa)

Lets first decide on the IP addressing scheme we want for the east and west VPC subnets.

Public Subnet
Private Subnet
NAT Instance VPC IP
NAT Instance Security Group Default Default
Private Instance Name WEST_INSIDE_1 EAST_INSIDE_1
Private Instance VPC IP
Private Instance Security Group West_Hostgroup East_Hostgroup

The first 3 items in the table above are most important – you cannot change the CIDR of a VPC once set. See diagram for an overview of the setup.

Site to Site VPN between AWS VPC subnets

Read ahead for detailed setup instructions. Unless mentioned otherwise, each of these needs to be done at both East and West regions. Note that the smallest possible instance type in VPC is a “m1.small” since Amazon does’nt allow running a t1.micro in the VPC (a t1.micro would have been more than enough for routing traffic though and cost much lesser :-( )

Create a VPC with Public and Private Subnets in US-EAST

  1. Login to AWS console and in VPC tab, start with VPC Wizard by selecting the option “VPC with Public and Private subnets”
  2. Set the CIDR, Public and Private subnets to values from the table (or your customized ones)
  3. With the wizard, you will have to create a NAT instance (create a dummy instance for now, we dont care what it is, we will be dropping this and building our own NAT instance out of a Ubuntu 11.10 AMI instead of using AWS supplied Amazon Linux AMI. Why? Because I’ve had problems with openswan crashing the Amazon Linux 3.0.0 kernel).
  4. Create VPC and promptly terminate the NAT instance it started up.
  5. Repeat the same process, but this time for the West VPC (Remember to swap the Region dropdown for us-west now :-) )

Create Public NAT instance in US-East

  1. Go back to US-EAST EC2 and spawn a new Ubuntu 11.10 64 bit instance in the east VPC in the subnet with IP specified as
  2. This would have got created in the VPC security group “Default” (Remember: VPC security groups are separate from plain EC2 security groups!)
  3. Assign an elastic IP address to this instance and set the Default security group to allow SSH from your management host
  4. Repeat the same process for US-WEST replacing the subnet and IP with corresponding US-West values.

I will assume for the purpose of this discussion that -

  • The instance IDs of the East and West NAT/Public instances are i-east4567 and i-west1234 respectively.
  • The Elastic IP addresses for the East and West NAT instances are and respectively.

Setup the public/NAT instances to allow IPv4 forwarding, and set iptables rules to route packets for the VPN

  1. Configure sysctl.conf to allow IPV4 forwarding (This is off by default) and also set params to suit Openswan. See gist for details.
  2. Setup script /etc/configure-pat.sh (see gist, save and set this to have permissions 500) and update /etc/rc.local to invoke it.
  3. Remember to modify configure-pat.sh to set the remote VPC subnet to for the East public instance and to for the West public instance! (This is the first iptables POSTROUTING rule in the script). This is essential to ensure that packets transfer between inside hosts at east and west is only routed but not NAT’ted!
  4. In the EC2 console, modify the “Source/Dest Check” property to disable it or if you prefer the command line, use ec2-modify-instance-attribute to change SrcDestCheck ! Do this now, if you forget this you may end up spending several minutes trying to debug everything from VPN access to basic internet access for the private subnet :-)
  5. Repeat the same process for US-WEST.

Sysctl.conf for NAT instance -

Configure-pat.sh for NAT instance (This is pretty much the same as the stock configure-pat.sh from a AWS NAT instance except for the additional iptables rule to prevent NAT of site to site VPN traffic):

Once you have these setup, make sure to run “sysctl -p” to setup kernel config and also run “/etc/configure-pat.sh” from command line as root.

Modify the “default” security group for both east and west VPCs

  • Modify the East “default” security group to allow all traffic from EIP of West NAT instance i.e. allow all from
  • Modify the West “default” security group to allow all traffic from EIP of East NAT instance i.e. allow all from
  • I am assuming here that you used the VPC security group “default” for east and west NAT instances, if you created custom security groups for these, then modify those accordingly.
  • IPSEC is hard enough to debug on “real” hardware without involving the complexity and multiple variables that EC2 and VPC add, start with simple  rules for starters and get more creative with firewall rules once base setup is in place and confirmed to work!

Setup Private/inside instances in the private subnets

  1. Start up an AMI of your choice in the US-EAST VPC private subnet ( and allot it IP under security group (newly created) “East_Hostgroup”
  2. Repeat step for US-WEST for VPC subnet with instance IP and instance in security group “West_Hostgroup”
  3. There is no “special” configuration for the Private subnet instances (outside of running any specific app that you may want)
  4. Modify the East and West Hostgroup security groups to allow all traffic from self, and all traffic from the VPC “default” security group.

Alter VPC Routing tables to set the NAT instance to be the default gateway

AWS does’nt like specifying routing properties within the Instance i.e. at the OS level. The recommended way to set routing properties is via use of the VPC routing tables.

  1. Go to the US-EAST VPC tab and create a new routing table (assuming name is rtb-east456).
  2. Remove any default route this routing table may have.
  3. Add a route for subnet (i.e. all destination traffic!) via Instance ID i-east4567 (i.e. Default route for this routing table must be via the US-East NAT instance). Save changes.
  4. Then in the associations tab for rtb-east456, add subnet “″ from the dropdown list.
  5. This will ensure that all access from the internal instances in subnet of this VPC will be routed through the VPC NAT instance.
  6. Login to the instance East_Inside_1 and confirm you can access the internet. If you cant, make sure the setup on the EAST NAT instance was fully done and the routing table for the East_Inside_1
  7. Repeat steps 1-6 for the West side setup as well to create a new routing table, set its default route to be the instance ID of VPC_West_NAT and associating the subnet with this new routing table.

Note: Multiple subnets can be associated with a single routing table. If you were to add more private subnets in the future, you could always associate them with the same routing table.

Setup Openswan on the public instance at US-EAST

  1. Install openswan and mtr via apt. (and while you are at it, also install mtr and traceroute on the private instances at East)
  2. Configure /etc/ipsec.conf as shown.
  3. Configure /etc/ipsec.d/eastwest.conf as shown below (This config is for the East side openswan setup, update west side setup!)
  4. We will be using a shared key for VPN setup, This wil be /etc/ipsec.secrets on both boxes with the key set to a long hard to guess string.
  5. Repeat 1-3 for VPC_West_NAT instance making sure to swap the IP and subnet values accordingly
  6. On both NAT instances, “service ipsec start” to establish the VPN.

IPSEC config (/etc/ipsec.conf)

Setup /etc/ipsec.d/eastwest.conf (Note that the gist below shows config for the us-east NAT instance, swap the EIP and subnet values to make it work for the us-west NAT instance)

Setup /etc/ipsec.secrets(Sample, do set a complex PSK !)


Confirm that-

  • Both public/NAT instances can access the internet! (If not, you probably forgot to map an Elastic IP to them)
  • The VPC security group applied to the NAT instances allows  all traffic from the EIP of the NAT instance on the other end.
  • East_Hostgroup allows all traffic from itself and local default security group.(Similarly for West_Hostgroup)
  •  Default security group at East VPC allows all traffic from East_Hostgroup (and similarly for West side setup too).
  •  Private instances can ping to the local NAT instance and vice verse for East & West.
  • Private instance can access internet resources (If you cant, then make sure src/dest check is disabled on NAT instance and make sure to run sysctl and configure-pat.sh on NAT instance and confirm that the private subnet has routing table set to the custom one that gateways through the NAT instance!)

If all of this checked out OK, then with ipsec service running, you should be able to ping from west side private instances to east private instances (and vice verse).

My observations/rants

  1. Let me run a t1.micro in the VPC! Throwing a m1.small for a basic NAT instance seems like an overkill.
  2. You allow VPCs to be linked to Cisco/Juniper routers etc but why not to each other? Should be a breeze really without having to jump through hoops like this… :-(
  3. A VPC subnet is restricted to a single Availability zone (AZ) i.e. us-east1a/b/c etc and not spread over us-east region. This means that if you want to have multiple database servers in a VPC for redundancy, then its best to add more private subnets – one per AZ and distribute instances across multiple private subnets rather than consolidating multiple instances in the same private subnet (this way your setup is better protected against single AZ failures).
  4. You may want to do the same for the NAT instances public subnet since that is also tied to a specific AZ and not over the region. This means that in this setup, if the AZ where the private subnet is located in goes down, then your VPN would be down. (Which brings me back to #2 about AWS allowing native VPC-to-VPC IPSEC setup which would remove the NAT instances as a single point of failure).
  5. It’ll be nice to have custom tags for VPC objects i.e. “Name=VPC_Pub_Sub_172_18_0″ rather than “subnet-0f2c4gh”…
  • mounif

    there is no mico instances in VPC

  • mounif

    There are no micro instances in VPC.

  • Srinivas

    So? Thats what I said… :)

  • Rob Kolosky

    There are micro instances in VPC now. Thank you for the walkthrough – exactly what I was looking for!

Powered by WordPress