Random thoughts of a warped mind…

February 12, 2014

Sync S3 buckets in parallel mode via concurrent threads

Filed under: All,Amazon EC2,EC2,Git,Ruby,S3 — Srinivas @ 18:13

A week back I realized one of my core S3 buckets at work (which we use for all a bunch of app uploads that are always needed) was a us-west-2 only bucket and not US-Standard. (Dont like that, When S3 gives you 11 9s why not get a US Standard bucket???). Considering that we had varnish in multiple regions with this bucket as the backend, I wanted to do two things -

1. Migrate all data from this bucket to a US-Standard bucket

2. Migrate all data from this bucket to a EU/Ireland bucket as well (coz I have app servers etc out there as well which need the same data – Did’nt want to come across the pond for every object we had to retrieve). Why? Reduced latency and reduced B/W costs (costs nothing when a EC2 instance in EU has to pull an object from a EU bucket).


August 15, 2013

Github – swap origin urls

Filed under: All,Development,Git — Srinivas @ 16:07

Ever pull from a git repo that was open and you had pull access (ONLY) to and then wanted to push back to it later when you were granted that access? Chances are that you cloned from the git:// url (Git protocol) and trying to push back to it throws errors like this.


August 13, 2013

Dump http requests in-transit with tcpdump

Filed under: All,Amazon EC2,Linux,Virtualization — Srinivas @ 14:57

Note to self -

tcpdump -A -s 0 ‘tcp port 80 and (((ip[2:2] – ((ip[0]&0xf)<<2)) – ((tcp[12]&0xf0)>>2)) != 0)’

Handy on haproxy/varnish boxes to see requests/responses in realtime for debugging on-the-fly… As opposed to having to dump to trace file and analyzing off-server with wireshark or similar…

August 2, 2013

Cloudfront woes – “Your request contains one or more invalid invalidation paths.” – Use custom regexp for URI::encode

Filed under: All,Amazon EC2,EC2,Linux,Ruby — Srinivas @ 12:27

AWS Cloudfront is a content delivery network part of Amazons EC2/AWS stack which lets you serve static assets from a source (S3 bucket or custom origin server) by caching it across numerous edge locations. Occassionally the underlying content can change which needs the cache to be refreshed – This is done via a Cloudfront cache invalidation request which specifies a distribution id and a list of paths to refresh (e.g. /index.html or /imgs/logo.png etc).


September 24, 2012

Get an Amazon load balancer running in <5 mins

Filed under: All,Amazon EC2,AWS ELB,EC2,Ruby — Srinivas @ 15:47

Heres a quick Ruby script to get an Amazon HTTP load balancer running in ~5 minutes (or lesser). The code in this github project assumes you have an AWS account backed up by a credit card and have your AWS Access Key ID and Secrets available – Paste them into the YML config file from the git repo.

Modify the region, node counts, AMI ID etc in the config YML to suit your requirements – Keep in mind that the user-data script assumes you are on a Ubuntu/Debianish distro and sets up a test CGI page on the instances based on that assumption.


Older Posts »

Powered by WordPress