Random thoughts of a warped mind…

September 17, 2014

Reconfig for service discovery

Filed under: Amazon EC2,Chef,Git,Linux,Redhat OpenShift,Ruby,Virtualization — Srinivas @ 09:49

Imagine a constantly changing fleet of servers… New servers (or virtual machines) going online as capacity is needed or being taken offline as load drops… Maybe you already use Chef/Puppet to bootstrap your servers and use their node attributes to populate N servers in a load balancer (Haproxy for example) configuration. 

Now adding/removing a new server (a backend, maybe a rails/tomcat server, whatever) would mean that chef-client has to run on all the haproxy boxes so they know about the new backend (or the one that went away). This would work if you  run chef-client every 5 minutes or so? But why? Chef/Puppet are primarily meant to bootstrap your servers and not to sync state. Enter Reconfig and service discovery.

(more…)

February 12, 2014

Sync S3 buckets in parallel mode via concurrent threads

Filed under: All,Amazon EC2,EC2,Git,Ruby,S3 — Srinivas @ 18:13

A week back I realized one of my core S3 buckets at work (which we use for all a bunch of app uploads that are always needed) was a us-west-2 only bucket and not US-Standard. (Dont like that, When S3 gives you 11 9s why not get a US Standard bucket???). Considering that we had varnish in multiple regions with this bucket as the backend, I wanted to do two things -

1. Migrate all data from this bucket to a US-Standard bucket

2. Migrate all data from this bucket to a EU/Ireland bucket as well (coz I have app servers etc out there as well which need the same data – Did’nt want to come across the pond for every object we had to retrieve). Why? Reduced latency and reduced B/W costs (costs nothing when a EC2 instance in EU has to pull an object from a EU bucket).

(more…)

August 15, 2013

Github – swap origin urls

Filed under: All,Development,Git — Srinivas @ 16:07

Ever pull from a git repo that was open and you had pull access (ONLY) to and then wanted to push back to it later when you were granted that access? Chances are that you cloned from the git:// url (Git protocol) and trying to push back to it throws errors like this.

(more…)

May 24, 2013

Sinatra app on Redhats OpenShift PaaS

Filed under: Development,EC2,Git,Linux,Redhat OpenShift,Ruby,Virtualization — Srinivas @ 02:08

Got to poke around a bit today into Redhats Open Shift PaaS… This is quite similar to Herokus platform in the sense you could pick up a “dev platform”, build an app locally and push it to the PaaS and run it from there…

For the impatient, rush over to the App on Openshift and  pull the code from github

(more…)

March 8, 2013

My chef cookbook to provision EBS PIOPS volumes at Amazon EC2

Filed under: Amazon EC2,Chef,EC2,Git,Linux,Ruby — Srinivas @ 20:21

Heres my Chef cookbook to provision PIOPS EBS Volumes at Amazon EC2 (PIOPS volumes can be requested with a specific IOPs count which is guaranteed to be provided, something plain vanilla EBS volumes dont/cant do).

This cookbook is intended to be a drop-in replacement for Opscodes standard AWS cookbook (except that the namespace used for the node attributes is different, so a little manual work/scripting would be required). Support for Elastic volumes and Load balancers coming shortly…

This cookbook uses the Fog gem so is more easily extendable (or extensible?) to add more AWS resources as needed… Fog v1.6.0 or higher will be auto-installed when this cookbook is invoked.

See Onepower_AWS  for details.

Things to note -

1. Not all AWS Availability zones support PIOPS volumes, see bin/test_piops_support.rb for figuring out which AZs support PIOPS volumes.

2. PIOPS volumes tend to cost more than plain vanilla EBS volumes – And for a real performance gain, PIOPS volumes must be used with instances launched with the ebs-optimized flag as thats what sets up instance specific QoS at AWS to sort of segregate dedicated I/O bandwidth for the instance to access a PIOPS volume (separate from the standard network interface bandwidth that normal EBS uses/shares).

Older Posts »

Powered by WordPress