Ansible philosophy for AWS

A devops guy recommended Musashi's book "Five Rings". Musashi was a Samurai who lived long - a rare event, apparently. It is a book on strategy. The book is very short but he explains that this is fine. The phrase "from one thing flows a thousand things" appears repeatedly. Learn one thing well and many things will flow from it.

I will go a step further and say that if many things flow from learning something, it is a good investment, and apply these two phrases to AWS and Ansible.

AWS is fundamentally programmable infrastructure. Study aws help on the command line and you will be able to do everything that is worth doing in AWS. The quality of an API often depends on the number of users and the level of support. There are tens of thousands of certified AWS developers and many more uncertified ones and most of those use the AWS cli for at least some of their work. And the cli is maintained by Amazon itself. That indicates a pretty reliable and well maintained API. It is good.

Ansible fundamentally describes the desired state of infrastructure and has a command to make reality correspond to the desired state. Ansible scripts are structured data, which aligns well with defining machine-readable state. It is promising.

Now for combining Ansible with AWS. Ansible has many modules that replicate part of the AWS API. That already tells you that you probably don't want to go anywhere near those modules. Unsurprisingly many of them are broken or incomplete. Furthermore their language doesn't correspond cleanly to the AWS API.

Happily there is a clean alternative. All the data sent and returned by the AWS cli comes as JSON blobs; structured data that can be compared easily with state defined in Ansible. By focusing on manipulating structured data in Ansible you can find clean and compact ways of testing whether the real AWS state corresponds to your intended state.  The json_query filter that implements JMESpath is your friend. Likewise when theory != reality you will be able to make theory == reality. The only two phrases that matter in Ansible are "Is it so?" and "Make it so!".

Thank you, Musashi.

Sed on blocks of text

A devops guy just asked me whether sed can work on blocks of text as opposed to just lines.  Sure it can.  Try this for fun:

man man | sed -rn '/^ *-/{h;:a;n;/^ *$/{bb};s/^ *//g;H;ba;:b;x;s/^ *//g;s/\n/ /g;p}'

Happy to explain how it works.  Please note that Mac's sed is broken; please use gsed instead!

React trumps Swift for startups

In an early stage startup it is very important to be able to move very swiftly and decisively, often under a highly constrained budget. In that context React wins hands down. What is wrong with Swift:

Consider a typical startup that produces an iPhone app first - it generates some buzz but it doesn’t go viral - the reaction is muted - because more than 50% of people who hear about it have an Android phone. The mathematics of viral infections applies - if the probability of transmission falls below a certain threshold the disease doesn’t become epidemic. Fail 1. Several months later an Android version has been produced. Windows phone and the rarer species are not even on the roadmap. However the initial wave of journalistic attention has passed, so virality is improved but now there is no wave to kick start it. Fail 2. The pain doesn’t stop there. A startup often has to make a sequence of small changes of plan - swiftly - yet now there are two apps to maintain and trying pivot both in rapid sync is extremely hard. Pain 3. Finally the death knell: The iPhone programmer isn’t seeing success and is being wooed by other companies, so leaves his or her share options behind and moves on to the next startup. The first startup is now in serious trouble. It didn’t win fast, so now it’s dead in the water.

React gets every one of the above points right. The codebase for iPhone, Android and Windows is largely identical, so when you go live you can push on all platforms simultaneously. You get a stronger viral effect, you keep all your apps in sync and you can make changes faster.

Secondly, it’s the economics. One React programmer is cheaper than an iPhone programmer + an Android programmer. The React programmer can probably also maintain your website with much the same codebase. So in the specific context of startups, React or a similar cross platform stack such as Meteor is almost always a slam dunk winner.

Personal mobile signal strength map

OpenSignal is a crowd sourced app that shows where mobile phone network coverage is good and where it ain't.  It gives individual users the tools to act rationally, including choosing a good network and proving when their network isn't living up to its commitments.  It is also not just a pointless platform for moaning about how bad life is because that same data is also available to network operators so that they can improve their network with a customer-centric sensor suite.  It doesn't deal with hypotheticals or with performance under test conditions but with real life and real usage by real people.  That's why I like it, apart from it's geek value, of course!

A theme that I have heard recently is mobile phone users wanting to play with their own data rather than just seeing the aggregate stats.  The raw data has always been available, or at least for as long as I have known OpenSignal, however what's missing is a good set of tools for people to crunch their own numbers.  I threw together some mapping software a while ago for mobile phone readings.  It sounds as if that will cover some of what's needed.  So here's the deal.  I've put my mapping code on github.io so you can visualise your data.  Now, I'm curious about one thing:  What is special about your personal data versus the general aggregate?  I'm interested because we can play a game of egg-hunt or rather "find the tower" and once we've found a tower I like making a note of what's on it, however that's just me and it doesn't strike me like the sort of thing a huge number of people would be interested in.  Why do YOU care about your personal signal data?  What do you do with it?  Answers on an envelope please, and have fun mapping your data!

BTW as you will see, the map could do with styling by someone with an artistic eye.  Contributions are welcome!

Regards, Bit.

Certificates: Make a chain file

I toasted a server recently.  I like doing that.  It's like cleaning out the pipes with a bongoberry smoothie.  Chuck the old cruft overboard and reinstall just the bits I love.  But I chucked out the certificate chain as well.  I still had the certificate itself but with no onward chain mobiles won't talk to my server.  Whoops.  Not actually a problem as I had been a good little boy and kept backups of my disks religiously every day so I could simply have recovered it.  But I took this as a wee bit of a challenge.  I haven't been in the security industry for a while now.  The amount of junk google returns when you search for openssl and chain file is impressive.  So could I make myself a chain file from scratch?  Here's how.

What's a chain file?  Its a sequence of certificates that connect my personal certificate to some self-proclaimed authority.

Given my certificate, how do I find the next certificate up the chain?  Dump the certificate and you should see an extension called "Authority Information Access" with the URL of at least one more, and usually just one, upstream certificate.  Download and repeat.  You may find that the downloaded certificate is a binary blob that you need to convert but that's all.

Finally concatenate those bits into a chain file.  Yay.  man 1:0 google.

By the way, jeg elsker https://www.ssllabs.com/ssltest/analyze.html

Rounding up to page size

I was surprised yesterday to see an experienced programmer check in a terribly contorted piece of code to round a byte size up to the next page boundary.  Furthermore it still managed to be wrong.  Here is how to do it:

size_t page_size = getpagesize(); // Power of 2
total_size = total_size + ((page_size-1)&-total_size);

There are entire books on the subject but if you have to learn formula after formula by heart a book probably won't do much other than fill your head with things that still won't quite be what you need in the heat of battle.  Much better to understand why it is obvious:  Numbers mod m form an additive group, so normal addition and subtraction rules apply.  You wish to add something to total_size to get zero mod m? If x+total_size == 0 then x == -total_size (mod m).  Using (page_size-1)&... is a way of saying (mod 2**n) that frees us from the rather unnatural convention in computers for how modulo is computed that would otherwise give us a negative value, thereby rounding down, as this code does:

size_t page_size = getpagesize(); // Power of 2
total_size = total_size + (-total_size)%page_size;

By the way, the same logic works modulo any base.  To round down to the nearest 100:

rounded = value +(-value)%100;

Have fun, and kick ass in the small things as well as the big ones! :-)

Yosemite DNS

Yosemite DNS seems to be fairly broken.

A small hint is when ping bitdivine.com fails, even though dig bitdivine.com succeeds and Chrome, which uses it's own DNS server rather than the system one, is fine.  It seems that apple has changed their DNS client from mDNSResponder to discoveryd and the latter is having some teething problems.

This worked for me:

Add Google's name servers to those listed in the system preferences (under System Preferences->Networking->Wifi->Advanced->DNS):

Restart discoveryd by running this in Terminal:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.discoveryd.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.discoveryd.plist

There might well be a better way.  It depends on what you trust.