Tuesday Tech Tip: Create Multiple Directories in One Line

Oftentimes you find yourself wanting to create a directory with multiple subdirectories. For example, when creating Ansible roles I almost typically always create the role and three to five different subdirectories underneath it.

This is actually quite easy to do on any posix compliant shell (although I have only tried bash and zsh).

This will yield the following directory structure.

Pretty simple but incredibly handy. This also works for many different arbitrary shell commands, give it a try!

Replacing Text in Nginx with sub_filter

Sometimes you find yourself in a weird predicament. A third party application that you’ve slapped nginx in front of insists on using internal IP addresses or ports despite your reverse proxy passing all the correct headers and other pieces required. Or maybe you’ve found yourself in the situation I found myself in last week where you have a third party internal application wanting to reference a file on a CDN. As luck would have it, that CDN has been deprecated and changing it requires rebuilding a jar where the script path is hardcoded and then repackaging the internal application that uses it. Long story short, we’ll soon be doing some yak shaving that could possibly take all day so how about we just use sub_filter instead and take a nap?

Use It

Thankfully most stock builds of nginx include ngx_http_sub_module by default. Using it is actually simple enough, just slap the following directive under your location or server directive in nginx.conf.

And that’s it! You can also use regular expression here as well. sub_filter_once indicates whether nginx should apply the replacement once or repeatedly. One gotcha you might have is that by default this only works with mime types of text/html. If you want to modify the mime-types that subfilter executes on set `sub_filter_types` to the desired type.

Give It a Try

I’ve put together a very simple demonstration of using subfilter for text replacement on Github.

Friday Functions: NPM Path

This week’s Friday function is probably the most simplistic in my zsh library. Every time I worked with node.js I always got a little annoyed with having to reference the command line apps referenced under ./node_modules/bin. Sure I could do an npm install -g but I don’t like mucking with my global environment on a per project basis. Python has virtualenvs which separate these concerns pretty well but to my knowledge node.js has no equivalent.

So my zshrc has contained this function for quite some time.

Now I just run npm path from the root of a node.js project to quickly add its bin directory to my path. I’m sure there are probably more elegant solutions these days… please let me know in the comments below! 🙂

Testing In Ansible

This is a topic I brushed up against yesterday and meant to blog about it at the end of the day but got a little busy. A lot of times when provisioning boxes locally in vagrant I’ve thought it would be incredibly useful to be able to automatically test the system to ensure all the expected bits are provisioned as expected.

I’ll probably throw together a nice public demo but the short and skinny is to include a final ansible provisioning step after the normal step that runs a test playbook of sorts against the system. For us we just dumped our test tags into our main roles and tag them as test. Then in vagrant we exclude test tagged tasks and then in the test phase we only run those tagged tasks. Below is an example for one of our services to test that two service processes are running and that the load balancer is also serving up responses that are the same as those running on the two processes.

I’ve also heard of other tools in this space like ServerSpec which may fit your bill if you’re not running ansible or are running some mixed environment. So far I think ansible fits well here but you’re definitely going to be a little limited due to the tests being in yaml. Although you could hypothetically write some custom modules or resort to shell wizardry if you need something more advanced.

I’m really excited about this… the idea we could have full test suites with each of our ansible roles that can verify a whole swath of aspects like expected ulimits and the like is GREAT.

Friday Functions: AWS ZSH Helper

This morning I’m going to go with a new recurring weekly post: Friday Functions! While some of it will aim to share my large inventory of zsh functions I’ve acquired over the years I’ll also be finding new additions if I run out of material. So it also serves to help me learn more!

This week’s function is probably only useful if you’re into AWS and use the awscli tool to interact with it from the command line. Using the awscli command direction can be quite verbose so some nice shortcuts are useful. I actually learned of this handy function from Kris’s awesome collection of zsh configuration and made a few small adaptions to it.

This is pretty useful. If you want to find all instances with http in the name you just run aws-instances-describe http.

Screen Shot 2016-03-31 at 6.14.42 PM

Or if you want to look for instances by a specific tag you can use the `-t` switch. For example, to find all instances with the worker_email role tag we can just run aws-instance-describe -t role worker_email. You can add -s to changed the filter to include the running state and like the actual call you can include multiple instances. So if you wanted to find all stopped instances with the taskhistory role you’d run aws-instance-describe -t role taskhistory -s stopped. The function sets this to default to running instances only since that’s what I’m looking for 99% of the time… looking for stopped or terminated instances is definitely the exception.

Hope this was interesting enough. Ideas, thoughts, comments or criticism are all welcome in the comments below! Let me know what you think! 🙂

Managing Dotfiles With Ansible

Yesterday I posted about managing our local configuration with Ansible and today I’m going to continue this path by putting my zsh configuration under configuration management.

Installing ZSH and oh-my-zsh

First up let’s install our preferred shell and customizations. For me this is zsh and oh-my-zsh. Up front I know that this is going to probably be a multi-step process so I’m going to create a local role to bundle up the tasks and templates we’ll be using. I create a new directory named roles and add it to the roles_path in our ansible.cfg. This new directory with our new role will look like the following.

For our initial set of tasks we’ll install zsh via homebrew, configure zsh as the current user’s shell and install oh-my-zsh.

You can see this change in its entirety here. Next up we’ll add a template for our zshrc that we can customize to our liking. To start we’ll grab the zshrc template from the oh-my-zsh git repository and save it to jamescarr.dotfiles/templates/zshrc.j2.

A good first piece of literal text to template out is the zsh theme and the plugins loaded up. We’ll define these with default variables under jamescarr.dotfiles/defaults/main.yaml.

You’ll notice here we’ll also be polite by backing up the existing zshrc file if it is present. A good benefit with this example is that we can now switch up the different pieces of our .zshrc configuration from our playbook by overriding the default variables.

You can find everything up to this point at this referenced commit in jamescarr/ansible-mac-demo.

Up Next

Obviously a topic like zsh customizations is a rather large undertaking deserving of its own post so tomorrow I’ll share some of my personal zsh functions and aliases that I find extremely useful to start off with.

Well did you find the useful? Did you run into any problems following along? Please let me know in the comments below!

Managing Your Macbook with Ansible

For a long time I’ve been a big believer in Infrastructure as Code and I have always wanted to use configuration management to provision my personal workstation and keep it constantly updated to an expected state. Boxen was one of the first tools I saw in this space and it even seemed like it might be comfortable since I was using Puppet at the time. However I never really had a lot of luck with it and the original aim of Boxen was actually lost on us at Zapier since we engineered a very nice docker-compose based setup that lets anyone begin running and hacking on zapier locally constrained by the time it takes to download the docker images for the first time.

That being said when we began transitioning from Puppet to Ansible last year and I naturally started using it locally to kind of whet my appetite a bit. Here’s a brief run down of how I’m currently using Ansible to manage my laptop and some notes on where to go next.

Getting Started

There are several guides out there on getting Ansible installed, the most authoritative being the instructions right on Ansible’s website. I won’t repeat those well written steps here.

Once that’s all done let’s run ansible --version and verify we’re running Ansible or above. If you’re visiting from the future then I will have to say that I am really unsure if this blog post will work with 3.0.0 or whatever release is out then. Keep in mind this post is written in 2016. 🙂

First up we’ll create a very simple Ansible playbook that just prints a hello world message to the console to ensure we have everything configured correctly.

Place this in a project directory (I name mine “personal-dev-laptop”) and run ansible-playbook main.yml. If all is configured correctly you’ll see a playbook run that executes a task that prints out “Hello World” to the console.


The most important piece to a provisioning system is the package management and Ansible is no different. Homebrew is the go to on OSX and thankfully Ansible has a pretty decent homebrew module for managing the state of different Homebrew packages. Let’s dip our toes in by adding a task to ensure macvim is installed and at the latest version.

The nice benefit here is that each time we run Ansible macvim will automatically get updated to the latest available package. However if we want to ensure a package is simply installed but don’t want to upgrade each time we run we can set the state to `present`. After awhile if we’ve worked with vim and decided that it’s just not for us and we’d prefer to use emacs instead we could just set macvim’s state to absent and emacs state to latest.

Taking It Further

Sure we can just keep adding our own tasks to install roles, perhaps even using a with_items iterator to include a big list of them but sooner or later we’re going to be duplicating a lot of work someone else has done. Which is a good time to introduce third party roles installed via ansible galaxy. There are most likely several good roles out there but my favorite so far is geerlinguy.homebrew. I usually put a requirements yaml file in the main root of my project with the module I want to use and the version I want to lock in.

Now to install this third party role we’ll run ansible-galaxy install -p vendor -r requirements.yaml. The -p switch will install it to a local directory named vendor so we don’t clobber the global include path and we can add that directory to our project’s .gitignore so that it isn’t stored in git. We also add an ansible.cfg to specify the role path for third party roles we’ll be using.

Now we also update our main.yaml to include a few changes. Firstly we want to include the new role we just imported and then we move the packages we want to install as variables that the homebrew role will utilize.

This time we’ll run with the -K switch since this role also ensures that homebrew is installed and will require sudo access to do so. Now I know what you’re thinking… you’re thinking “James is trying to hack my box!” and quickly closing your browser tab. Obviously you should never provide sudo without giving the source code a look over and the most important pieces will be the main task file and meta file where there could be dependent roles. After careful inspection we decide all is good and run ansible-playbook -K main.yml. Congratulations, you now have Spotify and iterm2 installed!

One small improvement to make before we move on is to extract these variables that are specifically for homebrew to their own var file. While it might seem silly now, sooner or later we might be using many roles that utilize many different variables and mixing them will lead to a lot of confusion. I personally like to name the variable files after the role they’re used for as illustrated below.

Managing OSX Settings

You can do a lot of tweaking to how your OSX behaves by using the osx_defaults module to manage OSX Defaults. There’s a lot of opportunities here but I’ll just leave a quick and dirty example to set our preferred screensaver below.

You could possibly even go as far as using this to manage various applications you have installed and possibly even setting registration keys for those applications. I haven’t even gotten to that point yet either so I’m not covering it here.

Further Reading

Well I hope this was good for you… it was good for me and helped me flesh out some of my current setup. I’m still on my path to learning how to best utilize ansible to manage my development environment so there’s definitely more to learn that I’ll continue to share as time progresses. I’m also not ignorant of a few other projects that aim to make working with ansible to manage development environments easier and one I’ve been looking at is Battleschool.

You can find the completed work for this blog post on github at jamescarr/ansible-mac-demo.

Crouching Supervisor, Hidden File Descriptor Setting

Here’s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances that are used to load balance to nodes in our RabbitMQ cluster. We’ve done this a lot of times before and set all the usual user settings required under limits.d to ensure proper file descriptors are allocated for the haproxy process. While creating this new role we also decided to use supervisor to supervise the haproxy process as it was previously observed in an older release that it didn’t automatically restart when it crashed (which in itself is a rarity).

Everything looked solid and we began throwing some traffic at the new balancer. Eventually we discovered something had gone horribly wrong! Tons of connection refused errors began showing up and the behavior exhibited was what one would expect if file descriptors weren’t being allocated correctly. Sure enough a quick look at /proc/<pid>/limits revealed that maximum open file descriptors were set to the very low value of 1024. We directed traffic back to the old balancer and began the investigation. How could this be? All of the settings were correct so why is it being set to 1024?

Supervisor was one new variable in the mix so I decided to begin pursuing the supervisor documentation and scanning for the number 1024 to see what might be tied to that. Sure enough, I came to discover the minfds setting. Let’s take a look at what the supervisor documentation has to say about this setting.

The minimum number of file descriptors that must be available before supervisord will start successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the supervisord process to satisfy minfds.

The hard limit may only be raised if supervisord is run as root. supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be obtained from the OS, so it’s useful to be able to specify a minimum value to ensure it doesn’t run out of them during execution. This option is particularly useful on Solaris, which has a low per-process fd limit by default.

Default: 1024

Well that doesn’t make much sense… if I’m reading this correctly it’s simply saying that the number specified is the minimum that should be available, right? The devil as they say is in the details. If we look at the documentation on setrlimit we’ll clearly see that this will actually set the limits without any reservations on what it currently is. The call basically is going to set max open files to whatever the value minfds is defined to in supervisor. Sure enough, as an experiment I set minfds in supervisor’s configuration to a higher number and after restarting supervisor the number of open file descriptors allocated to the haproxy process were greatly increased and reflected what minfds was set to.

In the end this pain also turned out to be unnecessary. While we had used supervisor because it was “what we know well” it turned out that the newer distribution we were releasing on already managed services via systemd which by default was also configured to respawn on failure.


Hopefully this story will prevent a similar trail of sorrow for others who may encounter the same situation!

TLDR; If you’re having supervisor supervise an application that is sensitive to max open file descriptors you’ll want to ensure minfds is set to match!


It is interesting what you learn when you have a five year old going through kindergarten and how much the observations you make seem to reflect on life in general. While I was standing in the hallway after dropping Caitlynn off for class I was looking at the different drawings with descriptions written out by each student in her class and took notice of the interesting ones. One kid likes to curl the end of each letter that seems to be nudging towards a beginner’s journey into calligraphy. Another student uses very distinctive block letters in his writing.

In the weeks that followed I began to notice something in the work Caitlynn was bringing home. One page she used letters that exactly mimicked those written by the student who used fancy curls at the end of her letters. Another week, she used block letters and within the same week had a paper that mixed both in a way that created something truly unique. As time progressed her writing keeps developing in ways that seem to incorporate what she has seen other students do and her drawings also follow what she observes and then evolves into something truly her own.

That’s when it really struck me, something very obvious but I never took the time out to think about it. A very large portion of our behavior, knowledge and actions are learned by mimicking others. We take the knowledge and experience of others and make it our own, using it as a base to build off of. As we encounter new situations and different approaches we continue to take those experiences and add them to our own experiences. Who we chose to surround ourselves with and who we look upon as our role models heavily influence who we turn out to be.

In engineering I find this very important as a way to acquire new knowledge. We’re influenced by the people we work with or even by reading the source code of projects we admire on Github. Whether it’s different tools like tmux or certain patterns and practices we observe others using we stand back and think “Huh, that’s clever.” and find someway to incorporate it into our behavior.

It is also important to keep in mind that just as we’re influenced by the behavior of others we too are influencers. That’s why it’s important to take the experiences we learn and find a way to share them, whether it’s a blog post or a simple Github project that shares a best practice you’ve learned. You have to ask yourself if you want your discoveries to be left in the dark or available for others to learn from?

Therefore I have decided to dust off this old blog and begin blogging again because I miss sharing meaningful content. I think there can be a lot of excuses as to why I’ve dropped the ball for so long to provide useful content here but the truth is none of those reasons matter. What does matter is that I’ve decided to challenge myself this year by posting as much as possible even if it’s just a summary of some small discovery I made that day. The blog has a new lease on life again so stay tuned!

Open a Github Issue From Slack!

The other day one of my co-workers opined that it’d be fantastic if we could open a GitHub issue from Slack. Fifteen minutes later the channel got to bask in the awesomeness… of this!

Read on to discover how to use Zapier (shameless plug: yes, I work on this) to whip this up quickly as well!

Opening the Issue

First up, we need to log in to Zapier and set up our first of two Zaps, the one that will create a new issue from Slack.

Now we’ll select our two services and the desired actions:

Next up, connect Slack and GitHub to Zapier.

When we get to step four, we’ll want to setup a custom filter so that we only trigger on Slack messages that contain !gh_issue.

At step five we’ll want to plug the values in to the GitHub issue from Slack. If you scroll back you’ll remember we used a specific format for our issue:

!gh_issue title(Junk Issue) description(Junk Issue!) repo(zapier/zapier-infra)

In Zapier-land, we extract those elements with parenthesis as variables. So when pulling from the trigger we get the raw text and the extracted variables as names like {{text__title}}, {{text__description}}, etc.

At step six we’ll load some samples.

Hrmph. All filtered out. Ah! We haven’t actually tried to create an issue from Slack. Let’s go do that now!

Now we go back to step six and refresh and we should see a new unfiltered sample, of which we can click “See filter sample” to view what will go to GitHub.

Looks good! Let’s go ahead and click “Test” and check that the GitHub issue was created on GitHub.

Great! Let’s go ahead and name this Zap!

But that’s only half the story. It’d also be nice if there was some notification in the channel that it had been created. Not 100% needed, but it would be nice!

The Webhook

So we have multiple ways we could approach this here:

  • Create a Zap that polls GitHub issues and alerts the channel of new issues
  • Setup a webhook through Zapier to push new issues instantly to Slack
  • Use the native Slack/GitHub integration on Slack to send the new issue notification

I’ll admit I didn’t have much luck using the native integration despite wanting it to work as it would have required the least amount of setup. Polling was easy to setup, but it means I can have anywhere from a 1 minute to a 15 minute delay from when I open the issue to when it is published back to Slack. So I opted for the webhook route.

The Webhook Trigger on Zapier is immensely powerful. You can use it to poll a URL, catch incoming webhooks, and even send webhooks back out to other services. It’s pretty raw but it gets the job done, and it gets it done instantly.

Like last time, for step one we will select our services: Webhook to Slack!

In step two, we’ll be given a webhook we can copy and paste to plug it into GitHub. Let’s navigate to GitHub really quick to add it.

In our repository settings page on GitHub, let’s add a new webhook.

By default this will fire on all events. We don’t want that, we want each issue.

This will be grayed out until an event fires, so let’s go back to Zapier and continue working on our Zap.

On step four, we’ll want add a custom filter so that the Zap will only trigger when issue action is equal to “open”. Otherwise this will fire whenever any activity takes place, such as opening and closing issues.

The first time through you may get a modal pop up prompting you to go create a new issue when you try to select a field. This is because webhooks are instant and require a user interaction to take place first. So go create an issue (manually or from Slack, it doesn’t matter) and follow the instructions to get it caught by Zapier. Now we can select the field we need and move on. 🙂

At step five it’s time to set up the channel the message will be sent to and what the message will be. I typically prefer to alert the channel of a new issue opened on a repository and then link to it.

There is also a field for Icon URL that can be used to plug in a specific icon for the Slack bot that broadcasts the message. I usually use a character of ours (Zapbot!) that is similar to Hubot, but Octocat fits well here too!

Now we’ll test the Zap and if all goes well, name it and set it live!

Whelp that wraps it up for us… hope you find these Zaps as useful as we have!