Ninja magic with AWS command line tools

AWS has one of the best CLIs of any cloud platform. The recent Python-based awscli release is just a pleasure to use because it outputs in JSON by default. This means you can use a tool like jq to put together a power tool-chain that acts on JSON data. For…

Log queries in Postgres on Mac OS X

Sometimes, it's useful to log queries during development in Postgres. Assuming you installed Postgres using brew install postgresql, here's how you configure logging. Open /usr/local/var/postgres/postgresql.conf in your favorite editor and add these lines: logging_collector = on log_directory = 'pg_log' log_statement = 'all' Restart Postgres…

Appuri is co-organizing the Seattle Spark Meetup

At my company, Appuri, we are excited about the possibilities Spark opens up, so we are putting our money where our mouth is. We are co-organizing the Seattle Spark Meetup. True to the spirit of Spark, this will be a community effort. We are partnering with the formidable Denny Lee…

Setting up Spark on IntelliJ with Maven

In a previous post, I talked about how to set up IntelliJ for Spark development using sbt. There is an alternative way to accomplish this using Maven. Simply add Maven to your IntelliJ project and add this snippet to your pom.xml: <dependencies> <dependency> <groupId…

Setting up IntelliJ for Spark

Appuri is hiring data scientists, Java/Scala developers as well as front-end devs proficient with modern frameworks like AngularJS. Please drop me a line at bilal at appuri dot com if you are interesting in learning more. As a “reborn” Java developer, I often find myself struggling with project setup…

How to Process a Million Songs in Seconds: With Shark

Appuri is hiring data scientists, Java/Scala developers as well as front-end devs proficient with modern frameworks like AngularJS. Please drop me a line at bilal at appuri dot com if you are interesting in learning more. TL;DR Shark gives SQL-like access to big data with the speed advantages…

How to process a million songs in seconds

Appuri is hiring data scientists, Java/Scala developers as well as front-end devs proficient with modern frameworks like AngularJS. Please drop me a line at bilal at appuri dot com if you are interesting in learning more. PS: This excellent blog post inspired us to try Spark with the 1…

My day with Windows Azure

Some of you might know that I used to work on Windows Azure. We are evaluating technology options for my startup, so I decided to host our public-facing website on Windows Azure. It was a little surreal to use your own product as an outsider. Here are some key takeaways…

Don't hire by the job description

I've written a couple of job descriptions recently for a program manager role my team is hiring for. Here's a random extract from the posting: "Ability to lead change and drive own initiatives..." "Eye for design..." "Cross-team skills..." I hate to break it to you, but even if you do…

Stupidly simple == simply stupid

I picked up a copy of Wes McKinney's excellent Python for Data Analysis a few days ago. The book recommends IPython, NumPy and matplotlib, which are usually easy enough to install on Linux. Surprisingly, I spent the better part of the morning getting all of these packages to install and…