The hype cycle is at a high for cloud computing, distributed “NoSQL” data storage, and high availability map-reducing eventually consistent distributed data processing frameworks everywhere. Back in the real world we know that these technologies aren’t a cure-all. But they’re not worthless, either. We’ll take a look behind the curtains and share some of our experiences working with these systems in production at SimpleGeo.
Our stack consists of Cassandra, HBase, Hadoop, Flume, node.js, rabbitmq, and Puppet. All running on Amazon EC2. Tying these technologies together has been a challenge, but the result really is worth the work. The rotten truth is that our ops guys still wake up in the middle of the night sometimes, and our engineers face new and novel challenges. Let us share what’s keeping us busy—the folks working in the wee hours of the morning—in the hopes that you won’t have to do so yourself.
Mike works on distributed storage at Urban Airship (by way of SimpleGeo). His job entails figuring out how to store and query large amounts of data in a horizontally scalable data store, while ensuring that the system runs smooth as butter and is easy to operate.
Before that, he spent 3 years at Yahoo! performing a variet of roles. Initially hired to the Ops Tools team to work on infrastructure projects such as DNS automation, he then went on to spread his time around on automating abuse/spam prevention, rebuilding the offline batch processing system, optimizing database queries, building out high-profile integrations, and rescuing kittens at Flickr.
When not writing code, Mike spends his time playing soccer, occasionally giving talks at conferences, and learning about how computers work.
Derek Smith is a recent graduate from Loyola Marymount Univiersity. With early guidance from a few CS professors, he began his Senior year with the iPhone SDK. As a student developer he earned a WWDC Student Scholarship. After college, Derek began working as an iPhone Dev consultant which lead him to his current position at SimpleGeo. At SimpleGeo, Derek was given the opportunity to explore different implementations for providing an augmented reality environment for the iPhone.
Paul is the Operations team lead at SimpleGeo where he works to build and maintain fully automated and fault-tolerant systems to support the company’s location platform. Before joining SimpleGeo, Paul brought systems automation to Digg, building out configuration management for 500+ servers. Since joining SimpleGeo, Paul has been focused on systems automation and integration while trying to maintain stability in the face of rapid growth and obscure Linux kernel bugs. In his off time, Paul enjoys tinkering with new technologies, playing tabletop RPGs, and chasing his toddler around the house.
Mike Malone is lead architect at SimpleGeo, where he works on building and integrating scalable systems that power the company’s location platform. Since joining SimpleGeo, Mike has been working to ensure operational continuity in the face of rapid growth, partial system failures, and traffic bursts. Before joining SimpleGeo, Mike helped build the microblogging web site Pownce, where he learned a lot about the technical and social difficulties of scaling an online community. After Pownce’s acquisition by Six Apart in 2008, Mike worked on the TypePad platform team, where he gained a great deal of experience building RESTful web services. In his spare time Mike enjoys tinkering with new technologies. When he’s not on the computer, you can probably find him hanging out with his girlfriend, Katie, and their friends at a good bar.
Comments on this page are now closed.