Category Archives: Airbrake News

1.8 Million Airbrake Gem Downloads

Airbrake hit one million gem downloads this week! Cheers and streamers! Fireworks! Champagne for everyone!

Our diligent frisbee packing, our international meet-ups, our long hours of coding into the night weren’t just for fun. You like us! You really like us! Or you just like being told when your site’s been bad. In any case, we’re excited that our family has grown so large, and excited to keep getting better and better.

To commemorate the Great Event, let’s take a retrospective walk down memory lane, shall we?

2008: Airbrake gets its humble beginning as Hoptoad, one of three (sort of) answers to the wide gaping gap in the market left by ExceptionNotifier for Ruby on Rails.

airbrake hoptoad

 

July 15, 2011: Hoptoad becomes Airbrake.

February 7, 2012:It’s publicly announced that Exceptional is acquiring Airbrake. The two combine their powers for good. Together, we strive for better, smarter exception-tracking, better developerment, better user experience, and a more valuable process for capturing and managing errors.

Simultaneously, our first international meetup as partners occurs in Berlin, Germany.

May 18 2012: The merger brings lots of new changes–good changes– including a frontend migration to Heroku to accommodate a rapidly growing user base.

September 7, 2012:We hit 500,000 downloads. It was a thrill at the time; little did we know we were to double that number in just five months’ time.

September 21, 2012: Airbrake makes the front page of GigaOm, both a marker of our dramatic progress, and a great piece of publicity.

February 15, 2013: We hit 1 million downloads! And that’s one million just as the newer incarnation, Airbrake. Over 800,000 Hoptoad downloads have slowly, step by step, leapfrogged us to this point.

Take a look at the commit history:

 

Additions and deletions per week:

 

All of which to say, THANK YOU! Thank you to everyone who’s spent time and effort maintaining or contributing to our gem, and all of our customers and users who have helped us improve the product. We have a lot of updates (JSON, Asynchronous) on the way and can’t wait to push forward! It’s been a wild ride, and we’re just getting started!

 

Airbrake & Exceptional Customers Infographic

Today we made the front page of GigOm. We’ve seen great growth in a last few months and completed major backend work on both products. We’ve been building a stable platform that will let us rebuild our notifications,  provide awesome search and a super blazing fast UI.

Today I’m happy to share these stats from the Airbrake and Exceptional user base.  A big thank you to all Airbrake + Exceptional users for making this possible.  Lot of amazing stuff is in the works.

@benarent and the @airbrake team.

Airbrake frontend migration to Heroku.

Airbrakers–thank you for your patience with our major May migration. You’ve asked us to share details of our new infrastructure and this will be the first of many posts where we peel off one aspect of our deployment and share our approach.

Today we’re going to talk about our Frontend.

From 2008 through 2011, the entire Airbrake codebase consisted of a monolithic Ruby on Rails application. Over time this application became optimized to handle an ever-increasing volume of customers.

Error growth since 2008

[ Graph showing increased numbers of exception processed by Airbrake. ]

At the end of 2011 we began a major backend rewrite that allowed the separation of the user-facing code (the “frontend”) and the mainly-api focusing code (the “backend”) that our 13 exception reporters use. Ruby, iOS, Coldfusion, Sinatra, .net, PHP, Android, Rack, Django, AppEngine, Redis, Merb, log4j, Chef, Erlang, Javascript, Java, and Drupal

Earlier this year we began testing our new backend with a subset of our users. At the same time, we looked for opportunities to allow us to make faster progress on our frontend. Giving us the ability to do things like WebHooks (in beta), GitHub Integration, Commenting, Better Debugging (Simulate Error) and more… (these all came to Airbrake in today’s release!)

Originally, we intended to host our new frontend (still Ruby on Rails) on Amazon EC2. We made this decision as our previous hosting facility was unable to keep up with our high-throughput needs–and our experience on Amazon in our other businesses (Exceptional and RightSignature) was positive in their ability to provision reliable service.

During our migration planning, we also brought up a staging server on Heroku (a PaaS that operates on Amazon). Amazingly–we had thought Airbrake to be sluggish when on our dedicated hosting provider’s servers; however, the Heroku app was significantly faster . It turns out that much of our performance was being wasted on calls within high-latency network calls to our support services (REDIS, Memcache, Legacy Mongo and our MySQL database). On Heroku (and therefore Amazon), these became low-latency, high-speed calls.

We spoke to the Heroku team and heard their commitment to making Heroku a platform that could handle Airbrake (at least our frontend) and we decided to use Heroku as one component of a robust deployment environment.

We have a commitment to demonstrating Heroku’s world class hosting environment by rocking Airbrake for their team and their most excellent users.

Oren Teich, Heroku COO.

 

With this support, we continued to focus on the components of our migration. We’ve added a pair of NGINX + HAProxy gateways on Amazon AWS XL instances (XL for network throughput). These serve Airbrake.io’s root domain. Requests to /notifier_api/v2/notices are routed to our new backend and the rest of the requests head onward to our frontend servers.

We launched two weeks ago with our own EC2 application pool as well as Heroku serving our application tier for our frontend. Unfortunately, our HAProxy configuration did not correctly account for session handling–and our ad-hoc attempts to remedy were causing more pain than relief–and during a fix attempt, we sent all frontend traffic to our Heroku application servers. We started to bump our Heroku dynos to match our usage and met demand at just over 75 dynos. We have identified a large number of our users have scripted access to /errors.xml and other popular endpoints. Having lots of these frequent requests necessitates a high dyno count. But, besides the dyno count, Heroku showed itself to be stably handling our application load (besides a few of these  that occurred as our initial 20 dynos were saturated).

We let this architecture settle in with our usage patterns and for the past two weeks we saw steadily faster-than-before access times for our users. We’ve been monitoring our support–and we have a few issues–which are mainly due to our new backend (next week!)–but our frontend has been blissfully consistent.

- Jonathan.

How Exceptional and Airbrake First Met.

Hear ye, hear ye! Sometimes rivers run backwards. Sometimes the sun disappears. Today two long-time competitors have become friends.

Airbrake+Exceptional joins forces.

That’s right. We could not be more excited to announce that the team behind Exceptional has acquired Airbrake. And–we’d like to give you the backstory.

 

The Beginning

Rewind the clock to 2008. You know the drill–we launched a website–and worried about finding user-facing issues before our users did. At the beginning of 2008, in the Ruby on Rails community, your only option was to install a plugin called ExceptionNotifier. This sent you an email on every website exception. If this wasn’t stuck in your junkmail folder–any site that saw traction also saw a stream of emails–many nearly duplicated–inbound with exceptions.

Exception Notifier 2006

Luckily three services were launched to solve this problem. You know two of them–Airbrake ( Hoptoad) and Exceptional. But you likely didn’t know about their ugly third cousin RightErrors.com.

All three services were offspring of Ruby on Rails service-companies. Airbrake came from thoughtbot. Exceptional came from Contrast. And RightErrors came from ELC Technologies.

contrast and thoughtbot team photo

OK. RightErrors didn’t officially launch. Mainly because while at ELC, Jonathan founded a bunch of new business concepts, and only a small fraction of them succeeded. In 2008 the new ELC businesses succeeding were RightScale and RightSignature–and they stole the thunder from any further development on RightErrors (or Farlanders, RightMirror, RightBoard and a few others :- ). Even though it didn’t launch, the seed was planted–not just for an error reporting service but for much more… hold that thought for now.

Growing Airbrake and Exceptional

The teams at Contrast and thoughtbot were finding that their side-business-error-handling-companies were getting great organic uptake.

“Scaling issues are often described as ‘great problems to have’—but Exceptional truly exploded and we often found ourselves staying up late to calm our newborn,”  jokes Eoghan, founder of Contrast.

And at thoughtbot,

“There were times we’d be getting slammed with 150,000 requests per minute, or with 500MB of session data, because some website was going crazy. Running Airbrake was like being under a constant DoS attack!”, exclaimed founder, Chad Pytel.

Getting hundreds of dollars a month on your service might feel like free money for the first few months–after 3 years, high-traffic scaling, supporting tens of thousands of customer support requests–the money stopped feeling “free.” And the whole endeavor became more of a distraction than a source of passion and inspiration. For Contrast, this was paramount…

Passion and 2011: By Jonathan.

In 2007, I moved to Ireland. I’ve 6 kids and an Irish wife–and there’s a romantic notion of living with the family in a quaint European city–like Dublin, Ireland. ELC had a solid team in Santa Barbara and I worked remotely + frequent visits. Anyone living in Ireland will tell you–it’s really a small Country. I managed to avoid the tech scene for the first two years in the Country. Then at the end of 2009 I got a phonecall from a random Irishman looking for space in Dublin for a Rails meetup. The caller was Paul Campbell. Many of you know Paul. He’s half of the creative force behind funconf as well as an early Contrast teammate and consultant to the stars of the Ruby on Rails community. We became instant friends.

Yes–this is all relevant–or equally irrelevant–but important for the telling of the story. But I can speed things up: 2010 was a year of change–I exited RightScale and started advising CloudKick; I sold ELC and started commuting one-day-a-week to Accel London. And in 2011 I moved to San Francisco with a small motley crew of teammates (an Irishman and Brit–who had been dating my au pair!) to mature my newest startup. As it happens–this one simmered rather than soared and by the end of the year fates collided.

Paul Campbell sent me a short note to meet up with Contrast’s founder Eoghan McCabe who was visiting San Francisco. Eoghan and I had great discussion about Exceptional and the future of development practices. And he shared that there was an opportunity for a new team for Exceptional.

“Jonathan introduced us to great suitors for the business. But their vision for the product remained at odds with our dreams for Exceptional. And planning to put the product completely in the hands of the new team, it was important to us that we were on the same page so that our customers would be properly taken care of.”

And so a deal was done–our team took up the Exceptional reigns–and freed from distraction, Eoghan’s team put-together the dream-launch of their new venture.

Airbrake

During 2011 the thoughtbot team also had a call to focus. They had launched CopyCopter, Trajectory and had a new project Apprentice.io brewing–all while running their thriving consultancy, leading best practices and maintaining Airbrake. And Airbrake was growing like crazy–attracting attention and needing attention.

copycopter trajectory apprentice.io

And late in 2011 the new Exceptional team met with thoughtbot to talk about future direction and potential cooperation. What happened next is exceptional. We started talking about the big picture behind our businesses. That the way we develop software has changed. That our tools are worse today than they were 10 years ago. And we were all passionate about seeing our products catalyze better development practices–and better lives–for developers.

For instance–how many of you are test-driven or BDD teams? How many use continuous integration? And how much do you monitor your code coverage–or keep builds from production from your build status? And if you’re doing such a good job covering your code and building your builds–why are you still using one of our services IN PRODUCTION?

Well–to us the answer is obvious. Yes, we want to be test-driven. Yes, we want to have our CI server always operational and blocking deploys. But in reality, only a fraction of teams have the diligence to implement these practices–and even those that do know that you will still find new exception-causing use cases once you let the mayhem of real life usage onto your product. And that is why you’re using us in production.

But that means that you are getting tremendous value from the exceptions you get from us. You can think of it as XDD (user eXperience Driven Development)–we give you a real-life cross-section of the areas of your codebase that need love and attention. Not mock-pain felt by mock tests–but real pain felt by real users. And by making these user experiences visible, you have a tangible agile metric–user experience pain–that you can track to help maintain the velocity that moves your features forward with the right level of end-user experience for your product.

And we don’t just need to be a metric for your iteration planning meetings. When we do collect an exception, we get all the environment details of what was happening when the exception occurred–and we can use this information to help you identify where in your code to focus AND to give you a starting point for the right kind of test to add to your test suite. This and much more vision came from our conversation with the thoughtbot team: graphing exception volume over time; complex filtering of exceptions (only see crawlers / only see logged in users); community-wide insight on exception resolution (gem combo causing common exceptions); better error page experience for users; and more.

By the end of the visit we had agreed to find a path for Airbrake and Exceptional to work together–and today we’re announcing that Exceptional has acquired Airbrake to focus efforts and bring the best next-generation development practices to our combined 75,000 development teams!

 

What’s Next

Firstly we want to thank Eoghan, Des and Ciaran and Chad, Dan, Matt, Joe, and Jon and all of the other contributors who brought to life Airbrake and Exceptional. You have been phenomenal to work with–we wholeheartedly support you and your new ventures. Yes, you do want to signup for Intercom.io right now to start hearing what it takes to find Premium users in your Freemium userbase. And–unless you miraculously have a glut of great developers and designers–you want to join Apprentice.io to have access to highly-trained up-and-coming talent. We are customers of both services.

And we want to thank Airbrake and Exceptional’s community. You are the heart and soul of the internet. Literally–you are Groupon and Square and Posterous and Patch and HubSpot and GrubWithUs and Pinterest and EngineYard and Uber and Monster and Friendster and Foursquare and Kickstarter and 74,990 others.

We have fantastic plans for 2012, but our first priority is to match the service and support you’ve come to expect. There are high-traffic, big data challenges and a tremendous variety of environments to support. This quarter we are committed to giving you exceptional support and building a strong foundation. Then each quarter this year we will be releasing major product updates that make our lives as developers better.

Upcoming Updates:

Along the way, we will be running outreach to our community. You’ll see us saying hi when you log in (thank you Intercom!) and you’ll see us in real life sponsoring local events. Here’s a brief calendar of where we’ll be and when:

  • Berlin, Germany – Today / Now! 
  • Dublin, Ireland – February
  • London, England – February (This Thursday)
  • Los Angeles – March
  • Krakow, Poland – April
  • Moscow, Russia – May
  • Beijing, China – June
  • Tokyo, Japan – July
  • Chicago – August
  • New York – Late 2012
  • San Francisco – Always

And more coming…

We hope to have the opportunity to meet and everyone of our users: to hear your feedback, to learn about your practices and creative passions–and share a pint.

Cheers!
@Ben , Stu , Colleen, @Jonathan, Helen, Fredrik and Teri

AirbrakeApp.com is now Airbrake.io

As you know we’ve had to migrate from our previous amphibious name and under pressure we moved immediately towards AirbrakeApp.com. Now that we’ve had time to explore availability, we’ve found a better, final home at: Airbrake.io.  You’ll see a new wildcard SSL cert* and domain name, but the service behind will work identically as before. We will continue to support gems pointing to airbrakeapp.com and hoptoadapp.com for the immediate future.

We’ve also migrated to an upgraded, speedy DNS service and added a dedicated Airbrake blog, airbrake.io ,  which you’ve found!

On the error-reporting side, you’ll see our error reporting libraries updated over the coming days–with our canonical Airbrake gem already updated. Please raise any support issues immediately to our attention at our support helpdesk or call us +1-888-479-8323. Yes–we’ve ramped up our support options and team to handle Airbrake’s growing user base. In fact, as we make these transitions, we wanted to share a few tidbits:

  • We are regularly handling over 100k requests per minute.
  • We’ve sent 35 Million emails since January 2010, and will send 4 Million more in the next 30 days.
  • We have handled 3 Billion Errors of which over 4 Million are unique.

On our team we’ve added a dedicated development team with a feature roadmap that includes your top requests. We’ll be writing an update with details later in the month as well as sharing some big news later in the year.

We are applying the final touches to our blog, please send your feedback and comments to Ben .

*Technical Footnote. The airbrake.io SSL is applied to a separate IP address  208.85.144.248. More information.