Saturday, January 23, 2010

Heroku: The Good, the Bad, and the Ugly

After over a year as a happy customer with A2 Hosting, having Kitsch'nware running on one of their inexpensive shared host plans, I decided it was time to move on. Nothing against A2--they've been great, service is solid, support is EXCELLENT, and the price is right. But it was really only ever meant as a staging/development server, for a few beta testers to use--I knew it wouldn't hold up under any sort of load. So I wanted to put it somewhere else, but had a few requirements:

  1. The new server needed to be dynamically scalable. Right now Kitsch'nware isn't being heavily used, but I'm almost ready to start opening it up to more users, and I want to be able to crank up the server power on the fly as necessary. The flip side is that right now it's still a low traffic site, so I don't want to pay a bunch of money for this flexibility until I actually need it. But when the time comes, I don't want to have to re-deploy or mess with anything. Heroku fit the bill--for Kitsch'nware's basic functionality, it may actually cost me a little less than I was paying at A2, which I didn't expect. (Another bonus is that New Relic's Bronze monitoring package is included for free, which would normally cost at least $50/month!) And scaling it up is as easy as going to their web page and moving a slider. Way cool!
  2. I didn't want to have to manage the server. I could have stuck with A2, and moved up to a virtual server, but I don't want to manage a Linux box on top of everything else. Kitsch'nware is a one man show right now, and I already have way too much work to do on it without having to worry about dealing with server details too. I just need to be able to deploy and run my Ruby on Rails app. Also moving to an A2 virtual server would have been a guaranteed higher cost, even if I wasn't using the extra capacity, and should things *really* take off later, there would have been another move to a dedicated server. Heroku makes this sort of thing really simple, and you don't pay for what you're not using.
  3. I wanted to have access to memcached. A2 doesn't offer that on their shared hosting, only on virtual/dedicated servers, which as I said, I didn't want. Heroku is currently beta-testing memcached, and it's supposed to be going into production soon, so I figured that would work. If I could get into the beta, cool (and I did), and if not I could wait a little longer for it.

So sorry, A2--you were great, but just didn't offer what I needed. Sorry!

I actually looked into Heroku when I originally put Kitsch'nware out into production. At the time (late 2008/early 2009?) it wouldn't run at all. I don't remember what the issue was, but I think I was using some RoR feature, or maybe some gem that wasn't supported. There was also the problem that SSL wasn't supported, and I don't think custom domain names were supported either (but I could be wrong about that). At any rate, it didn't meet my needs. Last October when I was at Aloha On Rails, I helped out with their RoR class and got to play around with Heroku again. Their deployment process was really slick, but I wasn't sure if Kitsch'nware would run on it. During the conference, Heroku's Blake Mizerany (@bmizerany) gave a great talk on Heroku, and I filed a bug to test it out. (BTW: A shout out to Blake to thank him for pinging me on Twitter when I posted I was having Heroku issues.)

Time went on, and I didn't try Heroku for a few reasons. First, because I was using Beanstalk's Subversion hosting for source control, and Heroku requires Git. The rest of the RoR world moved on to Git long ago, but I started with Subversion and Beanstalk for a number of reasons, and didn't have any compelling reason to switch. Second, I just hadn't needed to scale Kitsch'nware. So I plodded on--everything worked, no reason to mess with it. But recently I decided it was time to get Kitsch'nware ready to go to the next level, which meant making it scalable. Heroku looked like the best option, so step 1 was converting to Git. I set up a GitHub repo for Kitsch'nware, and after a few attempts got my SVN repo imported the way I wanted it. (Some articles I found via Google, such as this one helped greatly, especially with correcting the author history.) Note: Beanstalk has been testing a Git system, but I didn't think it was available yet, so jumped to GitHub. They contacted me on Twitter after that, telling me it was available if I wanted them to enable it on my account. If I'd known ahead of time, I might have done that, but now that I'm already switched I'm staying (even though I now have to pay for GitHub, where Beanstalk was free for my little repo). Sorry, guys!

Once that was done, I tried deploying it. Had to fix some things, like getting gems set up the way they do it (really cool, actually), switching from the Whenever gem I'd been using to set up cron jobs to Heroku's cron system (also very cool), and some other minor config/deploy types of changes. To my shock and glee, it worked!

So the next step was to switch my domain over, and get SSL set up--this is where things got ugly. To be fair, some of my headaches were caused by domain registar issues, which isn't Heroku's fault. (Let me preface all this by saying Heroku could use some work on their documentation--I'm going to try to outline everything I came across here in case somebody there sees this and wants to fix it.) Here's more or less what happened:

  1. I followed Heroku's custom domain instructions to get working. Their page could use to be reorganized--right at the beginning they recommend using their Zerigo DNS plug-in, but the instructions to do that are at the bottom of the page. Minor detail, but it seems like the recommended solution should be front and center. :) Anyway, it turned out my registrar (A2 hosting, I transferred the domain to them when I started using their hosting) has a really shitty system for changing DNS info. They just assume you'll be using their hosting if you're using them as a registrar, so what they allow you do is minimal. I was at least able to put in the Zerigo DNS info, but it didn't seem to work. I think the issue in the end was DNS caching/propagation, as I noticed using a different machine the next day that I was suddenly able to hit, even though I still couldn't from my MacBook Pro. A little time and rebooting the MacBook eventually fixed it.
  2. When I changed the DNS entries at A2, it apparently nuked extra DNS info I forgot was there, like MX entries. I use Google Apps for Kitsch'nware's email accounts so I get custom email addresses, but mail gets routed to Google rather than my web host. This was something I must have set up before moving to A2, and it just happened to keep working when I transferred the domain to them. (Previously I had it registered with 1&1, a registrar I've used for years that allows you to change whatever DNS settings you like.) I'm in the middle of transferring the domain back to 1&1, but things were slightly complicated because I'd set it up as privately registered. So hopefully that will be fixed soon, and I can get my custom Gmail up and running again, but for now all email has been dead going on a week, which pisses me off. Not Heroku's fault, though.
  3. I had things set up at A2 so you could go to either or, and either way you would get redirected to (which is the domain my SSL certificate is for). When I don't have my SSL cert installed, both domains seem to resolve correctly. Though I just removed the cert to try to repro this, and it didn't happen. :( Maybe this actually had something to do with my Heroku custom domains add-on settings? I switched it back and forth between and a few times while trying to get everything working. Right now it's set on, because that's what my SSL cert is for, but it means that anybody trying to hit just gets a lookup error. Lame. I'm hoping this can all be easily remedied by getting the DNS settings at 1&1 set up correctly (using an A record like Heroku mentions on their custom domains page), but until the domain transfer happens that domain is dead. It's possible adding to the custom domains list might make the URL work, but login won't work because of the SSL cert being for So not really Heroku's fault here, and I should be able to fix it once my domain is transferred, but it would be nice if Heroku had some sort of redirect set-up where you could just automatically send 'www' (or any subdomain maybe) to the root, or vice-versa to support a single SSL cert. (I also tried setting up the Heroku wildcard domains add-on, and it didn't work for me. Not sure if this was me trying to do something unsupported, me using it incorrectly, or the documentation being incomplete/wrong.)
  4. SSL. This turned into another head-scratching adventure. After digging up my cert and private key from A2, I used the Heroku command line tool to install the cert. At the time, I had my custom domain set up as '', so that caused some headaches. I finally got that sorted out, so the custom domain and the SSL cert match. So I thought everything was good, until I tried to log in with Safari instead of Firefox. All of a sudden I got a security error saying the cert wasn't signed by a valid authority. WTF? The cert is from GoDaddy (cheap, and they work!) and I never had any issues with it on A2. So I figured the problem was with my Heroku setup, not the cert itself. Did some Googling and found a gotcha when using GoDaddy certs with nginx servers (like Heroku) and Safari. Applied the fix from that page (combining the two cert files you get directly from GoDaddy, who don't provide nginx in their list of servers when you download the cert) and bam! things started to work. This is something that Heroku really needs to put on their SSL documentation page, so I'll see if a can at least make them aware of this bit.

    • Related note about Heroku SSL: Using SNI so you can run your own cert on their system means that XP users using IE will see a cert warning, which really sucks. I suspect Kitsch'nware will have more than a few users using XP/IE, so I'm not sure how I'll handle that yet. It isn't Heroku's fault at all--it's a Microsoft problem, and there is a fix, but it costs $100 a month at Heroku. :( It's just not important enough of an issue (yet) for me to do that, but this is something that worked just fine at A2 and I'm having to give up moving to Heroku.

Some other notes on Heroku's documentation:

  • Their documentation pages that talk about 3rd party services, like their New Relic page and Zerigo page don't mention whether you need to create an account with that 3rd party. I'm pretty sure you don't, though I have them anyway (I already had accounts on New Relic and Exceptional). I wound up setting up a Zerigo account I didn't need because when I couldn't get things working, I thought maybe that was required. This would be a nice thing to add to the documentation just to avoid any confusion.
  • When you do a Git push to Heroku, you can only push from the master branch. I had set up a 'heroku' branch to keep my A2/heroku versions separate during the transition, and kept trying to push it, but then my changes just wouldn't show up on the live site! I don't remember what error, if any, the Git push command gave me, but I was totally stumped until I did a little digging through the Heroku docs. Don't remember where I found it, but there was some little thing somewhere that mentioned this. It probably should be a little more prominent in the documentation, as I'm sure I'm neither the first, nor the last person to try this. :)

    UPDATE (01/24/10): According to @telinnerud: ' To push a branch other than to master to Heroku do git push heroku some_local_branch:master'. I haven't tried it myself, I'm just using master for Heroku now, and actually I don't need to maintain two branches as I've switched over completely.

So that's my adventure with Heroku thus far. It's been frustrating, but not really their fault in the end. Things are very close to working, and hopefully will be totally fixed once my domain transfer finally goes through. Heroku is a cool service, and I'm looking forward to being a happy customer. :)


Frank N. Stein said...

I have tertiary syphilis.

Daz said...

Thanks for the post. This will come in handy for me in the near future.

Walker said...

Keep in mind this post is pretty old now; Heroku has changed a lot. I've been running Kitsch'nware there with (pretty much) no issues all this time since they got a few things straightened out. Since then they've also added some great stuff like memcache. I'd wholeheartedly recommend Heroku now.