That is exactly what I ended up doing on all the Ubuntu-based machines
I had. It was easier to install from source so I had current revisions
of Ruby, and to manage all my gems by hand.

When rvm came out I switched to it and now it's my favorite way to go
on Mac OS and Ubuntu because it handles the grunt work of maintaining
multiple versions of Ruby, allows gem bundles, and makes it trivial to
switch from one version of Ruby to another when testing.