> $ gem install rubygems-update
> $ update_rubygems

From what I can tell the update script copies files from the gem dir to
the site_ruby dir. Is it safe to remove the gem once the update is
finished? If yes, consider it a feature request to clean up the gem and
the update script itself.