Are you using the Gem::Specifications#extesions method? in your
gemspec?  This builds the extension when you install the gem
automatically.  You just say

spec.extensions = ["ext/[whatever]/extconf.rb"]

in your gemspec