I'll skip what Gavin has already covered and I agree with.

> * This is going to be commercial software, and I do NOT intend to give
> away the source code.  Is their a way that Ruby programs can be
> securely compiled so that the source code is inaccessible?

It seems to me that the language lends itself to being quite well
obfuscated, but Ruby users at the moment are largely developers and
open-source fans, so there isn't such a thing (AFAIK) as an obfuscator, yet.
Could be fun to write, for an experienced rubyist.

If you want "secure compilation", I believe this limits you to using C or
C++. Java and .NET, for example, "compile" into bytecode for a virtual
machine that is tailored to run them, and decompilers are readily available
that will produce quite legible source. Although, with something like
Perl/Tk, you may be able to use their B::C module/perlcc program (see
perlcompile manpage) to compile to native code via C.

Finally, comments are a significant part of the value of source code. When
you compile your .NET or Java app, you will strip the comments. In lieu of
an obfuscator, a simple script to strip the comments before packaging your
app is half-way there.

> * For that matter, is there a way that Ruby programs can be compiled
> as binary executables?  On Windows, Linux, and Mac?

Not that I know of. But there should be.

> * I will be using a shareware-validation protocol not unlike
> http://www.shareit.com. Will Ruby be compatible with this?

Probably. What interface is required?

> * If I am porting the entire program to Ruby, rather than just using
> it as a frontend to my standalone Matlab programs, are there good Fast
> Fourier Transform libraries for Ruby?

I did a quick bit of research, and found what you are looking for exists in
Ruby: Ruby/GSL is a Ruby interface to the GNU Scientific Library (GSL).
See:
    http://raa.ruby-lang.org/list.rhtml?name=ruby-gsl-1
    http://www5d.biglobe.ne.jp/~ytsune/Ruby-GSL/index.html
    http://www5d.biglobe.ne.jp/~ytsune/Ruby-GSL/fft.html

Hope this is of some help,
Dave