> Hi Brian,
>   There is a module called 'webfetcher' in
> http://www.ruby-lang.org/en/raa.html

thanks for the tip, that looks like a good package and i will keep it in
mind for future projects.

my Crawler program is a lot less full-featured, and probably less robust too
(considering i just wrote it).  nevertheless i hope that it may be useful
for those with simple webcrawling needs.

-brian