Re: How to use ReXML "in the wild"?

< ^ > P N |<>|^_>< --- | ~ ...Help
Hi Kenneth,
> I'd very much like to use ReXML's XPATH features to extract info from
> Google's financial info pages, but find that Rexml chokes on the
> Javascript, here's the result of trying to read in a page with this
> bit of code:

Don't try that ;) REXML in the wild == epic FAIL. At this level, you might
want to try Hpricot or Nokogiri. At a bit higher level, scRUBYt!
You can read about web scraping in Ruby here (my most succesfull article
ever, was even mentioned in Learning Ruby from O'Reilly):

http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/

> Is there a good way to get around this problem? If, not, I guess it's
> back to regular expressions...

Web scraping with regular expressions is almost never a good idea.

Try scRUBYt!:

require 'rubygems'
require 'scrubyt'

data = Scrubyt::Extractor.define do
  fetch 'http://finance.google.com/finance?fstype=ii&q=NYSE:WAT'

  body '/html/body' do
    revenue '/div[4]/div[2]/table/tr[2]' do
      ending_9_27 '/td[2]'
      ending_6_28 '/td[3]'
    end

    gross_profit '/div[4]/div[2]/table/tr[2]' do
      ending_9_27 '/td[2]'
    end
  end
end

puts data.to_xml

output:

<root>
  <body>
    <revenue>
      <ending_9_27>386.31</ending_9_27>
      <ending_6_28>398.77</ending_6_28>
    </revenue>
    <gross_profit>
      <ending_9_27>386.31</ending_9_27>
    </gross_profit>
  </body>
</root>


HTH,
Peter
___
http://scrubyt.org
http://www.rubyrailways.com