Ara.T.Howard wrote:

> On Sat, 21 Feb 2004, Hal Fulton wrote:
> 
> 
>>Ara.T.Howard wrote:
>>
>>>don't limit yourself to scripting tasks!  i'm currenlty designing an entire
>>>near-realtime satelite image processing system in ruby!
>>
>>Ara,
>>
>>That's pretty cool. What can you tell us about this without violating some
>>kind of confidentiality policy? :)
> 
> 
> ha!  check out the url in my sig for general info...
> 
> i can't elaborate much at the moment as i'm slammed with a deadline but the
> _design_ so far is:
> 
>   - various job classes utilizing image processing classes that use guy's mmap
>     module (considering some stuff using narray).  our algorithims are all
>     custom so no packaged lib would work.  eg. we aren't doing an normal image
>     ops - we go through pixels and scanlines applying custom algorithims to
>     them to detect stuff (nighttime lights of the world).  this is the
>     scientist's bag - i avoid it and consider only values 0-255 and issues of
>     massive i/o.  typical files are 100mb each and explode into about 1.5 GB
>     out output.  we currently have programs that _require_ 2.5mb of ram.  it's
>     absolutely ridiculous..
> 
>   - an entire system built around sge (sun grid engine) and seriously heavy
>     duty netapps and mass storage units - lots of data
> 
> 
> essentially we process incoming satelite data and archive it.  in addition
> certain clients subscribe to have us do custom data processing/delivery for
> them.  the current system - isn't - it's a massive pile of scripts that call
> scripts that call scripts...
> 
>   1024.times{ puts "scripts that call scripts" }
> 
> ...
> 
> i jest not.
> 
> 
> the central problems are
> 
>   - the processing is the 'science'.  eg. it changes almost daily
>   - the volumes of data and cpu requirements present all sorts of challenges
>   - setting a 'subscription' (custom processing for a customer) requires a
>     ground-up approach.  there is _zero_ reusable code currently
>   - the logging is non-existent
>   - we can't use a database (politics)
>   - our sysads are horrible
> 
> my current work has been
> 
>   - figure out a persistence strategy.  i wanted to use pstore for alot of it
>     but couldn't since it didn't support locking over nfs.  see
>     
>       http://raa.ruby-lang.org/list.rhtml?name=posixlock
> 
>   - configurable work flows see
> 
>       http://raa.ruby-lang.org/list.rhtml?name=flow
> 
>   - controlling various external process (idl, c program, fortran, etc) in a
>     way that allows fine control over output see
> 
>       http://raa.ruby-lang.org/list.rhtml?name=session
> 
>   - quite a few in-house packages to deal with various data formats and do
>     various calculations see
> 
>       http://www.codeforpeople.com/lib/ruby/dmsp/
>       http://www.codeforpeople.com/lib/ruby/envi/
>       http://www.codeforpeople.com/lib/ruby/solpos/
>       http://www.codeforpeople.com/lib/ruby/stpjob/
> 
> all the above is a literal snapshot of what i'm doing - as is normal around
> here i haven't had much time for docs...  sorry.
> 
> 
> at the exact moment i'm working on a configurable set of jobs that generates
> fire products using nighttime satelite imagery.  it's basically a testbed for
> what the near-realtime system might be but i am still very much in the
> 'enumerate the problem hotspots and check them off' phase.  i still haven't
> been able to check them all off so my 'design' ( a strong term ) is very much
> in flux - suggestions (espcially in form of solved.tar.gz) are welcome!  ;-)
> 
> 
> a year ago i designed a bi-temporal database emulator (ruby classes) and a web
> interface to it.  that system was almost all ruby and is in production here
> are noaa. see
> 
>   http://www.fsl.noaa.gov/publications/forum/feb2003/2_03_MMeta.html
>   http://raa.ruby-lang.org/list.rhtml?name=btpgsql
> 
> and specifically
> 
>   http://www.codeforpeople.com/lib/ruby/btpgsql/btpgsql-0.2.4/doc/
> 
> for quite a long discussion of the database part.  the web interface is,
> unfortunately, viewable only from the intranet.  it's based on fastcgi and
> postgresql though.
> 
> 
> 
>>Also, there's a "real world Ruby" page in the wiki somewhere.  This might
>>make a nice addition.
> 
> 
> when time allows...
> 
> back to work...
> 
> cheers.
> 
> -a
> 
> 
I didn't understand half of it. But that is what makes it way cool! :-)