On Jun 29, 2009, at 2:00 PM, Mark Thomas wrote:
> I have a long-running batch job that I would like to speed up.
> Currently it uses only one CPU and the server I have in mind for this
> has 16 cores, and I want to take advantage of them.
>
> I'm thinking one of three possibilities:
>
> 1. JRuby, where the threads are native OS threads
> 2. A message queue (e.g. ActiveMQ + Stomp), where worker threads run
> as separate processes, thus using all cores.
> 3. A MapReduce implementation (e.g. hadoop)
>
> I would like to see if anyone has gone down this road and can weigh in
> on these options.

How difficult it your task? If you were able to reduce (heh) it to a  
MapReduce problem, you could use something like Skynet or Starfish.  
For even simpler forking, check out Ara Howard's threadify or my  
forkify for simple parallel processing.

- Lee