On Fri, 29 Sep 2006, Krishna Dole wrote:

> Hi all,
> If anyone is willing, I'd be grateful for some advice on the forking
> job scheduler I've written. It works fine in simple tests, but does
> not feel elegant. On IRC kbrooks recommended an asynchronous main
> loop, but i don't understand how to implement that in this situation.
> The first version I wrote used threads, but several sources
> recommended fork instead. I have also considered just using the shell
> command 'ps' to see how many jobs are running, launching more as
> needed.
>
> The basic requirements:
> - Each job is a long-running external process (taking a day or more)
> and all jobs require a different amount of time to run (so
> asynchronous launching will be needed).
> - I want to keep N jobs running at all times (N = 4 in the example below)

no need to reinvent the wheel!  ;-)

   http://www.linuxjournal.com/article/7922
   http://codeforpeople.com/lib/ruby/rq/
   http://codeforpeople.com/lib/ruby/rq/rq-2.3.4/README

download ruby queue (rq) and run it locally.  it does this and much, much more

# setup a work q

   harp:~ > rq ./q create
   ---
   q: /home/ahoward/q
   db: /home/ahoward/q/db
   schema: /home/ahoward/q/db.schema
   lock: /home/ahoward/q/lock
   bin: /home/ahoward/q/bin
   stdin: /home/ahoward/q/stdin
   stdout: /home/ahoward/q/stdout
   stderr: /home/ahoward/q/stderr


# start a daemon processs that will run 4 jobs at a time

   harp:~ > rq ./q start  --max_feed=4


# submit a job

   harp:~ > rq ./q submit echo foobar
   ---
   -
    jid: 1
    priority: 0
    state: pending
    submitted: 2006-09-29 08:49:46.814603
    started:
    finished:
    elapsed:
    submitter: jib.ngdc.noaa.gov
    runner:
    stdin: stdin/1
    stdout:
    stderr:
    pid:
    exit_status:
    tag:
    restartable:
    command: echo foobar

# wait a bit

# check the status

   jib:~ > rq ./q list 2
   ---
   -
    jid: 2
    priority: 0
    state: finished
    submitted: 2006-09-29 08:49:50.839391
    started: 2006-09-29 08:50:09.282754
    finished: 2006-09-29 08:50:09.798060
    elapsed: 0.515306
    submitter: jib.ngdc.noaa.gov
    runner: jib.ngdc.noaa.gov
    stdin: stdin/2
    stdout: stdout/2
    stderr: stderr/2
    pid: 721
    exit_status: 0
    tag:
    restartable:
    command: echo barfoo


# view the stdout

   jib:~ > rq ./q stdout 2
   barfoo


there is a command-line interface plus programming api - so you can almost
certainly accomplish whatever it is you need to do with zero or very little
coding on your part.


kind regards.

-a
-- 
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei