On Fri, May 27, 2011 at 1:44 AM, Roger Pack <rogerpack2005 / gmail.com> wrote=
:
>
> yeah best bet is to spawn off multiple processes.

Though, I wonder if this is an improvement over the serial uploading.
Unless the upload rate is limited to a fraction of the total upload
speed for each process wanting to upload data (either on the uploading
or the receiving end), two files uploaded in parallel will use half*
the bandwidth, and thus upload just as slow or fast as the two files
in series.

Additionally, the program complexity rises: instead of keeping tabs on
successful uploads of one file, it's now n uploads that have to be
monitored and redone on failure.

But since engineering challenges are fun, I'd do it like this (if I had to)=
:

- Create an upload queue server containing all unfinished uploads.
- Spawn several worker processes that check the queue for the next
available upload (beware: race condition. A simple block-and-backoff
strategy would be sufficient to prevent that, I think), and mark their
assigned uploads.
- Once the upload is finished, the worker processes send the all
clear, and the upload gets removed from the queue.
- If a worker process cannot finish its upload for whatever reason,
the upload gets marked as unfinished again.
- If a worker finishes its upload, it marks the queue item for deletion.

Rails can use delayed_job <http://rubygems.org/gems/delayed_job> that
does something similar, so go ahead and steal what you can.


* Actually a little less due to a touch more overhead, but I doubt
that that is a significant factor for 99% of cases.

--=20
Phillip Gawlowski

A method of solution is perfect if we can forsee from the start,
and even prove, that following that method we shall attain our aim.
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0-- Leibnitz