On Sunday 09 November 2008 01:12 pm, Robert Klemme wrote:
> Sorry to disappoint you but this amount of copying won't be really 
fast 
> regardless of programming language.  You do not mention what a 
"source" 
> in your case is, what operating systems are involved and what 
transport 
> media you are intending to use (local, network).  If you need to 
> transport using a network in my experience tar with a pipe works 
pretty 
> well.  But no matter what you do, the slowest link will determine your 
> throughput: you cannot go faster than network speed or the speed that 
> your "sources" can read or write.
> 
> Here's the tar variant, since you copy images I assume data is 
> compressed and does not need compression (on your favorite Unix shell 
> prompt):
> 
> $> ( cd "$source" && tar cf - . ) | ( ssh user@target "cd '$target' && 
> tar xf - )
> 
> If you can physically move the source disk to the target host and then 
> do a local copy with cp -a that's probably the fastest you can go - 
> unless the physical takes ages (e.g. to the moon or other remote 
locations).

I agree with Robert, but before I saw his response I did some 
calculations.  Assuming all the images are the same size (about 200 
KB), moving 4,211 of them in 47 seconds is a data rate close to 18 
MB/sec.--that's faster than a 100 mb/sec Ethernet, not counting any 
overhead due to collisions.

That's pretty fast for most channels.  Are you moving data from one disk 
to another on the same computer?  Or over a high speed connection 
between two computers?  What is the raw hardware speed of the 
interconnect?

I wouldn't be too worried about the 13 hours, you've got a lot of data 
to move.

Randy Kramer
-- 
I didn't have time to write a short letter, so I created a video 
instead.--with apologies to Cicero, et.al.