On Tue, 29 Nov 2005, Robert Klemme wrote:

> Hugh Sasse wrote:
> > On Tue, 29 Nov 2005, Robert Klemme wrote:
> >
> >> If Hugh is using ActiveRecord intensively with a database then it's
> >> most likely that he'll see no positiv performance effect from
> >> compiling it with more aggressive optimization.
> >>
> >> In fact it's likely that careful optimization on the database side
> >> will yield better results.  This can be as easy as creating some
> >> indexes - but might be much more complicated - depending on the
> >> bottleneck.  (Often it's IO and this might have several reasons,
> >> from sub optimal execution plans to slow disks / controllers.)
> >>
> > At the moment my script to populate the tables is taking about an
> > hour.  Anyway it's mostly ruby I think, because it spends most of
> > the time setting up the arrays before it populates the db with them.
> 
> How did you measure that?

By eye! :-)  The code doesn't access the database at all until the last
part, and it doesn't get there till about 45 mins.  But to be
honest, this is so slow it isn't worth benchmarking to get the
milliseconds.
     555    1676   17179 /home/hgs/csestore_meta/populate_tables2.rb
I could post the script if you like. I've not profiled it to find
out where the slow bits are because it would take about 5 hours
going by previous slowdowns when profiling.
> 
> > Besides that, I'm fairly new to database work, so I'm trying to
> > optimize what I know about before I start fiddling with the db.
> 
> Um, although I can understand your waryness with regard to the unknown -
> you may completely waste your time.  IMHO you should first determine the
> cause of the slowness and then find a solution.  If you optimize something
> that just takes 10% of the whole running time you'll never seen an
> improvement of more than 10%...

this is true.  This script isn't run very often though, so if it
takes an hour I can live with it. Taking 5 hours to profile it once would
be too long.
> 
> Another option to get masses of data into a database is to use some form
> of bulk insert / bulk load.  Depending on your database there are probably
> several options.

MySQL.  Part of the problem is that this script is also for
updating, based on new data.  If the db is empty it just inserts,
else it updates.  Easy enought in ActiveRecord.
> 
> > Slow disks/controllers (+ lots of users) could be a factor, the
> > machine is 5.5 years old.
> 
> Could be.  If possible spend it at least more mem.

Thanks.  I'm trying to get something done about that.
> 
> Kind regards
> 
>     robert
> 
        Thank you,
        Hugh
> 
>