Tom Machinski wrote:
> On 10/28/07, ara.t.howard <ara.t.howard / gmail.com> wrote:
>> i'm reading this as
>>
>>    - need query
>>    - need readonly
>>    - need sorting
>>    - need fast
>>    - need server
>>
>> and thinking: how isn't this a readonly slave database?  i think that
>> mysql can either do this with a readonly slave *or* it cannot be done
>> with modest resources.
> 
> The problem is that for a perfectly normalized database, those queries
> are *heavy*.
> 
> We're using straight, direct SQL (no ActiveRecord calls) there, and
> several DBAs have already looked into our query strategy. Bottom line
> is that each query on the normalized database is non-trivial, and they
> can't reduce it to less than 0.2 secs / query. As we have 5+ of these
> queries per page, we'd need one MySQL server for every
> request-per-second we want to serve. As we need at least 50 reqs/sec,
> we'd need 50 MySQL servers (and probably something similar in terms of
> web servers). We can't afford that.
> 
> We can only improve the queries TTC by replicating data inside the
> database, i.e. de-normalizing it with internal caching at the table
> level (basically, that amounts to replicating certain columns from
> table `bars` in table `foos`, thus saving some very heavy JOINs).
> 
> But if we're already de-normalizing, caching and replicating data, we
> might as well create another layer of de-normalized, processed data
> between the database and the Rails servers. That way, we will need
> less MySQL servers, output requests faster (as the layer would hold
> the data in an already processed state), and save a much of the
> replication / clustering overhead.
> 
> -Tom
> 
> 
MapReduce and Starfish?