If by "optimization" you mean compression, see here:
http://www.simplesystems.org/RMagick/doc/comtasks.html#compressing

If you mean color reduction, see here:
http://www.simplesystems.org/RMagick/doc/image3.html#quantize