Issue #6836 has been updated by luislavena (Luis Lavena). File improve-require-and-file-expand_path-windows.v2.diff added =begin usa (Usaku NAKAMURA) wrote: > Hello, > > If the performance problem is in 'require' and 'load', change only them > and be stayed File.expand_path the same behavior. > Can't do so? Thank you Usa for your feedback. Based on Hiroshi experiment I've worked in an updated patch (attached) This new patch shows the same performance boost on (({require})) without breaking backward compatibility of (({File.expand_path})) trunk: ruby 2.0.0dev (2012-08-23 trunk 36786) [i386-mingw32] Rehearsal ------------------------------------------------------ core_require_empty 1.264000 3.151000 4.415000 ( 4.446254) --------------------------------------------- total: 4.415000sec user system total real core_require_empty 1.154000 3.229000 4.383000 ( 4.432253) Rehearsal ------------------------------------------------------- core_require_nested 1.248000 3.447000 4.695000 ( 4.707269) ---------------------------------------------- total: 4.695000sec user system total real core_require_nested 1.467000 3.214000 4.681000 ( 4.699268) patched: ruby 2.0.0dev (2012-08-23 trunk 36786) [i386-mingw32] Rehearsal ------------------------------------------------------ core_require_empty 0.593000 0.936000 1.529000 ( 1.595091) --------------------------------------------- total: 1.529000sec user system total real core_require_empty 0.624000 0.936000 1.560000 ( 1.577090) ruby 2.0.0dev (2012-08-23 trunk 36786) [i386-mingw32] Rehearsal ------------------------------------------------------- core_require_nested 0.843000 0.998000 1.841000 ( 1.855106) ---------------------------------------------- total: 1.841000sec user system total real core_require_nested 0.764000 1.061000 1.825000 ( 1.838106) This also improves Rails and Rake startup time From (trunk, mid-size Rails app): V:\enki>timer ruby script\rails runner -e production "p $LOADED_FEATURES.size" 1135 real 22.710 system 11.013 user 11.575 V:\enki>timer rake -T ... real 8.868 system 0.031 user 0.000 To (patched): V:\enki>timer ruby script\rails runner -e production "0" real 10.735 system 1.716 user 8.923 V:\enki>timer rake -T ... real 3.068 system 0.015 user 0.015 Updated the Gist that contains the benchmark and patch: https://gist.github.com/3242245 Considering the size of the patch and to make it more easy to review, I've pushed to my fork on GitHub the individual commits and explanation of each change here: https://github.com/luislavena/ruby/compare/improve-require-and-file-expand_path Looking forward for your comments and feedback. Thank you for your time. =end ---------------------------------------- Bug #6836: Improve File.expand_path performance in Windows https://bugs.ruby-lang.org/issues/6836#change-28999 Author: luislavena (Luis Lavena) Status: Assigned Priority: Normal Assignee: nobu (Nobuyoshi Nakada) Category: core Target version: 2.0.0 ruby -v: ruby 2.0.0dev (2012-08-04 trunk 36616) [i386-mingw32] =begin (original write up in markdown here: https://gist.github.com/3242245) == Background While investigating the performance issues of (({File.expand_path})) on Windows, Usaku Nakamura and Nobuyoshi Nakada on [ruby-core:39504] pointed out that due security concerns, accessing files on Windows required normalized paths. This was covered in the security update of March 2008, WEBrick file-access vulnerability [1]. After closer inspection of WEBrick code (mentioned by the security update), I noticed it uses (({File.expand_path})) to perform the heavy lifting of path normalization in the request. The code around this can be inspected in (({prevent_directory_traversal}))[2] and (({shift_path_info}))[3] methods. This approach performs a hit into the filesystem, contrary to its implementation in any other operating system. (({File.expand_path})) is heavily used by (({require})), which result in slow application startup, depending on the application size or number of gems it depends on. Stepping back for a second, we can see that the security issue is around WEBrick and the way it determines (({path_info})) absoluteness. It is also clear that to solve WEBrick security issue, a tax has been applied to the entire Ruby ecosystem, penalizing startup performance. With Hiroshi Shirosaki's help, we worked on a patch that: * Limit filesystem hit only to WEBrick, using Windows' GetLongPathName [4]. * Use a Windows-specific API to normalize paths * Improve encoding support. What started as an experiment named Fenix [5] has shown great results on a variety of systems. This patch has been integrated in TheCodeShop [6] releases of Ruby 1.9.3 and tested by Ruby-Core developers Hiroshi Shirosaki and myself. == Performance To demonstrate the benefits of this patch, I've used measurements [7] project and both (({core_require_empty})) and (({core_require_nested})) workloads, obtaining the following results: ruby 2.0.0dev (2012-08-03 trunk 36611) [i386-mingw32] Rehearsal ------------------------------------------------------ core_require_empty 1.186000 3.385000 4.571000 ( 4.676267) --------------------------------------------- total: 4.571000sec user system total real core_require_empty 1.217000 3.385000 4.602000 ( 4.643266) Rehearsal ------------------------------------------------------- core_require_nested 1.514000 3.760000 5.274000 ( 5.305303) ---------------------------------------------- total: 5.274000sec user system total real core_require_nested 1.466000 3.713000 5.179000 ( 5.233300) And with patch applied: ruby 2.0.0dev (2012-08-03 trunk 36611) [i386-mingw32] Rehearsal ------------------------------------------------------ core_require_empty 0.765000 1.077000 1.842000 ( 1.887603) --------------------------------------------- total: 1.842000sec user system total real core_require_empty 0.717000 1.123000 1.840000 ( 1.887603) Rehearsal ------------------------------------------------------- core_require_nested 0.717000 1.670000 2.387000 ( 2.480405) ---------------------------------------------- total: 2.387000sec user system total real core_require_nested 0.890000 1.528000 2.418000 ( 2.496004) Benchmarks were performed all on the same hardware and OS: * CPU: Core 2 Duo T7500 @ 2.20Ghz * RAM: 4GB * HDD: 1.5GB RAMdisk (ImDisk) * OS: Windows 7 Ultimate x64 All tests associated (both File and WEBrick ones) pass. Additional tests that exercise specific aspects of new function were added. Patch has been tested also against: * Visual Studio build of Ruby * Ubuntu 12.04 * Mac OSX And the patch didn't affect either build or tests of it. === Real life impact: Rails The biggest Ruby project affected by this is Rails applications. An empty Rails application on startup requires more than 700 files from different gems: V:\emptyapp>ruby script\rails runner -e production "p $LOADED_FEATURES.size" 772 When benchmark startup using w32time [8]: V:\emptyapp>timer ruby script\rails runner -e production "0" real 7.285 system 4.539 user 2.683 And patched Ruby: V:\emptyapp>timer ruby script\rails runner -e production "0" real 2.620 system 0.873 user 1.700 (best result taken from several warm ups). Now, a mid-sized application like Enki [9] which loads 1146 files in production mode, result in: V:\enki>timer ruby script\rails runner -e production "p $LOADED_FEATURES.size" 1146 real 22.620 system 11.497 user 11.076 Almost ((*23 seconds*)), compared to patched version: V:\enki>timer ruby script\rails runner -e production "0" real 11.013 system 1.981 user 8.938 This change also improves performance of (({rake})) inside Rails, from: V:\enki>timer rake -T ... real 8.689 system 0.015 user 0.000 To: V:\enki>timer rake -T ... real 3.307 system 0.000 user 0.031 Making normal operations more accessible. Looking forward for your thoughts on these changes. Thank you. [1] http://www.ruby-lang.org/en/news/2008/03/03/webrick-file-access-vulnerability/ [2] https://github.com/ruby/ruby/blob/trunk/lib/webrick/httpservlet/filehandler.rb#L242-263 [3] https://github.com/ruby/ruby/blob/trunk/lib/webrick/httpservlet/filehandler.rb#L330-337 [4] http://msdn.microsoft.com/en-us/library/windows/desktop/aa364980.aspx [5] https://github.com/luislavena/fenix [6] http://thecodeshop.github.com/ [7] https://github.com/jonforums/measurements [8] https://github.com/thecodeshop/w32time [9] https://github.com/xaviershay/enki =end -- http://bugs.ruby-lang.org/