山本です。

>fnmatchに合わせてはどうでしょう？

こんな感じでいいでしょうか。でも、よく見るとコメントがありますね。

修正後：

  E:\ruby-cvs\ruby>miniruby -ve "puts File.fnmatch('[', '[')"
  ruby 1.9.0 (2004-02-18) [i386-bccwin32]
  false

  E:\ruby-cvs\ruby>touch [
  Touch  Version 4.2  Copyright (c) 1998 Borland International

  E:\ruby-cvs\ruby>miniruby -ve "puts Dir.glob('[')"
  ruby 1.9.0 (2004-02-18) [i386-bccwin32]


Index: dir.c
===================================================================
RCS file: /ruby/ruby/dir.c,v
retrieving revision 1.107
diff -u -w -b -r1.107 dir.c
--- dir.c	16 Feb 2004 01:33:12 -0000	1.107
+++ dir.c	18 Feb 2004 04:33:54 -0000
@@ -916,15 +916,8 @@
 	switch (c) {
 	  case '?':
 	  case '*':
+	  case '[':
 	    return 1;
-
-	  case '[':	/* Only accept an open brace if there is a close */
-	    open++;	/* brace to match it.  Bracket expressions must be */
-	    continue;	/* complete, according to Posix.2 */
-	  case ']':
-	    if (open)
-		return 1;
-	    continue;
 
 	  case '\\':
 	    if (escape && !(c = *p++))