Here is a solution:

I have recently sucessfully compiled ruby under AIX 4.1.5!

First:
gmake realclean
(To remove the results of your unsuccesfull tries)

I removed the occurences of -brtl from the file in configure and 
configure.in, since it did not link correctly.

I used 
configure --without-gcc
(since xlc runs faster and produces stabler and faster code than gcc.)

I also had the same problem with ISDIGIT as you. I created a loadmap
file by not linking to ruby.a, but linking all object files.
After that, I recognized that the symbol is is in the "dln.c"

I replaced ISDIGIT by isdigit there. (Matz has included <ctypes.h> but it 
does not contain ISDIGIT but only isdigit in my version of AIX...,
perhaps I should have include ruby.h instead. I don't know. Ask Matz!)

(If you haven't removed -brtl from configure, then you get some errors
while linking.)

The next problem was the curses extension.

I had two undefined symbols.
I changed the makefile to linke ncurses instead of curses and
added the flag -L/usr/local/lib, since "libncurses.a" was located there.

I also had some linking problem with the socket extension.
I could only solve it by deleting the flag -DHAVE_RESOLVE, since it 
always gave syntax error in resolve.h (I think the configure.h and defines.h
are in error, but I don't have the time to trace it back.)
I think that some of the functionality ist lost, but it compiles and links 
at least.

After it compiled OK.

I also tried to compile it under 4.3.3, but there were always an error
in <sys/types.h> (even in the first .c file) so I did not try it any more,
but I think it should be fixed by someone with more knowledge about ruby.

Best Regards, Christian