Gameboy Development Forum

Exactly what are you trying to fit inside the case? Do I understand you correctly if I think you're planning to mold the plastic yourself for the new case? Or are you going to reuse the old case?
I don't think you'll be able to find a smaller regulator that does the same job. However, the regulator is a switching mode regulator with both buck and boost. This means that it can accept a wide range of input voltages. In a lab test I did it could accept anything form 1V up to 7V. however when feeding it with 3-4 V or lower (Don't remember the exact threshold) the screen was too pale to be useful.
If you're going for a Prosound, you can put the jack vertically on the left side of the regulator - there's enough space for that. For anything else, the pocket on the right side shouldn't be underestimated. Remove the speaker and the headphone jack for more space. (Again, if you're going for a Prosound mod, these are kind of unnecessary)
You could also mode the battery compartment. Either by reducing the total number of batteries to three or perhaps even 2. They'll drain faster when you do that though. Or you could replace the battery compartment with a rechargeable battery solution. That way you'll trim away a pretty big amount of space. The downside is that you'll have to design the battery circuit yourself. But if you can get 3.3 V or more, that'll probably be enough to power the GB's regulator correctly.