The Rules engines compiler sometimes generates classes for the snippets of java semantics that you may use in a rule package. This allows the semantic java code to run as fast as regular code (as it all ends up as class files, which the runtime JIT can do its magic with).
The downside of this, is the increased load cost when building rules, and the associated memory overhead.
As typically you could have lots of rules, that means lots of small classes.
Lots of small classes mean that the JVM is loading lots of small classes. Some JVMs (like Suns Hotspot and IBMs) have a special part of memory (called PermSpace - permanent space) where loaded classes are kept. Sometimes you may run out of this, especially if there are lots of other utilities loading classes on the fly. You can increase the PermSpace from the default settings quite easily, please refer to OutOfMemoryExceptions for tips on this.
It is becoming more common for applications in Java (and scripting lanaguages) to do classworking and generate/load classes on the fly, hence the increased incidence of this error (Hibernate, AOP, Spring - they all do classworking like this). JRockit JVM from BEA allocated classes on the heap, like any other object, so I don't think it would have this problem.