Recently we migrated from JBOSS 4.X to 5.1.0 EAP. I am listing below some of the problems faced and fixes done during the migration.
Also listed are the various setting we used during the migration. I found answers to all the problems I faced by going through various posts on forums here and elsewhere. I am consolidating below. Hope this helps!

• JAVA_OPTS setting:
  -Xms128M -Xmx512M -XX:MaxPermSize=256M

• Adding custom deployment folder

       Open <JBOSS_INSTALLATION_FOLDER>/jboss-as/server/default/conf/bootstrap/profile.xml. Under bean name "BootstrapProfileFactory" 's property applicationURIs"       
              add <value>file:/<YOUR DEPLOYMENT FOLDER></value>

      Example:<value>file:/trunk/OASYS/jboss_deploy_eskm_batch/skm/skmweb</value>

     Also add the following entry in 
     <JBOSS_INSTALLATION_FOLDER>/jboss-as/server/default/conf/bootstrap/ vfs.xml.
     (Under bean name "VFSCache" 's property "permanentRoots") 

                    <entry>

                          <key>file:/<YOUR DEPLOYMENT FOLDER></key>
                          <value><inject bean="VfsNamesExceptionHandler"/></value>
                   </entry>
                   I was running into constant out of memory issues. The above entry along with JAVA_OPTS setting
                   seemed to help.Keep posted if you find a better solution.

• Changes in ejb-jar.xml   
  

    Modified tags acknowledge-mode and message-driven-destination tags to the following:
    <activation-config>
        <activation-config-property>
            <activation-config-property-name>
                       acknowledgeMode
            </activation-config-property-name>
            <activation-config-property-value>
                      AUTO_ACKNOWLEDGE
            </activation-config-property-value>
            </activation-config-property>
            <activation-config-property>
               <activation-config-property-name>
                    destinationType
               </activation-config-property-name>
               <activation-config-property-value>
                     javax.jms.Queue
               </activation-config-property-value>
             </activation-config-property>
     </activation-config>

•   JBoss.xml validation problems:
  

         Open <JBOSS_INSTALLATION_FOLDER>/jboss-as/ server/ default/ deployers/ metadata-deployer-jboss-beans.xml.
         Add the following entry under 'bindingClassesByLocations' property of bean name 'SchemaResolverConfig' and
         <!--JBossMetaData--> comment : 
        <entry>
           <key>jboss_5_1.xsd</key>
           <value>org.jboss.metadata.ejb.jboss.JBoss50MetaData</value>
        </entry>

• java.lang.IllegalStateException: Expected either '.jar' or '.jar/' at the end of 1.0       

       Fixed the above issue by removing jaxb-xjc.2.2.1.1 from deployment and used the one provided by JBOSS.

• Your JAXP provider oracle.xml.jaxp.JXDocumentBuilderFactory@951bb0 does not support XML Schema. Are you running on Java 1.4 with Apache Crimson? Upgrade to Apache Xerces (or Java 1.5) for full XSD support.

     Fixed the above by adding the following system properties during JBoss startup in run.bat / run.sh.
    -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
    -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl

• ClassLoaderManager Unexpected error during load of:org.w3c.dom.UserDataHandler java.lang.LinkageError: loader constraint violation: loader (instance of org/jboss/classloader/spi/base/BaseClassLoader) previously initiated loading for a different type with name "org/w3c/dom/UserDataHandler"

        You get this error when you have same class present in your CLASSPATH twice and these classes are of different versions. I had

        xmlbeans2.4.0.jar in our deployment that caused this issue. Fixed the above linkage problem by removing this class.

• Adding JBOSS messaging queue

         Added an entry in <YOUR_JBOSS_INSTALLATION_FOLDER>/jboss-as/ server/ default/ deploy/ messaging/ destinations-service.xml
    
      <mbean code="org.jboss.jms.server.destination.QueueService"
        name="jboss.messaging.destination:service=Queue,name=Myqueue"
        xmbean-dd="xmdesc/Queue-xmbean.xml">
        <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
        <depends>jboss.messaging:service=PostOffice</depends>
      </mbean>

•    Miscellaneous

          Removed  xercerImpl.jar from our deployment and used the one present in JBOSS 5.1.0 EAP.

Vishal