6 Replies Latest reply on Sep 27, 2012 8:40 AM by gavinlam

    JBoss 6 - Invalid User-issue

    james4711

      Hello everyone!

       

      Hope somebody can help me:

       

      Since some time I have the issue that I receive sometimes (and that is the mostly strange thing) the following exception:

       

      javax.ejb.EJBAccessException: Invalid User

          at

      org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:161)

          at

      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

          at

      org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)

       

       

      What is strange: The application runs for several days, even several weeks .. and suddenly the error occurs. A JBoss-restart solves the issue and everything is fine again.

       

      Maybe it has something to do with security-changes we made (as the error just occured some time after these changes) regarding this description https://community.jboss.org/wiki/SecureTheJmxConsole:

      What we did: We modified the following files to make sure that for jmx-console and web-console login a password is forced:

      /opt/jboss/jboss-6.0.0.Final/common/deploy/jbossws-console.war/WEB-INF/jboss-web.xml

      /opt/jboss/jboss-6.0.0.Final/common/deploy/jbossws-console.war/WEB-INF/web.xml

      /opt/jboss/jboss-6.0.0.Final/common/deploy/jmx-console.war/WEB-INF/jboss-web.xml

      /opt/jboss/jboss-6.0.0.Final/common/deploy/jmx-console.war/WEB-INF/web.xml

      /opt/jboss/jboss-6.0.0.Final/server/default/conf/props/jmx-console-users.properties

      /opt/jboss/jboss-6.0.0.Final/server/default/conf/props/jmx-console-roles.properties

       

      Again .. the strange thing is that everything works well for a several, not defined periode of time. And after a restart everthing is fine again (and we also can't reproduce the error but we can just wait until this happens again).

       

      Hope somebody has some ideas.

       

      Thanks!

       

      KR,

       

      James

        • 1. Re: JBoss 6 - Invalid User-issue
          steve.j.hall

          Hi James

           

          I am interested in your configuration.  I have been having a very similar problem with Active Directory integration using the LdapExtLoginModule.  Our team has traced most of these issues to actual account locks on Active Directory.  However we still have a few "interesting" issues.

           

          Are you using pure file based user and authorization?  users.properties and roles.properties?  I was unclear about your configuration based on what you write above.

           

          My configuration...

           

          I have both web and swing client applications using EJB3.  I authenticate web users by individual user: myaccount, youraccount, hisaccount, heraccount, etc.  I authenticate the swing app -- approximately 375 instances -- using a common user and password that are embedded in the application.  Lets call this account the applicationaccount.

           

          We were having massive account lockout issues and the associated "invalid user" reports in our exception handling stack traces.  Again many of these issues were able to be traced to show that Active Directory had either the AD bind account or the acutal user account, or the applicationaccount locked. 

           

          I have tried to help on the swing app by using a two authentication providers and marking them both as sufficient.  My login config now looks like this

           

          <application-policy name = "other">

                <authentication>

                    <login-module code = "org.jboss.security.auth.spi.LdapExtLoginModule" flag = "sufficient">

                     <module-option name="java.naming.factory.initial">com.sun.jndi.ldap.LdapCtxFactory</module-option>

                      <module-option name="java.naming.provider.url">ldap://LDAPServer.mydomain.mycomapny.net:389</module-option>

                      <module-option name="java.naming.security.authentication">simple</module-option>

                      <module-option name="java.naming.referral">follow</module-option>

                      <module-option name="bindDN">bindaccount</module-option>

                      <module-option name="bindCredential">bindaccountpassword</module-option>

                      <module-option name="baseCtxDN">DC=mydomain,DC=mycompany,DC=net</module-option>

                      <module-option name="baseFilter">(sAMAccountName={0})</module-option>

                      <module-option name="rolesCtxDN">DC=mydomain,DC=mycompany,DC=net</module-option>

                      <module-option name="roleFilter">(sAMAccountName={0})</module-option>

                      <module-option name="roleNameAttributeID">cn</module-option>

                      <module-option name="roleAttributeID">memberOf</module-option>

                      <module-option name="roleAttributeIsDN">true</module-option>

                      <module-option name="roleRecursion">2</module-option>

                      <module-option name="allowEmptyPasswords">false</module-option>

                      <module-option name="searchScope">SUBTREE_SCOPE</module-option>

                      <module-option name="debug">true</module-option>

                  </login-module>

                  <login-module code="org.jboss.security.auth.spi.UsersRolesLoginModule" flag="sufficient">

                      <module-option name="usersProperties">tek-ejb-client-users.properties</module-option>

                      <module-option name="rolesProperties">tek-ejb-client-roles.properties</module-option>

                  </login-module>

                 </authentication>

              </application-policy>

              </authentication>

            </application-policy>

           

          After this configuration I had the applicationaccount in both Active Directory and in the tek-ejb-client-users.properties / tek-ejb-client-roles.properties files

           

          I tested this configuration by locking the bindaccount on Active Directory.  So in theory I could not bind against AD and thus could not authenticate the applicationaccount using AD.  In my test case, I was able to authenticate and authorize the applicationaccount using the backup file based provider.

           

          I am still seeing some "invalid user" errors even on the swing client.  So this is curious.

           

          What facts can you add?

          • 2. Re: JBoss 6 - Invalid User-issue
            steve.j.hall

            I spent some time on this issue and have some important data to share.  See notes here:  https://community.jboss.org/wiki/JBossLDAPInvalidUserProblems

            • 3. Re: JBoss 6 - Invalid User-issue
              steve.j.hall

              Hi James,

               

              Not sure if you are still having this problem.  I spent some time on this recently.  I have documented my findings and a work around here:

               

              https://community.jboss.org/wiki/JBossLDAPInvalidUserProblems

               

              Regards.

              Steve

              • 4. Re: JBoss 6 - Invalid User-issue

                Hi Steve,

                 

                I'm having this problem as well but when I click on the link you provided. It says that I'm not authorized to view that page.

                 

                Can you post your findings elsewhere?

                 

                Thanks,


                Gavin

                • 5. Re: JBoss 6 - Invalid User-issue
                  steve.j.hall

                  I learned a bit more and was going to make an update, as part of what I wrote could have some bad side effects.  Anyway, it appears someone did not like what I documented on this site and has removed it.   I cannot get to it either.

                   

                  The basic idea is this...  JBoss is open source so we can get the source for the LdapExtLoginModule.  JBoss also allows us to define and install our own custom login module. This is documented and not too difficult. Combining these two ideas, one can get the JBoss code, add some logging and debugging information and install it in the  application server.  Which is what I did.

                   

                  Once I had that working, I also recognized that JBoss has a cache of the LDAP lookups.  So if you want to exercise your LDAP you must shut caching off.  To shut caching off

                   

                  find the file server\default\deploy\legacy-conf-service.xml

                   

                  find the section for

                      <mbean code="org.jboss.security.plugins.JaasSecurityManagerService" name="jboss.security:service=JaasSecurityManager">

                   

                  the set the following two values to zero

                   

                   

                  <attribute name="DefaultCacheTimeout">0</attribute>


                  <attribute name="DefaultCacheResolution">0</attribute>

                   

                  Write a test case that will exercise your login.  In my case I have a secured EJB.  So I wrote a client application that called a method on the EJB in a loop with a 1 second pause between executions.  Now I am authenticating every call against my LDAP.

                   

                  Now I can really see it fail.  I start getting about one failure every 2 minutes.

                   

                  At this point, I have enough logging to see that my root problem is a socket timeout on my LDAP -- Active Directory.

                   

                  It appeard to us that we had one bug in the finally block of this method.

                  rolesSearch(InitialLdapContext ctx, SearchControls constraints, String user, String userDN, int recursionMax, int nesting)  throws NamingException

                   

                  First the code makes this call

                   

                  NamingEnumeration results = ctx.search(rolesCtxDN, roleFilter, filterArgs, constraints);

                   

                  In the finally block is has

                  if( results != null){  results.close(); }

                   

                  The results.close method is where the connection exception is being thrown.  So, what I found is that the LDAP was authenticating, fetching all the roles, and we were encountering an exception after successfully doing all the difficult and important work.

                   

                  The documentation for the close method shows this:

                   

                  Closes this enumeration.  After this method has been invoked on this enumeration, the enumeration becomes invalid and subsequent invocation of any of its methods will yield undefined results. This method is intended for aborting an enumeration to free up resources. If an enumeration proceeds to the end--that is, until hasMoreElements() or hasMore() returns false-- resources will be freed up automatically and there is no need to explicitly call close().

                   

                  So, it seems reasonable to change the code to this:

                   

                  if( results != null && results.hasMore())

                                  try{

                                      results.close();

                                  } catch (Exception e){

                                      log.error("Error closing naming enumeration", e);

                                  }

                           }

                   

                  This seems to resolve one problem.

                   

                  Next, we see that there is a significant performance problem in the method

                  protected void rolesSearch(InitialLdapContext ctx, SearchControls constraints,

                                  String user, String userDN, int recursionMax, int nesting)

                                  throws NamingException

                   

                  To determine this, we put Microsoft Network Monitor between the JBoss Server and the LDAP and found that if the role is a DN   -- login-config.xml file has <module-option name="roleAttributeIsDN">true</module-option>   --   then for a user with n roles, this method makes at least n + 1 network round trips.  Essentially the program gets a list of roles from the LDAP but a role is in a form that shows the entire DN (Distinguished Name).  The DN includes the part we are interested in the CN, which is the role.  Rather than parsing the role portion out of the DN in code, the login module makes a call to the LDAP for each role simply to ask the LDAP to return the CN portion the from the DN.  So if I have n roles this is n socket connections to the LDAP.  In our case, this turned out to be one of the most frequent causes of failure, as it is not uncommon for a user to have in excess of 30 roles as our LDAP is configured such that it returns all roles for a user, not just the ones that pertain to our Java EE applications.  With this much load, our LDAP would either not accept the connection, or would simply time out.  Accordingly we would get an Invalid User error.  The solution that seems to be working is to refactor this code to parse the CN out of the DN, thus eliminating a large number of calls to the LDAP.  This took about 10 minutes to code.  So it's not a hard problem to solve.  After this change our network round trips drop, we authenticate much more quickly, and we seem to stop having the Invalid User problems.  Now, when I look at Network Monitor I see that the network calls are down to bind using bind account, bind using the user's account (validates user and password), then one call to fetch the list of user roles.

                   

                  I would note, for a general purpose solution the JBoss approach may make sense, as it's a robust approach that should generally work for any organization, whereas my approach is tailored to my organization.  However, the reality is that the out of the box solution is quite inefficient.

                   

                  I would also add that there is another apparent probelm in the LdapExtLoginModule:

                   

                  When any socket timeout or socket connection exception is thrown, it prevents the validatePassword method from returning normally one of {true, false}, which in turn prevents  a second backup provider from being able to fulfill the request.  In my testing I found that trying to provide a second LDAP or file based backup provider failed when a socket connection exception was encountered.

                   

                  So, if you are having this problem, use the LdapExtLoginModule as a starting point for your own custom provider.  Customize the provider to your needs.  The JBoss provided LdapExtLoginModule is perhaps over generalized for your needs and can be improved.

                   

                  Good Luck,

                  Steve

                  • 6. Re: JBoss 6 - Invalid User-issue

                    Thank you for posting your solution. I'm sure it will be very helpful to the community. I have discovered that my problem is different. It turned out to be a credentials issue.