2 Replies Latest reply on Oct 29, 2015 10:03 AM by ymartin

    RHQ web interface: 503 service not available after ORA-00904 for storagenode VERSION

    ymartin

      Hello,

       

      I have upgraded a RHQ 4.4 instance to 4.13.1 running on Oracle XE. The upgrade process has finished with success and data migration was succesful too.

       

      But after server start up, there is no way to enter the web interface. The POST request to "/portal/sessionAccess" fails with 503 Service Unavailable. To confirm, the JBoss welcome page at "localhost:7080" also says the service not ready yet.

       

      RHQ runs Java 7u79 64 bits. The "server.log" contains:

       

      15:07:58,648 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-6-thread-1) SQL Error: 904, SQLState: 42000

      15:07:58,649 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-6-thread-1) ORA-00904: "STORAGENOD0_"."VERSION": invalid identifier

      15:07:58,654 ERROR [org.jboss.as.ejb3.invocation] (pool-6-thread-1) JBAS014134: EJB Invocation failed on component StorageNodeManagerBean for method public abstract org.rhq.core.domain.util.PageList org.rhq.enterprise.server.cloud.StorageNodeManagerRemote.findStorageNodesByCriteria(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.criteria.StorageNodeCriteria): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet

      [...]

      Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet

          at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.10.Final.jar:4.2.10.Final]

      [...]

      Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "STORAGENOD0_"."VERSION": invalid identifier

          at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491) [ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]

          at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:107)

          at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)

          at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) [hibernate-core-4.2.10.Final.jar:4.2.10.Final]

       

      Is there any Oracle backend issue with RHQ 4.13.1 or is the upgrade process failed at some step ?

      How should I diagnose to get it fixed ?

        • 1. Re: RHQ web interface: 503 service not available after ORA-00904 for storagenode VERSION
          jayshaughnessy

          Hi Yves,

           

          The upgrade from RHQ4.4 (pre-storage-node, pre-rhqctl) to 4.13.1 is fairly complicated, so make sure you carefully followed the upgrade instructions at Upgrading RHQ - RHQ - Project Documentation Editor.

           

          Your error would indicate that the rhq_storage_node db table is not [fully] defined or possibly is corrupt.  It seems perhaps that the database update did not succeed during your upgrade, I'm not sure.  I'd probably recommend starting the upgrade process over.  It's not clear what went wrong.  Was this a single-server update, or multi-node HA?

          • 2. Re: RHQ web interface: 503 service not available after ORA-00904 for storagenode VERSION
            ymartin

            Hello Jay

             

            My RHQ setup is fairly simple, no HA. I have followed instructions and upgrade the server database and agent properly, always checking progress and all logs on disk, everything has succeeded. A storage node has been created successfully during the process and I run the data migration, which lasted 20 minutes without error or warning.

             

            When accessing the administration page, the RHQ_STORAGE_NODE.VERSION column was lacking and the table contains no row. The following message give me a clue:

             

            16:26:45,244 WARN  [org.rhq.enterprise.server.storage.StorageClientManager] (pool-6-thread-1) Storage client subsystem wasn't initialized. The RHQ server will be set to MAINTENANCE mode. Please verify

            that the storage cluster is operational.: java.lang.IllegalStateException: There is no storage node metadata stored in the relational database. This may have happened as a result of running dbsetup or d

            eleting rows from rhq_storage_node table. Please re-install the storage node to fix this issue.

             

            As I have just created the storage node in the process, I did the mistake to skip the "upgrade storage schema" step (telling me it is useless). So the following command fixed it:

             

            # ./bin/rhqctl upgrade --storage-schema

            16:37:15,207 INFO  [org.jboss.modules] JBoss Modules version 1.3.0.Final-redhat-2

            16:37:16,266 INFO  [org.rhq.server.control.command.Upgrade] Updating RHQ Storage Cluster schema

            16:37:16,276 INFO  [org.rhq.server.control.command.Upgrade] The RHQ Storage Cluster schema update is running

            16:37:19,588 INFO  [org.jboss.modules] JBoss Modules version 1.3.0.Final-redhat-2

            16:37:24,657 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to check storage schema compatibility.

            16:37:28,380 INFO  [org.rhq.cassandra.schema.VersionManager] Completed storage schema compatibility check.

            16:37:29,265 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] Persisting to database new storage nodes for values specified in server configuration property [rhq.storage.nodes]

            16:37:29,384 INFO  [org.rhq.enterprise.server.installer.Installer] Database setup is complete.

             

            Thank you for your help

            Yves