3 Replies Latest reply on Jun 1, 2010 4:21 AM by chammp

    Quartz Deadlocks

    aehrensberger

      We are using Quartz with  Spring and our configuration is throwing deadlocks when quartz has more than 1  thread configured. I'm starting to believe that it's because we don't have our  quartz configured correctly with Spring, but I can't find enough documentation  on how to configure the two to play nicely.  I'm also starting to believe that Quartz is not going to be our solution, unfortunately, because of this issue.

       

      We are running tests under heavy load, starting as many as 1000 jobs in a matter of minutes, all scheduled to be picked up by 50 quartz threads after 10 second delays.

       

      We are running on both  Windows and Linux environments - pointing at MSSQL and Oracle DBs. With both OS,  using either DB, we can throw the following deadlock errors...

       

      2010-03-17 18:52:31,737 [] [] ERROR [DocFinityScheduler_Worker-42]  core.ErrorLogger core.ErrorLogger (QuartzScheduler.java:2185) - An error occured  while marking executed job complete. job= 'BPM.6e41a6567f0000020100362a51dc7a50'

       

      org.quartz.JobPersistenceException: Couldn't remove trigger: Transaction  (Process ID 87) was deadlocked on lock resources with another process and has  been chosen as the deadlock victim. Rerun the transaction. [See nested  exception: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process  ID 87) was deadlocked on lock resources with another process and has been chosen  as the deadlock victim. Rerun the transaction.]
      at  org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1469)at  org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2978)at  org.quartz.impl.jdbcjobstore.JobStoreSupport$39.execute(JobStoreSupport.java:2962)  at  org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3713)at  org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3747) 
      at  org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3709)at  org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2958)at  org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1727)at  org.quartz.core.JobRunShell.run(JobRunShell.java:273)
      at  org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534)

       

      Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction  (Process ID 87) was deadlocked on lock resources with another process and has  been chosen as the deadlock victim. Rerun the transaction.
      at  com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(Unknown  Source at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(Unknown  Source) at  com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown  Source) at  com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown  Source) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source) at  com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown Source)  at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown  Source)at  com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown Source)  at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown  Source)at  com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)  at  org.quartz.impl.jdbcjobstore.StdJDBCDelegate.deleteSimpleTrigger(StdJDBCDelegate.java:1820)  at  org.quartz.impl.jdbcjobstore.JobStoreSupport.deleteTriggerAndChildren(JobStoreSupport.java:1345  at  org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1453  ... 9 more

       


      I have attached as many files as I could think of that relate to the issue.  Can anybody help me out here?   Do I have it  configured incorrectly to not use a JobStoreCMT versus a JobStoreTX? Do I have  it configured incorrectly in terms of threads or something else?

       

      I'm  really at a loss. We've been scouring the web for this stuff for documentation  for two days and are still scratching our heads.

        • 1. Re: Quartz Deadlocks
          kukeltje

          jBPM does not use quartz at all, so I think you might be mistaken (or I am, but then enlighten me where quartz is used)

          • 2. Re: Quartz Deadlocks
            aehrensberger

            Sorry to be misleading - no, jbpm doesn't use quartz, and I understand that.  But in reading this forum, I've seen multiple people that have suggested quartz for scheduling purposes in conjunction with jbpm.

             

            That said, we followed that advice and have the stated problem.  I've posted to some quartz forums, but there is no movement on those forums, so I figured I'd try the jbpm forum to see if others have had similar issues - OR - to see if other have different scheduling suggestions?

            • 3. Re: Quartz Deadlocks

              We're suffering from the same deadlock-disease here. Did you find the cause for this behaviour? Or any workaround?