We had this same issue a few years back on an older version of JBoss. I assume you have <transaction:ejb-transaction/> specified in your components.xml file?
First off, the reason this is happening is the transaction has already been started by the time start() method has been called on your bean. Because of this, you can't change the timeout.
You can do 1 of 2 things:
1) Get rid of <transaction:ejb-transaction/> in components.xml and control the transactions yourself by delegating to a SLSB and starting the transaction with the specfied timeout.
2) If you can't do #1 ( we couldn't do #1 ), I have another solution which I can send you.