tomjenkinson: Is there a way to have the transaction reaper thread set a cancelation flag instead of canceling the active transaction directly from a non-application thread? The idea being that the cancelation flag may be noticed from the application thread and acted on. If the application thread doesn't notice the cancel flag, the transaction doesn't timeout but instead completes. smarlow: hi, at the moment that feature isn't there tomjenkinson: from my read of the xopen distributed tx spec, we are allowed to implement this however we like (from a background thread or app thread) tomjenkinson: I was consulting on a Hibernate case JBPAPP-10515 that I suggested we should talk about ^ tomjenkinson: if we agree that the above should be possible, I'd like to create a jira for it and get it scheduled. tomjenkinson: should we discuss further on email? smarlow: sure, just looking another case atm sorry * clebert (~clebert@redhat/jboss/clebert) has joined #jbossts * pfrobinson_ (~paul@redhat/jboss/pfrobinson) has joined #jbossts * pfrobinson_ has quit (Quit: pfrobinson_) smarlow: hi, I am back now, I have read your suggestion again and I am not positive what you are wanting to achieve smarlow: you want the reaper to mark a transaction as "timed out", I can understand that part of the request tomjenkinson: I was just finishing the email ;) yes its more what happens on the app thread side smarlow: I am not so clear what you want to do there on the next app thread side call into the transaction.getStatus() or any other method, I want JBossTM to do the tx cancel action from there instead of from the reaper thread tomjenkinson: so that the transaction sync afterCompletion is called from the app thread tomjenkinson: and JPA can do its "not thread safe" clearing of the persistence context smarlow: what I think you really are looking at is being able to handle timeouts without a transaction reaper, is that right? the fact that the reaper is marking them as timed out is somewhat of an "implementation detail" in your design really, we could just check the timeout using system.currentTimeMillis in certain calls to the transaction such as commit/rollback smarlow: essentially, you want me to check the timeout during commit/rollback and not use the transaction reaper at all tomjenkinson: sure, that would be a different way to do the same thing tomjenkinson: I'm also wondering what others are doing in this area tomjenkinson: like I said, the xopen spec allows us to do whatever we like (IMO) but the JPA spec doesn't. smarlow: ok, right as you say its not specced that it has to be a different thread processing the timeout, JBTM has always done it that way so clearly the change needs a good UC, I am just going to look at JBPAPP-10515 to see if that provides the UC smarlow: re JBPAPP-10515 I don't think you can guarantee what thread afterCompletion is called in tomjenkinson: well, the reaper transaction thread is one example of a non-application thread. are there other examples that you can give me? smarlow: I don't think that there is any restriction on using a transaction in different threads, so maybe JTS propagated calls to other servers is another example I mean that also call after completion smarlow: I think with JTA any thread can complete the transaction, so any thread would be calling afterCompletion too smarlow: just grepping the internet for "multiple thread JTA" tomjenkinson: lets continue on email, so we can involve others and build our fact database up over time. :) smarlow: jta spec "Multiple threads may concurrently be associated with the same global transaction." tomjenkinson: I did see that also smarlow: so that means one thread starts the transaction, enlists JPA, hands the transaction to another thread which calls commit and therefore afterCompletion in this new thread smarlow: I think relying on the thread in afterCompletion is unsafe * ochaloup has quit (Quit: Konversation terminated!) smarlow: I can tell you from the top of my head afterCompletion is invoked in an undefined context tomjenkinson: well, this is why I wanted to bring you into the discussion, so you could tell me my idea is crazy or not :) smarlow: sure, does JBPAPP-10515 explain why they need the thread association in afterCompletion? SessionImpl.clear() is called in afterCompletion, I assume that uses a thread local? tomjenkinson: basically, the application state in the persistence context needs to be cleared and the JPA EntityManager way to do that is not thread safe thread locals are not involved, the synchronization has a reference to EntityManager/session smarlow: make AbstractEntityManagerImpl$1.afterCompletion(int) set a flag something else can clear later? smarlow: essentially your design, but implemented in the application code rather than TM tomjenkinson: the problem with that is the later may not happen or in time, if the transaction actually completes before we see the flag smarlow: the transaction won't complete smarlow: the reaper has the transaction locked by this point smarlow: when you try to commit the transaction you will get an error saying: "The transaction is not active!" smarlow: catch that and then check this flag and then clear the session smarlow: oh, I see what you mean smarlow: QueryImpl.getResultList() might fail or something? smarlow: that is only because the session was cleared by the reaper smarlow: we are talking about stopping that though smarlow: I think this should work just fine smarlow: make your sync set a flag but not do anything to the session tomjenkinson: yeah, currently we sort of fail fast because we notice that internal storage has been nulled out unexpectedly and we longjmp smarlow: exactly smarlow: what I am suggesting is you don't clear the session until you detect the failure in your completion code smarlow: though that could be pretty poor too what transaction related locks are held when the after completion code is invoked? smarlow: by the time the afterCompletion fires we have rolled back the resources smarlow: after completion is probably the right place to do what you are doing smarlow: you probably more need to make hibernate thread safe smarlow: but I can appreciate that is harder tomjenkinson: its not that its hard, its just not required by the jpa spec, so other persistence providers may not work correctly smarlow: the problem is you cannot be certain that afterCompletion is being called after you commit, or if it is being called by the reaper smarlow: as I see it there is no bug here in JBTM, you agree with that right? the bug is that this persistence provider is assuming the same thread that registered the resource is going to complete the transaction, right? tomjenkinson: this is a design flaw imo smarlow: in jpa or JBTM? well, JBTM/xopen came first, so it appears to be in jpa tomjenkinson: but like I said before, I also want to know what other systems do smarlow: yeah, exactly, there is a very good reason to do it in another thread smarlow: the locks will be released immediately rather than waiting for a potentially wedged XAR smarlow: I suspect other JPA are thread safe I agree, its always good to release locks before calling something that might block tomjenkinson: so the investigation continues :) smarlow: well, we have gotten to the bottom of it, I guess now it is finding the right solution smarlow: here is a UC that won't work for the JPA as it stands (even without timeout) smarlow: AS1:EJB1 enlists JPA(XAR1) then calls AS2:EJB2 over JTS smarlow: AS2:EJB2 enlists JPA(XAR2) this is done by a jacorb request processor thread at random smarlow: AS1:EJB1 calls commit on the transaction, JPA(XAR1) afterCompletion is handled in the same thread as before - this is fine - but then the call to AS2 to commit the transaction goes over a different jacorb request processor, now afterCompletion in AS2 is done by a different thread and there is nothing anyone can do to stop it - mwahh hah hah haaaahhhhhh smarlow: (mad scientist cackle optional) when AS1 calls AS2, the persistence context doesn't get propagated (not that it matters for this uc). yeah, I see how a different thread calls aftercompletion in ^ tomjenkinson ping nmcl: hi tomjenkinson just browsing the discussion above. The specifications are clear: once the timeout goes off we are supposed to abort the transaction. Not set a flag and let the transaction continue to run. tomjenkinson though I may have missed some of the conversation ;) nmcl: so it is specced that we have to use a different thread, so much the better, I think smarlow agrees that changing the reaper isn't really a goer even before you confirmed that, but that will definitely clarify that tomjenkinson it is NOT the role of a container to rollback a transaction once the timeout has gone off. It IS the role of the transaction system to do this. tomjenkinson and you're right that doing anything in afterCompletion that relies on a) the thread being the same thread that was in beforeCompletion, or b) there being a transaction context active on the thread, is wrong. tomjenkinson afterCompletion calls are always best-effort and a transaction manager can ignore them if it wants (of course we don't, but I'm just saying.) nmcl: yeah, I already gave smarlow two uc where it certainly wouldn't nmcl: true, e.g. crash tomjenkinson +1 tomjenkinson, nmcl: could we expose (via extension) to the afterCompletion that we are in a timeout case? smarlow: walk the thread stack smarlow: I think the best thing is to make JPA thread safe though smarlow tomjenkinson anything we do outside of the spec means we won't be runnable anywhere else. tomjenkinson: I'll investigate what the other persistence providers do for this case (if they are thread safe) smarlow so is the real reason that JPA is not thread safe (just reading Tom's last comment)? nmcl: the JPA specification states that the persistence context is not thread safe smarlow but what precisely does it mean by "not thread safe"? nmcl: basically jpa doesn't expect afterCompletion from a different thread, I have explained to smarlow how this can happen even in a none-timeout scenario nmcl: so afterCompletion fires, it cleans up their session, but their query manager can't tolerate that tomjenkinson because it's a different thread? nmcl: effectively nmcl: the JPA spec doesn't require our implementation to be thread unsafe of course. but the application exposed EntityManager is described as thread unsafe. this is the same EntityManager that we need to clear the state of (via EntityManager.clear()) from the afterCompletion tomjenkinson and I should have added … different from what? The application thread? The thread that registered the resource in the first place? nmcl: its not really the fact its a different thread, that is why I said "effectively" nmcl: basically afterCompletion cleans up their session nmcl: but their code isn't expecting this to have happened smarlow since only one thread can register a given resource, why tie the clean to the thread? Why isn't the resource id and, maybe, the transaction id together sufficient to uniquely identify something? nmcl: so it throws an NPE/similar cos Session.clear nuked some vital bits it doesn't check before using nmcl: so as you can see, its not the thread per se, its more the fact that afterCompletion happens at a point they aren't expecting tomjenkinson ok. obviously some context (no pun intended) that I missed in the conversation initially. nmcl: maybe so nmcl: we currently are doing the clean from whichever thread invokes the afterCompletion smarlow: I think that is good, I think the rest of JPA needs to understand that that can happen smarlow: and handle it tomjenkinson: how ever we decide to handle this, I want to ensure that application state is safe smarlow: naturally tomjenkinson: I'll send email about this, like I promised. I need to work on a different issue now :) tomjenkinson, nmcl: thanks for helping with ^ smarlow: nps smarlow no problem