AI OnAI Off
Indexing processor state is Closed.
ChangeNotificationDatabase (underlying mechanism for catalog content indexing queue) calls (used by InDatabaseQueue<T>) produce deadlocks in database: deadlocks may occur during different calls like GetStatus(), Enqueue(), Dequeue().
After that I have to reindex full catalog (that takes me about 40 minutes) because indexing proccessor becomes "Invalid"
But that is not working in my production solution.
Could some take a look into deadlocks issue?
As I see the problem is inside dbo.ChangeNotificationXXX stored procedures that are being called in concurrent manner and each dbo.ChangeNotificationXXX procedure makes updates to several tables (processors, connections, queue it self).
Probably I have outdated database(stored procedures like dbo.ChangeNotificationXXX)?
My MSSQL version is: 11.0.2100.60
Edited,
Jun 02, 2014 21:17
I have just setup following experiment:
Input:
- Commerce Manager Web application = App1(stopped)
- CMS Front site = App2 (stopped)
- "Catalog Index" processor invalid
Actions:
1. Start App1
- App1 starts to do notification "heartbeats"
2. Start App2
- App2 starts to do notification "heartbeats"
At some moment App1 and App2 do concurrent heartbeats and deadlock occurs, as a result App1 stops doing "heartbeats" (in other cases it can produce another error, like invalidating of valid processor)
Please check trace(SQL) and app logs:
Edited,
Jun 03, 2014 7:42
Could someone take a look? My first post about the problem was more than month ago:)
Jun 03, 2014 7:44
Have you created a support case for this problem?
We are also experiencing the probleme with Indexing processer is closed. I haven't seen the ERROR log from the QueueWrapper, but I guess it could still be the same problem.
My experience is, that EPiServer looks at these forum posts, but doesn't necessary take action on them. However the support cases are monitored and action is taken at some point.
Regards
Anders
Jun 04, 2014 15:38
Hi Andres,
QueueWrapper is my custom class that wraps default implementation from EPiServer framework and does logging if error occurs. Default implementation does not log errors, so you even can't know that they do happen:)
About support case, I personally haven't, but another guy from our dev team has I suppose.
Jun 04, 2014 18:26
Closed state = need to reboot IIS because you can invalidate/recover in such case (what I definitely want to avoid)
Here is some log:
2014-04-23 23:43:48,729 [5] ERROR Jula.Common.Events.QueueWrapper: wrapped error
System.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 71) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at EPiServer.Events.ChangeNotification.EventQueue.ChangeNotificationDatabase.GetStatus(Func`1 connectionFactory, Guid connectionId)
at EPiServer.Events.ChangeNotification.EventQueue.InDatabaseQueue`1.<GetStatus>b__0(Guid connectionId)
at EPiServer.Events.ChangeNotification.EventQueue.InDatabaseQueue`1.GetStatus()
at Jula.Common.Events.QueueWrapper.Safe[T](Func`2 action)
ClientConnectionId:04443c6b-6cc4-421e-88b9-74e368fab74d
Jula.Common.Events.QueueWrapper - is custom class that wraps InDatabaseQueue<T> from commerce f/m and just adds logging abilities.
I suppose this error makes indexing process closed.
Is it a bug? or some configuration issue?