Service broker provides automatic poison message detection, which disables the service queue upon five consecutive transaction rollbacks in trying to receive messages from the queue. However, an application should not rely on this feature for normal processing. Because automatic poison message detection stops the queue, this halts all processing for the application until the poison message is removed. Instead, an application should attempt to detect and remove poison messages as part of the application logic.
Typically, the message processing application can choose among the following four strategies:
A. Returns the poison message back to the service queue by rolling back the transaction in hope that the message can be successfully consumed next time when retried.
B. Preserves the message, the context and the encountered error to some permanent storage, such as a database table, commits the RECEIVE, and continues normally.
C. Ends the conversation with an error.
D. Uses SQL server event notification to ask for manual intervention when queue is disabled.
Option A above is good at handling failures caused by temporary conditions. For instance, a purchase order placed by an user can't be processed if the user profile has not existed yet because, say, user profile request processing that's happening in parallel takes more time than the purchase order; once that processing is complete, the purchase order processing is expected to be through without a problem. Another example could be that the processing transaction is chosen as the victim to rollback in the case of a deadlock; after such a deadlock is detected and removed, the message can then be successfully consumed.
However, there are situations where processing the same message will forever fail. One example may be that the message itself is not self-contained, say a purchase order can't be fulfilled because that credit card holder name and home address given don't match what's on file at the bank. A second example could be that a user profile can't be established because the account name chosen has already taken by an existing user. In these cases, approach B then sounds more suitable for the application to log the message content as well as the context in which it was happening so the situations can be further examined to see if the processing logic can be improved to handle these exception cases normally. Indeed, the two examples mentioned here have actually surfaced deficiencies in the application code because both scenarios can well be successfully processed if the message processing code is revised. For a more vivid example how inappropriate application error handling can lead to poison messages, check out Remus' blog here.
Care must be taken when continuing to the next message without the current one been processed. In a stateful session, skipping messages may cause the conversation into an illegal state and hence invalidates the rest of the dialog. If this is the case, the application then can't afford ignoring one message; thus ending the dialog with error as suggested by option C becomes the right thing to do. As an example, say a live meeting service that is unicasting an ongoing presentation, it had already received slide #2 and was waiting for the content of slide #3. But it then found it couldn't process the slide #3 message because of invalid media format. If it simply ignores them, the slide #3 worth of presentation simply becomes dumb to the receiver. So terminating the session with an error probably is the most appropriate.
Depending on what it is doing, an application can well use a combination of what is illustrated the above. For example, it can allow a failed receive to retry three seconds later. Further, if the retry is still unsuccessful, it then can choose to log the message, commit the RECEIVE transaction and continue to the next one.
As the last resort, application programmers may want not to do anything in their code but fully depend on the default mechanism built in service broker for poison message detection and handling. As formerly stated, the mechanism alone is not good enough. But combining it with SQL server event notification at least can request for administrator's manual intervention when a poison message is detected.
Once a BROKER_QUEUE_DISABLED event notification is defined, a notification message of the following format will be sent by service broker to the specified event notification queue when the service queue is disabled due to poison messages:
<EVENT_INSTANCE>
<EventType>BROKER_QUEUE_DISABLED</EventType>
<PostTime>2008-06-27T12:30:18.357</PostTime>
<SPID>53</SPID>
<DatabaseID>6</DatabaseID>
<TransactionID/>
<NTUserName/>
<NTDomainName/>
<HostName/>
<ClientProcessID/>
<ApplicationName/>
<StartTime>2008-06-27T12:30:18.168</StartTime>
<ObjectID>53575229</ObjectID>
<ServerName>junan02</ServerName>
<LoginSid/>
<EventSequence>7844</EventSequence>
<IsSystem>1</IsSystem>
</EVENT_INSTANCE>
To identify which service queue this notification is fired for, first use the database id to find out the database name:
SELECT name AS database_name
FROM sys.databases
WHERE database_id = 6
GO
(1 row(s) affected)
SsbTestDBTarget
Then switch to that database, and get the service queue name using the object id:
USE SSBTestDBTarget
GO
SELECT name AS service_queue_name
FROM sys.service_queues
WHERE object_id = 53575229
GO
(1 row(s) affected)
Service1Queue
The SQL script below shows in detail how to create an event notification service to receive notification messages when a user queue being watched gets disabled. A stored procedure is defined to process the posted notification by emailing the administrator about which queue is disabled and ask him/her to jump in to identify and eliminate the problem so normal message processing on the queue can be resumed.
-- create the event notification queue to receive queue_disabled events
CREATE QUEUE [QueueDisabledNotifQueue];
GO
-- create the event notification service for receiving queue_disabled events
CREATE SERVICE [QueueDisabledNotifService]
ON QUEUE [QueueDisabledNotifQueue]
([http://schemas.microsoft.com/SQL/Notifications/PostEventNotification]);
GO
-- create queue-disabled event notification to watch on 'Service1Queue'
CREATE EVENT NOTIFICATION [QueueDisabledNotif]
ON [Service1Queue]
FOR BROKER_QUEUE_DISABLED
TO SERVICE [QueueDisabledNotifService], 'current database'
GO
-- define the stored proc to process queue_disabled notifications
CREATE PROCEDURE [QueueDisabledNotifHandler] AS BEGIN
DECLARE @messageType SYSNAME
DECLARE @conversationHandle UNIQUEIDENTIFIER
DECLARE @messageBody XML
WHILE(1=1)BEGIN
BEGIN TRANSACTION
WAITFOR (
RECEIVE TOP(1)
@messageType=message_type_name,
@messageBody=message_body,
@conversationHandle=conversation_handle
FROM [QueueDisabledNotifQueue]), TIMEOUT 5000
IF( @@ROWCOUNT = 0 ) BEGIN
COMMIT TRANSACTION -- rollback is inappropriate here as it'll be counted by queue disabling logic for poison message detection
BREAK
END -- if( @@rowcount ... )
IF( @messageType = 'http://schemas.microsoft.com/SQL/Notifications/EventNotification' ) BEGIN
DECLARE @cmd NVARCHAR(MAX)
SET @cmd = 'dbo.sp_send_dbmail
@profile_name="Administrator",
@recipients="joesmith@my-company.com",
@body="CAST(@messageBody as NVARCHAR(MAX)",
@subject="Queue Disabled Detected";'
EXEC (@cmd)
END -- if( @messageType ... )
COMMIT TRANSACTION
END -- while(1=1)
END
GO
-- kick off queue disabled notification processing
ALTER QUEUE [QueueDiabledNotifQueue]
WITH ACTIVATION (
STATUS = ON,
PROCEDURE_NAME = QueueDisabledNotifHandler,
MAX_QUEUE_READERS = 1,
EXECUTE AS SELF )