Multiple Moderation Approval Requests

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4.

Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term).

Here is the process we saw:

1. Message requiring moderation sent.
2. Approval request sent to moderator.
3. Moderator approves request
4. Approval request sent to moderator
5. Moderator approves request
6. Repeat request and approval process a few more or a lot more times.

This process was happening even though we had not moved any mailboxes to Exchange 2016 yet. 

When searching, there were very few references to this issue on the Internet or support forums. However, there were a few suggestions that were consistent:

• Ensure arbitration mailboxes are moved to Exchange 2016 (one of these stores the messages until they are approved).
• Delete and recreate rules.
• Move moderators mailbox to Exchange 2016.
• Restart transport services.

All of these things were done but we still had issues. However, when both mailboxes were on Exchange 2016, the approvals on messages seemed to only happen twice. This was better than the random number from before.

I reviewed the message tracking logs for errors and didn't see any. In the logs, each time an approval was received, the message was released for delivery, but then promptly moderated again. However, the second attempt to approve worked.

All of my initial testing was done using inbound messages. So, I tried some scenarios with both mailboxes on Exchange 2016, and these were my results:

• Inbound messages routing through Exchange 2010 first - Two approvals required
• Outbound messages routing out through Exchange 2010 - One approval only
• Inbound messages routing through Exchange 2016 only - One approval only

Based on these results, we can see that Exchange 2010 coexistence definitely plays a role, because when Exchange 2010 is not part of the inbound routing the issue doesn't occur. This at least provided confidence that after migration was complete the issue would not be persist.

The other item that needed to be addressed was the Exchange 2016 CU4. Microsoft releases updates in matched sets and Exchange 2016 CU4 was one step behind Exchange 2010 SP3 RU17. In the hope that having both at the same update level would fix it, we applied Exchange 2016 CU 5. We did the update to CU 5 but there was not change.

Final testing indicated that any message that entered the Exchange organization through Exchange 2010 was subject to double approval. This happened if the message came in from external or was generated by a mailbox in Exchange 2010. The location of the moderator mailbox did not make a difference.

So, to minimize the issue:

• Move all inbound message routing to Exchange 2016 sooner rather than later. This includes Internet mail and applications that send messages to be moderated.
• Move mailboxes that generate the most messages to be moderated first. Once the source is in Exchange 2016 the problem is mediated.

I should also note that there was a red herring in the application event log. We saw this error:

Event ID 1051, MSExchange Extensibility
Warning, MExRuntime
 Agent 'Approval Processing Agent' caused an unhandled exception 'SmtpResponseException: 250 2.1.5 APPROVAL.ApprovalRequestUpdated; approval request updated successfully' while handling event 'OnCreatedMessage'

However, review of the logs indicated that the error was present before the issue appeared. So it appears to be noise rather than useful information.