I've hit an issue recently that has me a bit stumped. (sorry for the really long post!!)
We're in the middle of an intra-forest migration project which is going fine, we're migrating batches of users on a nightly basis and haven't had any issues to date.
This past weekend however I was doing unrelated work to promote some new domain controllers as part of lifecycle work. The promotion was going fine until replication started complaining about a specific user object not being consistent in the local copy of the database as which point replication of one read-only partition fails to complete.
So we're migrating users from domains child1, child2, child3 etc into parent.local. The user object that shows up in the replication error was migrated out of child1.parent.local into parent.local around 30 days ago and went without a hitch. The DC's being promoted are in parent.local, all DCs are GCs in the forest and the only partition in the forest that won't replicate in to the new DC is child1.parent.local. When the event occurs it retries replication and after a while the KCC kicks in and creates additional replication links out to other DCs these succeed in replicating in all partitions except child1.parent.local.
I can't find any remnants of the user object in the child1 domain, I can see infrastructureUpdate object was created within child1's Infrastructure container at the point of migration (which I think is what its meant to do) and none of the established DCs have any replication issues.
On promotion, the object in question replicates into the new DC fine from its partners in parent.local and I can update the object on the new DC or an existing parent1 DC and the changes replicate fine.
I've attempted 3 promotions, all 3 showed the issue, the first I demoted and logged a call with MS, their initial suggestion was to move the object in question in parent.local to another OU. So this past weekend I promoted another 2 DCs, the 1st hit the issue, I moved the object on another parent.local DC to another OU and the issue resolved itself, replication completed on the new DC and I moved the object back to its original OU and the DC appears fine, its acting as a GC. On the second newly promoted DC (promoted 24hours later) again I hit the issue, I moved the object as before and unfortunately the replication issue continued. I tried moving it on both on the newly promoted DC and other DCs in parent.local and although the moves were fine the same issue continued to report (though the DN of the object updating to reflect the move) and in the end I demoted out the server.
Has anyone come across this before? Any pointers?
Thanks all for any suggestions in advance!
The event text is listed below:-
1/8/2015 11:10:37 AM Error newdc.parent.local 1084 Microsoft-Windows-ActiveDirectory_DomainService Replication NT AUTHORITY\ANONYMOUS LOGON Internal event: Active Directory Domain Services could not update the following object with changes received from the following source directory service. This is because an error occurred during the application of the changes to Active Directory Domain Services on the directory service. Object: CN=user444,OU=OurUsers,DC=parent,DC=local Object GUID: d4454402-8844-4d44-9550-060b14451699 Source directory service: 923d123a-4daa-4505-f66e-f28042bef29d._msdcs.parent.local Synchronization of the directory service with the source directory service is blocked until this update problem is corrected. This operation will be tried again at the next scheduled replication.
User Action Restart the local computer if this condition appears to be related to low system resources (for example, low physical or virtual memory). Additional Data Error value: 8443 The replication operation encountered a database inconsistency.
2108 Microsoft-Windows-ActiveDirectory_DomainService Replication NT AUTHORITY\ANONYMOUS LOGON This event contains REPAIR PROCEDURES for the 1084 event which has previously been logged. This message indicates a specific issue with the consistency of the Active Directory Domain Services database on this replication destination. A database error occurred while applying replicated changes to the following object. The database had unexpected contents, preventing the change from being made. Object: CN=user444,OU=OurUsers,DC=parent,DC=local Object GUID: d4454402-8844-4d44-9550-060b14451699 Source directory service: 923d123a-4daa-4505-f66e-f28042bef29d._msdcs.parent.local