{"id":4160,"date":"2018-07-14T16:42:15","date_gmt":"2018-07-14T12:42:15","guid":{"rendered":"https:\/\/rakhesh.com\/?p=4160"},"modified":"2018-07-14T16:50:47","modified_gmt":"2018-07-14T12:50:47","slug":"aside-under-the-hood-with-dags","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/exchange\/aside-under-the-hood-with-dags\/","title":{"rendered":"[Aside] Under the Hood with DAGs"},"content":{"rendered":"<p>Watching this Ignite 2015 video: <a href=\"https:\/\/channel9.msdn.com\/Events\/Ignite\/2015\/BRK4105\">Under the Hood with DAGs<\/a>, by <a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/\">Tim McMichael<\/a>.<\/p>\n<p>Adding some links here to supplement the video:<\/p>\n<ul>\n<li><a href=\"https:\/\/blogs.msdn.microsoft.com\/clustering\/2012\/11\/21\/tuning-failover-cluster-network-thresholds\/\">Tuning Failover Cluster Network thresholds<\/a> &#8211; useful when you have stretched DAGs<\/li>\n<li><a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2012\/05\/30\/exchange-2010-the-mystery-of-the-9223372036854775766-copy-queue\/\">The mystery of the 9223372036854775766 copy queue\u2026<\/a> &#8211; never had this one but good info.\n<ul>\n<li>Basically, the cluster registry keeps track of the last log number per database and also the timestamp. When a node wants to see its copy queue length (i.e. how behind it is in terms of processing the logs) it can compare this log number with the log number it has actually processed. Sometimes, however, some node might be having issue updating the cluster registry or reading the cluster registry and so they fall behind in terms of receiving updates. In such cases the last log number will match what they have processed, but it is actually outdated info and so if the Exchange Replication service on the server hosting the passive copy notices that the timestamp is <span style=\"text-decoration: underline;\">12 minutes ago<\/span> it puts its database copy into self-protection mode. This is done by putting the copy queue length (a.k.a. CQL) manually as 9 quintillion (the maximum a 64-bit integer can be). No one can actually have such large a copy queue length so it&#8217;s as good number to choose.<\/li>\n<li>The video suggests rebooting each node until you find one which might be holding updates. But the link above suggests a different method.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>DAC<\/h3>\n<p>Came across some Datacenter Activation Coordination (DAC) from Tim&#8217;s blog: <a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2012\/05\/21\/part-1-my-databases-do-not-automatically-mount-after-i-enabled-datacenter-activation-coordination\/\">part 1<\/a>, followed by a series of posts you can see at the end of part 1.<\/p>\n<blockquote>\n<p align=\"justify\">DAC mode works by using a bit stored in memory by Active Manager called the Datacenter Activation Coordination Protocol (DACP). DACP is simply a bit in memory set to either a 1 or a 0. A value of 1 means Active Manager can issue mount requests, and a value of 0 means it cannot.<\/p>\n<p align=\"justify\">The starting bit is always 0, and because the bit is held in memory, any time the Microsoft Exchange Replication service (MSExchangeRepl.exe) is stopped and restarted, the bit reverts to 0. In order to change its DACP bit to 1 and be able to mount databases, a starting DAG member needs to either:<\/p>\n<ul>\n<li>\n<div align=\"justify\">Be able to communicate with any other DAG member that has a DACP bit set to 1; <em>or<\/em><\/div>\n<\/li>\n<li>\n<div align=\"justify\">Be <em>able to communicate with all DAG members that are listed on the StartedMailboxServers<\/em> list.<\/div>\n<\/li>\n<\/ul>\n<\/blockquote>\n<p>The bit I italicized is important. If you read his blog post you&#8217;ll see why. If DAC is activated and you are starting up a previously shutdown DAG, even though the DAG might have quorum it will not start up if some members are still offline. (I had missed that when <a href=\"https:\/\/docs.microsoft.com\/en-us\/exchange\/high-availability\/database-availability-groups\/dac-mode\">reading about DAC earlier<\/a>). To summarize it succinctly from <a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2012\/05\/30\/part-2-datacenter-activation-coordination-and-the-file-share-witness\/\">part 2<\/a> of his series:<\/p>\n<blockquote>\n<p>Remember, with DAC mode enabled, different rules apply for mounting databases on startup. The starting DAG member must be able to participate in a cluster that has quorum, and it must be able to communicate with another DAG member that has a DACP value of 1 or be able to communicate with all DAG members listed on the StartedMailboxServers list.<\/p>\n<\/blockquote>\n<p>Here&#8217;s highlights from some of the interesting posts in Tim&#8217;s series:<\/p>\n<ul>\n<li><a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2013\/02\/03\/part-4-datacenter-activation-coordination-and-the-prevention-of-split-brain\/\">Part 4<\/a> has info on the steps to do a datacenter switchover and the cmdlets available when DAC is enabled. Essentially: you 1) <code>Stop-DatabaseAvailabilityGroup<\/code> with <code>\u2013configurationOnly:$TRUE<\/code> switch for the site that is down &#8211; this marks the servers in the site that is down as down, 2) <code>Stop-Service CLUSSVC<\/code> on the nodes in the site that is up, and finally 3) <code>Restore-DatabaseAvailabilityGroup<\/code> specifying the site that is up. This <a href=\"https:\/\/docs.microsoft.com\/en-us\/exchange\/high-availability\/manage-ha\/datacenter-switchovers\">Microsoft doc on datacenter switchovers<\/a> is worth reading side-by-side. It contains info on both DAC and non-DAC scenarios so watch out for that.<\/li>\n<li><a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2013\/01\/27\/part-5-datacenter-activation-coordination-how-do-i-force-automount-consensus\/\">Part 5<\/a> has info on how to use the <code>Start-DatabaseAvailabilityGroup<\/code> cmdlet to set the DACP bit as 1 on a specified server thus bringing up the DAG by forcing a consensus.<\/li>\n<li><a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2013\/02\/04\/part-6-datacenter-activation-coordination-who-has-a-say\/\">Part 6<\/a> is an interesting story. A nice edge case of DAC being enabled and graceful shutdown.<\/li>\n<li><a href=\"https:\/\/blogs.technet.microsoft.com\/timmcmic\/2013\/08\/18\/part-8-datacenter-activation-coordination-stop-in-the-name-of-dag\/\">Part 8<\/a> is another interesting story on what happens due to a typo in a cmdlet.<\/li>\n<\/ul>\n<p>Very briefly, the <a href=\"https:\/\/docs.microsoft.com\/en-us\/exchange\/high-availability\/database-availability-groups\/dac-mode#other-benefits-of-dac-mode\">DAC cmdlets<\/a>:<\/p>\n<ul>\n<li><code>Stop-DatabaseAvailabilityGroup<\/code> &#8211; mark a specified server, or all server in a specified AD site, as down. Use the <code>-ConfigurationOnly<\/code> switch to mark the server as down in AD only but not actually do anything on the server(s). Need to use this switch if the servers are already offline but AD is up and accessible in that site. This cmdlet also forces a sync of AD across sites so the information is propagated.<\/li>\n<li><code>Start-DatabaseAvailabilityGroup<\/code> &#8211; same as above, but mark as up. Can use the <code>-ConfigurationOnly<\/code> switch to not really do anything but only mark in AD.<\/li>\n<li><code>Restore-DatabaseAvailabilityGroup<\/code> &#8211; it evicts any stopped servers, it can configure the DAG to use an alternate witness server, and it brings up the DAG after doing this. This cmdlet can only be used against a DAG with DAC enabled.<\/li>\n<\/ul>\n<h3>Dynamic Quorum<\/h3>\n<p>Came across dynamic quorum from the videos (wasn&#8217;t previously aware of it). Am being lazy and will put in some screenshots from the video:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4166\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?resize=660%2C374&#038;ssl=1\" alt=\"\" width=\"660\" height=\"374\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?w=1920&amp;ssl=1 1920w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?resize=300%2C170&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?resize=768%2C435&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?resize=1024%2C580&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?resize=400%2C227&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-1.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>The highlighted part is the key thing.<\/p>\n<p>Remember that quorum is defined as &#8220;<code>(the number of votes)\/2 + 1<\/code>&#8220;. Each node (or witness) typically has a single vote, and <code>(number of votes)\/2<\/code> is rounded down (i.e. 7\/2 = 3.5, rounded down to 3).<\/p>\n<p>With dynamic quorum once a node (or set of nodes) fail, and if the remaining set of nodes form a quorum (note &#8211; they <em>have<\/em> to form quorum), then the required quorum of the cluster is adjusted to reflect the remaining number of nodes.<\/p>\n<p>Take a look at the scenario below:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4168\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?resize=660%2C371&#038;ssl=1\" alt=\"\" width=\"660\" height=\"371\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?w=1920&amp;ssl=1 1920w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?resize=768%2C432&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?resize=1024%2C576&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?resize=150%2C84&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?resize=400%2C225&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-2.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>We have two data centers. 6 nodes + a witness, so initially the quorum was 7\/2 + 1 = 4.<\/p>\n<p>The link between the two data centers goes down. Data center B has 3 nodes, which is below the quorum of 4 so all 3 nodes shutdown. Data center A has 3 nodes + witness, thus meeting the quorum and it stays up.<\/p>\n<p>At this point if any further node in data center A goes down, they will fall below the quorum and the cluster will shutdown. To avoid such a situation is where dynamic quorum comes in. With dynamic quorum (introduced since Server 2012) when the nodes in data center A form quorum, the new quorum requirements is 4\/2 + 1 = 3.<\/p>\n<p>If a node goes down in data center A, leaving 2 nodes + a witness, since they meet the new quorum of 3 the cluster stays up. The quorum then gets revised to be 2\/2 + 1 = 2. If yet another node goes down, the remaining node + witness still meets the new quorum of 2 and so the cluster continues to stay up.<\/p>\n<p>Another slide:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4170\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?resize=660%2C373&#038;ssl=1\" alt=\"\" width=\"660\" height=\"373\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?w=1912&amp;ssl=1 1912w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?resize=768%2C434&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?resize=1024%2C578&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?resize=400%2C226&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-3.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>Two data centers, 2 nodes + 1 node, no witness; the quorum is therefore 3\/2 + 1 = 2.<\/p>\n<p>One of the nodes in data center A goes down. Since the number of remaining nodes meets quorum, the cluster can stay up. <span style=\"text-decoration: underline;\">But<\/span> since there is no fail share witness each node cannot be given an equal vote (I wasn&#8217;t aware of this). Thus the cluster service picks up one of the nodes (the one with the lowest node ID) and gives it a vote of 0. The node in data center A has a vote of 0. The new quorum is thus 1\/2 + 1 = 1.<\/p>\n<p>If the link between the two data centers goes down, the node in data center B stays up <em>even though the node in data center A too could have formed quorum<\/em>! Nothing wrong with it, just an edge case to keep in mind as chances are you probably wanted data center A to remain up as that&#8217;s why you provisioned two nodes there in the first place.<\/p>\n<p>Now for a variant in which there is a witness:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4172\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?resize=660%2C372&#038;ssl=1\" alt=\"\" width=\"660\" height=\"372\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?w=1916&amp;ssl=1 1916w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?resize=768%2C433&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?resize=1024%2C577&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?resize=400%2C225&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-4.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>So two data centers, 2 nodes + 2 nodes, 1 witness in data center A; the quorum is therefore 5\/2 + 1 = 3.<\/p>\n<p>As before, one of the nodes in data center A goes down. Since there are 3 nodes + witness remaining, they meet quorum and the cluster continues. The new quorum is 4\/2 + 1 = 3. Again, the data center link goes down. <em>Everything goes down<\/em>! :) Why? Coz no one has a clear majority. Each side has 2 votes, not the 3 required.<\/p>\n<p>Interestingly I have this setup at work. So a critical thing to keep in mind is that <em>if I were to update &amp; reboot the witness or one of the nodes in data center A (my preferred data center), and the WAN link were to go down &#8211; I could lose the cluster<\/em>! No such problems if I update &amp; reboot a node in data center B and the link goes down, as data center A has the majority. Funny, it&#8217;s like you must keep the witness in the less preferred data center.<\/p>\n<p>Windows Server 2012R2 improves upon dynamic quorum by adding dynamic witness.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4174\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?resize=660%2C373&#038;ssl=1\" alt=\"\" width=\"660\" height=\"373\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?w=1912&amp;ssl=1 1912w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?resize=768%2C434&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?resize=1024%2C578&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?resize=400%2C226&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-5.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>So if the number of votes is odd, the witness vote is removed. And if the witness is offline or failed, then too it is removed (that includes reboots too, right?).\u00a0<\/p>\n<p>Now things get tricky.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4176\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?resize=660%2C373&#038;ssl=1\" alt=\"\" width=\"660\" height=\"373\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?w=1912&amp;ssl=1 1912w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?resize=768%2C434&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?resize=1024%2C578&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?resize=400%2C226&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-6.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>Going back to the previous example: so two data centers, 2 nodes + 2 nodes, 1 witness in data center A; the quorum is therefore 5\/2 + 1 = 3.<\/p>\n<p>As before a node data center A goes down (the picture is a bit incorrect as I skipped some intermediate slides), the remaining nodes have quorum so the cluster stays put. The new quorum is 4\/2 + 1 = 3. <span style=\"text-decoration: underline;\">But<\/span> since the number of nodes is now ODD, cluster service removes the witness from the vote calculations. So the new quorum turns out to be 3\/2 + 1 = 2. At this point if the link goes down, the nodes in data center B have quorum and so they form a cluster while the remaining node in data center A is shut down. So unlike the Server 2012 case, which had no dynamic witness, the whole cluster does not go down!<\/p>\n<p>Now, going back to the case where one of the nodes (not witness) had its vote removed as there were only one node in each data center, I mentioned that the node with the lowest ID gets removed. The next two slides talk about that, including how to select a node in a cluster that we&#8217;d preferentially like to remove the vote of in such situations.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4179\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?resize=660%2C373&#038;ssl=1\" alt=\"\" width=\"660\" height=\"373\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?w=1912&amp;ssl=1 1912w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?resize=768%2C434&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?resize=1024%2C578&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?resize=400%2C226&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-7.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4180\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?resize=660%2C373&#038;ssl=1\" alt=\"\" width=\"660\" height=\"373\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?w=1912&amp;ssl=1 1912w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?resize=300%2C169&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?resize=768%2C434&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?resize=1024%2C578&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?resize=150%2C85&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?resize=400%2C226&amp;ssl=1 400w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2018\/07\/Dynamic-Quorum-8.png?w=1320&amp;ssl=1 1320w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>At this point I&#8217;d also like to link to <a href=\"https:\/\/docs.microsoft.com\/en-us\/windows-server\/storage\/storage-spaces\/understand-quorum\">this Microsoft doc on cluster quorum<\/a>. Am going to quote some parts from there as they explain well and I&#8217;d like to keep it here as as reference to myself.<\/p>\n<blockquote>\n<h3 id=\"how-cluster-quorum-works\" class=\"heading-with-anchor\">How cluster quorum works<\/h3>\n<p>When nodes fail, or when some subset of nodes loses contact with another subset, surviving nodes need to verify that they constitute the <em>majority<\/em> of the cluster to remain online. If they can\u2019t verify that, they\u2019ll go offline.<\/p>\n<p>But the concept of <em>majority<\/em> only works cleanly when the total number of nodes in the cluster is odd (for example, three nodes in a five node cluster). So, what about clusters with an even number of nodes (say, a four node cluster)?<\/p>\n<p>There are two ways the cluster can make the <em>total number of votes<\/em> odd:<\/p>\n<ol>\n<li>First, it can go <em>up<\/em> one by adding a <em>witness<\/em> with an extra vote. This requires user set-up.<\/li>\n<li>Or, it can go <em>down<\/em> one by zeroing one unlucky node\u2019s vote (happens automatically as needed).<\/li>\n<\/ol>\n<\/blockquote>\n<p>I didn&#8217;t know about point 2 until watching this video.<\/p>\n<p>Worth bearing in mind that this also applies in the case of the witness being lost. <em>So any time your witness is offline the cluster service automatically zeroes the vote of one of the nodes<\/em>. If you have 2 nodes in each data center + a witness in one data center, and you reboot the witness &#8211; that is fine. One of the nodes will have its vote zeroed out, but there&#8217;s no impact and when the witness returns the zeroed out node gets its vote back. But if during the time your witness is rebooting you also have a network outage between the two data centers, then the data center with majority nodes (i.e. not the data center containing the node whose vote was zeroed) wins and the cluster fails over there.<\/p>\n<p>Some more:<\/p>\n<blockquote>\n<h3 id=\"dynamic-witness\" class=\"heading-with-anchor\">Dynamic witness<\/h3>\n<p class=\"\">Dynamic witness toggles the vote of the witness to make sure that the <em>total number of votes<\/em> is odd. <em>If there are an odd number of votes, the witness doesn&#8217;t have a vote. If there is an even number of votes, the witness has a vote<\/em>. Dynamic witness significantly reduces the risk that the cluster will go down because of witness failure. The <em>cluster decides whether to use the witness vote based on the number of voting nodes that are available in the cluster<\/em>.<\/p>\n<p>Dynamic quorum works with Dynamic witness in the way described below.<\/p>\n<h3 id=\"dynamic-quorum-behavior\" class=\"heading-with-anchor\">Dynamic quorum behavior<\/h3>\n<ul>\n<li>If you have an <strong>even<\/strong> number of nodes and no witness, <em>one node gets its vote zeroed<\/em>. For example, only three of the four nodes get votes, so the <em>total number of votes<\/em> is three, and two survivors with votes are considered a majority.<\/li>\n<li>If you have an <strong>odd<\/strong> number of nodes and no witness, <em>they all get votes<\/em>.<\/li>\n<li>If you have an <strong>even<\/strong> number of nodes plus witness, <em>the witness votes<\/em>, so the total is odd.<\/li>\n<li>If you have an <strong>odd<\/strong> number of nodes plus witness, <em>the witness doesn&#8217;t vote<\/em>.<\/li>\n<\/ul>\n<\/blockquote>\n<p>Am pretty sure am going to forget all this a few days from today so I&#8217;ll <a href=\"https:\/\/docs.microsoft.com\/en-us\/windows-server\/storage\/storage-spaces\/understand-quorum\">re-link to the docs again<\/a> as it goes into more detail and has examples etc.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Watching this Ignite 2015 video: Under the Hood with DAGs, by Tim McMichael. Adding some links here to supplement the video: Tuning Failover Cluster Network thresholds &#8211; useful when you have stretched DAGs The mystery of the 9223372036854775766 copy queue\u2026 &#8211; never had this one but good info. Basically, the cluster registry keeps track of &hellip; <a href=\"https:\/\/rakhesh.com\/exchange\/aside-under-the-hood-with-dags\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">[Aside] Under the Hood with DAGs<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[258,86],"tags":[559,186,837,623,187],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/4160"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=4160"}],"version-history":[{"count":0,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/4160\/revisions"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=4160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=4160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=4160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}