Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

vCenter appliance stuck on “Requesting Information” when joining domain

Posted: February 27th, 2014 | Tags: , , | Category: Virtualization | §

I tried joining my freshly installed vCenter Server appliance to the domain and it was stuck on “Requesting information”. Rebooted, tried again, same results.

Then while rebooting I looked at the console to see if there were any error messages. Noticed some messages about certification errors and SSL handshakes failing. Then I remembered I had changed the appliance name from the default to something else, and also assigned a static IP etc. The renaming wen’t fine, but could it be that certificates had to be regenerated manually with the new name?

Sure enough, yes! In the appliance go to Admin and change the radio box for “Certification regeneration enabled” to Yes. Reboot, and now you’ll see messages on the console indicating that certificates are being regenerated due to a name change. Login, try joining to the domain, and now it works!

Creating a Server 2012 Failover Cluster for iSCSI target

Posted: February 25th, 2014 | Tags: , , , , | Category: Windows | §

This post is about setting up a Server 2012 R2 failover cluster that acts as an iSCSI target server.

I have four servers: WIN-DATA01, WIN-DATA02, WIN-DATA03, WIN-DATA04. I will be putting WIN-DATA03 & WIN-DATA04 in the cluster. As you know clusters required shared storage so that’s what WIN-DATA01 and WIN-DATA02 is for. In a real world setup WIN-DATA01 & WIN-DATA02 are your SAN boxes whose storage you want to make available to clients. Yes you could have clients access the two SAN boxes directly, but by having a cluster in between you can provide failover. Plus, Windows now has a cool thing called Storage Pools which let you do software RAID sort of stuff.

Prepare the iSCSI target server

First step, prepare the iSCSI target servers that will provide storage for the cluster. The steps for this are in my previous post so very briefly here’s what I did:

Repeat the above for the other server (you can login to that server and issue commands, or do remotely like I did below). Everything’s same as above except for the addition of the -ComputerName switch.

Excellent!

Now let’s move on to the two servers that will form the cluster.

Add shared storage to the servers

Here we add the two iSCSI targets created above to the two servers WIN-DATA03 & WIN-DATA04.

Login to one of the servers, open Server Manager > Tools > iSCSI Initiator.

Targets

Easiest option is to enter WIN-DATA01 in the Target field and click Quick Connect. That should list the targets on this server in the Discovered targets box. Select the ones you want and click Connect.

Another option is to go to the Discovery tab.

Discovery tab

Click Discover Portal, enter the two server names (WIN-DATA01, WIN-DATA02), refresh (if needed), and then the first tab will automatically show all targets on these two servers. Select the ones you want and click Connect as before.

The GUI sets these connections as persistent (it calls them “Favorite Target”) so they are always reconnected when the server reboots.

You can also use PowerShell to add these connections though that isn’t as easy as this point and click. Instructions are in my earlier post so here they are briefly:

Unlike the GUI, PowerShell does not mark these targets as persistent so I have to specify that explicitly when connecting.

Prepare the shared storage

The shared storage we added is offline and needs to be initialized. You can do this via the Disk Management UI or using PowerShell as below. The initializing bits need to be done on any one server only (WIN-DATA03 or WIN-DATA04) but you have to make the disks online on both servers.

Create the cluster

Login to one of the servers that will form the cluster, open Server Manager, go to Tools > Failover Cluster Manager and click Create Cluster on the right side (the Actions pane). This launches the Create Cluster Wizard.

Click Next on the first screen, enter the server names on the second …

Adding Servers to Cluster

… do or don’t do the validation tests (I skipped as this is a lab setup for me), give a name for the cluster and an IP address, confirm everything (I chose to not add all eligible storage just so I can do that separately), and that’s it.

Couple of things to note here:

  • If your cluster servers don’t have any interfaces with DHCP configured, you will also be prompted for an IP address. Otherwise a DHCP address is automatically assigned. (In my case I had an interface with DHCP).
  • A computer object with the cluster server name you specify is created in the same OU as the servers that make up the cluster. You can specify a different OU by giving the full name as the cluster name – so in the example above I would use “CN=WIN-CLUSTER01,OU=Clusters,OU=Server,DC=rakhesh,DC=local” to create the object in the Clusters OU within the Servers OU. Would have been good if the wizard mentioned that.
  • Later, when you add roles to this cluster server, it creates more virtual servers automatically. These are placed in the same OU where the cluster server object is so you must give this object rights to add/ remove computers in that OU. So it’s best you have a separate OU which you can delegate rights to. I used the Delegation Control Wizard to give the WIN-CLUSTER01 object full control over Computer Objects in the rakhesh.local/Servers/Clusters OU.

    5

Instead of the Create Cluster Wizard one can use PowerShell as below:

Configuring the cluster

A cluster has many resources assigned to it. Resources such as disks, networks, and its name. These can be seen in the summary page of the cluster or via PowerShell.

Cluster Resources

Network

I am interested in changing the IP address of my cluster. Currently it’s taken an IP from the DHCP pool and I don’t want that. Being a lab setup both my servers had a NAT interface to connect to the outside world and so the cluster is currently picking up an IP from that. I want it to use the internal network instead.

If I right click on IP address resource I can change it.

IP Address

In my case I don’t want to use this network itself so I have to go to the Networks section in the UI …

Networks

… where I can see there are two networks specified, with one of them having no cluster use while the other is configured for cluster & client use, so I right click on the first network and …

Network Settings

… enable it for cluster access, then I right click on the second network and …

Network Settings

… rename it (for my reference) as well as disable it from the cluster.

Now if I go to the resources section and right click the IP address, I can select the second network and assign a static IP address.

Here’s how to do the above via PowerShell.

To change the network name use the Get-ClusterNetwork to select the network you want (the result is an object) and directly assign the new name as a value:

One would expect to be able to set IP addresses too via this, but unfortunately these are read-only properties (notice it only has the get method; properties which you can modify will also have the set method):

To set the IP address via PowerShell we have to remember that we went to the Cluster Resource view to change it. So we do the same here. Use the Get-ClusterResource cmdlet.

That doesn’t totally help. What we need are the parameters of the resource, so we pipe that to the Get-ClusterParameter cmdlet.

Perfect! And to modify these parameters use the Set-ClusterParameter cmdlet.

To take the Cluster IP Address resource offline & online use the Stop-ClusterResource and Start-ClusterResource cmdlets:

Although it doesn’t say so, you have to also stop and start the Cluster Name resource for the name to pick up the new IP address.

Lastly, to change whether a particular network is used for cluster communication or not, one can do the same technique that was used to change the host name. Just use the Role property. I am not sure what the values for it are, but from my two networks I can see that a value of 0 means it is not used for cluster communication, while a value of 3 means it is used for cluster communication and client traffic.

The GUI is way easier than PowerShell for configuring this network stuff!

Storage (Disks)

Next I want to add disks to my cluster. Usually all available disks get added by default, but in this case I want to do it manually.

Right click Failover Cluster Manager > Storage > Disk and select Add Disk. This brings up a window with all the available disks. Since this is a cluster not every disk present on the system can be used. The disk must be something visible to all members of the cluster as it is shared by them.

Add Disk

Via PowerShell:

To add these disks to the cluster pipe the output to the Add-ClusterDisk cmdlet. There’s no way to select specific disks so you must either pipe the output to a Where-Object cmdlet first to filter the disks you want, or use the Get-Disk cmdlet (as it lets you specify a disk number) and pipe that to the Add-ClusterDisk cmdlet.

(I won’t be adding these disks to my cluster yet as I want to put them in a storage pool. I’ll be doing that in a bit).

Quorum

Quorum is a very important concept for clusters (see an earlier post of mine for more about quorum).

The cluster which I created is currently in the Node Majority mode. (Because I haven’t added and Disk or File Share witnesses to it). That’s not a good mode to be in so let’s change that.

Go to the Configure Cluster Quorum Settings as in the screenshot below, click Next …

Quorum

… choose the second option (Select the quorum witness) and click Next …

Witness

… in my case I want to use a File Share witness so I choose that option and click Next …

File Witness

… create a file share someplace, point to that, and click Next …

File Share

… click Next and then Finish.

Now it’s configured.

Storage (Pool)

(If you plan to create a storage pool add an extra shared disk to your cluster. I created a new target on the WIN-DATA02 & WIN-DATA01 server (no need for doing on both but I just did so it’s consistent), mapped another virtual disk to it, and used the iSCSI initiator on WIN-DATA03 & WIN-DATA04 to map it. Storage Pools on Failover Clusters require a minimum of three disks).

I want to use Storage Spaces (they are called Storage Pools in Failover Cluster Manager). Storage Spaces is a new feature in Server 2012 (and Windows 8) and it lets you combine disks and create virtual disks on them that are striped, mirrored, or have parity (think of it as software RAID).

We can create a new Storage Pool by right clicking on Failover Cluster Manager > Storage > Pools and select New Storage Pool.

Pool

Click Next, give the Pool a name …

Pool Name

… select the disks that will make up the pool (notice that the disks shown below are the iSCSI disk that are visible to both nodes; neither disk is actually on the local computer, they are both from a SAN box someplace (in this case the WIN-DATA01 & WIN-DATA02 servers from where we created these before) …

Pool Disks

… and click Create.

Right click on the pool that was created and make virtual disks on that. These virtual disks are software RAID equivalents. (The pool is a placeholder for all your disks. The virtual disks are the logical entities you create out of this pool). Here’s the confirmation page of the pool I created:

Pool Virtual Disk

Once the disk is created, be sure to not uncheck the “Create a volume when this wizard closes” check box. If you did uncheck, you’ll have to go to Server Manager > File and Storage Services > Volumes, click on TASKS and select New Volume. The virtual disk is just a disk, what we have to do now is create volumes on it.

Below is a screenshot of the volume I created. I chose ReFS for no particular reason, and assigned the full space to the volume. Also, the screenshot shows that I didn’t assign a drive letter. That’s incorrect. I did just that I forgot to do that when taking this screenshot. (Note that I assigned the drive letter R. This will be used later).

Volume

Via PowerShell:

As with the GUI, once the disk is created a cmdlet has to be run to create a volume on the disk. It’s probably the New-Volume cmdlet, but it’s throwing errors in my case and I am too tired to investigate further so I’ll skip it for now.

Add Roles

Finally lets add the iSCSI target server role.

Right click Failover Cluster Manager > Roles and select Configure Roles.

Click Next, select iSCSI Target Server, and click Next. This steps assumes the iSCSI Target Server role is installed on both nodes of the cluster. If not, install it via Server Manager or PowerShell.

Give the role (the virtual server that hosts the role) a name and IP address …

ISCSI Target Server

… select storage (the previously created volume; if you missed out on creating the volume go back and do it now), click Next, Next, and that’s it.

At this point it’s worth taking a step back to understand what we have done. What we did just now is create a virtual server (a role in the cluster actually) and assigned it some storage space. One might expect this storage to be the one that’s presented by the iSCSI server as a target, but no, that’s not the case. Think of this server and its storage as similar to the WIN-DATA01 and WIN-DATA02 servers that we dealt with initially. What did we do to set these as an iSCSI target? We created a target, created virtual iSCSI disks, and assigned mappings to them. That’s exactly what we have to do here too!

Ideally one should be able to use the GUI and do this, but Server Manager seems to have trouble communicating with the newly created WIN-DATA server. So I’ll use PowerShell instead.

And that’s it! Now I can add this target back to the WIN-DATA01 & WIN-DATA02 servers if I want (as those are the Initiator IDs I specified) and whatever I write will be written back to their disks via this clustered iSCSI target. I am not sure if the mirroring will happen across both servers though, but this is all just for fun anyways and not a real life scenario.

Lastly …

Before I conclude here’s something worth checking out.

The WIN-DATA virtual server is currently on the WIN-DATA04 node. This means WIN-DATA04 is the one who is currently providing this role, WIN-DATA03 is on standby.

If I login to WIN-DATA04 and check its disks I will see the mirrored volume I created:

If I do the same on WIN-DATA03 I won’t see the volume.

If I right click on the role in Failover Cluster Manager, select Move > Select Node, and select the new node as WIN-DATA03, then this becomes the new active node. Now if I check the volumes and disks of both servers the information will be the other way around. WIN-DATA03 will have the volume and disk, WIN-DATA04 won’t have anything!

This is how clustering ensures both servers don’t write to the shared storage simultaneously. Remember, the shared storage is just a block device. It doesn’t have any file locking nor is it aware of who is writing to it. So it’s up to the cluster to take care of all this.

Also …

Be sure to add WIN-CLUSTER01 and WIN-DATA to DNS with the correct IPs. If you don’t do that Server Manager and other tools won’t be able to resolve the name.

There’s more …

This post is just a tip of the iceberg. There’s so many more cool things you can do with iSCSI, Clustering, and Storage Spaces in Server 2012 so be sure to check these out elsewhere!

Server 2012 add/ remove initiator IDs for an iSCSI target

Posted: February 24th, 2014 | Tags: , , | Category: Windows | §

Once you create an iSCSI target on Windows Server 2012 there doesn’t seem to be a way to add/ remove initiator IDs via the GUI. But you can use PowerShell for it. Set-IscsiServerTarget is your friend.

Bear in mind this replaces the existing list with the new one.

A cool thing about most of these iSCSI cmdlets is that they work remotely too. So one can add a -ComputerName xxx switch to work with a remote computer.

Notes on iSCSI and Server Core 2012

Posted: February 24th, 2014 | Tags: , , , | Category: Windows | §

For the Initiator (client)

Prerequisites

  1. Be sure to enable the enable the MSiSCSI service and set its startup type to Automatic.

  2. Ensure the iSCSI initiator outgoing & incoming rules are allowed on the firewall.

Connecting to a Target

To connect to a target you have to connect to a target portal first. The portal lets you enumerate the targets available.

After connecting, list the targets thus:

Do this for each portal you are aware of:

Note that Get-IscsiTarget now lists targets from all the portals.

To connect to a specific target, use the Connect-IscsiTarget cmdlet:

Viewing the disks

To view the disks available, use the Get-Disk cmdlet.

Numbers 2 & 3 are the iSCSI disks.

As far as the OS is concerned these are regular disks. To use them: 1) Change their status to online, 2) Initialize the disk, and 3) Partition & Format.

If you have many disks (physical or via iSCSI) you can also use the Get-IscsiConnection and Get-IsciSession to select just the iSCSI connections you want, and filter these to the Get-Disk cmdlet.

Making a session persistent across reboots

By default iSCSI connections are not persistent across reboots (note the IsPersistent : False above). To make a connection persistent there area two options:

  1. When connecting to the iSCSI target the first time use the -IsPersistent switch.

  2. If you want to make an existing session persistent, pipe it to the Register-IcsiSession cmdlet.

To make a session non-persistent, pipe it to the Unregister-IscsiSession cmdlet.

Disconnecting a Target

If you would like to disconnect a session use the Disconnect-IscsiTarget cmdlet.

For the Target (server)

  1. Create a Target.

    When creating the target it is very important to specify the initiator IDs. The cmdlet doesn’t prompt for these if you don’t mention (unlike the UI which doesn’t go ahead unless initiator IDs are specified). If the initiator IDs are missing no one can see this target.

    Initiator IDs can be specified via IQNs, IP addresses, IPv6 addresses, DNS name, and MAC addresses. Notice the ‘IPN:xxx’ bit above? Replace IQN with IPAddress, IPv6Address, DNSName, and MACAddress if you are specifying initiator IDs using these, followed by the address or name. For instance:

    If there’s only one initiator ID it can be specified as it is. If there multiple, separate them with commas. There’s no need to put them inside @(…) as above – that’s just to make it explicit to the reader that the input is an array. Separating by commas will have the same effect irrespective of the @(…) notation.

    It’s also worth pointing out that if multiple initiators are allowed to access a target, they will all be able to read/write to the target, but expect corruption. There are exceptions of course.

  2. Create an iSCSI virtual disk which will back the LUN presented by the target.

  3. Create multiple virtual disks if needed.
  4. Map these virtual disks to the target.

And that’s it. Initiators can now connect to the target. In the example above, since the target has two LUNs, initiators will see two disks after connecting.

NIC teaming in Windows Server 2012

Posted: February 21st, 2014 | Tags: , | Category: Windows | §

Windows Server 2012 includes NIC teaming as part of the OS now so there’s no need for additional software. I find that pretty cool. It’s quite easy to setup too. You can do via the Server Manager GUI or use PowerShell. Just one PowerShell cmdlet, that’s it!

So what is NIC teaming?

Scenario 1: Say your server has one network card (NIC) and is connected to one switch port. If the NIC fails for any reason, your server is kaput – offline! Similarly if the switch fails for any reason, your server is kaput.

Scenario 2: Say your server has two 1GbE NICs. That is, two NICs that supports 1Gbps bandwidth each. And say your server runs two network intensive applications. Both applications use 1Gbps bandwidth each. Even though you have two NICs of 1Gbps each, you can’t use them together and so there’s no way to balance your two applications over the two NICs. You are stuck with just 1Gbps bandwidth.

These are two areas where NIC teaming can help.

NIC teaming simply means you combine your NICs into one team. (Windows Server 2012 can combine 32 such NICs into one team. And these NICs can be from different hardware providers too!) Before Windows Server 2012 you needed third party software to do NIC teaming – often from the network card manufacturer. But if you have Windows Server 2012 this is built into the OS.

The two usage scenerios above are called Link Balancing (LB) and FailOver (FO) – hence another name for NIC teaming is LBFO.

Link Balancing means you team your NICs into one and traffic is balanced between the two NICs. If you had two 1GbE NICs, the net effect is not as though you had a 2GbE NIC. No, the NICs are still independent in terms of bandwidth, but since you can use both NICs you are able to get more bandwidth. If you have an application on your server communcating with some client, all traffic between them related to a particular “flow” goes over the same NIC. But traffic to other clients can go over the other NIC, and also traffic to the same client but unrelated to the prior flow can go over the other NIC. There is a reason for this. If traffic for a particular flow went over both NICs, you could have packets appearing out of order (maybe one NIC has a faster path than the other) and that leads to an overhead of reassembling these packets.

Failover means you team your NICs into one and if one NIC fails traffic isn’t interrupted as it goes through the other NIC. This also helps in cases where the switch that a NIC is connected to, or the cable between them, goes down. When that NIC goes offline, other NICs will take over.

Now, NIC teaming isn’t a one sided job. Although the NICs may be teamed, as far as the network switch(es) they connect to is(are) concerned, they are still disparate. The switch(es) see one IP address, associated with multiple MAC addresses – on different ports/ switch – and so you need to involve the switch(es) too in your plan of NIC teaming. That said, you may choose to not involve the switches too if you want, but that has its own implications.

If you decide to involve the switch, there are two ways of going about with the teaming. One is a static way in which your switch admin designates the ports that the multiple NICs connect to as part of a team, and then the switch will take care of things. This has a drawback in that if you move the NICs to any other switch port, teaming breaks unless the network admin reconfigures the new ports on the switch.

The other way is an automatic way, in which if your switch(es) support the Link Aggregration Control Protocol (LACP; usually referred to also as IEEE 802.3ad or 802.1ax) you can use any ports on the switch and the ports will automatically figure out they are part of a team. Of course, if you decide to use LACP you have to use LACP switches and also tell the bits in the OS responsible for teaming that the switch knows LACP so it can behave accordingly. LACP works by sending some frames (called LACPDUs) from NIC to the switch to determine which ports are part of the same team and then combining these ports into a single link.

It’s important to note here that teaming which involves the switch, usually called “switch dependent” teaming, requires all the NICs to the be connected to the same switch (but don’t quote me on this as I am not a networks guy and this is just my understanding!).

If you decide not to involve the switch, usually called “switch independent” teaming, you can use multiple switches as the switches don’t know anything and all the action happens at the NIC end. This has some implications though, in that traffic from the switch to your teamed NICs will not be distributed as the switch doesn’t know of the multiple NICs. As far as the switch(es) knows your teamed NIC interface has one MAC address – that of one of the NICs – and so that NIC is the one which will get all the incoming traffic.

(As an aside, this is why if we decide to take a NIC out of the team and use it independently, it is a good idea to disable the team and enable. If the NIC we are taking out has the same MAC address as the team, it won’t work because the switch won’t be aware of it. Disabling and enabling the team will cause the teamed NIC to pick up a new MAC address from the remaining NICs).

When setting up NIC teaming in Windows Server 2012 you can choose from the options above.

Apart from deciding how the teaming is setup, we have to also select what algorithm is used for load balancing (the LB in LBFO). Windows Server 2012 gives you 2 options if you use the GUI, and 5 options if you use PowerShell. Let’s look at these now.

HyperVPort

  • This is used in case your Windows Server 2012 box functions as the host for a bunch of VMs. Each VM only sees one NIC – the teamed NIC – but internally, the host assigns each VM to a particular NIC. (Say you have two NICs in the team, and 5 VMs. VMs 1,3,5 could be on NIC 1, VMs 2,4 could be on NIC 2).
  • Obviously, selecting such a load balancing algorithm means any particular VM is always limited by the bandwidth of the NIC it is internally assigned to. It can’t aggregate over the other NICs to increase its bandwidth.
  • If you do switch independent teaming, the switch will send incoming traffic for each VM to the NIC assigned to it as the switch is aware these are different VMs. If you do switch dependent teaming, the incoming traffic is anyways distributed.

Address Hashing

  • This option creates a hash based on various components of the packet, and assigns all packets with that hash to a particular NIC. The components chosen for the hash vary. You can choose to use the source and destination IP addresses and TCP & UDP ports. You can choose to use the source and destination IP address. Or you can choose to use the source and destination MAC address. (If you choose the first option, for instance, and the packet is say an IPSec packet and hence has no TCP & UDP port, the second option is automatically used. Similarly if you choose the second option and that cannot be used for any reason, the last option is used).
  • If you do switch independent teaming, since the NIC used keeps varying the teamed NIC uses the MAC address of one of the NICs and that’s the MAC address the switch(es) the NICs connect to is aware of. So all incoming traffic from the switch to NICs is sent to that particular NIC whose MAC address is used. If you do switch dependent teaming, the incoming traffic is anyways distributed.
  • When using the GUI, the default option is the first one (use source and destination IP addresses and TCP & UDP ports).
  • When using PowerShell too, the default option is the first one (it is called TransportPorts). The second option is called IPAddresses and the third option is called MacAddresses.

Dynamic

I couldn’t find much information on this. This is an option available only in PowerShell (it is called Dynamic) and it seems to be an algorithm that automatically load balances between NICs depending on their load. When using switch independent teaming, incoming traffic is not distributed and is sent to a particular NIC; when using switch dependent teaming incoming traffic is distributed.

That’s all!

Clusters & Quorum

Posted: February 20th, 2014 | Tags: , , , | Category: Virtualization, Windows | §

I spent yesterday refreshing my knowledge of clusters and quorum. Hadn’t worked on these since Windows Server 2003! So here’s a brief intro to this topic:

There are two types of clusters – Network Load Balancing (NLB) and Server Clusters.

Network Load Balancing is “share all” in that every server has a full copy of the app (and the data too if it can be done). Each server is active, the requests are sent to each of them randomly (or using some load distributing algorithm). You can easily add/ remove servers. Examples where you’d use NLB are SMTP servers, Web servers, etc. Each server in this case is independent of the other as long as they are configured identically.

Server Clusters is “share nothing” in that only one server has a full copy of the app and is active. The other servers are in a standby mode, waiting to take over if the active one fails. A shared storage is used, which is how standby servers can take over if the active server fails.

The way clusters work is that clients see one “virtual” server (not to be confused with virtual servers of virtualization). Behind this virtual server are the physical servers (called “cluster nodes” actually) that make up the cluster. As far as clients are concerned there’s one end point – an IP address or MAC address – and what happens behind that is unknown to them. This virtual server is “created” when the cluster forms, it doesn’t exist before that. (It is important to remember this because even if the servers are in a cluster the virtual server may not be created – as we shall see later).

In the case of server clusters something called “quorum” comes into play.

Imagine you have 5 servers in a cluster. Say server 4 is the active server and it goes offline. Immediately, the other servers detect this and one of them becomes the new active server. But what if server 4 isn’t offline, it’s just disconnected from the rest of the group. Now we’ll have server 4 continuing to be active, but the other servers can’t see this, and so one of these too becomes the active server – resulting in two active servers! To prevent such scenarios you have the concept of quorum – a term you might have heard of in other contexts, such as decision making groups. A quorum is the minimum number of people required to make a decision. Say a group of 10 people are deciding something, one could stipulate that at least 6 members must be present during the decision making process else the group is not allowed to decide. A similar concept applies in the case of clusters.

In its simplest form you designate one resource (a server or a disk) as the quorum and whichever cluster contains that resource sets itself as the active cluster while the other clusters deactivate themselves. This resource also holds the cluster database, which is a database containing the state of the cluster and its nodes, and is accessed by all the nodes. In the example above, initially all 5 servers are connected and can see the quorum, so the cluster is active and one of the servers in it is active. When the split happens and server 4 (the currently active server) is separated, it can no longer see the quorum and so disables the cluster (which is just itself really) and stops being the active server. The other 4 servers can still see the quorum, so they continue being active and set a new server as the active one.

In this simple form the quorum is really like a token you hold. If your cluster holds the token it’s active; if it does not you disband (deactivate the cluster).

This simple form of quorum is usually fine, but does not scale to when you have clusters across sites. Moreover, the quorum resource is a single point of failure. Say that resource is the one that’s disconnected or offline – now no one has the quorum, and worse, the cluster database is lost. For this reason the simple form of quorum is not used nowadays. (This mode of quorum is called “Disk Only” by the way).

There are three alternatives to the Disk Only mode of quorum.

One mode is called “Node Majority” and as the name suggests it is based on majority. Here each node has a copy of the cluster database (so there’s no single point of failure) and whichever cluster has more than half the nodes of the cluster wins. So in the previous example, say the cluster of 5 servers splits into one of 3 and 2 servers each – since the first cluster has more than half of the nodes, that wins. (In practice a voting takes place to decide this. Each node has a vote. So cluster one has 1+1+1 = 3 votes; cluster two has 1+1 = 2 votes. Cluster one wins).

Quorums based on majority have a disadvantage in that if the number of nodes are even then you have a tie. That is, if the above example were a 6 node cluster and it split into two clusters of 3 nodes each, both clusters will deactivate as neither have more than half the nodes (i.e. neither have the quorum). You need a tie breaker!

It is worth noting that the effect of quorum extends to the number of servers that can fail in a cluster. Say we have a 3 node cluster. For the cluster to be valid, it must have at least 2 servers. If one server fails, the cluster still has 2 servers so it will function as usual. But if one more server fails – or these two servers are disconnected from each other – the remaining cluster does not have quorum (there’s only 1 node, which is less than more than half the nodes) and so the cluster stops and the servers deactivate. This means even though we have one server still running, and intuitively one would expect that server to continue servicing requests (as it would have in the case of an NLB cluster), it does not do so in the case of a server cluster due to quorum! This is important to remember.

Another mode is called “Node & Disk Majority” and as the name suggests it is a mix of the “Node Majority” and “Disk Only” modes. This mode is for clusters with an even number of modes (where, as we know, “Node Majority” fails) and the way it works is that the cluster with more than half the nodes and which also contains the resource (a disk, usually called a “disk witness”) designated as quorum is the active one. Essentially the disk witness essentially acts as the tie breaker. (In practice the disk witness has an extra vote. So a cluster with 6 nodes will still require more than 3 nodes to consider it active and so if it splits into 3 nodes each, when it comes to voting one of the clusters will have (3+1=) 4 votes and hence win quorum).

In “Node & Disk Majority” mode, unlike the “Disk Only” mode the cluster database is present with all the nodes and so it is not a single point of failure either.

The last mode is called “Node & File Share Majority” and this is a variant of the “Node Majority” mode. This mode too is for clusters with an even number of nodes, and it works similar to “Node & Disk Majority” except for the fact that instead of a resource a file share is used. A file share (called a “file witness” in this case) is selected on any server – not necessarily a part of the cluster – and one node in the cluster locks a file on this share, effectively telling others that it “owns” the file share. So instead of using a resource as a tie breaker, ownership of the file share is used as the tie breaker. (As in the “Node & Disk Majority” mode the node that owns the file share has an additional vote as it owns the file share). Using the previous examples, if a cluster with 6 nodes splits into 3 nodes each, whichever cluster has the node owning the file share will win quorum while the other cluster will deactivate. If the cluster with 6 nodes splits into clusters of 4 and 2 nodes each, and say the 2 node cluster owns the file share, it will still lose as more than half the nodes are in the 4 node cluster (in terms of votes the winning cluster has 4 votes, the losing cluster has 3 votes). When the cluster deactivates, the current owner will release the lock and a node in the new cluster will take ownership.

An advantage of the “Node & File Share Majority” mode is that the file share can be anywhere – even on another cluster (preferably). The file share can also be on a node in the cluster, but that’s not preferred as if the node fails you lose two votes (that of being the file share owner as well as the node itself).

Here are some good links that contain more information (Windows specific):

At work we have two HP LeftHand boxes in a SAN cluster. The only quorum mode used by this is is the “Node Majority” one, which as we know fails for an even number of nodes, and so HP supplies a virtual appliance called “Failover Manager” that is installed on the ESX hosts and is used as the tie breaker. If the Failover Manager is powered off and both LeftHands are powered on together, suppose both devices happen to come on at the same time a cluster is not formed as neither has the quorum. To avoid such situations the Failover Manager has to be present when they power on, or we have to time the powering on such that one of the LeftHands is online before the other.

Using Physical Disks with VMware workstation

Posted: February 20th, 2014 | Tags: | Category: Virtualization, Windows | §

I want to try out the new iSCSI target feature of Windows Server 2012. This feature only works with virtual disks however, not physical disks, so I’ll have to create a VHD file in the machine and use that as a target. No problemo, just that the server itself is running from a virtual disk (as it’s a VM running on VMware Workstation) and it kind of feels odd to have a virtual disk inside another virtual disk – all that overhead is going to make it slow! So I decided to pass a physical disk instead, to the server, and let it create the VHD file on that.

First, decide if you are going to allocate a disk or a partition to the virtual machine. I was going to allocate a partition, so I cleared up some space, made an empty partition on it, edited the settings of the virtual machine, and added this partition as a physical disk. One thing to keep in mind is to NOT choose the “Independent disks” mode in settings. I selected this mode initially as I didn’t want the disk to be used when making snapshots, but this had the unintended effect of making the disk read-only to the guest. I am guessing this is a bug. Independent mode doesn’t really apply to physical disks anyway as they bypass the hypervisor and so can’t be snapshotted.

After adding the disk, power on the virtual machine. The disk will be present, but you’ll have to get it ready before being able to use it, so here’s what I did. Note that I am using Windows Server 2012 Core so diskpart is all I have.

The formatting fails! This is because the disk is shown as read-only for some reason so we have to tell diskpart to clear that attribute. Do the following –

And now it formats and we can use the disk! Thanks to this KB article where I read about clearing disk attributes.

Cloning Server Core 2012 using VMware Workstation

Posted: February 18th, 2014 | Tags: , , | Category: Virtualization, Windows | §

Pretty straightforward, but it’s not obvious whether cloning the machine will also sysprep. It does not. So after cloning the machine, don’t forget to sysprep it.

  1. Ensure the VM is powered off.
  2. Right click the VM > Manage > Clone.
  3. The cloning wizard is launched. Go ahead with it. I chose the full clone option.
  4. After cloning power up the machine. At this point it is identical to the original so it’s best the original is kept powered off to avoid any conflicts.
  5. Login to the machine, go to C:\Windows\System32 run sysprep.exe. Keep the default system cleanup action but click Generalize so the SIDs etc are recreated. I selected to reboot under Shutdown Options.
  6. That’s pretty much it. Upon reboot you are given the option of logging in with the local accounts; upon logging in you have to change the password.
  7. Now set the IP, name, etc as usual. Enjoy!

These instructions apply to all versions of Windows. Server Core 2012 is just what I used it with.

Initial Server 2012 Core configuration using PowerShell

Posted: February 18th, 2014 | Tags: , , , | Category: PowerShell, Windows | §

Just for future reference to myself.

  1. Install Server Core 2012 as usual.
  2. Login. Change the IP address thus:
  3. Specify DNS server addresses:
  4. Rename the computer:
  5. Restart the computer so the name change takes effect.
  6. Update: Rename the computer & join the domain (thanks to Daniel Streefkerf (check out his blog, if you like my blog posts you’ll surely enjoy his!) for pointing this out – since PowerShell 3.0 the Add-Computer cmdlet has a -NewName switch that lets you rename and add/ move a computer):

Update: If you want to do NIC teaming, do the following after step 1.

Now continue with assigning IP address etc, but use the newly created Team adapter instead.

Running VMware ESXi within Hyper-V

Posted: February 17th, 2014 | Tags: , | Category: Virtualization | §

VMWare ESXi can run within Hyper-V, but there are some gotchas.

First thing to remember is that even if you get this working Hyper-V does not expose the CPU hardware virtualization features to guests so ESXi will complain that hardware virtualization is missing. ESXi will install and be able to run 32-bit guests, but it won’t do be able to run 64-bit OSes. (There is a lot of overhead virtualizing a 64-bit guest, much of which can be eliminated by hardware virtualization. That’s why you can’t run a 64-bit guest without hardware virtualization). As an aside, the ESXi hypervisor can expose hardware virtualization features to its guests so it’s easier to run Hyper-V inside ESXi and have 64-bit VMs inside it.

When booting up ESXi 5.5 within Hyper-V (on my Windows 8.1 and Windows 8 machines) the ESXi installer gets up to the stage of “Relocating modules and starting up the kernel” and is stuck. I read on some forum posts that it’s to do with an increased display memory requirement for ESXi 5.5 and so I tried installing ESXi within VirtualBox but that didn’t help either. (VirtualBox lets you change the display memory. To use VirtualBox, however, you’ll have to uninstall the Hyper-V role from the host machine. As long as Hyper-V is installed VirtualBox can’t see the hardware virtualization features and so is limited to one virtual CPU for the guest – which won’t do for ESXi). I also tried older versions of ESXi – 5.1 and 5.0 – with similar results (instead of being stuck the screen would blank out).

Back to Hyper-V I tried increasing the amount of RAM but that didn’t help either. Finally I came across this blog post which details a workaround for Dell servers – with the exact symptoms that I was having in Hyper-V – and that worked for me. Sort of. Now the screen would blank out instead of being stuck. However, this time when I increased the RAM allocated to ESXi to 4GB it worked!

That isn’t the end though. Although the installer now starts, it complains about the lack of a network card. Turns out ESXi does not have drivers for the Hyper-V network adapters (legacy or non-legacy). You can, however, get drivers for the legacy adapter and build a custom install CD (thanks to this blog post which seems to be down now; original instructions are at this, this, and this forum posts, followed by additional configuration as per this forum post to put the ESXi guest network adapter in promiscuous mode for networking to work). Once you do all that, finally ESXi is able to install within Hyper-V!

p.s. Link to self with a forum post by someone who read this post and contacted me. Summarizes the steps succintly.

Unable to access to VMware Workstation guest from the host?

Posted: February 17th, 2014 | Tags: , | Category: Virtualization, Windows | §

Yesterday I setup an ESXi guest within a VMware Workstation host. Connected the guest to a custom network and ensured that a virtual adapter for this network is created on the host. Assigned IPs etc but whatever I do I couldn’t get the host and the guest to ping each other. Disabled firewalls, fiddled with the routes, no luck!

Finally I realized I had enabled a VLAN on the ESXi guest management interface. Doh! Even though both are on the same subnet, placing them on different VLAN means they are on a different network from each other and so obviously can’t communicate with each other unless you have a router in between!

I tried to set the VMware virtual adapter on the host to a different VLAN but couldn’t find any tools for doing. There are 802.1q drivers for supported physical adapters (and also for the “e1000” virtual network adapter for guests) that let you assign VLANs but I couldn’t find anything for the virtual adapter on the host side.

Microsoft Account and shared folders

Posted: February 4th, 2014 | Tags: | Category: Windows | §

If you are using a Microsoft Account on your computer, it’s worth keeping in mind that traditional shared folders won’t work. By traditional shared folders I mean from other computers in the same workgroup as yours, but without a Microsoft Account.

I had two machines – a Windows 8 client, and Windows Server 2008 R2 server. Client had an account with username “rakhesh” which I later converted to a Microsoft Account. Since the username still appears in many places as “<MACHINENAME>>\rakhesh” I figured I should be able to access my shared folders on the client, from the server, by logging in as “<MACHINENAME>\rakhesh”. But no, that does not work. Finally I had to create a new local account on the client and use that for shared folders.

The reverse too applies. At home I have two Windows 8 clients. They are not in a HomeGroup, just a regular workgroup. Both have the same user account but one of them is converted to a Microsoft Account. The shared folders are on the machine with a local account and whenever I try accessing these from the machine with a Microsoft Account it fails. I worked around that by mapping the folders to a drive and specifying the local account credentials then. I believe one can also use the Credential Manager for this but I haven’t tried it yet.

The new Paper app from Facebook

Posted: February 4th, 2014 | Tags: , , | Category: Gadgets | §

Just tried out the new Paper app from Facebook. Blown away by its design and interface. I think I might finally start following my Facebook newsfeed again.

The cool thing about the Paper app is that it gives your Facebook stories equal footing as other stories. Previously I used to use Flipboard, Google+, or Twitter to follow stories while Facebook was just for following news and pics from my friends. This usually led to the Facebook app being rarely opened as general news was more important to me than friends news. Moreover the linear interface while fine wasn’t conducive for the media rich content that my friends were sharing.

The Paper app changes all that. It lets you subscribe to various categories (called Headlines) such as Tech, Pop Culture, etc and also shows your Facebook feed as another category (the first in fact). Moreover, the stories are shown in a Flipboard style interface – both layout and gestures – and that’s very good for quickly flipping through all the stories (news as well as friend updates). Now I don’t have to choose between these two. One app let’s me read both, share to Facebook easily, and even save to apps like Instapaper and Pocket for reading later.

I think the cool realisation that Facebook struck upon here is that rather than having an app that was solely about reading and posting to Facebook, it made an app that is just about ensuring users spend their time in that app. As a side effect of that users automatically end up spending time on their Facebook feeds too. And based on the stories users read, like, and share – which they might not have done otherwise if they were reading the stories in another app – Facebook is able to capture all these signals about its users. Which is all the more useful to Facebook.

Check it out!