Playing with XenServer in my testlab (basically as a VM in VMware Workstation hah!) I ran into trouble while creating a Machine Catalog via Citrix Desktop Studio. I forget the exact message but it was about lack of resources. I could see that in the create catalog process it was creating a snapshot and making a copy VM, powering it on and off successfully, and then it was failing. I kept an eye on my storage during this and saw that indeed it was exceeding the allocated space. I had thought it would do thin provisioning but in retrospect I realize XenServer never asked me about thick or thin when I added my iSCSI storage. Hmm.
Well turns out that for iSCSI XenServer has only thick provisioning. You get thin provisioning only if you are using the ext3 filesystem or NFS. Since iSCSI uses LVM, bummer!
Here’s a forum post on how to identify if your SR is thick or not.
Regarding thin provisioning – it is only for locally attached storage (which can use ext3) or NFS. Block attached storage is thick.
Before I realized all this I had spent some Googling on how to create a thin provisioned SR (Storage Repository). I felt that maybe it’s a GUI restriction and I can workaround by using the CLI. Turns out I was wrong. Here’s an article that explains SRs in XenServer anyway. It’s a good read. Here’s an article just on enabling thin provisioning for ext3 SRs via the CLI.
While on the topic of storage, this is something I wanted to blog about earlier but never got around to. When using SMB/ CIFS shares, XenServer only supports NTLMv1. Here’s instructions on using NTLMv2.
smbclient is a good tool to test SMB connects from a XenServer. Example:
[root@xs7-02 /]# smbclient -L //10.10.1.12/Files -U rockylabs\rakhesh -m SMB3
Enter rockylabsrakhesh's password:
session setup failed: NT_STATUS_LOGON_FAILURE
That seems to work, but I get a logon failure. This is because I didn’t put the username in quotes.
[root@xs7-02 /]# smbclient -L //10.10.1.12/Files -U 'rockylabs\rakhesh' -m SMB3
Enter rockylabs\rakhesh's password:
Domain=[ROCKYLABS] OS= Server=
Sharename Type Comment
--------- ---- -------
ADMIN$ Disk Remote Admin
C$ Disk Default share
D$ Disk Default share
IPC$ IPC Remote IPC
I have no idea what the three commands below except that they are to do with mounting an SMB/ CIFS share on a XenServer permanently. I had noted these commands as part of my would be blog post, but it’s been a while now and I forget. Sometime when I get around to doing SMB3 or NTLMv2 with XenServer again I hope to refer to these again and better explain. I don’t want to spend too much time on XenServers now and get sidetracked …
xe sr-create name-label=data02_ISO_Store type=ISO shared=true device-config:location=//data02.rockylabs.zero/Files device-config:username='rockylabs\xssmb' device-config:password='Password1' device-config:options='-t cifs'
xe-mount-iso-sr //data02.rockylabs.zero/Files -o username=xssmb,password=Password1,domain=rockylabs,vers=3
# I don't think this command really works - not sure
xe-mount-iso-sr //data02.rockylabs.zero/Files -o username=xssmb,password=Password1,domain=rockylabs,sec=ntlmv2
After issue the above commands I think the shared folder is mounted only on one host in the pool. But right clicking on it and doing a repair will get it mounted on all hosts in the pool.
XenServer 7.0 and above support SMB for VM disk storage too. Prior versions support SMB only for ISO storage.
These are some notes to myself on the Windows security subsystem.
On Windows, the Local Security Authority (LSA) is a subsystem that is responsible for security of the system. The LSA runs as a process called the LSA Subsystem Service (LSASS; you can find it as
c:\Windows\System32\lsass.exe) and takes care of two tasks: (1) authentication and (2) enforcing local security policies on system.
For authentication the LSA makes uses of Security Support Providers (SSPs) that provide various authentication protocols. SSPs are Dynamic Link Libraries (DLLs) that offer the authentication protocol to applications that wish to make use of it. They expose a Security Service Provider Interface (SSPI) API which applications can make use of without knowing about the underlying protocol. (Generic Security Service Application Program Interface (GSSAPI or GSS-API) is an IETF standard that defines an API for programs to access security services. SSPI is is a proprietary variant of GSSAPI).
In a way this post ties in with other things I have been reading about and posted recently. Stuff like encryption ciphers and Active Directory. On domain joined machines for instance, LSA uses Active Directory, while on non-domain joined machines LSA uses Security Accounts Manager (SAM). Either case the LSA is a critical component.
It is possible to create custom SSPs to support new protocols. Microsoft includes the following SSPs (may not be an exhaustive list).
- Provides the Kerberos authentication protocol. This is the protocol of choice in Windows.
- Kerberos cannot be used with non-domain joined systems.
- More about Kerberos in a later post. I plan to cover it as part of Active Directory.
NTLM — LM, NTLM, and NTLMv2 (
- Provides the NTLM authentication protocol.
- LM == LAN Manager (also called as LAN MAN). It’s an old way of authentication – from pre Windows NT days. Not recommended any more.
- NTLM == NT LAN Manager. Is a successor to LM. Introduced with Windows NT. Is backward compatible to LAN MAN. It too is not recommended any more. It’s worth pointing out that NTLM uses RC4 for encryption (which is insecure as I previously pointed out).
- NTLMv2 == NT LAN Manager version 2. Is a successor to NTLM. Introduced in Windows 2000 (and in Windows NT as part of SP4). It is the current recommended alternative to LM and NTLM and is the default since Windows Vista.
- Although Kerberos is the preferred protocol NTLM is still supported by Windows.
- Also, NTLM must be used on standalone systems as these don’t support Kerberos.
- NTLM is a challenge/ response type of authentication protocol. Here’s how it works roughly:
- The client sends its username to the server. This could be a domain user or a local user (i.e. stored in the server SAM database). Notice that the password isn’t sent.
- To authenticate, the server sends some random data to the client – the challenge.
- The client encrypts this data with a hash of its password – the response. Notice that the hash of the password is used as a key to encrypt the data.
- If the username is stored in the server SAM database, the hash of the password will be present with the username. The server simply uses this hash to encrypt its challenge, compares the result with the response from the client, and if the two match authenticates the client.
- If the username is not stored in the server SAM database, it sends the username, the challenge, and response to a Domain Controller. The Domain Controller will have the password hash along with the username, so it looks these up and performs similar steps as above, compares the two results, and if they match authenticates the client.
- Here are some interesting blog posts on NTLM security:
- NTLM Challenge Response is 100% Broken – talks about vulnerabilities in NTLM & LM, and why it’s better to use NTLMv2.
- The Most Misunderstood Windows Security Setting of All Time – about the
HKLM\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel registry key (mentioned in the above blog post too) which affects how Windows uses NTLMv2. By default Vista and above only send NTLMv2 reponses but accept LM, NTLM, and NTLMv2 challenges. This post also goes into how NTLMv2 performs the challenge/ response I mention above differently.
- NTLMv2 uses a different hash function (HMAC-MD5 instead of MD4).
- NTLMv2 also includes a challenge from the client to the server.
- It is also possible to use NTLM for authentication with NTLMv2 for session security.
- Rehashing Pass the Hash – a blog post of Pass the Hash (PtH) which is about stealing the stored password hash (in memory) from the client and using that to authenticate as the client elsewhere (since the hash is equivalent to the password, getting hold of the hash is sufficient). This post also made me realize that LM/ NTLM/ NTLMv2 hashes are unsalted – i.e. the password is hashed and stored, there’s no extra bit added to the password before salting just to make it difficult for attackers to guess the password. (Very briefly: if my password is “Password” and its hashed as it is to “12345”, all any attacker needs to do is try a large number of passwords and compare their hash with “12345”. Whichever one matches is what my password would be! Attackers can create “hash tables” that contain words and their hashes, so they don’t even have to compute the hash to guess my password. To work around this most systems salt the hash. That is, the add some random text – which varies for each user – to the password, so instead of hashing “Password” the system would hash “xxxPassword”. Now an attacker can’t simply reuse any existing hashtables, thus improving security).
- A good blog post that illustrates Pass the Hash.
- A PDF presentation that talks about Pass the Hash.
- Windows 8.1 makes it difficult to do Pass-the-Hash. As the post says, you cannot eliminate Pass-the-Hash attacks as long as the hash is not in some way tied to the hardware machine.
- When you login to the domain, your computer caches a hash of the password so that you can login even if your Domain Controller is down/ unreachable. This cache stores an MD4 hash of the “MD4 hash of the password + plus the username”.
- If all the above isn’t enough and you want to know even more about how NTLM works look no further than this page by Eric Glass. :)
- This is a psuedo-SSP. Also called the Simple and Protected GSS-API Negotiation Mechanism (SPNEGO).
- It lets clients and servers negotiate a protocol to use for further authentication – NTLM or Kerberos. That’s why it is a psuedo-SSP, it doesn’t provide any authentication of its own.
- Kerberos is always selected unless one of the parties cannot use it.
- Also, if an SPN (Service Principal Name), NetBIOS name, or UPN (User Principal Name) is not given then Kerberos will not be used. Thus if you connect to a server via IP address then NTLM will be used.
- Provides the Credential Security Support Provider (CredSSP) protocol.
- This allows for user credentials from a client to be delegated to a server for remote authentication from there on. CredSSP was introduced in Windows Vista.
- Some day I’ll write a blog post on CredSSP and PowerShell :) but for now I’ll point to this Scripting Guy blog post that gives an example of how CredSSP is used. If I connect remotely to some machine – meaning I have authenticated with it – and now I want to connect to some other machine from this machine (maybe I want to open a shared folder), there must be some way for my credentials to be passed to this first machine I am connected to so it can authenticate me with the second machine. That’s where CredSSP comes into play.
- CredSSP uses TLS/SSL and the Negotiate/SPNGO SSP to delegate credentials.
- More about CredSSP at this MSDN article.
- Provides the Digest protocol. See these TechNet articles for more on Digest authentication and how it works.
- Like NTLM, this is a challenge/ response type of authentication protocol. Mostly used with HTTP or LDAP.
- There is no encryption involved, only hashing. Can be used in conjunction with SSL/TLS for encryption.
- Provides the SSL/ TLS authentication protocols and support for Public Key Infrastructure (PKI).
- Different versions of Windows have different support for TLS/SSL/ DTLS because the SChannel SSP in that version of Windows only supports certain features. For instance:
- More about SChannel at this TechNet page.
- Used when visiting websites via HTTPS.
- Used by domain joined machines when talking to Domain Controllers – for validation, changing machine account password, NTLM authentication pass-through, SID look-up, group policies etc.
- Used between domain machines and Domain Controllers, as well as between Domain Controllers. In case of the latter secure channel is also used for replication. Secure channels also exist between DCs in different trusted domain.
- Upon boot up every domain machine will discover a DC, authenticate its machine password with the DC, and create a secure channel to the DC. The Netlogon service maintains the secure channel.
- Every machine account in the domain has a password. This password is used to create the secure channel with the domain.
- Upon boot up every domain machine will discover a DC, authenticate its machine password with the DC, and create a secure channel to the DC.
- This is a good post to read on how to find if secure channel is broken. It shows three methods to identify a problem –
NLTest, PowerShell, WMI – and if the secure channel is broken because the machine password is different from what AD has then the
NLTest /SC_RESET:<DomainName> command can reset it.
- A note on the machine password: machine account passwords do not expire in AD (unlike user account passwords). Every 30 days (configurable via a registry key) the Netlogon service of the machine will initiate a password change. Before changing the password it will test whether a secure channel exists. Only after creating a secure channel will it change the password. This post is worth reading for more info. These password changes can be disable via a registry key/ group policy.
- More on how SSL/TLS is implemented in SChannel can be found at this TechNet page.