Subscribe via Email

Subscribe via RSS


Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

AppV – Empty package map for package content root

Had an interesting problem at work yesterday about which I wish I could write a long and interesting blog post, but truthfully it was such a simple thing once I identified the cause.

We use AppV for streaming applications. We have many branch offices so there’s a DFS share which points to targets in each office. AppV installations in each office point to this DFS share and thanks to the magic of DFS referrals correctly pick up the local Content folder. From day-before, however, one of our offices started getting errors with AppV apps (same as in this post), and when I checked the AppV server I found errors similar to this in the Event Logs:

The DFS share seemed to be working OK. I could open it via File Explorer and its contents seemed correct. I checked the number of files and the size of the share and they matched across offices. If I pointed the DFS share to use a different target (open the share in File Explorer, right click, Properties, go to the DFS tab and select a different location target) AppV works. So the problem definitely looked like something to do with the local target, but what was wrong?

I tried forcing a replication. And checked permissions and used tools like dfsrdiag to confirm things were alright. No issues anywhere. Restarting the DFS Replication service on the server threw up some errors in the Event Logs about some AD objects, so I spent some time chasing up that tree (looks like older replication groups that were still hanging around in AD with missing info but not present in the DFS Management console any more) until I realized all the replication servers were throwing similar errors. Moreover, adding a test folder to the source DFS share correctly resulted it in appearing on the local target immediately – so obviously replication was working correctly.

I also used robocopy to compare the the local target and another one and saw that they were identical.

Bummer. Looked like a dead end and I left it for a while.

Later, while sitting through a boring conference call I had a brainwave that maybe the AppV service runs in a different user context and that may not be seeing the DFS share? As in, maybe the error message above is literally what is happening. AppV is really seeing an empty content root and it’s not a case of a corrupt content root or just some missing files?

So I checked the AppV service and saw that it runs as NT AUTHORITY\NETWORK SERVICE. Ah ha! That means it authenticates with the remote server with the machine account of the server AppV is running on. I thought I’d verify what happens by launching File Explorer or a Command Prompt as NT AUTHORITY\NETWORK SERVICE but this was a Server 2003 and apparently there’s no straightforward way to do that. (You can use psexec to launch something as .\LOCALSYSTEM and starting from Server 2008 you can create a scheduled task that runs as NT AUTHORITY\NETWORK SERVICE and launch that to get what you want but I couldn’t use that here; also, I think you need to first run as the .\LOCALSYSTEM account and then run as the NT AUTHORITY\NETWORK SERVICE account). So I checked the Audit logs of the server hosting the DFS target and sure enough found errors that the machine account of the AppV server was indeed being denied login:

Awesome! Now we are getting somewhere.

I fired up the Local Security Policy console on the server hosting the DFS target (it’s under the Administrative Tools folder, or just type secpol.msc). Then went down to “Local Policies” > “User Rights Assignment” > “Access this computer from the Network”:

secpolSure enough this was limited to a set of computers which didn’t include the AppV server. When I compared this with our DFS servers I saw that they were still on the default values (which includes “Everyone” as in the screenshot above) and that’s why those targets worked.

To dig further I used gpresult and compared the GPOs that affected the above policy between both servers. The server that was affected had this policy modified via  GPO while the server that wasn’t affected showed the GPO as inaccessible. Both servers were in the same OU but upon examining the GPO I saw that it was limited to a certain group only. Nice! And when I checked that group our problem server was a member of it while the rest weren’t! :)

Turns out the server was added to the group by error two days ago. Removed the server from this group, waited a while for the change across the domain, did a gpupdate on the server, and tada! now the AppV server is able to access the DFS share on this local target again. Yay!

Moral of the story: if one of your services is unable to access a shared folder, check what user account the service runs as.

Extract secret keys from Two-Factor Authentication (TFA) QR codes

Got me Pebble Time yesterday! Yay. Found a cool app for Two-Factor Authentication codes called QuickAuth (it’s open source too, amazing!). 

The app requires you to enter the secret keys for your Two-Factor Authentication sites. Unfortunately I never saved these when I set up TFA on my devices. I was smart enough to save the QR code for each site and this way I was always able to add new devices by just scanning the saved QR code, but now I had to enter the secret key and I was stuck. 

Enter another open source project Zebra Crossing (zxing). This is a library for processing QR codes and they have an Android app called Barcode Scanner. Get this app, scan the QR code, and you get an output that starts with otp://. That’s the secret key you want. Enter this into QuickAuth. 

If you don’t want to download the app there’s also an online interface to upload a QR code and decode. Nice!

p.s. In case it helps anyone – on the face of it there seems to be no easy way to delete a key/ site once you enter it into QuickAuth. Later I realized if I long press the select button on the pebble when it shows a code I get many options. One of these lets you delete the key/ site. 

Notes on Windows LSA, Secure Channel, NTLM, etc.

These are some notes to myself on the Windows security subsystem. 

On Windows, the Local Security Authority (LSA) is a subsystem that is responsible for security of the system. The LSA runs as a process called the LSA Subsystem Service (LSASS; you can find it as c:\Windows\System32\lsass.exe) and takes care of two tasks: (1) authentication and (2) enforcing local security policies on system.

For authentication the LSA makes uses of Security Support Providers (SSPs) that provide various authentication protocols. SSPs are Dynamic Link Libraries (DLLs) that offer the authentication protocol to applications that wish to make use of it. They expose a Security Service Provider Interface (SSPI) API which applications can make use of without knowing about the underlying protocol. (Generic Security Service Application Program Interface (GSSAPI or GSS-API) is an IETF standard that defines an API for programs to access security services. SSPI is is a proprietary variant of GSSAPI). 

In a way this post ties in with other things I have been reading about and posted recently. Stuff like encryption ciphers and Active Directory. On domain joined machines for instance, LSA uses Active Directory, while on non-domain joined machines LSA uses Security Accounts Manager (SAM). Either case the LSA is a critical component. 

It is possible to create custom SSPs to support new protocols. Microsoft includes the following SSPs (may not be an exhaustive list).

Kerberos (Kerberos.dll)

  • Provides the Kerberos authentication protocol. This is the protocol of choice in Windows. 
  • Kerberos cannot be used with non-domain joined systems. 
  • More about Kerberos in a later post. I plan to cover it as part of Active Directory. 

NTLM — LM, NTLM, and NTLMv2 (Msv1_0.dll)

  • Provides the NTLM authentication protocol.
    • LM == LAN Manager (also called as LAN MAN). It’s an old way of authentication – from pre Windows NT days. Not recommended any more. 
    • NTLM == NT LAN Manager. Is a successor to LM. Introduced with Windows NT. Is backward compatible to LAN MAN. It too is not recommended any more. It’s worth pointing out that NTLM uses RC4 for encryption (which is insecure as I previously pointed out).
    • NTLMv2 == NT LAN Manager version 2. Is a successor to NTLM. Introduced in Windows 2000 (and in Windows NT as part of SP4). It is the current recommended alternative to LM and NTLM and is the default since Windows Vista.
  • Although Kerberos is the preferred protocol NTLM is still supported by Windows.
  • Also, NTLM must be used on standalone systems as these don’t support Kerberos. 
  • NTLM is a challenge/ response type of authentication protocol. Here’s how it works roughly:
    • The client sends its username to the server. This could be a domain user or a local user (i.e. stored in the server SAM database). Notice that the password isn’t sent. 
    • To authenticate, the server sends some random data to the client – the challenge
    • The client encrypts this data with a hash of its password – the response. Notice that the hash of the password is used as a key to encrypt the data. 
    • If the username is stored in the server SAM database, the hash of the password will be present with the username. The server simply uses this hash to encrypt its challenge, compares the result with the response from the client, and if the two match authenticates the client. 
    • If the username is not stored in the server SAM database, it sends the username, the challenge, and response to a Domain Controller. The Domain Controller will have the password hash along with the username, so it looks these up and performs similar steps as above, compares the two results, and if they match authenticates the client.  
  • Here are some interesting blog posts on NTLM security:
    • NTLM Challenge Response is 100% Broken – talks about vulnerabilities in NTLM & LM, and why it’s better to use NTLMv2.
    • The Most Misunderstood Windows Security Setting of All Time – about the HKLM\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel registry key (mentioned in the above blog post too) which affects how Windows uses NTLMv2. By default Vista and above only send NTLMv2 reponses but accept LM, NTLM, and NTLMv2 challenges. This post also goes into how NTLMv2 performs the challenge/ response I mention above differently.
      • NTLMv2 uses a different hash function (HMAC-MD5 instead of MD4).
      • NTLMv2 also includes a challenge from the client to the server. 
      • It is also possible to use NTLM for authentication with NTLMv2 for session security.
    • Rehashing Pass the Hash – a blog post of Pass the Hash (PtH) which is about stealing the stored password hash (in memory) from the client and using that to authenticate as the client elsewhere (since the hash is equivalent to the password, getting hold of the hash is sufficient). This post also made me realize that LM/ NTLM/ NTLMv2 hashes are unsalted – i.e. the password is hashed and stored, there’s no extra bit added to the password before salting just to make it difficult for attackers to guess the password. (Very briefly: if my password is “Password” and its hashed as it is to “12345”, all any attacker needs to do is try a large number of passwords and compare their hash with “12345”. Whichever one matches is what my password would be! Attackers can create “hash tables” that contain words and their hashes, so they don’t even have to compute the hash to guess my password. To work around this most systems salt the hash. That is, the add some random text – which varies for each user – to the password, so instead of hashing “Password” the system would hash “xxxPassword”. Now an attacker can’t simply reuse any existing hashtables, thus improving security).
      • A good blog post that illustrates Pass the Hash.
      • A PDF presentation that talks about Pass the Hash.
      • Windows 8.1 makes it difficult to do Pass-the-Hash. As the post says, you cannot eliminate Pass-the-Hash attacks as long as the hash is not in some way tied to the hardware machine.
    • When you login to the domain, your computer caches a hash of the password so that you can login even if your Domain Controller is down/ unreachable. This cache stores an MD4 hash of the “MD4 hash of the password + plus the username”.
    • If all the above isn’t enough and you want to know even more about how NTLM works look no further than this page by Eric Glass. :)

Negotiate (secur32.dll)

  • This is a psuedo-SSP. Also called the Simple and Protected GSS-API Negotiation Mechanism (SPNEGO). 
  • It lets clients and servers negotiate a protocol to use for further authentication – NTLM or Kerberos. That’s why it is a psuedo-SSP, it doesn’t provide any authentication of its own. 
  • Kerberos is always selected unless one of the parties cannot use it. 
  • Also, if an SPN (Service Principal Name), NetBIOS name, or UPN (User Principal Name) is not given then Kerberos will not be used. Thus if you connect to a server via IP address then NTLM will be used. 

CredSSP (credssp.dll)

  • Provides the Credential Security Support Provider (CredSSP) protocol. 
  • This allows for user credentials from a client to be delegated to a server for remote authentication from there on. CredSSP was introduced in Windows Vista.
  • Some day I’ll write a blog post on CredSSP and PowerShell :) but for now I’ll point to this Scripting Guy blog post that gives an example of how CredSSP is used. If I connect remotely to some machine – meaning I have authenticated with it – and now I want to connect to some other machine from this machine (maybe I want to open a shared folder), there must be some way for my credentials to be passed to this first machine I am connected to so it can authenticate me with the second machine. That’s where CredSSP comes into play. 
  • CredSSP uses TLS/SSL and the Negotiate/SPNGO SSP to delegate credentials. 
  • More about CredSSP at this MSDN article

Digest (Wdigest.dll)

  • Provides the Digest protocol. See these TechNet articles for more on Digest authentication and how it works
  • Like NTLM, this is a challenge/ response type of authentication protocol. Mostly used with HTTP or LDAP. 
  • There is no encryption involved, only hashing. Can be used in conjunction with SSL/TLS for encryption.

SChannel (Schannel.dll)

  • Provides the SSL/ TLS authentication protocols and support for Public Key Infrastructure (PKI). 
  • Different versions of Windows have different support for TLS/SSL/ DTLS because the SChannel SSP in that version of Windows only supports certain features. For instance:
  • More about SChannel at this TechNet page
  • Used when visiting websites via HTTPS.
  • Used by domain joined machines when talking to Domain Controllers – for validation, changing machine account password, NTLM authentication pass-through, SID look-up, group policies etc.
    • Used between domain machines and Domain Controllers, as well as between Domain Controllers. In case of the latter secure channel is also used for replication. Secure channels also exist between DCs in different trusted domain. 
    • Upon boot up every domain machine will discover a DC, authenticate its machine password with the DC, and create a secure channel to the DC. The Netlogon service maintains the secure channel. 
    • Every machine account in the domain has a password. This password is used to create the secure channel with the domain. 
      • Upon boot up every domain machine will discover a DC, authenticate its machine password with the DC, and create a secure channel to the DC. 
    • This is a good post to read on how to find if secure channel is broken. It shows three methods to identify a problem – NLTest, PowerShell, WMI – and if the secure channel is broken because the machine password is different from what AD has then the NLTest /SC_RESET:<DomainName> command can reset it.
      • A note on the machine password: machine account passwords do not expire in AD (unlike user account passwords). Every 30 days (configurable via a registry key) the Netlogon service of the machine will initiate a password change. Before changing the password it will test whether a secure channel exists. Only after creating a secure channel will it change the password. This post is worth reading for more info. These password changes can be disable via a registry key/ group policy
  • More on how SSL/TLS is implemented in SChannel can be found at this TechNet page.

Microsoft Account and shared folders

If you are using a Microsoft Account on your computer, it’s worth keeping in mind that traditional shared folders won’t work. By traditional shared folders I mean from other computers in the same workgroup as yours, but without a Microsoft Account.

I had two machines – a Windows 8 client, and Windows Server 2008 R2 server. Client had an account with username “rakhesh” which I later converted to a Microsoft Account. Since the username still appears in many places as “<MACHINENAME>>\rakhesh” I figured I should be able to access my shared folders on the client, from the server, by logging in as “<MACHINENAME>\rakhesh”. But no, that does not work. Finally I had to create a new local account on the client and use that for shared folders.

The reverse too applies. At home I have two Windows 8 clients. They are not in a HomeGroup, just a regular workgroup. Both have the same user account but one of them is converted to a Microsoft Account. The shared folders are on the machine with a local account and whenever I try accessing these from the machine with a Microsoft Account it fails. I worked around that by mapping the folders to a drive and specifying the local account credentials then. I believe one can also use the Credential Manager for this but I haven’t tried it yet.