Had an interesting problem at work yesterday about which I wish I could write a long and interesting blog post, but truthfully it was such a simple thing once I identified the cause.
We use AppV for streaming applications. We have many branch offices so there’s a DFS share which points to targets in each office. AppV installations in each office point to this DFS share and thanks to the magic of DFS referrals correctly pick up the local Content folder. From day-before, however, one of our offices started getting errors with AppV apps (same as in this post), and when I checked the AppV server I found errors similar to this in the Event Logs:
Empty package map for package content root [\\domain.local\dfs\Content]
The DFS share seemed to be working OK. I could open it via File Explorer and its contents seemed correct. I checked the number of files and the size of the share and they matched across offices. If I pointed the DFS share to use a different target (open the share in File Explorer, right click, Properties, go to the DFS tab and select a different location target) AppV works. So the problem definitely looked like something to do with the local target, but what was wrong?
I tried forcing a replication. And checked permissions and used tools like
dfsrdiag to confirm things were alright. No issues anywhere. Restarting the DFS Replication service on the server threw up some errors in the Event Logs about some AD objects, so I spent some time chasing up that tree (looks like older replication groups that were still hanging around in AD with missing info but not present in the DFS Management console any more) until I realized all the replication servers were throwing similar errors. Moreover, adding a test folder to the source DFS share correctly resulted it in appearing on the local target immediately – so obviously replication was working correctly.
I also used robocopy to compare the the local target and another one and saw that they were identical.
robocopy "\\local\target" "\\remote\target" /e /l /xd "\\local\target\DfsrPrivate"
Bummer. Looked like a dead end and I left it for a while.
Later, while sitting through a boring conference call I had a brainwave that maybe the AppV service runs in a different user context and that may not be seeing the DFS share? As in, maybe the error message above is literally what is happening. AppV is really seeing an empty content root and it’s not a case of a corrupt content root or just some missing files?
So I checked the AppV service and saw that it runs as
NT AUTHORITY\NETWORK SERVICE. Ah ha! That means it authenticates with the remote server with the machine account of the server AppV is running on. I thought I’d verify what happens by launching File Explorer or a Command Prompt as
NT AUTHORITY\NETWORK SERVICE but this was a Server 2003 and apparently there’s no straightforward way to do that. (You can use
psexec to launch something as
.\LOCALSYSTEM and starting from Server 2008 you can create a scheduled task that runs as
NT AUTHORITY\NETWORK SERVICE and launch that to get what you want but I couldn’t use that here; also, I think you need to first run as the
.\LOCALSYSTEM account and then run as the
NT AUTHORITY\NETWORK SERVICE account). So I checked the Audit logs of the server hosting the DFS target and sure enough found errors that the machine account of the AppV server was indeed being denied login:
Logon failure; the user has not been granted the requested logon type at this computer.
Awesome! Now we are getting somewhere.
I fired up the Local Security Policy console on the server hosting the DFS target (it’s under the Administrative Tools folder, or just type
secpol.msc). Then went down to “Local Policies” > “User Rights Assignment” > “Access this computer from the Network”:
Sure enough this was limited to a set of computers which didn’t include the AppV server. When I compared this with our DFS servers I saw that they were still on the default values (which includes “Everyone” as in the screenshot above) and that’s why those targets worked.
To dig further I used
gpresult and compared the GPOs that affected the above policy between both servers. The server that was affected had this policy modified via GPO while the server that wasn’t affected showed the GPO as inaccessible. Both servers were in the same OU but upon examining the GPO I saw that it was limited to a certain group only. Nice! And when I checked that group our problem server was a member of it while the rest weren’t! :)
Turns out the server was added to the group by error two days ago. Removed the server from this group, waited a while for the change across the domain, did a
gpupdate on the server, and tada! now the AppV server is able to access the DFS share on this local target again. Yay!
Moral of the story: if one of your services is unable to access a shared folder, check what user account the service runs as.