Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

[Aside] NetScaler – CLI Networking

Just putting these two here as a reference to myself (no idea why coz I am sure I’ll just Google and find them later when I need to :p)

As an aside (to this aside):

  • The NetScaler config is stored as ns.conf at /nsconfig
  • Older versions have a .0, .1, .2, etc suffixed to the filename. 
  • Backups are stored in /var/ns_sys_backup.
  • More info on backups etc

Notes on Master Image Preparation (PVS & MCS)

Reading some links on creating a Master Image; here’s notes to myself regarding that. These are the links I refer to:

Note these are very rough/ brief. These really are rough notes to myself as I am trying to make organize my mind. 

In case of PVS: Master Target Device – this is the VM whose image is used  to create a virtual hard disk (vDisk). This vDisk is what PVS uses to stream to its VMs.

Unlike MCS, the Master Target Device does not have to be a VM. PVS works against both VMs and physical machines. It does not care about the compute; all it does is look at the machine and create a vDisk by capturing its contents. You network boot into the machine you want to capture, and PVS creates an image by streaming its contents to the PVS server to create a vDisk. 

The Master Target Device or its disk can be removed after vDisk creation.

Here’s my understanding of the order in which to do stuff:

1. Install the OS

2. Install Updates and Device Drivers; including integration tools such as VMware Tools, XenServer Tools, etc. 

In case of MCS:

3. Install any applications (optional) & VDA & domain join (optional) & shutdown the machine. 

4. Add it to MCS to create a catalog. Recommended that we take a snapshot and point MCS to this snapshot. Else MCS will make its own snapshot (and we can’t change the snapshot name). 

In the case of PVS:

3. Install any applications & VDA. (No shutdown!) (No domain join!)

4. Install the Target Device software (this is the Provisioning Services target device software).

5. Launch the imaging wizard. (No need for snapshots either as we are capturing it to a new vDisk). 

5a. Note: The Target Device name can be different from the name of the machine/ VM. 

6. After the vDisk is created we use a VM (existing or new one) to network boot and use this vDisk. Then we can domain join etc. PVS has a lot more steps. Check out http://www.carlstalhood.com/pvs-master-device-convert-to-vdisk/

Components of the Provisioning Services target device software include an imaging wizard to capture the image, a NIC filter driver used for streaming images from the PVS server to the target devices (remember: the target device software is not used only for capturing the image – i.e. the master target device, but also by the target devices after an OS is loaded), a virtual disk to store the OS and applications (again, used by the target devices). There’s also a system tray utility. 

[Aside] NetScaler VPX Express limitations etc.

Reading about NetScaler VPX as we are looking at implementing VPX Express in our site as part of a POC. 

  1. VPX Express is limited to 5  Mbps (as opposed to say 1 Gbps for VPX-1000). 
  2. The license is free but you have to keep renewing annually.
  3. The Edition is NetScaler Standard. This and this are two links I found that explain the difference between various editions. 
    1. tl; dr version: Standard is fine for most uses. 
  4. The Gateway part supports 5 concurrent user connections. 
  5. You cannot vMotion or XenMotion VPX. 
  6. This is an excellent blog post on VPX, MPX, and others. Worth a read. 

Changing Delivery Controller addresses on VDA

This weekend the NetScaler gateway on one of our offices failed, rendering the office users unable to connect to their resources. I wanted to repoint all those users to the Delivery Controllers of a different office where the NetScaler was working, and ask users to temporarily connect via that. 

I had to two two things basically. 1) Change a registry key to repoint the VDA to the other Delivery Controllers. 2) Restart the VDA service (these machines were running 5.6(!!) so it’s called the Workstation Agent). 

Brief notes on NetScaler and Citrix StoreFront

I spent the last two days intermittently trying to set up NetScaler and Citrix StoreFront in my home lab. It was a mixed bag partly due to my nature of just jumping into something and figuring it out as I go along :) but compounded by the fact that while there’s a lot of documentation on the Internet they seem to be either outdated or don’t explain the big picture of what we are trying to achieve. (Or maybe I am just slow in picking it up – wouldn’t put that possibility aside!)

Anyways. Here’s a couple of stuff in no particular order. 

This PDF is the official documentation on setting up NetScaler with Citrix StoreFront. It’s a good one – lots of screenshots etc. Page 19 onwards seems to be outdated though with the latest version of NetScaler that I have – 11.1 53.11. 

When I started this exercise though I was on a much older version of NetScaler – 10.5 56.22. I started setting up the NetScaler as a gateway (e.g. http://ctxstorefront.myfqdn) to my internal StoreFront servers (http://data01.myfqdn and http://data02.myfqdn), and my steps were more or less along the lines of the PDF above which I discovered later. That wasn’t a success though coz when I’d connect to the NetScaler gateway it gave some errors (I forget what now). Digging into this I realized that the wizard also creates a load balanced virtual server on the NetScaler and that was showing as down. Dug into that a bit and found out that the underlying monitor probes to the two StoreFronts were failing. If I separately created services representing my StoreFronts and attach the same monitor (basically, a monitor of type STOREFRONT, with the correct StoreName etc) it fails. 

That was good to know coz I learnt a bit about the monitor probes. :) I found the following in my NetScaler logs:

That’s same as what the GUI was telling me but I was additionally able to find what error code was being generated (thanks to this KB article):

So it runs a Perl script basically. Nice. Went through that script and here’s the relevant section:

So it tries to access http(s)://<my delivery controller>/Citrix/<my Store>/discovery and gets a 200 error. Odd that it gets it, because if I try to probe that URL via cURL from NetScaler, I get a 404 error (which is sort of correct; I get a 403 error via IE and that’s the correct one I think):

Anyhoo, that stumped me for a while until I found forum/ blog post where they said upgrading to a newer version of NetScaler supposedly fixes this. So I went down that route, and yes, it helped. 

After upgrading though my UI changed. :)

The new UI is very different. But that’s good I think, coz it forced me think at least on what exactly am I trying to achieve here; and to take a step backward and understand what is going on. So here’s my understanding:

  • We have StoreFronts. :) That’s the thing you actually connect to. 
  • We can have a group of StoreFronts for High Availability. You can configure each of these independently or you can keep them in sync from the UI itself. 
  • For each StoreFront we need to specify a base URL.
    • For a single StoreFront it’s easy – http://data01 or whatever (where data01 is my StoreFront server name in this case).
    • But what about when I have a group of StoreFronts? If I am keeping them in sync via the UI itself, the base URL I define on one of them will be pushed out to all. So all my StoreFronts will have a base URL of http://data01 even though they might be called data01, data02, etc. That’s a problem coz all my clients will only connect to the first one as that’s where DNS will point clients to.
  • To avoid the above situation for multiple StoreFronts we need a common base URL which can be load balanced across all. A regular DNS round-robin situation won’t work coz we also need clients to stick on with whichever StoreFront they connect to, and also some form of monitoring to ignore StoreFronts that are down would be good to. So this is where the NetScaler first comes in!
  • Create a Virtual Server on the NetScaler that will load balance the various StoreFront services we define on it. Simple stuff – just HTTP or HTTPS services that are load balanced. Add the STOREFRONT type monitor. And a Persistence of type COOKIEINSERT. That’s all. (Oh, and add SSL certs etc if you are using HTTPS). 
    • I haven’t gone through this link but am putting it here as a reference to my future self – pretty sure I must have missed something when setting up things now. Also, that link goes into some scenarios such as where we do SSL termination at the NetScalers. 

What I realized as I thought about this is that this Virtual Server I create above is the main thing. Going by my URLs above, I should have http(s)://ctxstorefront.myqdn load balance among http(s)://data01 and http(s)://data02. Forget about the whole external access stuff for now. 

Once I do this and ensure things are working (they do), now I can think about external access. What does the NetScaler need to do there? It doesn’t need to do any load balancing coz that’s already setup; so all it needs to do is provide a VPN gateway sort of service for external clients to connect to! So that means a new Virtual IP on the NetScaler. Create this using the wizard in the “Integrate with Citrix Products” > “XenApp and XenDesktop”. This will basically create a Virtual Server in the NetScaler Gateway section (note: not in the Load Balancing section). As part of configuring, we have to point this Virtual Server to a StoreFront. Here use the Load Balanced StoreFront Virtual Server that we created on the NetScaler above – this is where it all ties in! 

I could use the same internal URL for the external access too I guess and use split DNS – not sure, because I do have to specify this on the StoreFronts and I haven’t tried/ thought of any side effects – but in my case I simply decided to go with a different URL for the external access. Specify that for the certificate and Virtual Server etc, add that to DNS, and now I can access the StoreFronts externally too via the NetScaler. 

If I were to go to the Virtual Server in the Gateway section there’s no obvious mapping from this one to the internal load balance Virtual Server. But there is. It’s in the policies section. That has a Published Application pointing to the load balanced URL (including the Store name (actually, there’s two policies – one for the Web another for the Receiver, to capture the different names)) so that’s how the traffic flow works. Users hit the gateway, the gateway does the authentication etc and checks its policies, finds one that offers uses the StoreFront published application at such and such URL, and it looks that up (it is with itself) and thus hits the load balanced Virtual Server, and so on … 

Finally! I have some idea of how this all ties in together. :)

PVS Console XenDesktop Setup Wizard crash and timeout

If you get the following error when running the “XenDesktop Setup Wizard” from PVS console (and a catalog is created with no machines):

Or you get the following error (and again a catalog is created but no machines):

Or you get neither of these but the PVS Console simply freezes when you click OK/ Next at one of the steps and it throws an error about timeout and being unable to connect to the remote server (sorry no screenshot, forgot to take it when I got the error and now I can’t replicate it) …

The solution is simple! Install the PVS Console on your Delivery Controller server and run the wizard from there. For some reason that seems to do the trick. Thanks to this forum post

Citrix breaks after removing the root zone from your DNS server?

Two years ago I had removed the root zone on our DNS servers at work. Coz who needs root zones if your DC is only answering internal queries, i.e. for zone sit has. Right?

Well, that change broke our Citrix environment. :) Users could connect to our NetScaler gateway but couldn’t launch any resource after that. 

Our Citrix chaps logged a call with our vendor etc and they gave some bull about the DNS server not responding to TCP queries etc. Yours truly wasn’t looking after Citrix or NetScalers back then, so the change was quietly rolled back as no one had any clue why it broke Citrix. 

Fast forward to yesterday, I had to do the change again coz now we want our DNS servers to resolve external names too – i.e. use root hints and all that goodness! I did the change, and Citrix broke! Damn. 

But luckily now Citrix has been rolled into our team and I know way more about how Citrix works behind the scenes. Plus I keep dabbling with NetScalers, so I am not totally clueless (or so I’d like to think!). 

I went into the DNS section of the NetScaler to see what’s up. Turns out the DNS virtual server was marked as down. Odd, coz I could SSH into the NetScaler and do name lookups against that DNS virtual server (which pointed to my internal DC basically). And yes, I could do dig +notcp to force it to do UDP queries only and nothing was broken. So why was the virtual server marked as down?!

I took a look at the monitor on the DNS service and it had the following:

Ok, so what exactly does this monitor do? Click “Edit Monitor” – nothing odd there – click on “Special Parameters” and what do I find? 

Yup, it was set to query for the root zone. Doh! No wonder it broke. 

I have no idea why the DNS monitor was assigned to this service. By default DNS-UDP has the ping-default monitor assigned to it while DNS-TCP has the tcp-default monitor assigned to it.  Am guessing that since our firewall block ICMP from the NetScalers to the DCs, someone decided to use the DNS monitor instead and left it at the default values of monitoring for the root zone. When I removed the root zone that monitor failed, the DNS virtual server was marked as down, and the NetScaler could no longer resolve DNS names for the resources users were trying to connect to. Hence the STA error above. Nice, huh!

Fix is simple. Change the query in the DNS monitor to a zone your DNS servers. Preferably the zone your resources are in. Easy peasy. Made that change, and Citrix began working. 

As might be noticeable from the tone of the post, I am quite pleased at having figured this out. Yes, I know it’s not a biggie … but just, it makes me happy at having figured it coz I went down a logical path instead of just throwing up my hands and saying I have no idea why the DNS service is down or why the monitor is red etc. So I am pleased at that! :)

 

[Aside] NetScaler newnslog files

Some links to myself on the newnslog files (these are binary log files; high precision; need a tool called nsconmsg to view them). 

A typical format of the command is like this:

The <operation> can be one of these (this is just a copy-paste from nsconmsg -?):

The newnslog files are rotated every 2 days (or a certain number of events if I remember correctly). The older ones can be accessed by putting a path to that file (e.g. /var/nslog/newnslog.28.tar.gz in the command above). This will extract the file and show the logs. The Citrix page says we have to extract the logs first, but am guessing that’s old info. 

That’s all for now. Will add more to this post later …

NetScaler/ Exchange RPC – TCP syn sent, reset received

At work one of my colleagues is setting up NetScalers as load balancers for our new Exchange environment. He is replicating the existing setup but found that the RPC 60001 & 60002 Service Groups on the NetScalers were being marked as down. Curious, I took a look.

After SSH-ing into the NetScaler I could see the following via show serviceGroup <serviceGroupName>:

My colleague too had seen this and pointed me to a good blog post from Citrix on what the reset codes mean. That blog post is a good one (that’s why I am linking it here, as a reference to myself) but I don’t think he was looking at the trace via a NetScaler trace so we had no idea of the codes. (Speaking of which, here’s a good post on NetScaler and Wireshark. Here’s a KB article on how to collect traces from NetScaler. And here’s a KB article on how to collect traces from the CLI. Whilst I have briefly read them, I haven’t tried them out currently). 

Back to the issue at hand. I could see that the individual servers (Exchange 2010 Client Access) were up on RPC 135 and HTTPS, but only RPC 60001 & 60002 were down. I decided to do a portQry against a server in the older environment and compare against the new. Here’s the relevant bits from an older server:

As expected, something is listening on ports 60001 and 60002. When I tried the same against the new server, however, there was nothing listening on either of these ports. I searched the output based on the UUIDs and found the port numbers were different:

So that’s why the NetScalers were getting a reset. Nothing was listening on those ports! Solution is simple. Configure these RPC ports as static.

That’s all! :)

[Aside] NetScaler SSL

Just putting in these links as bookmarks to myself for future. I kinda followed them while I was trying to change my NetScaler certs (kinda followed, coz I didn’t find these links when I Googled initially, so I just went ahead and figured it out by trying; but later I came across these and thought it would be a good idea to link them here). 

Notes on DNS servers & NetScaler

I must begin with a link to this forum post where someone explains the various DNS types on a NetScaler. A must-read. 

Now on to a bunch of screenshots and notes from me as I was just looking around with NetScalers and DNS. I have realized over time that my way of picking up stuff is by just doing it. A typical approach of reading about something and then trying it out doesn’t seem to work for me. (a) I get sleepy during reading and (b) that results in me never getting to the trying out stage. Instead, I seem to work better by just trying to begin with, succeed or break stuff in the process, and then go back and read or blog etc. about it. No hubris here that I am one of a kind :) am sure there’s more people who work this way – just that I too am like them. 

A negative with this approach is that I must have a test lab where I can try things out. So there’s the additional effort required from me in terms of having a place where I can just break stuff. That’s probably the only negative thing I can think of about my approach. Oh, and it also takes up additional time when I want to pick up something – because first I have to set the environment up (e.g. when I was trying to pick up NSX last month) and then spend time just doing things and making/ breaking stuff in the process. 

Anyways – end of digression. Back to NetScalers and DNS. 

On a NetScaler, under the Traffic Management > DNS > NameServers is where you define DNS servers. 

 

You create names servers by clicking on the “Add” button. That gives a new screen like thus:

I’ll start off the with the “Local” checkbox because it’s a very important one. Funny how it’s just there as a checkbox but it completely changes everything else! 

If you tick “Local” what it means is that the NetScaler acts as a DNS server responding to queries from clients. 

  • Thus the IP address you specify will be a Virtual IP on the NetScaler, where you can query for DNS replies. 
  • The records you can query are what will be defined on the NetScaler, under the Records section. 
  • The NetScaler can only act as a UDP based nameserver.

If you don’t tick “Local” then the NetScaler acts as a client. It won’t respond to any DNS queries. 

  • Thus the IP address you specify are what the NetScaler will contact for its own DNS queries. 
    • From the forum post I linked to above: NetScaler will monitor this IP address via ping from the NSIP (and not the SNIP).
  • Note: These IP address do not belong to the NetScaler. 
  • The IP addresses + DNS port combo cannot be defined on the NetScaler in the Load Balancing > Services section. You’ll get a “Resource already exists” error in that case. 
  • The IP addresses + DNS port combo can be defined in Service Groups. And can thus be used in load balancing etc. But as pointed out above, they cannot be defined as services. 

When creating a name server it is possible to use an existing DNS virtual server if one is already defined. The caveat with this is that only UDP is allowed. It is not possible to add a TCP or UDP/ TCP entry. In fact, the only options one gets in the drop down menu are UDP only DNS load balancer virtual services. (From the forum post: in this case the NetScaler will monitor the virtual server from its SNIP). 

It’s good to have TCP (or UDP/ TCP) servers in case of larger responses. In fact, when the NetScaler is acting as a load balancer for other DNS servers (this mode is called DNS proxy) it’s pretty much recommended to have TCP as an option too. 

If, say, the NetScaler is defined with only a UDP based DNS server (as in the screenshot below) then queries will fail if the DNS responses are large and require a TCP connection. 

This brings me to one more point. If we are creating a virtual server DNS just for the NetScaler’s internal use, we don’t need to define an IP address for it. The Name Server I have above actually does not have any virtual IP on the NetScaler. 

So – to summarize: 

  • In the Name Servers section we can set the NetScaler to act as a DNS server for a zone it has.
    • This is UDP only. 
    • This is not load balancing. i.e. not a virtual server. 
  • In the Name Servers section we can also point the NetScaler to other DNS servers the NetScaler itself can use. 
    • If an IP address is specified, it can be both UDP and TCP, and the NetScaler monitors it via ping from the NSIP.
    • If a virtual server (see next point) is specified, it is UDP only, and the NetScaler monitors it via ping from the SNIP.
      • The virtual server created for such internal use can be set in non-addressable mode (i.e. not IP address).
  • In the Virtual Servers section it is possible to define a DNS service. The NetScaler will then act as a DNS server. 
    • This is load balancing. The NetScaler doesn’t host any zones. 
    • The NetScaler will cache results though and serve from those if required.
    • The NetScaler does not use this internally. But it can be set to use this internally, if thus defined in the Name Servers section.
    • This is for both UDP and TCP. 
    • This is also known as a DNS proxy. 

I think that’s about the gist of it. I have skipped GSLB for now. Once again, pointing to the useful forum post. It’s a great one! 

[Aside] https://127.0.0.1 Citrix Store SSL discovery error

Due to a goof up on my part in my test lab, I was encountering this error. Found this forum post which helped me fix it; after which I realized the error was happening coz of a configuration error on my part. No point going into what my mistake was (in short – I have two servers that act as both StoreFront and Delivery Controller; the base URL of the StoreFront is one of the servers but I was also trying to access the StoreFront via the other and it worked but kept erroring (it worked coz the IIS website is there; and it errored coz I am not supposed to access it via that URL) so that was me being silly). 

Thank you Internet!

Time to setup NetScalers in my test environment so I can access the StoreFront via them. Which is what I should have done in the first place to load balance between these two servers. 

Refresher to myself StoreFront and Delivery Controller authentication

In a previous post I had written about the flow of communication between Citrix Storefront and Delivery Controllers during user authentication. Here’s some more based on a Citrix blog post I am reading. 

Here’s what I had written in my previous post:

There’s a couple of steps that happens when a user logs in to access a Citrix solution. First: the StoreFront authenticates the user against AD. Or if the user is accessing remotely, the NetScaler gateway authenticates the user and passes on details to the StoreFront. Then the StoreFront passes on this information to the Delivery Controller so the latter can give a list of resources the user has access to. The Delivery Controllers in turn authenticate the user AD. The Delivery Controller then sends a list of resources the user has access to, to the StoreFront, which sends this on to the user’s Citrix Receiver or Browser. This is when the user sees what is available to them, and can select what they want.

When the user selects what they want, this is information is passed on to the StoreFront, which then passes the info to the Delivery Controller – who then finds an appropriate host that can fulfill the requirement and sends this information to the StoreFront. 

Emphasis mine. The Storefront communicates with the Delivery Controller using the XML Service. 

Here’s a list of authentication methods supported by the Storefront. 

When the Storefront communicates the user authentication information to the Delivery Controller, it may or may not include the password too (sent in clear-text) in this communication. If “User name and password” or “Pass-through from NetScaler” is selected, then the password is included. If “Domain pass-through” or “Smart card” is selected, then the password is not. The blog post doesn’t say anything about these, but I think “SAML Authentication” (used for ADFS) will not include the password, while “HTTP Basic” will. 

The StoreFront and Delivery Controller communicates twice (the two times I emphasized above). The first time is when the user authenticates and the StoreFront sends this information to the Delivery Controller to get a list of resources. The second time is when the user makes a selection and this information is passed on to the Delivery Controller so that an appropriate host can be selected. In both instances the password could be sent from the StoreFront to the Delivery Controller.

Notes on MCS disks

Primer 1. Primer 2. MCS Prep overview (good post, I don’t refer to all its points below). 

  • MCS creates a snapshot of the master VM you specify, but if you specify a snapshot it will not create another one. 
  • This snapshot is used to create to create a full clone. A full snapshot, so to say. 
    • This way the image used by the catalog is independent of the master VM. 
    • During the preparation of this full snapshot an “instruction disk” is attached to the VM that is temporarily created using the full snapshot. This disk enables DHCP on all interfaces of the full snapshot; does some KMS related tasks; and runs vDisk inventory collection if required.
  • This full snapshot is stored on each storage repository that is used by Desktop Studio. 
    • This full snapshot is shared by all VMs on that storage repository. 
  • Each storage repository will also have an identity disk (16 MB) per VM.
  • Each storage repository will also have a delta/ difference disk per VM.
    • This is thin provisioned if the storage supports it.
    • Can increase up to the maximum size of the VM.

Remember my previous post on the types:

  • Random.
    • Delta disk is deleted during reboot. 
  • Static + Save changes.
    • Changes are saved to a vDisk. 
    • Delta disk not used?
  • Static + Dedicated VM.
    • Delta disk is not deleted during reboot. 
    • Important to keep in mind: if the master image in the catalog is updated, existing VMs do not automatically start using it upon next reboot. Only newly created dedicated VMs use the new image. 
    • The delta disk is deleted when the master image is updated and existing VMs are made to use the new image (basically, new VMs are created and the delta disk starts from scratch; user customizations are lost). 
    • Better to use desktop management tools (of the OS) to keep dedicated VMs up to date coz of the above issue. 
  • Static + Discard changes.
    • Delta disk is deleted during reboot. 

A post on sealing the vDisk after changes. Didn’t realize there’s so many steps to be done. 

MCS choices (RAM cache & Disk cache)

Just a reminder to myself …

When creating a Desktop based Machine Catalog here are my choices:

If I choose Random then I get the option to allocate some of my RAM towards a cache, and also create a disk cache. RAM cache means all data is written to RAM first and then to disk as RAM fills up. And disk cache is like the Write Cache disk in PVS – you can specify a separate disk (maybe local to the host, or SSD storage) where data is written to.

Important to keep in mind here that the actual VM disk will not have any data written to it. All data writes either goes to the RAM cache or Disk cache. First RAM cache, then Disk cache. Both are optional; best to have both (or at least don’t do RAM cache only unless you have oodles or RAM!).

Read this post – it’s a good one. Also, check out the official post from Citrix introducing this feature in XenDesktop 7.9. MCS (Machine Creation Services) that makes use of RAM or Disk cache is known as MCSIO (Machine Creation Services Storage Optimization (beats me how that acronym works! :p)).

MCS VMs have two disks apart from the OS base disk – an identity disk and a delta disk. MCSIO VMs too have the identity disk and delta disk, but the delta disk is only used for maintenance tasks. Hence my comment above that when using either of these cache options, the size you allocate for these is your write cache/ delta disk. 

If I choose static I have three further options. 

If I go with static + save changes to a personal vDisk, I don’t get the option for cache disk etc. I can only choose my vDisk letter and size. 

 If I go with static + create a dedicated VM, again I don’t get any option for cache disk; I can only choose the copy mode (i.e. a linked clone or a full clone). 

If I go with static + discard all changes, then I get the option for cache disk and RAM allocation towards cache. Basically, static + discard is similar to random. You are not storing any changes, so it makes sense to use cache (RAM and/ or write cache). 

In the case of Server OS, I don’t have any choices (it’s always random) and I get the option for cache disk and RAM allocation.

MCSIO is only for non-persistent experiences.