 © Rakhesh Sasidharan
|
Posted: January 23rd, 2015 | Tags: encoding | Category: Asides, Linux BSD | § Wish I had come across this during my PuTTY days!
The TL;DR summary is that by default PuTTY uses an ASCII encoding while most Linux and other OSes uses UTF-8 encoding. It’s because of this mismatch that manpages and such sometimes appear with â characters. Change the PuTTY encoding and find happiness!
Posted: October 22nd, 2013 | Tags: download, encoding, headers, internet, net, web, webclient | Category: PowerShell | § The system.net.webclient class can be used to deal with web pages.
To download and display pages this class has couple of methods:
DownloadData downloads the page and displays it as an array of bytes.
|
|
# define a new object PS> $web = New-Object system.net.webclient # download an address PS> $web.DownloadData("http://rakhesh.com/about") 60 33 68 ... # just for kicks: this is how you can download the page as a byte array, cast each byte to a character # and join all these characters to get the entire page PS> ($web.DownloadData("http://rakhesh.com/about") | %{ [char]$_ }) -join "" <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html ... |
DownloadString downloads the page and displays it as one long string.
|
|
# define a new object PS> $web = New-Object system.net.webclient # download and return as a single string PS> $web.DownloadString("http://rakhesh.com/about","about.html") |
DownloadFile downloads the page and saves it to a file name you specify.
|
|
# define a new object PS> $web = New-Object system.net.webclient # download to a file PS> $web.DownloadFile("http://rakhesh.com/about","about.html") |
The class also has properties you can set to be used while downloading a page. For instance:
QueryString to specify pairs of query parameters and their values. For example: to do a Google search for the word “rakhesh” one can fetch the page http://www.google.com/search?q=rakhesh. This q=rakhesh is a query string, with q being a parameter and rakhesh being a value to the parameter. To do the same via the system.net.webclient class one would do the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# define a new object PS> $web = New-Object system.net.webclient # check its current query string PS> $web.QueryString.GetEnumerator() # add a new query string PS> $web.QueryString.Add("q","rakhesh") # confirm addition - just to show it's possible PS> $web.QueryString.GetEnumerator() q PS> $web.QueryString.Get("q") rakhesh # now fetch the web page to a file PS> $web.DownloadFile("http://www.google.com/search","results.html") # if you examine the results.html file you'll see it has done the search |
Headers to specify pairs of headers that can be set when requesting the web page:
|
|
PS> $web = New-Object system.net.webclient # set the user-agent string to "PowerShell Script" PS> $web.Headers.Add("user-agent", "PowerShell Script") |
Credentials to specify credentials for accessing the web page:
|
|
PS> $web = New-Object system.net.webclient PS> $web.Credentials = Get-Credential |
ResponseHeaders to view the headers received in response.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
PS> $web = New-Object system.net.webclient # no response headers initially PS> $web.ResponseHeaders # visit a page PS> $web.DownloadFile("http://rakhesh.com","temp.html") # now we have response headers PS> $web.ResponseHeaders Connection Keep-Alive Vary Accept-Ranges Content-Length Cache-Control Content-Type Date Expires Last-Modified Server # view one of the headers PS> $web.ResponseHeaders.Get("Expires") Tue, 22 Oct 2013 12:35:13 GMT # use a pipe to view all the headers PS> $web.ResponseHeaders.GetEnumerator() | %{ "$_ -> $($web.ResponseHeaders.Get($_))" } Connection -> keep-alive Keep-Alive -> timeout=10 Vary -> Accept-Encoding,Accept-Encoding,Cookie Accept-Ranges -> bytes Content-Length -> 464244 Cache-Control -> max-age=3, must-revalidate, proxy-revalidate Content-Type -> text/html; charset=UTF-8 Date -> Tue, 22 Oct 2013 12:35:10 GMT Expires -> Tue, 22 Oct 2013 12:35:13 GMT Last-Modified -> Mon, 21 Oct 2013 15:14:25 GMT Server -> Apache |
There are other properties and methods too, the above are what I had a chance to look at today.
Posted: April 28th, 2013 | Tags: encoding, file manipulation, ip address, out-file | Category: PowerShell | § If you find that Windows ignores your hosts file try the following: open the file, select save as, and while saving it change encoding from Unicode (or whatever it is) to ANSI. Be sure to select the file type as “All files” so Notepad doesn’t append a “.txt” to the file name (and double check after saving as sometimes Notepad still appends a “.txt”).
How is this PowerShell related?
I was using Out-File to put something into the hosts file. Windows expects the hosts file to be in ANSI encoding – and seems to ignore it otherwise – but Out-File uses Unicode encoding by default; so the net result is that Windows ignores the resulting hosts file. Even if I removed all the entries in it and made fresh ones for testing, since the file is still in Unicode encoding Windows ignores it. Finally, thanks to a great post I figured a workaround which led to realizing the problem.
Moral of the story: when writing to the hosts file using Out-File specify the encoding as ASCII. Like thus:
|
|
PS> Out-File -Encoding "ASCII" $hostsfile |
And if you are a PowerShell geek, you can avoid the longish method above of changing the hosts file type to ANSI through something like this:
|
|
PS> $blah = Get-Content $hostsfile PS> $blah | Out-File -Encoding "ASCII" $hostsfile |
Easy peasy!
|