The System.Net.Webclient class

The system.net.webclient class can be used to deal with web pages.

To download and display pages this class has couple of methods:

DownloadData downloads the page and displays it as an array of bytes.

# define a new object
PS> $web = New-Object system.net.webclient

# download an address
PS> $web.DownloadData("https://rakhesh.com/about")
60
33
68
...

# just for kicks: this is how you can download the page as a byte array, cast each byte to a character
# and join all these characters to get the entire page
PS> ($web.DownloadData("https://rakhesh.com/about") | %{ [char]$_ }) -join ""
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html
...

# define a new object

PS> $web = New-Object system.net.webclient

# download an address

PS> $web.DownloadData("https://rakhesh.com/about")

...

# just for kicks: this is how you can download the page as a byte array, cast each byte to a character

# and join all these characters to get the entire page

PS> ($web.DownloadData("https://rakhesh.com/about") | %{ [char]$_ }) -join ""

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html

...

DownloadString downloads the page and displays it as one long string.

# define a new object
PS> $web = New-Object system.net.webclient

# download and return as a single string
PS> $web.DownloadString("https://rakhesh.com/about","about.html")

# define a new object

PS> $web = New-Object system.net.webclient

# download and return as a single string

PS> $web.DownloadString("https://rakhesh.com/about","about.html")

DownloadFile downloads the page and saves it to a file name you specify.

# define a new object PS> $web = New-Object system.net.webclient # download to a file PS> $web.DownloadFile("https://rakhesh.com/about","about.html")

1
2
3
4
5

# define a new object
PS> $web = New-Object system.net.webclient

# download to a file
PS> $web.DownloadFile("https://rakhesh.com/about","about.html")

The class also has properties you can set to be used while downloading a page. For instance:

QueryString to specify pairs of query parameters and their values. For example: to do a Google search for the word “rakhesh” one can fetch the page http://www.google.com/search?q=rakhesh. This q=rakhesh is a query string, with q being a parameter and rakhesh being a value to the parameter. To do the same via the system.net.webclient class one would do the following:

# define a new object
PS> $web = New-Object system.net.webclient

# check its current query string
PS> $web.QueryString.GetEnumerator()

# add a new query string
PS> $web.QueryString.Add("q","rakhesh")

# confirm addition - just to show it's possible
PS> $web.QueryString.GetEnumerator()
q
PS> $web.QueryString.Get("q")
rakhesh

# now fetch the web page to a file
PS> $web.DownloadFile("http://www.google.com/search","results.html")

# if you examine the results.html file you'll see it has done the search

# define a new object

PS> $web = New-Object system.net.webclient

# check its current query string

PS> $web.QueryString.GetEnumerator()

# add a new query string

PS> $web.QueryString.Add("q","rakhesh")

# confirm addition - just to show it's possible

PS> $web.QueryString.GetEnumerator()

PS> $web.QueryString.Get("q")

rakhesh

# now fetch the web page to a file

PS> $web.DownloadFile("http://www.google.com/search","results.html")

# if you examine the results.html file you'll see it has done the search

Headers to specify pairs of headers that can be set when requesting the web page:

PS> $web = New-Object system.net.webclient # set the user-agent string to "PowerShell Script" PS> $web.Headers.Add("user-agent", "PowerShell Script")

1
2
3
4

PS> $web = New-Object system.net.webclient

# set the user-agent string to "PowerShell Script"
PS> $web.Headers.Add("user-agent", "PowerShell Script")
Credentials to specify credentials for accessing the web page:

PS> $web = New-Object system.net.webclient PS> $web.Credentials = Get-Credential

1
2

PS> $web = New-Object system.net.webclient
PS> $web.Credentials = Get-Credential

ResponseHeaders to view the headers received in response.

PS> $web = New-Object system.net.webclient

# no response headers initially
PS> $web.ResponseHeaders

# visit a page
PS> $web.DownloadFile("https://rakhesh.com","temp.html")

# now we have response headers
PS> $web.ResponseHeaders
Connection
Keep-Alive
Vary
Accept-Ranges
Content-Length
Cache-Control
Content-Type
Date
Expires
Last-Modified
Server

# view one of the headers
PS> $web.ResponseHeaders.Get("Expires")
Tue, 22 Oct 2013 12:35:13 GMT

# use a pipe to view all the headers
PS> $web.ResponseHeaders.GetEnumerator() | %{ "$_ -> $($web.ResponseHeaders.Get($_))" }
Connection -> keep-alive
Keep-Alive -> timeout=10
Vary -> Accept-Encoding,Accept-Encoding,Cookie
Accept-Ranges -> bytes
Content-Length -> 464244
Cache-Control -> max-age=3, must-revalidate, proxy-revalidate
Content-Type -> text/html; charset=UTF-8
Date -> Tue, 22 Oct 2013 12:35:10 GMT
Expires -> Tue, 22 Oct 2013 12:35:13 GMT
Last-Modified -> Mon, 21 Oct 2013 15:14:25 GMT
Server -> Apache

PS> $web = New-Object system.net.webclient

# no response headers initially

PS> $web.ResponseHeaders

# visit a page

PS> $web.DownloadFile("https://rakhesh.com","temp.html")

# now we have response headers

PS> $web.ResponseHeaders

Connection

Keep-Alive

Vary

Accept-Ranges

Content-Length

Cache-Control

Content-Type

Date

Expires

Last-Modified

Server

# view one of the headers

PS> $web.ResponseHeaders.Get("Expires")

Tue, 22 Oct 2013 12:35:13 GMT

# use a pipe to view all the headers

PS> $web.ResponseHeaders.GetEnumerator() | %{ "$_ -> $($web.ResponseHeaders.Get($_))" }

Connection -> keep-alive

Keep-Alive -> timeout=10

Vary -> Accept-Encoding,Accept-Encoding,Cookie

Accept-Ranges -> bytes

Content-Length -> 464244

Cache-Control -> max-age=3, must-revalidate, proxy-revalidate

Content-Type -> text/html; charset=UTF-8

Date -> Tue, 22 Oct 2013 12:35:10 GMT

Expires -> Tue, 22 Oct 2013 12:35:13 GMT

Last-Modified -> Mon, 21 Oct 2013 15:14:25 GMT

Server -> Apache

There are other properties and methods too, the above are what I had a chance to look at today.