curl – rakhesh.com

Not a biggie but in case it helps anyone.

I wanted to download all episodes of the excellent “My Dad Wrote a Porno” podcast for posterity. I couldn’t find any way of doing this so here’s what I ended up doing.

First I found the RSS feed. I noticed that it contains the actual audio file in enclosure tags.

Cool, so I just need to read these for a start. I can do that via curl.

curl -s http://rss.acast.com/mydadwroteaporno | grep -o '<enclosure url="[^"]*'

1	curl -s http://rss.acast.com/mydadwroteaporno \| grep -o '<enclosure url="[^"]*'

This gives me all the links thus:

<enclosure url="https://media.acast.com/mydadwroteaporno/bestofbookfour/media.mp3
<enclosure url="https://media.acast.com/mydadwroteaporno/mydadwroteachristmasporno3/media.mp3
<enclosure url="https://media.acast.com/mydadwroteaporno/mydadwroteachristmasporno1/media.mp3

<enclosure url="https://media.acast.com/mydadwroteaporno/bestofbookfour/media.mp3

<enclosure url="https://media.acast.com/mydadwroteaporno/mydadwroteachristmasporno3/media.mp3

<enclosure url="https://media.acast.com/mydadwroteaporno/mydadwroteachristmasporno1/media.mp3

I was able to extract just the URL via a modification to the above snippet to match the beginning double quotes:

curl -s http://rss.acast.com/mydadwroteaporno | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$'

1	curl -s http://rss.acast.com/mydadwroteaporno \| grep -o '<enclosure url="[^"]' \| grep -o '[^"]$'

Now all I needed to do was download these and also rename the “media.mp3” to be the directory name from the path. The following did that:

for i in $(curl -s http://rss.acast.com/mydadwroteaporno | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$'); do 
    url=$i
    outfile=$(echo $i | sed 's|https://media\.acast\.com/mydadwroteaporno/||' | sed 's|/media||')
    wget -q $url -O $outfile
done

for i in $(curl -s http://rss.acast.com/mydadwroteaporno | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$'); do

url=$i

outfile=$(echo $i | sed 's|https://media\.acast\.com/mydadwroteaporno/||' | sed 's|/media||')

wget -q $url -O $outfile

done

I use sed to strip out the domain name and also do the word “media”. What remains is the part of the path I am interested in.

I needed to download OneGet for the previous post. I had the URL, but rather than open a browser as I would usually do I used PowerShell to download the file:

PS> Invoke-WebRequest http://oneget.org/oneget.zip -OutFile oneget.zip

1	PS> Invoke-WebRequest http://oneget.org/oneget.zip -OutFile oneget.zip

Nifty, eh! If you omit the -OutFile you get the headers and such:

PS> Invoke-WebRequest http://oneget.org/oneget.zip


StatusCode        : 200
StatusDescription : OK
Content           : {80, 75, 3, 4...}
RawContent        : HTTP/1.1 200 OK
                    Content-MD5: /bez4IkFAORz1WWm80bHnw==
                    x-ms-request-id: d6df72e2-0001-0017-0896-11458a000000
                    x-ms-version: 2009-09-19
                    x-ms-lease-status: unlocked
                    x-ms-blob-type: BlockBlob
                    Content...
Headers           : {[Content-MD5, /bez4IkFAORz1WWm80bHnw==], [x-ms-request-id, d6df72e2-0001-0017-0896-11458a000000],
                    [x-ms-version, 2009-09-19], [x-ms-lease-status, unlocked]...}
RawContentLength  : 906946



PS> Invoke-WebRequest http://oneget.org/oneget.zip | select -ExpandProperty Headers

Key                                                         Value
---                                                         -----
Content-MD5                                                 /bez4IkFAORz1WWm80bHnw==
x-ms-request-id                                             d6dfadf7-0001-0017-3e96-11458a000000
x-ms-version                                                2009-09-19
x-ms-lease-status                                           unlocked
x-ms-blob-type                                              BlockBlob
Content-Length                                              906946
Content-Type                                                application/octet-stream
Date                                                        Wed, 25 Feb 2015 21:07:42 GMT
ETag                                                        0x8D1CE392F8BDAF4
Last-Modified                                               Fri, 14 Nov 2014 15:45:44 GMT
Server                                                      Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

PS> Invoke-WebRequest http://oneget.org/oneget.zip

StatusCode : 200

StatusDescription : OK

Content : {80, 75, 3, 4...}

RawContent : HTTP/1.1 200 OK

Content-MD5: /bez4IkFAORz1WWm80bHnw==

x-ms-request-id: d6df72e2-0001-0017-0896-11458a000000

x-ms-version: 2009-09-19

x-ms-lease-status: unlocked

x-ms-blob-type: BlockBlob

Content...

Headers : {[Content-MD5, /bez4IkFAORz1WWm80bHnw==], [x-ms-request-id, d6df72e2-0001-0017-0896-11458a000000],

[x-ms-version, 2009-09-19], [x-ms-lease-status, unlocked]...}

RawContentLength : 906946

PS> Invoke-WebRequest http://oneget.org/oneget.zip | select -ExpandProperty Headers

Key Value

--- -----

Content-MD5 /bez4IkFAORz1WWm80bHnw==

x-ms-request-id d6dfadf7-0001-0017-3e96-11458a000000

x-ms-version 2009-09-19

x-ms-lease-status unlocked

x-ms-blob-type BlockBlob

Content-Length 906946

Content-Type application/octet-stream

Date Wed, 25 Feb 2015 21:07:42 GMT

ETag 0x8D1CE392F8BDAF4

Last-Modified Fri, 14 Nov 2014 15:45:44 GMT

Server Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

Lastly, just for kicks, to get all elements of a certain tag from a website:

PS> (Invoke-WebRequest http://rakhesh.com).AllElements | ?{ $_.tagname -eq "h2" } | ft outerText

outerText
---------
PowerShell as wget/ curl
Notes of PowerShell DSC course, OneGet & Chocolatey
Desired State Configuration (DSC)
OneGet and friends
Notes on using Git from behind a firewall, tunneling SSH through a proxy, Git credential helpers, and so on …
Batch file to add BUILTIN\Administrators group to a folder of roaming profiles
To view a file or folder owner use dir /q
Beware of takeown and recursively operating
New features in BlackBerry OS 10.3.1 (picture heavy)
AD Domain Password policies
Using Git and Dropbox (along with GitHub but working tree in Dropbox)
Setting it up
Changing things
Changing things (part 2)
Wrapping up
Roaming profile permissions and versions
How to move/ separate the .git folder out of your working tree
Using Git and Dropbox (along with GitHub)
Thoughts on BBC One’s “The Missing”
Setting up a test lab with Azure (part 1)
Step 1: Create an affinity group
Step 2: Set up the network
Step 3: Create a storage account
Step 4: Create a VM
Step 5: Set up self-signed certificate for PowerShell remoting
Step 6: Create a domain and promote this machine to DC
Replicate with repadmin
Authorizing a DHCP server in a Child Domain requires Enterprise Admin rights (or a delegation)
Windows 10 Cortana is not available if the region is not US
Windows 10 activation
Notes on Dynamic DNS updates, DNS Scavenging, etc.
Scavenging
Quickly rename a bunch of files via PowerShell

PS> (Invoke-WebRequest http://rakhesh.com).AllElements | ?{ $_.tagname -eq "h2" } | ft outerText

outerText

---------

PowerShell as wget/ curl

Notes of PowerShell DSC course, OneGet & Chocolatey

Desired State Configuration (DSC)

OneGet and friends

Notes on using Git from behind a firewall, tunneling SSH through a proxy, Git credential helpers, and so on …

Batch file to add BUILTIN\Administrators group to a folder of roaming profiles

To view a file or folder owner use dir /q

Beware of takeown and recursively operating

New features in BlackBerry OS 10.3.1 (picture heavy)

AD Domain Password policies

Using Git and Dropbox (along with GitHub but working tree in Dropbox)

Setting it up

Changing things

Changing things (part 2)

Wrapping up

Roaming profile permissions and versions

How to move/ separate the .git folder out of your working tree

Using Git and Dropbox (along with GitHub)

Thoughts on BBC One’s “The Missing”

Setting up a test lab with Azure (part 1)

Step 1: Create an affinity group

Step 2: Set up the network

Step 3: Create a storage account

Step 4: Create a VM

Step 5: Set up self-signed certificate for PowerShell remoting

Step 6: Create a domain and promote this machine to DC

Replicate with repadmin

Authorizing a DHCP server in a Child Domain requires Enterprise Admin rights (or a delegation)

Windows 10 Cortana is not available if the region is not US

Windows 10 activation

Notes on Dynamic DNS updates, DNS Scavenging, etc.

Scavenging

Quickly rename a bunch of files via PowerShell

Got to love it when you can do such cool things with Windows/ PowerShell so easily! A few years ago I could have barely imagined such a thing would be possible on Windows.

Update: While I am at it, here’s how you can quickly download a bunch of files from a website. In this case I want to download all mp3 files from a music website (for example purposes!). Through trial and error I know the links have the text “download” in them, so I can filter for such links and then download each of these to an appropriate file:

(Invoke-WebRequest http://downloadming.nu/badlapur-2015-mp3-songs).Links | ?{ $_.outerText -eq "download" } | select href | %{ Invoke-WebRequest $_.href -OutFile $(Split-Path -Leaf $_.href) }

1	(Invoke-WebRequest http://downloadming.nu/badlapur-2015-mp3-songs).Links \| ?{ $_.outerText -eq "download" } \| select href \| %{ Invoke-WebRequest $_.href -OutFile $(Split-Path -Leaf $_.href) }

Or a variant:

# I expand the "href" hashtable beforehand in this case
(Invoke-WebRequest http://downloadming.nu/badlapur-2015-mp3-songs).Links | ?{ $_.outerText -eq "download" } | select -ExpandProperty href | %{ Invoke-WebRequest $_ -OutFile $(Split-Path -Leaf $_) }

1 2	# I expand the "href" hashtable beforehand in this case (Invoke-WebRequest http://downloadming.nu/badlapur-2015-mp3-songs).Links \| ?{ $_.outerText -eq "download" } \| select -ExpandProperty href \| %{ Invoke-WebRequest $_ -OutFile $(Split-Path -Leaf $_) }

If you use a different site you’ll have to modify the filters accordingly, but it’s pretty straight-forward …

Tag: curl

Downloading all episodes of a podcast

PowerShell as wget/ curl