Summary: Microsoft Scripting Guy, Ed Wilson, talks about using the Windows PowerShell BITS module to download large files.
Microsoft Scripting Guy, Ed Wilson, is here. One of the cool things about my job—OK, so there are many cool things about my job—but one of the things that I just geek out on, is access to the daily builds of Windows. All day, the developers are writing code for Windows and the testers are testing the builds. At night, they are all checked-in; and all night long, we build Windows. Boom, the next day, the timer dings, and we have a fresh build of Windows.
As you may recall, the Scripting Wife and I live in Charlotte, North Carolina, and I work from home. I rarely go into the office on the Microsoft campus. In fact, the last time I was there, it was to upgrade my corpnet attached laptop from a dogfood edition of Windows 8 to the RTM bits. Testing THAT process REQUIRED a physically hardwired LAN connection. But normally, via Direct Access, I am at home in my office working away, and I rarely notice or miss not being in the office in Charlotte. (Oh, I MISS the two hour commute. REALLY, I do).
I have absolutely the fastest commercially available Internet connection in my home office (currently clocked at a little better than dual-band ISDN), so I need to make my connections count. When I was downloading the daily builds, I would select the Windows 8 32-bit, Windows 8 64-bit, Windows Server 2012, and the RSAT tools for Windows 8 (32-bit and 64-bit). This ended up being nearly 12 gigabytes of files to download. With an unreliable Internet connection, it can be very frustrating to be in the middle of a download when, BOOM, the link drops, and you have to start all over. Major bummer! Major bummer indeed!
I solved my download problem by using the BITS module. (Now, this is not a completely new idea for me, I have been using BITS for years. In fact, I was teaching a Windows PowerShell class in Edinburgh, Scotland, and downloading the daily builds of Windows Vista. I would start the download in the Microsoft office there, teach class all day, pause my download, go to the hotel, and start the download back up. I would go to dinner, and usually, by the time I was back in the hotel, the download had completed, and I could create the virtual machines and begin my testing. Back in those days, I used a VBScript script that contained a number of calls to the BITS admin utility.
BITS is worth a bit of trouble because it implements checkpoint restarts, automatic bandwidth throttling, and a number of other very useful features.
Use the BITS PowerShell module to download large files
I use the following script to create a CSV file that I use to control the way that BITS works. It lists each file to download and provides a destination for each file.
The first thing I do is specify the path for my CSV file. It is always the same file name in the same file location. Next, I specify the path to the daily builds for the server, x86 and 64-bit builds. These four lines of code are shown here.
$csvPath = "C:\fso\files.csv"
$serverPath = "\\dailybuilds\amd64fre\iso\iso_server_eval_en-us"
$clientPath = "\\dailybuilds\x86fre\iso\iso_enterprise_eval_en-us"
$client64 = "\\dailybuilds\amd64fre\iso\iso_enterprise_eval_en-us"
If the target CSV file already exists, I rename it and provide the file with a random name. In this way, I can look back and see previous versions of the download file if required. Besides, it is a pretty cool use for the Get-Random cmdlet as shown here.
if(Test-Path $csvPath)
{ Rename-Item -Path $csvPath -NewName ("{0}.{1}.txt" -f $csvpath, (get-random).tostring()) }
I now need to add the header for my CSV file. The column headings are Source and Destination, and the code is shown here.
"source, Destination" | Out-File -FilePath $csvPath -Encoding ascii
I now need to produce a list of files to copy. The cool thing is that the Get-ChildItem cmdlet accepts an array of paths, and therefore I simply supply the three paths as an array to the cmdlet, and pipe the gather fileinfo objects down the line. This line of code is shown here.
Get-ChildItem $serverPath,$clientPath,$client64 |
Now I use the Foreach-Object cmdlet to walk through the collection and I pick up the FullName property (the full name includes the complete path to the file). Here is the cool part. I create a string. The string includes the complete path to the source file, a comma, and the complete path for the destination file. I then pipe this string to the Out-File cmdlet. I am, in effect, rolling my own CSV file. I will admit it is a bit of FrankenCode, but hey, it is my own code that I use for my own purposes (besides, K. Schulte made a comment on a recent posting that he loves FrankenCode—I am always happy to oblige).
ForEach-Object { "$($_.fullname),C:\data\win8Server\$($_.name)" } |
Out-File -FilePath $csvPath -Encoding ascii -Append
I have now created my input file that I will use for the BITS commands. The complete script is shown here.
CreateBitsDownloadFile.ps1
$csvPath = "C:\fso\files.csv"
$serverPath = "\\dailybuilds\amd64fre\iso\iso_server_eval_en-us"
$clientPath = "\\dailybuilds\x86fre\iso\iso_enterprise_eval_en-us"
$client64 = "\\dailybuilds\amd64fre\iso\iso_enterprise_eval_en-us"
if(Test-Path $csvPath)
{ Rename-Item -Path $csvPath -NewName ("{0}.{1}.txt" -f $csvpath, (get-random).tostring()) }
"source, Destination" | Out-File -FilePath $csvPath -Encoding ascii
Get-ChildItem $serverPath,$clientPath,$client64 |
ForEach-Object { "$($_.fullname),C:\data\win8Server\$($_.name)" } |
Out-File -FilePath $csvPath -Encoding ascii -Append
Starting and monitoring the download
My next Windows PowerShell script is not even a script. I just type a few Windows PowerShell commands. First I import the module (I know that Windows PowerShell 3.0 automatically imports modules, but it is a bit slow, and I can easily type ipmo *bit* very fast. So I import the BITS module and start the BITS transfer.
A couple of things…I use the Import-CSV cmdlet to import my CSV file, and I pipe the information to the Start-BitsTransfer cmdlet. I specify an Asynchronous transfer and a RetryInterval of 60 seconds. For everything else, I go with the defaults. One thing that is really confusing is the priority parameter. The first time I did this, I chose HIGH because I thought that would mean a high priority. Unfortunately, the highest priority is Foreground. The ordering goes like this: Foreground, High, Normal, Low. Luckily Foreground is the default, so I just leave it there. It works for me, but keep in mind that it will suck your pipe dry. So if, for instance, your wife is trying to watch the PowerScripting Podcast while you are doing this, you WILL hear about it. These two commands are shown here.
Import-Module *bits*
Import-Csv C:\fso\files.csv | Start-BitsTransfer -Asynchronous -retryInterval 60
To monitor the bits jobs, I use the following command, where I look for jobs that are not transferred.
Get-BitsTransfer | ? { $_.jobstate -ne 'transferred'}
If I want to look at the statistics of my download jobs, I use the command shown here.
Get-BitsTransfer | ? { $_.jobstate -ne 'transferred'} |
select jobid, jobstate,
@{Label="percent Complete"; EXPRESSION={($_.BytesTransferred/$_.BytesTotal*100)} }
Occasionally a BITS job will stall. But hey, this is why I use BITS in the first place. So to restart stalled jobs, I use the command shown here.
Get-BitsTransfer | ? jobstate -ne transferred | Resume-BitsTransfer -Asynchronous
If a BITS job continues to error out, it is time to get rid of it. So I use this command to delete those jobs.
Get-BitsTransfer | ? jobstate -eq error | Remove-BitsTransfer
When everything is percolating nicely, it is time to go watch an old Perry Mason rerun on my other computer. So I can keep an eye on things (and know if I have time to watch a second episode), I came up with this code. It refreshes every five minutes and lets me know the current status of the jobs.
1..1000 | ForEach-Object {
Clear-Host ; "last update $(get-date)"
Get-BitsTransfer |
Where-Object { $_.jobstate -ne 'transferred'} |
select jobid, jobstate,
@{Label="percent Complete"; EXPRESSION={($_.BytesTransferred/$_.BytesTotal*100)} }
sleep 300 }
WooHoo, the jobs are done. Well, how did my bandwidth do today? I always like to know if it would have been quicker to drive two hours to the Microsoft office in Charlotte to make my download. So I came up with this command to display the statistics.
Get-BitsTransfer |
% { "Job id {0} transfered {1} in {2} total minutes" -f
$_.jobid, $_.bytesTransferred,
[int](New-TimeSpan -Start $_.CreationTime -End $_.TransferCompletionTime).totalMinutes }
When it is done, I am not yet done. I have to complete my transfers. Without this command, all you will have is a bunch of non-usable files. So to complete the BITS transfer, use the Complete-BitsTransfer cmdlet as shown here.
Get-BitsTransfer | Complete-BitsTransfer
Well, I hope you find these commands useful. This is not complete documentation on the BITS cmdlets. In fact, I did not cover half of them. But these are the ones I use on a daily basis, and hence, for me anyway, they are the most useful. I worked them out over the course of the last couple of years whilst getting my daily ration of dogfood.
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy