I recently had the job of copying ten’s of thousands of IIS log files, each one at least 100MB, from one Azure Storage account to another. Using something simple like CloudBerry to copy the file just wasn’t going to cut it as it copies the file first to the local client, then pushes it back into Azure, not efficient at all.
A quick bit of digging and I discovered that the Azure PowerShell cmdlet Start-AzureStorageBlobCopy allows you to trigger a copy Azure to Azure, which runs very quickly, it will even allow you to copy an entire container from one storage account to another; what it won’t allow you to do is pass a filter in so only copies files matching the filter.
So here’s a function that I wrote to get that functionality, with some progress bars and timers for added effect 🙂
Function Start-AzureStorageBlobContainerCopy ( [Parameter(Mandatory=$true)][String]$srcStorageAccountName, [Parameter(Mandatory=$true)][String]$destStorageAccountName, [Parameter(Mandatory=$true)][String]$SrcStorageAccountKey, [Parameter(Mandatory=$true)][String]$DestStorageAccountKey, [Parameter(Mandatory=$true)][String]$SrcContainer, [Parameter(Mandatory=$true)][String]$DestContainer, [String]$filter = "" ) { Import-Module Azure $srcContext = New-AzureStorageContext -StorageAccountName $srcStorageAccountName -StorageAccountKey $SrcStorageAccountKey $destContext = New-AzureStorageContext -StorageAccountName $destStorageAccountName -StorageAccountKey $DestStorageAccountKey $timeTaken = measure-command{ if ($filter -ne "") { $blobs = Get-AzureStorageBlob -Container $SrcContainer -Context $srcContext | ? {$_.name -match $filter} } else { $blobs = Get-AzureStorageBlob -Container $SrcContainer -Context $srcContext } } Write-host "Total Time to index $timeTaken" -BackgroundColor Black -ForegroundColor Green $i = 0 $timeTaken = measure-command{ foreach ($blob in $blobs) { $i++ Write-Progress -Activity:"Copying..." -Status:"Copied $i of $($blobs.Count) : $($percentComplete)%" -PercentComplete:$percentComplete $copyInfo = Start-AzureStorageBlobCopy -ICloudBlob $blob.ICloudBlob -Context $srcContext -DestContainer $DestContainer -DestContext $destContext -Force Write-host (get-date) $copyInfo.name } } write-host Write-host "Total Time $timeTaken" -BackgroundColor Black -ForegroundColor Green } Start-AzureStorageBlobContainerCopy -srcStorageAccountName "<src Storage>" -SrcStorageAccountKey "<src key>" -SrcContainer "<src Container>" ` -destStorageAccountName "<dest Storage>" -DestStorageAccountKey "<dest key>" -DestContainer "<dest Container>" ` -filter "<filter>"
THIS POSTING AND CODE RELATED TO IT ARE PROVIDED “AS IS” AND INFERS NO WARRANTIES OR RIGHTS, USE AT YOUR OWN RISK
Steven Askwith 