Prototyping with PowerShell

Requirements, specification, design and implementation

Requirement

We have two NuGet servers. The primary server is locally hosted for the developers. The second server is an Azure DevOps NuGet feed for contractors typically working outside the firewall.

A scheduled task synchronizing the two servers runs hourly, copying any new packages from the local server to the cloud, and any new pre-release packages from the cloud.

A bug caused some proprietary packages to be copied to the cloud, and they had to be deleted.

Specification

A script was written to use the NuGet command line interface (CLI) application to list the packages, and to delete them.

Afterwards I saw that the packages still showed up in the Azure DevOps feed. Looking at the documentation lead to this page.

Azure DevOps package deletion

After deleting a few packages I decided to automate. There were hundreds of packages. Automating actions on the web site is done by using the REST API. Getting started is fairly obvious.


Artifact Detail

Drilling down, I get to the Artifact Details – Get Packages page.

Artifact Details – Get Packages

I have worked with REST APIs before, so I know that I will be using the Invoke-RestMethod cmdlet. It has a rather terrifying list of parameters. Fortunately we only need three: -Uri, -Method and -Headers.

The -Headers parameter provides the credentials. This consists of two parts, a Personal Access Token (PAT) generated by Azure DevOps:

Create PAT
New Token
Create new PAT
Copy the PAT

Copy the token and save it somewhere safe. In this case I am embedding it in a PowerShell function:

function Get-AzureDevOpsCredential {
	param(
		[string]$Token = '0u8kfiagblr2d5gz097uwvp2rl9f7bnfcyjjmk4a9kow9sn4ui2t',
		[string]$UserEmail = 'joglethorpe@ecentric.co.za'
	)

	@{Authorization=("Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $UserEmail,$Token))))")}
}
$auth = Get-AzureDevOpsCredential
$org = 'epsdev'
Invoke-RestMethod -Uri "https://feeds.dev.azure.com/$org/_apis/packaging/Feeds/TestFeed/packages?api-version=5.0-preview.1" -Method GET -Headers $auth
count value
----- -----
   79 {@{id=2b0c842e-204c-4698-9c90-0c55dfd5eef5; normalizedName=backofficeaudit.logging; name=BackOfficeAudit.Logging; protocolType=NuGet; url=https://feeds.dev.azure.com/epsdev/_apis/Packaging/...

I refine the script to get a list of the packages and versions I want to delete:

$uri = "https://feeds.dev.azure.com/$org/_apis/packaging/Feeds/TestFeed/packages?protocolType=NuGet&packageNameQuery=Ecentric.UPG.Entity.EntityDB&includeAllVersions=True&api-version=5.0-preview.1"
$pkg = Invoke-RestMethod -Uri $uri -Method GET -Headers $auth
$pkg.value
$pkg.value.versions
id             : 3f53f107-fe7c-4403-b0c6-708833c48bf9
normalizedName : ecentric.upg.entity.entitydb
name           : Ecentric.UPG.Entity.EntityDB
protocolType   : NuGet
url            : https://feeds.dev.azure.com/epsdev/_apis/Packaging/Feeds/85a7e085-89e4-401b-a758-31fddd1dd507/Packages/3f53f107-fe7c-4403-b0c6-708833c48bf9
versions       : {@{id=6eb17396-68a5-4873-b9aa-e36b295ae629; normalizedVersion=0.2.181; version=0.2.181; isLatest=True; isListed=True;
                 storageId=27C537932E0859EFD657141FC2FDE0E14073502691C28C2297BC95A0EC0262A200; views=System.Object[]; publishDate=2019-04-08T15:46:27.67005Z},
                 @{id=4660b946-b17a-41c5-89d1-adca9e29a5ea; normalizedVersion=0.2.178; version=0.2.178; isLatest=False; isListed=True;
                 storageId=F5EBB39EB7FD34EBF5A28DB31ED505C1B0DA59F68CD94668491845E535E171A100; views=System.Object[]; publishDate=2019-04-05T13:55:06.5019964Z},
                 @{id=99340196-4f93-4cc7-8390-20509886913c; normalizedVersion=0.2.177; version=0.2.177; isLatest=False; isListed=True;
                 storageId=F5526525AFCFC9495AED24D5421ACBC9CF63BD48F587ACF32E49AF181F0EFC3400; views=System.Object[]; publishDate=2019-04-05T13:54:50.0169655Z},
                 @{id=37ec02b2-e048-4a68-be71-c76db845e3c5; normalizedVersion=0.2.176; version=0.2.176; isLatest=False; isListed=True;
                 storageId=5AECA5990B14D2A12631C52D4121D2C8DB2A35BF2C8FF6B7DDB013FC04BC616900; views=System.Object[]; publishDate=2019-04-05T13:54:33.6320533Z}...}
_links         : @{self=; feed=; versions=}
id                : 6eb17396-68a5-4873-b9aa-e36b295ae629
normalizedVersion : 0.2.181
version           : 0.2.181
isLatest          : True
isListed          : True
storageId         : 27C537932E0859EFD657141FC2FDE0E14073502691C28C2297BC95A0EC0262A200
views             : {@{id=c69bbc8c-0958-4e69-b91b-e2cc63c1001a; name=Local; url=; type=implicit}}
publishDate       : 2019-04-08T15:46:27.67005Z

id                : 4660b946-b17a-41c5-89d1-adca9e29a5ea
normalizedVersion : 0.2.178
version           : 0.2.178
isLatest          : False
isListed          : True
storageId         : F5EBB39EB7FD34EBF5A28DB31ED505C1B0DA59F68CD94668491845E535E171A100
views             : {@{id=c69bbc8c-0958-4e69-b91b-e2cc63c1001a; name=Local; url=; type=implicit}}
publishDate       : 2019-04-05T13:55:06.5019964Z

Now it is a simple matter of providing a list of the packages to be deleted, and to do a script to list them. The next matter is to delete each version.

Returning to the REST API, I see nothing to help in the Artifact documentation. Just below is the Artifacts Package Types sidebar, which I expand to NuGet:

Delete NuGet Package Version

Following the link, I find the necessary information to delete every package version.

Now I have all the information I need to do the job. More important, I have a working prototype. This may be sufficient. If I want to pass this task on to an administrator without PowerShell experience, I will need to make this into a utility – a script or command.

I start with the script as a specification of what needs to be done. The big plus, is that it works, and will require no further research by the developer.

Design and Implementation

I have already described how to create a PowerShell Script Module from a script. In summary:

  • Create a PowersShell project, including
    • This script
    • Empty Script and Manifest modules
    • A Build script
    • Scripts and Tests folders
  • Add a reference to the module from the script
  • Refactor the script
    • Select code to extract to a function
    • Write a basic test script for the function
    • Extract the code into the function
    • Test the extracted code
    • Continue refactoring by
      • Extracting code
      • Eliminating duplications

A NuGet List item parser – Part 2: Including Pre-release packages

In Part 1 I reorganised my NuGet source synchronization tool’s code to make it testable, with the specific aim of allowing pre-release packages to be synchronized. In particular, the affected code was isolated into this function:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$parts = $ver.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
			}
		}
	}
}

This is a challenge because the pre-release part of the version starts with a hyphen, followed by a series of dot separated identifiers after that. Each identifier consists of letters, digits and hyphens.

A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92.

https://semver.org/#spec-item-9

For our purposes, the pre-release label is whatever follows the hyphen.

Creating the first tests

This is the Pester test that we created previously:

if (Get-Module SyncNuGetRepos -All) {
	Remove-Module SyncNuGetRepos
}
Import-Module "$PSScriptRoot\..\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1" -WarningAction SilentlyContinue
Describe "Parse-PackageItem" {
	Context "Exists" {
		It "Runs" {
			Parse-PackageItem
		}
	}
}

First I add a test for the non-pre-release case. This would exist if TDD was done from the beginning.

Context "Non pre-release Item parts" {
		$parts = Parse-PackageItem -Package 'package 1.0.123'
	It "Given 'package 1.0.123', <property> should be <value>" -TestCases @(
		@{ property = 'id'; value = 'package' }
		@{ property = 'major'; value = '1' }
		@{ property = 'minor'; value = '0' }
		@{ property = 'patch'; value = '123' }
		@{ property = 'prerelease'; value = '' }
	) {
		param ($property, $value)
		"$($parts.$property)" | should -Be $value
	}
}
  Context Non pre-release Item parts
    [+] Given 'package 1.0.123', 'id' should be 'package' 69ms
    [+] Given 'package 1.0.123', 'major' should be '1' 23ms
    [+] Given 'package 1.0.123', 'minor' should be '0' 14ms
    [+] Given 'package 1.0.123', 'patch' should be '123' 15ms
    [+] Given 'package 1.0.123', 'prerelease' should be <empty> 14ms

The tests pass.

The good thing is that we can easily modify this test for the other scenarios we want to test. For the next test I am adding a simple pre-release label:

	Context "Simple pre-release Item parts" {
			$parts = Parse-PackageItem -Package 'package 1.0.123-PreRelease'
		It "Given 'package 1.0.123', <property> should be <value>" -TestCases @(
			@{ property = 'id'; value = 'package' }
			@{ property = 'major'; value = '1' }
			@{ property = 'minor'; value = '0' }
			@{ property = 'patch'; value = '123' }
			@{ property = 'prerelease'; value = 'PreRelease' }
		) {
			param ($property, $value)
			"$($parts.$property)" | should -Be $value
		}
	}

As expected, this failed:

  Context Simple pre-release Item parts
    [-] Error occurred in Context block 173ms
      FormatException: Input string was not in a correct format.
      PSInvalidCastException: Cannot convert value "123-PreRelease" to type "System.Int32". Error: "Input string was not in a correct format."
      RuntimeException: Cannot convert value "123-PreRelease" to type "System.Int32". Error: "Input string was not in a correct format."
      at Parse-PackageItem, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1: line 109
      at <ScriptBlock>, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\Tests\Parse-PackageItem.tests.ps1: line 25
      at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1\Functions\Describe.ps1: line 161

The immediate error was the integer conversion, suggesting that the patch should be fixed. Instead, the pre-release label modifies the entire version, not just the patch. I change

		[string]$ver = $idVer[1]
		$parts = $ver.Split('.')

becomes

		[string]$ver = $idVer[1]
		$verPreRel = $ver.Split('-')
		[string]$verParts = $verPreRel[0]
		[string]$preRel = ''
		if ($verParts.Length -eq 2) {
			$preRel = $verPreRel[1]
		}
		$parts = $verParts.Split('.')

and I assign $preRel to the prerelease property:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$verPreRel = $ver.Split('-')
		[string]$verParts = $verPreRel[0]
		[string]$preRel = ''
		if ($verPreRel.Length -eq 2) {
			$preRel = $verPreRel[1]
		}
		$parts = $verParts.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
				prerelease = $preRel
			}
		}
	}
}

I’m cheating a little. The test failed, and I had to fix a typo.

The next test verifies that the pre-release part can consist of dot-separtated identifiers:

	Context "Dot-separated identifier pre-release Item parts" {
			$parts = Parse-PackageItem -Package 'package 1.0.123-Pre.Release'
		It "Given 'package 1.0.123', <property> should be <value>" -TestCases @(
			@{ property = 'id'; value = 'package' }
			@{ property = 'major'; value = '1' }
			@{ property = 'minor'; value = '0' }
			@{ property = 'patch'; value = '123' }
			@{ property = 'prerelease'; value = 'Pre.Release' }
		) {
			param ($property, $value)
			"$($parts.$property)" | should -Be $value
		}
	}

The test passes. The next test is for the pre-release part containing a hyphen:

	Context "Hyphened identifier pre-release Item parts" {
			$parts = Parse-PackageItem -Package 'package 1.0.123-Pre-Release'
		It "Given 'package 1.0.123', <property> should be <value>" -TestCases @(
			@{ property = 'id'; value = 'package' }
			@{ property = 'major'; value = '1' }
			@{ property = 'minor'; value = '0' }
			@{ property = 'patch'; value = '123' }
			@{ property = 'prerelease'; value = 'Pre-Release' }
		) {
			param ($property, $value)
			"$($parts.$property)" | should -Be $value
		}
	}

As expected, this failed:

    [-] Given 'package 1.0.123', 'prerelease' should be 'Pre-Release' 13ms
      Expected strings to be the same, but they were different.
      Expected length: 11
      Actual length:   0
      Strings differ at index 0.
      Expected: 'Pre-Release'
      But was:  ''
      -----------^
      60: 			"$($parts.$property)" | should -Be $value
      at <ScriptBlock>, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\Tests\Parse-PackageItem.tests.ps1: line 60

The value was empty because of $ver.Split(‘-‘) returning more than two parts. I check the String.Split documentation and see that I can make this $ver.Split(‘-‘, 2). After I make the change, the test passes.

Now my function looks like this:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$verPreRel = $ver.Split('-', 2)
		[string]$verParts = $verPreRel[0]
		[string]$preRel = ''
		if ($verPreRel.Length -eq 2) {
			$preRel = $verPreRel[1]
		}
		$parts = $verParts.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
				prerelease = $preRel
			}
		}
	}
}