Managing complex PowerShell applications

Background

I inherited responsibility for an archaic build and release framework when my predecessor moved on to leading an application development team. It was heavily customized mostly using PowerShell scripts invoked from command scripts.

The build and release tasks were all scripted in an enormous PowerShell script. My priority was to make this script understandable. It had a large number of functions followed by extensive control logic ending with a large switch based on its task parameter.

A single script was a reasonable solution to the problem of deploying any changes to developers and build servers, but maintenance was a horror. It best suited me to use Windows Explorer to index the functions, and to use Notepad++ for viewing them.

I was very new to PowerShell, and identifying the functions, and their bodies struck me as beyond my ability. My natural tendency is to write a parser, and my colleague is a regex wizard. I was not happy with either approach.

My research lead me to Microsoft’s PowerShell parser: System.Management.Automation.Language.Parser. Getting it to parse a PowerShell script is straightforward:

[System.Management.Automation.Language.Token[]]$tokens=$null
[System.Management.Automation.Language.ParseError[]]$errors=$null
[System.Management.Automation.Language.ScriptBlockAst]$script=$null
$script=[System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$tokens, [ref]$errors)

I spent some time poking around the syntax tree, and discovered that the function definitions are all top level statements. I first walked the statement list and this bit of PowerShell:

$script.EndBlock.Statements | ? { $_.Name -eq 'Out-FormattedXml' } | % { $_.Extent }

yields:

File                : C:\Dev\GTMBilling\artifact\Process_Dependencies.ps1
StartScriptPosition : System.Management.Automation.Language.InternalScriptPosition
EndScriptPosition   : System.Management.Automation.Language.InternalScriptPosition
StartLineNumber     : 7416
StartColumnNumber   : 1
EndLineNumber       : 7423
EndColumnNumber     : 2
Text                : Function Out-FormattedXml {
                        param (
                                [xml]$Xml,
                                [string]$FilePath
                        )
                        $Xml = $Xml.OuterXML.Replace(' xmlns=""','') # I don't know where this comes from
                        Format-XMLIndent $Xml -Indent 2 | Out-File $FilePath -Encoding utf8
                      }
StartOffset         : 200999
EndOffset           : 201232

So to get the functions I just need to reference the Name and Text properties.

I packaged this into one of two scripts to extract the functions from a named script, and a second one to put them back, using a template of the script stripped of the functions.

Later

I asked my predecessor to stand in for me when I was on leave, and he asked me to document the new version of the script. Specifically, he asked me for the dependencies between the functions. There are 278 of them, and it was not a job I wanted to do.

The AST of the script was the perfect tool to get this information. What I needed was a hash on the function names of the functions. Each record had two lists added of References and ReferencedBy links to other objects. Links from the script itself were also useful, so I made it a list of Executables instead of Functions.

Finding the references depended on searching the body of the function for commands invoking a function. I followed my natural inclination of walking through the semantic tree, but the program quickly became unmanageable.

The developers had followed the Visitor pattern on the AST. The calls are easy to locate using FindAll on each statement in the function definition to look for commands and check each if it is one of the functions. With the caller and callee in hand, the Referenced and ReferencedBy links are set.

What next?

My inexperience with PowerShell made me look for classes. It was an educational experience worth sharing.

I am revisiting this project to also show calls between modules.

Prototyping with PowerShell

Requirements, specification, design and implementation

Requirement

We have two NuGet servers. The primary server is locally hosted for the developers. The second server is an Azure DevOps NuGet feed for contractors typically working outside the firewall.

A scheduled task synchronizing the two servers runs hourly, copying any new packages from the local server to the cloud, and any new pre-release packages from the cloud.

A bug caused some proprietary packages to be copied to the cloud, and they had to be deleted.

Specification

A script was written to use the NuGet command line interface (CLI) application to list the packages, and to delete them.

Afterwards I saw that the packages still showed up in the Azure DevOps feed. Looking at the documentation lead to this page.

Azure DevOps package deletion

After deleting a few packages I decided to automate. There were hundreds of packages. Automating actions on the web site is done by using the REST API. Getting started is fairly obvious.


Artifact Detail

Drilling down, I get to the Artifact Details – Get Packages page.

Artifact Details – Get Packages

I have worked with REST APIs before, so I know that I will be using the Invoke-RestMethod cmdlet. It has a rather terrifying list of parameters. Fortunately we only need three: -Uri, -Method and -Headers.

The -Headers parameter provides the credentials. This consists of two parts, a Personal Access Token (PAT) generated by Azure DevOps:

Create PAT
New Token
Create new PAT
Copy the PAT

Copy the token and save it somewhere safe. In this case I am embedding it in a PowerShell function:

function Get-AzureDevOpsCredential {
	param(
		[string]$Token = '0u8kfiagblr2d5gz097uwvp2rl9f7bnfcyjjmk4a9kow9sn4ui2t',
		[string]$UserEmail = 'joglethorpe@ecentric.co.za'
	)

	@{Authorization=("Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $UserEmail,$Token))))")}
}
$auth = Get-AzureDevOpsCredential
$org = 'epsdev'
Invoke-RestMethod -Uri "https://feeds.dev.azure.com/$org/_apis/packaging/Feeds/TestFeed/packages?api-version=5.0-preview.1" -Method GET -Headers $auth
count value
----- -----
   79 {@{id=2b0c842e-204c-4698-9c90-0c55dfd5eef5; normalizedName=backofficeaudit.logging; name=BackOfficeAudit.Logging; protocolType=NuGet; url=https://feeds.dev.azure.com/epsdev/_apis/Packaging/...

I refine the script to get a list of the packages and versions I want to delete:

$uri = "https://feeds.dev.azure.com/$org/_apis/packaging/Feeds/TestFeed/packages?protocolType=NuGet&packageNameQuery=Ecentric.UPG.Entity.EntityDB&includeAllVersions=True&api-version=5.0-preview.1"
$pkg = Invoke-RestMethod -Uri $uri -Method GET -Headers $auth
$pkg.value
$pkg.value.versions
id             : 3f53f107-fe7c-4403-b0c6-708833c48bf9
normalizedName : ecentric.upg.entity.entitydb
name           : Ecentric.UPG.Entity.EntityDB
protocolType   : NuGet
url            : https://feeds.dev.azure.com/epsdev/_apis/Packaging/Feeds/85a7e085-89e4-401b-a758-31fddd1dd507/Packages/3f53f107-fe7c-4403-b0c6-708833c48bf9
versions       : {@{id=6eb17396-68a5-4873-b9aa-e36b295ae629; normalizedVersion=0.2.181; version=0.2.181; isLatest=True; isListed=True;
                 storageId=27C537932E0859EFD657141FC2FDE0E14073502691C28C2297BC95A0EC0262A200; views=System.Object[]; publishDate=2019-04-08T15:46:27.67005Z},
                 @{id=4660b946-b17a-41c5-89d1-adca9e29a5ea; normalizedVersion=0.2.178; version=0.2.178; isLatest=False; isListed=True;
                 storageId=F5EBB39EB7FD34EBF5A28DB31ED505C1B0DA59F68CD94668491845E535E171A100; views=System.Object[]; publishDate=2019-04-05T13:55:06.5019964Z},
                 @{id=99340196-4f93-4cc7-8390-20509886913c; normalizedVersion=0.2.177; version=0.2.177; isLatest=False; isListed=True;
                 storageId=F5526525AFCFC9495AED24D5421ACBC9CF63BD48F587ACF32E49AF181F0EFC3400; views=System.Object[]; publishDate=2019-04-05T13:54:50.0169655Z},
                 @{id=37ec02b2-e048-4a68-be71-c76db845e3c5; normalizedVersion=0.2.176; version=0.2.176; isLatest=False; isListed=True;
                 storageId=5AECA5990B14D2A12631C52D4121D2C8DB2A35BF2C8FF6B7DDB013FC04BC616900; views=System.Object[]; publishDate=2019-04-05T13:54:33.6320533Z}...}
_links         : @{self=; feed=; versions=}
id                : 6eb17396-68a5-4873-b9aa-e36b295ae629
normalizedVersion : 0.2.181
version           : 0.2.181
isLatest          : True
isListed          : True
storageId         : 27C537932E0859EFD657141FC2FDE0E14073502691C28C2297BC95A0EC0262A200
views             : {@{id=c69bbc8c-0958-4e69-b91b-e2cc63c1001a; name=Local; url=; type=implicit}}
publishDate       : 2019-04-08T15:46:27.67005Z

id                : 4660b946-b17a-41c5-89d1-adca9e29a5ea
normalizedVersion : 0.2.178
version           : 0.2.178
isLatest          : False
isListed          : True
storageId         : F5EBB39EB7FD34EBF5A28DB31ED505C1B0DA59F68CD94668491845E535E171A100
views             : {@{id=c69bbc8c-0958-4e69-b91b-e2cc63c1001a; name=Local; url=; type=implicit}}
publishDate       : 2019-04-05T13:55:06.5019964Z

Now it is a simple matter of providing a list of the packages to be deleted, and to do a script to list them. The next matter is to delete each version.

Returning to the REST API, I see nothing to help in the Artifact documentation. Just below is the Artifacts Package Types sidebar, which I expand to NuGet:

Delete NuGet Package Version

Following the link, I find the necessary information to delete every package version.

Now I have all the information I need to do the job. More important, I have a working prototype. This may be sufficient. If I want to pass this task on to an administrator without PowerShell experience, I will need to make this into a utility – a script or command.

I start with the script as a specification of what needs to be done. The big plus, is that it works, and will require no further research by the developer.

Design and Implementation

I have already described how to create a PowerShell Script Module from a script. In summary:

  • Create a PowersShell project, including
    • This script
    • Empty Script and Manifest modules
    • A Build script
    • Scripts and Tests folders
  • Add a reference to the module from the script
  • Refactor the script
    • Select code to extract to a function
    • Write a basic test script for the function
    • Extract the code into the function
    • Test the extracted code
    • Continue refactoring by
      • Extracting code
      • Eliminating duplications

Downloading an Azure DevOps Universal Package from the Command Line

Automating Azure DevOps

There are several ways in which Azure DevOps can be automated with scripts. Here I show how to use the AZ command line tool retrieve an artifact created by a build pipeline. Understanding how to use a CLI tool is the first requirement for automation.

The Azure DevOps CLI provides access to external assets such as build and release pipelines, work items, repositories, and build artifact feeds.

There are also the REST APIs that give access to the DevOps site assets.

Scenario

The Azure DevOps build pipeline can create build artifacts after a successful build. I want to manually download a Universal Package posted by a build.

Finding the package

On the Azure DevOps site, I navigate to the Artifact Packages


Navigating to the package

I select the package to see the details

The package details

Unfortunately the command to download the package is invalid, as the VSTS CLI tool has been deprecated. Some searching leads to Azure DevOps CLI in the Visual Studio Marketplace. Unlike the VSTS CLI, the DevOps functionality is an extension of the AZ CLI tool.

Following the help information, the correct command is

az artifacts universal download \
  --org "https://dev.azure.com/epsdev/" \
  --feed "TestRelease" \
  --name "epsbilling" \
  --version "1.0.0" \
  --path .

Before I can use this command, I need to log in

az login --allow-no-subscriptions -u ████@█████.com -p ████████

and at the end I need to log out of Azure

az logout

Thoughts

I find this integration into the Azure CLI annoying. The security model is intended for Azure applications which does not meet the needs of DevOps automation. Particularly annoying is the need to embed passwords in the scripts.

Azure DevOps supports the creation of Personal Access Tokens (PATs), with specific permissions. Something like that is required.

My workaround is to create user variables with the username and password, e.g. -u $env:artifactUserName -p $env:artifactPassword.

A NuGet List item parser – Part 2: Including Pre-release packages

In Part 1 I reorganised my NuGet source synchronization tool’s code to make it testable, with the specific aim of allowing pre-release packages to be synchronized. In particular, the affected code was isolated into this function:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$parts = $ver.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
			}
		}
	}
}

This is a challenge because the pre-release part of the version starts with a hyphen, followed by a series of dot separated identifiers after that. Each identifier consists of letters, digits and hyphens.

A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92.

https://semver.org/#spec-item-9

For our purposes, the pre-release label is whatever follows the hyphen.

Creating the first tests

This is the Pester test that we created previously:

if (Get-Module SyncNuGetRepos -All) {
	Remove-Module SyncNuGetRepos
}
Import-Module "$PSScriptRoot\..\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1" -WarningAction SilentlyContinue
Describe "Parse-PackageItem" {
	Context "Exists" {
		It "Runs" {
			Parse-PackageItem
		}
	}
}

First I add a test for the non-pre-release case. This would exist if TDD was done from the beginning.

Context "Non pre-release Item parts" {
		$parts = Parse-PackageItem -Package 'package 1.0.123'
	It "Given 'package 1.0.123', <property&gt; should be <value&gt;" -TestCases @(
		@{ property = 'id'; value = 'package' }
		@{ property = 'major'; value = '1' }
		@{ property = 'minor'; value = '0' }
		@{ property = 'patch'; value = '123' }
		@{ property = 'prerelease'; value = '' }
	) {
		param ($property, $value)
		"$($parts.$property)" | should -Be $value
	}
}
  Context Non pre-release Item parts
    [+] Given 'package 1.0.123', 'id' should be 'package' 69ms
    [+] Given 'package 1.0.123', 'major' should be '1' 23ms
    [+] Given 'package 1.0.123', 'minor' should be '0' 14ms
    [+] Given 'package 1.0.123', 'patch' should be '123' 15ms
    [+] Given 'package 1.0.123', 'prerelease' should be <empty&gt; 14ms

The tests pass.

The good thing is that we can easily modify this test for the other scenarios we want to test. For the next test I am adding a simple pre-release label:

	Context "Simple pre-release Item parts" {
			$parts = Parse-PackageItem -Package 'package 1.0.123-PreRelease'
		It "Given 'package 1.0.123', <property&gt; should be <value&gt;" -TestCases @(
			@{ property = 'id'; value = 'package' }
			@{ property = 'major'; value = '1' }
			@{ property = 'minor'; value = '0' }
			@{ property = 'patch'; value = '123' }
			@{ property = 'prerelease'; value = 'PreRelease' }
		) {
			param ($property, $value)
			"$($parts.$property)" | should -Be $value
		}
	}

As expected, this failed:

  Context Simple pre-release Item parts
    [-] Error occurred in Context block 173ms
      FormatException: Input string was not in a correct format.
      PSInvalidCastException: Cannot convert value "123-PreRelease" to type "System.Int32". Error: "Input string was not in a correct format."
      RuntimeException: Cannot convert value "123-PreRelease" to type "System.Int32". Error: "Input string was not in a correct format."
      at Parse-PackageItem, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1: line 109
      at <ScriptBlock&gt;, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\Tests\Parse-PackageItem.tests.ps1: line 25
      at DescribeImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1\Functions\Describe.ps1: line 161

The immediate error was the integer conversion, suggesting that the patch should be fixed. Instead, the pre-release label modifies the entire version, not just the patch. I change

		[string]$ver = $idVer[1]
		$parts = $ver.Split('.')

becomes

		[string]$ver = $idVer[1]
		$verPreRel = $ver.Split('-')
		[string]$verParts = $verPreRel[0]
		[string]$preRel = ''
		if ($verParts.Length -eq 2) {
			$preRel = $verPreRel[1]
		}
		$parts = $verParts.Split('.')

and I assign $preRel to the prerelease property:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$verPreRel = $ver.Split('-')
		[string]$verParts = $verPreRel[0]
		[string]$preRel = ''
		if ($verPreRel.Length -eq 2) {
			$preRel = $verPreRel[1]
		}
		$parts = $verParts.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
				prerelease = $preRel
			}
		}
	}
}

I’m cheating a little. The test failed, and I had to fix a typo.

The next test verifies that the pre-release part can consist of dot-separtated identifiers:

	Context "Dot-separated identifier pre-release Item parts" {
			$parts = Parse-PackageItem -Package 'package 1.0.123-Pre.Release'
		It "Given 'package 1.0.123', <property&gt; should be <value&gt;" -TestCases @(
			@{ property = 'id'; value = 'package' }
			@{ property = 'major'; value = '1' }
			@{ property = 'minor'; value = '0' }
			@{ property = 'patch'; value = '123' }
			@{ property = 'prerelease'; value = 'Pre.Release' }
		) {
			param ($property, $value)
			"$($parts.$property)" | should -Be $value
		}
	}

The test passes. The next test is for the pre-release part containing a hyphen:

	Context "Hyphened identifier pre-release Item parts" {
			$parts = Parse-PackageItem -Package 'package 1.0.123-Pre-Release'
		It "Given 'package 1.0.123', <property&gt; should be <value&gt;" -TestCases @(
			@{ property = 'id'; value = 'package' }
			@{ property = 'major'; value = '1' }
			@{ property = 'minor'; value = '0' }
			@{ property = 'patch'; value = '123' }
			@{ property = 'prerelease'; value = 'Pre-Release' }
		) {
			param ($property, $value)
			"$($parts.$property)" | should -Be $value
		}
	}

As expected, this failed:

    [-] Given 'package 1.0.123', 'prerelease' should be 'Pre-Release' 13ms
      Expected strings to be the same, but they were different.
      Expected length: 11
      Actual length:   0
      Strings differ at index 0.
      Expected: 'Pre-Release'
      But was:  ''
      -----------^
      60: 			"$($parts.$property)" | should -Be $value
      at <ScriptBlock&gt;, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\Tests\Parse-PackageItem.tests.ps1: line 60

The value was empty because of $ver.Split(‘-‘) returning more than two parts. I check the String.Split documentation and see that I can make this $ver.Split(‘-‘, 2). After I make the change, the test passes.

Now my function looks like this:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$verPreRel = $ver.Split('-', 2)
		[string]$verParts = $verPreRel[0]
		[string]$preRel = ''
		if ($verPreRel.Length -eq 2) {
			$preRel = $verPreRel[1]
		}
		$parts = $verParts.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
				prerelease = $preRel
			}
		}
	}
}

A NuGet List item parser – Part 1: Making legacy code testable

I am building a tool to synchronize two NuGet servers. One is hosted on a development server, and the other is in the cloud.

I need to add support for Pre-Release packages. In the first of two parts, I refactor the code to make it testable.

The existing code

This tool started its life as a script, which I started refactoring to avoid duplicate code. This tool needs to find the packages already in each source. The code is still scripty, Particularly when selecting the packages to be synchronized. There are a lot of legacy packages that should not go to the cloud server.

In the middle I have this code to put the package labels into a more useful structure:

[string]$pkg = $_
$idVer = $pkg.Split(' ')
if ($idVer.Length -eq 2) {
	$id = $idVer[0]
	if (-not $package.ContainsKey($id)) {
		[string]$ver = $idVer[1]
		$parts = $ver.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $pkg
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
			}
		}
	}
}

What the code sees is something like this:

NuGetShared 0.1.132

It also has to work with something like this:

NuGetProjectPacker 0.1.72-Dependencies

The PreRelease tag, -Dependencies in this example, can contain any combination of letters, digits and hyphens after the initial hyphen.

Preparation

I am going to do this as a professional programmer working on legacy code should. The code must be tested thoroughly, and the Test Driven Development (TDD) methodology demands that I write a failing test for any new functionality, then get the test to work.

In an earlier blog I described how I structure a PowerShell script module project. For testing I create a Tests subfolder. I will be using Pester to run my tests, so the naming convention of the files is important. I use the PowerShell Tools extension to Visual Studio which does some of the work for me. My new function will be called Parse-PackageItem. Parse is not an approved verb, but there does not appear to be an alternative.

Creating the test

Visual Studio creates this test for me, and I add a call to the new function without any parameters:

Describe "Parse-PackageItem" {
	Context "Exists" {
		It "Runs" {
			Parse-PackageItem
		}
	}
}

I run it, and it fails as expected

Describing Parse-PackageItem

  Context Exists
    [-] Runs 1s
      CommandNotFoundException: The term 'Parse-PackageItem' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
      at <ScriptBlock&gt;, C:\VSTS\ContinuousIntegration\SyncNuGetRepos\SyncNuGetRepos\Tests\Parse-PackageItem.tests.ps1: line 4

I start fixing the problem by adding the empty function in the Scripts folder. I could make it visible by dot-sourcing it:

. .\Scripts\Parse-PackageItem.ps1

but that will cause problems later. After I build my project, it is in my script module:

Import-Module "$PSScriptRoot\..\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1"

After I run the test, I get the annoying warning:

[WARNING] The names of some imported commands from the module 'SyncNuGetRepos' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
I suppress the warning by adding the Warning Action parameter
Import-Module "$PSScriptRoot\..\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1" -WarningAction SilentlyContinue

I also need to allow the test to rerun, and not re-import the module. That causes problems in Pester if there are multiple instances. The right thing to do is to remove it first. We always want to run our tests against the latest version of the module.

if (Get-Module SyncNuGetRepos -All) {
	Remove-Module SyncNuGetRepos
}
Import-Module "$PSScriptRoot\..\bin\Debug\SyncNuGetRepos\SyncNuGetRepos.psm1" -WarningAction SilentlyContinue
Describe "Parse-PackageItem" {
	Context "Exists" {
		It "Runs" {
			Parse-PackageItem
		}
	}
}

I now turn my attention to the function, and add it:

function Parse-PackageItem {
	
}

I build and run the test which succeeds.

Moving the body of code to the function

The function needs one parameter, -Package, which will replace the $pkg variable in the code

function Parse-PackageItem {
	param(
		[string]$Package
	)
}

I copy the code, paste it into the function, and rename $pkg:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		if (-not $package.ContainsKey($id)) {
			[string]$ver = $idVer[1]
			$parts = $ver.Split('.')
			if ($parts.Length -eq 3) {
				[int]$major = $parts[0]
				[int]$minor = $parts[1]
				[int]$patch = $parts[2]
				new-object -TypeName PSCustomObject -Property @{
					package = $Package
					id = $id
					version = $ver
					major = $major
					minor = $minor
					patch = $patch
				}
			}
		}
	}
}

At this point I discover a mistake: I have a hash table $package that I copied into the function. It is used as an optimization, to avoid creating the package object if it already exists. I need to move the test out of the function. The small functional change will be that I occasionally create then ignore the package object, instead of not creating it. I can live with that.

The function is now like this:

function Parse-PackageItem {
	param(
		[string]$Package
	)
	$idVer = $Package.Split(' ')
	if ($idVer.Length -eq 2) {
		$id = $idVer[0]
		[string]$ver = $idVer[1]
		$parts = $ver.Split('.')
		if ($parts.Length -eq 3) {
			[int]$major = $parts[0]
			[int]$minor = $parts[1]
			[int]$patch = $parts[2]
			new-object -TypeName PSCustomObject -Property @{
				package = $Package
				id = $id
				version = $ver
				major = $major
				minor = $minor
				patch = $patch
			}
		}
	}
}

and the code it replaces looks like this:

$pkg = Parse-PackageItem -Package $_
if ($pkg -and -not $package.ContainsKey($pkg.id)) {
	$pkg
}

Note that function is fault tolerant, and does not create the package object if the label does not match the desired pattern.

Summary

Quite a lot of work has gone into this before even starting on the “real” work. It is worth it, because the resulting code is simpler. It is simpler because a large block of tangled code has been simplified by removing a cohesive part of the code that did not belong there. The extracted function is exactly the part of the tool that I want to change.

The testing script will justify its existence when I have added tests for edge cases for what it already does, and for the new functionality.

Having made a start, I will continue, adding tests for the rest of the tool’s functionality.

The script could have been good enough. They often are. It is when things start getting complicated that this approach is justified. The script that I had partially developed was a working prototype of some of what I want to achieve.

Parameterizing Scripts with PowerShell Data Files

In my previous blog I introduced the Get-Blacklist function which turned a string array into a hashtable for quick comparisons:

function Get-Blacklist {
	param(
		[string[]]$Blacklist
	)
	$blacklistHash = @{}
	$Blacklist | % {
		$blacklistHash[$_] = $true
	}
	$blacklistHash
}

and invoked it thus:

. .\Scripts\Get-Blacklist.ps1
@blacklist = Get-Blacklist -Blacklist 'Powershell','Sandbox','sqlccJamie','Graphics','CentinelTest','Backoffice Service Portal'

As a rule I prefer not to embed literal constants like this in my scripts, especially as they go into production. Values like these are prone to change much more than the pure code, and will likely be maintained in future by administrators with little PowerShell experience.

There are many configuration file formats that PowerShell handles well, but the PowerShell Data File format is particularly useful. It is the format used for module manifests, i.e. .psd1 files. Here is one for the blacklist:

@{
    # Team Blacklist - Teams not to be sychronized between NuGet servers
    TeamBlacklist = @('Powershell','Sandbox','sqlccJamie','Graphics','CentinelTest','Backoffice Service Portal')
}

I’ll be adding more than one of these, hence the longer name. We need to import it before we can use it:

$par = Import-PowerShellDataFile -Path "$PSScriptRoot\SyncPackages.psd1"

# Get Azure DevOps project and repos
$repoNames = Get-RepoNames -Blacklist $par.TeamBlacklist

There we go!

I made use of the built-in variable $PSScriptRoot pointing to the script being executed. It was introduced in PowerShell 3.0.

Building a script module

Like I said in the previous blog, I like to write short functions and keep each in its own file in source control. The problem is then to make these functions accessible. The usual approach is to dot-source them. I’m not particularly wild about the idea, but let’s have a look at that first.

Sample code

This example is taken from a utility to synchronize two NuGet servers. Only some of the packages will be synchronized. I use a blacklist to filter the desired packages.

I usually start a project like this with a monolithic script. That could be sufficient, but in this instance I wanted to reuse the code identifying the packages on each server. Once I had that working well enough against one server, I refactored the code into functions. These I saved in individual scripts.

The easy way: use dot-sourcing

Here is a simple example of how it could be done with dot-sourcing. First a code block.

@blacklist = @{}
@blacklist['Powershell'] = $true
@blacklist['Sandbox'] = $true
@blacklist['sqlccJamie'] = $true
@blacklist['Graphics'] = $true
@blacklist['CentinelTest'] = $true
@blacklist['Backoffice Service Portal'] = $true

This is the list of projects to be ignored, and I wanted to use a hash for the purpose. The first refactoring is to extract the names and use a loop:

@blacklist = @{}
'Powershell','Sandbox','sqlccJamie','Graphics','CentinelTest','Backoffice Service Portal' | % {
	@blacklist[$_] = $true
}

I now refactored this into a function:

function Get-Blacklist {
	param(
		[string[]]$Blacklist
	)
	$blacklistHash = @{}
	$Blacklist | % {
		$blacklistHash[$_] = $true
	}
	$blacklistHash
}

which would be invoked thus:

. .\Scripts\Get-Blacklist.ps1
@blacklist = Get-Blacklist -Blacklist 'Powershell','Sandbox','sqlccJamie','Graphics','CentinelTest','Backoffice Service Portal'

I like doing the dot-sourcing near the first usage of the function while I am still busy refactoring the code. The dot sourcing line can then be moved together with the function to a new location. This can be done differently later. I am putting the scripts into a separate folder so that they can be manipulated as a group, for example by dot-sourcing them all:

ls .\Scripts\*.ps1 -Recurse | % {
    $path = $_.FullName
    . %path
}

What is not to like about dot-sourcing

The problem for me comes with deploying the code. The fewer files and folders the better. The simplest deployment is as of single script that contains all the functions as well as the top level code – a single .ps1 file. When there are multiple scripts to be deployed, it becomes more convenient to package shared functions in a script module – a single .psm1 file. To be more professional, this should be accompanied by a manifest module with metadata like descriptions, versions and visibility rules – a .psd1 file.

The visibility rules are another thing to dislike about dot-sourcing: everything is visible.

Packaging the module

My script is SyncPackages.ps1. My plan is to deploy it with the SyncNuGetRepos script module. Everything happens in the SyncNuGetRepos folder, and I put my script there. I create an empty file called SyncNuGetRepos.psm1. Later I can put other stuff there. Also in the folder I have the Scripts subfolder with my functions.

Now I create my Build.ps1 script that creates the module. The new module is put in the bin subfolder. It is all very simple:

[string]$text = gc .\SyncNuGetRepos.psm1 | Out-String
ls .\Scripts\*.ps1 -Recurse | % {
	$path = $_.FullName
	$fn = gc $path | Out-String
	$text = "$text
$fn
"
}
if (-not (Test-Path .\bin)) {
	mkdir .\bin | Out-Null
}

$text | Out-File .\bin\SyncNuGetRepos.psm1 -Encoding utf8

We need to import the module when running the script, so we add the code to the beginning of SyncPackages.ps1:

if (Get-Module SyncNuGetRepos -All) {
    Remove-Module SyncNuGetRepos
}
if (Test-Path .\bin\SyncNuGetRepos.psm1) {
    Import-Module .\bin\SyncNuGetRepos.psm1
} else {
    Import-Module .\SyncNuGetRepos.psm1
}

The idea is to use the module we have just built from the bin folder. When we deploy the thing, the script and the module will be in the same folder. Of course we need to ensure that the context is also pointing to the folder.