Faster build pipeline using cache task in Azure DevOps

Av Ove Lartelius 16 juni 2020 Lästid: 2 minuter

Speed up your YAML build pipeline in Azure DevOps with the Cache task.

Sometimes the builds are taking a long time. Often the NuGet restore/npm install takes a couple of minutes to download all packages that you use in your solution. One way to speed up that process is to use the “new” Cache task in Azure DevOps.

Pipeline caching (https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops)

It took me a long time to understand how you actually used this task (yes I know, I´m kind of solve…). But when I found the example in jraps20 build.yml(https://gist.github.com/jraps20/ea29de8b45493dcfa0a1ec5b0af82337) I finally understood how it works and how to do this.

So, what you could do is add 2 tasks. The first task is a PowerShell task creates a hash of the content in packages.config. That means that as long you don´t change and version of NuGet packages, and/or adding/removing NuGet packages, the hash will be the same. That will decide if the cached packages folder should be used or if it should download Nuget packages into the packages folder.

The second task is actual cache task that use the hash in the cachekey to figure out if there is any cached “packages” folder to reuse.

Note: The Cache task (Cache files between runs) is only supported when you are using YAML.

Before I implemented the caching my YAML file started like this:

trigger:

batch: true

branches:

include:

- develop

variables:

- group: DXP-variables

stages:

- stage: Build

jobs:

- job: DevInte

pool:

name: Azure Pipelines

vmImage: 'windows-latest'

demands:

- npm

- msbuild

- visualstudio

- vstest

variables:

solution: '**/*.sln'

buildPlatform: 'Any CPU'

buildConfiguration: 'Release'

steps:

- task: NuGetToolInstaller@0

displayName: 'Use NuGet 5.*'

inputs:

versionSpec: '5.*'

- task: NuGetCommand@2

displayName: 'NuGet restore'

inputs:

command: 'restore'

restoreSolution: '$(Solution)'

feedsToUse: 'config'

nugetConfigPath: 'NuGet.config'

And after implemented the caching tasks it looks like this:

trigger:

batch: true

branches:

include:

- develop

variables:

- group: DXP-variables

stages:

- stage: Build

jobs:

- job: DevInte

pool:

name: Azure Pipelines

vmImage: 'windows-latest'

demands:

- npm

- msbuild

- visualstudio

- vstest

variables:

solution: '**/*.sln'

buildPlatform: 'Any CPU'

buildConfiguration: 'Release'

steps:

- task: PowerShell@2

displayName: 'Calculate and save packages.config hash'

inputs:

targetType: 'inline'

script: |

# generates a hash of all packages.config and saves each on a single line on 'hash.txt'

Get-FileHash -Algorithm MD5 -Path (Get-ChildItem packages.config -Recurse) >> hash.txt

Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\hash.txt"

workingDirectory: '$(System.DefaultWorkingDirectory)'

- task: CacheBeta@0 # speed up builds by caching packages folder

inputs:

key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\hash.txt # hash map generated in previous step

path: $(Build.SourcesDirectory)\packages

cacheHitVar: 'nuget_cache_hit' # variable that can be checked to see if it was successful

displayName: Cache nuget packages

continueOnError: true

- task: NuGetToolInstaller@0

displayName: 'Use NuGet 5.*'

inputs:

versionSpec: '5.*'

- task: NuGetCommand@2

displayName: 'NuGet restore'

inputs:

command: 'restore'

restoreSolution: '$(Solution)'

feedsToUse: 'config'

nugetConfigPath: 'NuGet.config'

Result

You never really know how much this will fast up you build. But an example from my last project made the build pipeline become around 1.5 minutes faster. (If I have no packages.config changes of course). And of course, it depends on how many NuGet packages you use etc.

Example before:

In our case in this project we use around 115 NuGet packages and the folder weight around 400Mb. It takes around 2 minutes (if lucky) to download these packages.

Example after:

And with the cache task we managed to get the same task to run in 30 seconds.

NPM

I have not given you any example of how to do this with NPM packages. Here is an example of YAML that cache both NuGet and NPM packages.

trigger:

batch: true

branches:

include:

- develop

variables:

- group: DXP-variables

- name: webProjectDirectory

value: '$(Build.SourcesDirectory)\Customer.Web'

stages:

- stage: Build

jobs:

- job: DevInte

pool:

name: Azure Pipelines

vmImage: 'windows-latest'

demands:

- npm

- msbuild

- visualstudio

- vstest

variables:

solution: '**/*.sln'

buildPlatform: 'Any CPU'

buildConfiguration: 'Release'

steps:

- task: PowerShell@2

displayName: 'Calculate and save packages.config hash'

inputs:

targetType: 'inline'

script: |

          # generates a hash of all packages.config and saves each on a single line on 'packagesconfighash.txt'

          Get-FileHash -Algorithm MD5 -Path (Get-ChildItem packages.config -Recurse) >> packagesconfighash.txt

Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\packagesconfighash.txt"

workingDirectory: '$(System.DefaultWorkingDirectory)'

- task: CacheBeta@0 # speed up builds by caching packages folder

inputs:

        key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\packagesconfighash.txt # hash map generated in previous step

path: $(Build.SourcesDirectory)\packages

cacheHitVar: 'nuget_cache_hit' # variable that can be checked to see if it was successful

displayName: Cache nuget packages

continueOnError: true

- task: NuGetToolInstaller@0

displayName: 'Use NuGet 5.*'

inputs:

versionSpec: '5.*'

- task: NuGetCommand@2

displayName: 'NuGet restore'

inputs:

command: 'restore'

restoreSolution: '$(Solution)'

feedsToUse: 'config'

nugetConfigPath: 'NuGet.config'

- task: PowerShell@2

displayName: 'Calculate and save package.json hash'

inputs:

targetType: 'inline'

script: |

          # generates a hash of all package.json and saves each on a single line on 'packagejsonhash.txt'

          Get-FileHash -Algorithm MD5 -Path (Get-ChildItem package.json -Recurse) >> packagejsonhash.txt

Write-Host "Hash File saved to $(System.DefaultWorkingDirectory)\packagejsonhash.txt"

workingDirectory: '$(System.DefaultWorkingDirectory)'

- task: CacheBeta@0 # speed up builds by caching node_modules folder

inputs:

        key: nuget|1|$(Agent.OS)|$(Build.SourcesDirectory)\packagejsonhash.txt # hash map generated in previous step

path: $(WebProjectDirectory)\node_modules

cacheHitVar: 'npm_cache_hit' # variable that can be checked to see if it was successful

displayName: Cache npm packages

continueOnError: true

- task: Npm@1

displayName: 'NPM Set progress false (speedup)'

inputs:

command: custom

verbose: false

customCommand: 'set progress=false'

- task: Npm@1

displayName: 'npm install'

inputs:

workingDir: '$(WebProjectDirectory)'

verbose: false

- task: Npm@1

displayName: 'npm run build'

inputs:

command: custom

workingDir: '$(WebProjectDirectory)'

verbose: false

customCommand: 'run build'

I hope this helps!

Vi vill gärna höra vad du tycker om inlägget

Ove Lartelius

Backend-utvecklare | OMVP

Läs alla blogginlägg av Ove Lartelius