Azure

Azure DevOps Multi-Stage Pipelines: Require Stage Approval

In last week’s post, we covered taking our existing build pipeline and making it a multi-stage Pipeline with a build stage and a deploy stage. This week we are going to add another stage to our pipeline for production. Since we don’t want the production stage deployed before it has been through QA we will need to hold the stage until it is verified ready, which is what this post is going to be about. If you haven’t read last week’s post, Azure DevOps Pipelines: Multi-Stage Pipelines, you might want to start there before reading the rest of this post if you are new to multi-stage pipelines.

 

Add an Environment

In order to require approval on a stage is to associate it with and environment and add the approval requirement to the environment. In Azure DevOps under Pipelines select Environments and then click the Create environment button.

On the New environment dialog fill in a Name. If you had actual resources associated with the environment they can be added to provide traceability, but in this example, we are going to stick with the None option.

Require Approval for an Environment

Now that the resource has been created on its details page we can use the three dots to open the menu and click Approvals and checks.

On the next screen click the +button in the upper right corner and then from the lists of check select Approvals and then click Next. As you can see from the partial list in the screenshot the range of check available for approvals is massive.

The next dialog is used to select the uses or groups who should be able to perform approves for the environment. When your approves are set click Create to finish.

Use an Environment in a Pipeline

If you are following along from a previous post not that the Deploy stage has been renamed to QA to make the Pipeline results clearer.

Before:
- stage: Deploy
  jobs:
  - job: Deploy
    steps:
      - script: echo Fake deploying code

After:
- stage: QA
  jobs:
  - job: DeployQa
    steps:
      - script: echo Deploying to QA

Now we are going to add a new stage for our production environment. Notice that instead of a normal job we are using a deployment job which enables us to specify our desired environment. Deployment jobs have a ton more features than we are using so make sure and check out the docs to see what other options are available.

- stage: Production
  jobs:
  - deployment: DeployProduction
    environment: 'Production'
    strategy:
     runOnce:
       deploy:
         steps:
          - script: echo Deploying to Production

Save and run the Pipeline and we will look at how this new state presents differently than the Build and QA stages.

Pipeline Results

As you can see in the following screenshot the results of the Pipeline run have has a section notifying that the Production stage can’t run until it has been reviewed. Also, notice in the Stages section that the Production stage shows a Waiting status.

Clicking the Review button will show a dialog that will allow you to approve or reject the stage that is waiting.

Wrapping Up

The extra layer of options provided environments enables most of the scenarios that I missed when I first started playing with multi-stage Pipelines. Having the build and release steps for an app in source control and the added ability to vary them by branch makes this worth it even if I do have to deal with more YAML.

Azure DevOps Multi-Stage Pipelines: Require Stage Approval Read More »

Azure DevOps Pipelines: Multi-Stage Pipelines

The last couple of posts have been dealing with Release managed from the Releases area under Azure Pipelines. This week we are going to take what we were doing in that separate area of Azure DevOps and instead make it part of the YAML that currently builds our application. If you need some background on how the project got to this point check out the following posts.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML
Azure DevOps Pipelines: Naming and Tagging
Azure DevOps Pipelines: Manual Tagging
Azure DevOps Pipelines: Depends On with Conditionals in YAML
Azure DevOps Pipelines: PowerShell Task
Azure DevOps Releases: Auto Create New Release After Pipeline Build
Azure DevOps Releases: Auto Create Release with Pull Requests

Recap

The current setup we have uses a YAML based Azure Pipeline to build a couple of ASP.NET Core web applications. Then on the Release side, we have basically a dummy release that doesn’t actually do anything but served as a demo of how to configure a continuous deployment type release. The following is the current YAML for our Pipeline for reference.

name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r)

resources:      
  repositories: 
  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: 'Get-ChildItem -Path Env:\'

  - template: buildCoreWebProject.yml@Shared
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

- job: WebApp2
  displayName: 'Build WebApp2'
  condition: and(succeeded(), eq(variables['BuildWebApp2'], 'true'))
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2
      
- job: DependentJob
  displayName: 'Build Dependent Job'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2

  steps:
  - template: buildCoreWebProject.yml@Shared
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1Again

- job: TagSources
  displayName: 'Tag Sources'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2
  - DependentJob
  condition: |
    and
    (
      eq(dependencies.WebApp1.result, 'Succeeded'),
      in(dependencies.WebApp2.result, 'Succeeded', 'Skipped'),
      in(dependencies.DependentJob.result, 'Succeeded', 'Skipped')
    )
 
  steps:
  - checkout: self
    persistCredentials: true
    clean: true
    fetchDepth: 1

  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: |
        $env:GIT_REDIRECT_STDERR` = '2>&1'
        $tag = "manual_$(Build.BuildNumber)".replace(' ', '_')
        git tag $tag
        Write-Host "Successfully created tag $tag" 

        git push --tags
         Write-Host "Successfully pushed tag $tag"     

      failOnStderr: false

The above setup works great, but in April of this year, Azure Pipelines got the concept of multi-stage Pipelines which gives us the ability to manage the Release side of things in the same YAML as our builds and allows releases to be source controlled and different per branch in the same way that builds in YAML can be.

Simplified Build YAML

The above is the full YAML for our sample builds, which is a lot of code. The following is a paired down version that we will be using for the rest of this post that only builds WebApp1 and should help the changes stand out.

name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r)

resources:      
  repositories: 
  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: 'Get-ChildItem -Path Env:\'

  - template: buildCoreWebProject.yml@Shared
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

Adding Stages

Stages are an extra layer of grouping that help divide a Pipeline similar to how jobs work except at a higher level. Jobs are a group of Steps, but Stages are a group of Jobs. In the following YAML, you can see that our existing jobs have been grouped under a Build stage and a new Release stage has been added.

name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r)

resources:      
  repositories: 
  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

variables:
  buildConfiguration: 'Release'

stages:
- stage: Build
  jobs:
  - job: WebApp1
    displayName: 'Build WebApp1'
    pool:
      vmImage: 'ubuntu-latest'

    steps:
    - task: PowerShell@2
      inputs:
        targetType: 'inline'
        script: 'Get-ChildItem -Path Env:\'

    - template: buildCoreWebProject.yml@Shared
      parameters:
        buildConFiguration: $(buildConfiguration)
        project: WebApp1.csproj
        artifactName: WebApp1

- stage: Deploy
  jobs:
  - job: Deploy
    steps:
      - script: echo Fake deploying code

When adding stages watch your whitespace it is easy to miss spacing in your existing code when wrapping them in stages.

Results

After running the Pipeline with the above changes you will see on the Pipeline’s summary page that it will display the results of each stage.

In the detailed view of a specific Pipeline run, there will now be a Stages tab that shows the results by stage. If you hit the expander on a stage it will also give you an option to rerun a stage if you ever have that need.

Wrapping Up

Hopefully, this will help you get a jump start on setting up your own multi-stage Pipelines. While I’m still not in love with YAML it is nice to have builds and releases in source control with the ability to vary by branch when you have the need. This setup works great if you want all your stages to run every time. A follow-up post will look at how to make a stage that requires approval.

Azure DevOps Pipelines: Multi-Stage Pipelines Read More »

Azure DevOps Releases: Auto Create Release with Pull Requests

Last week we covered auto-creating Release when a build completes. This week we are going to cover how to create a release when a build from a pull request completes. This setup would be helpful for verification of changes before the actual make it into a releasable branch. The following posts will help you catch up if you’re new to the series.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML
Azure DevOps Pipelines: Naming and Tagging
Azure DevOps Pipelines: Manual Tagging
Azure DevOps Pipelines: Depends On with Conditionals in YAML
Azure DevOps Pipelines: PowerShell Task
Azure DevOps Releases: Auto Create New Release After Pipeline Build

Build Validation Branch Policy

Before we can have a Release created with a pull request we have to make sure that the pull request process does a build. I’m going to review how to do this quickly, for more info see my Branch Policies post. To do this we are going to head over to the Repos section of Azure DevOps. In the Branches section on the branch we want a build on a pull request for select the three dots and then click Branch policies.

On the repo settings page scroll down to the Build Validation section and click the + button to add a build to the pull request process.

The Add build policy dialog has a few options, but we are taking the defaults. Do note that if you have multiple Build pipelines that you make sure and adjust that option to the correct build. Click Save when you’re done.

Release Changes to allow Pull Request Trigger

Based on how we set up our Release to trigger when a build complete you might think that using the same build to validate a Pull Request would automatically trigger a new Release, but that isn’t the case. In most cases, this is actually a good thing since you wouldn’t want to deploy change before they have been reviewed. In the case we are trying to cover our release is to a QA environment so the requested changes can be verified before they make it into a releasable branch.

To enable a Release to be created from a pull request we need to head over to the Pipeline > Release area in Azure DevOps. Once there with the release in question selected click the Edit button.

In the Artifacts, section click the lightning bolt to edit the continuous deployment triggers.

Near the middle of the dialog, we want to Enable the Pull request trigger. Doing this will also require you to enter Target Branch Filters which are the branches that will be allowed to trigger a release when they are a target of a pull request.

Next, we need to enable our sample stage to be deployed for releases based on pull requests. Click the lightning bolt on the left side of the stage to edit the pre-deployment conditions.

On the dialog that shows Enable the Pull request deployment setting.

After closing the dialog make sure and Save the release.

Results

To show the results I created a new branch with a small change and created a PR into the master branch. From the Pull Request, we can click the View all checks button to see the status of the required build.

On the Checks dialog, you can see at the bottom that our sample release ran successfully.

If you click on the release you can see in the artifacts section that the files being used are from the pull request’s merge branch, not the branch being PRed or the target branch.

Wrapping Up

Building and releasing on a pull request opens up a lot of options, especially around making sure your code is verified before making it into a release branch.

Azure DevOps Releases: Auto Create Release with Pull Requests Read More »

Azure DevOps Releases: Auto Create New Release After Pipeline Build

I hit a snag with my planned post for this weeks and decided to hope back over to Azure DevOps and show how to create a Release and have that Release be triggered anytime a build Pipeline complete successfully. This post is going to be using the same Azure DevOps Project as all the previous posts in this series which are linked below.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML
Azure DevOps Pipelines: Naming and Tagging
Azure DevOps Pipelines: Manual Tagging
Azure DevOps Pipelines: Depends On with Conditionals in YAML
Azure DevOps Pipelines: PowerShell Task

Creating a Release

In Azure DevOps to create a release go to Pipelines and then Releases. Since this is our project’s first Release we have a New pipeline button to click to start the creation process.

The New pipeline button will start the creation process by showing a Select a template dialog. Since our release isn’t really going to do anything yet we are going to click the Empty job option.

Next, we want to add the artifacts from our build Pipeline to this Release. Click Add an artifact to start the process.

The Add an artifact dialog will show which allows us a lot of options on the source of the artifact. For this setup, we are going to use a Source type of Build since our artifacts are the result of an Azure DevOps Build. The next option we need to select is Source which is where we select which Azure DevOps Build Pipeline we want to use artifacts from. In this sample case, we only have a single option. Once the Source is selected a couple more options will show up, but we are taking the default values for those and just clicking the Add button.

Now that we have the basic release setup click the Save button. There will be a prompt for a folder. Either enter a folder name or leave blank to keep the release in the root and then click OK to complete the save.

Auto Create After Pipeline Build

Now that we have our basic Release Pipeline we are going to set up a continuous deployment trigger on the artifact so that any time a new build is completed. In the Artifacts area click the Lighting Bolt on the artifact the trigger should be on, we only have one option in our sample.

A dialog will show different trigger options. We are going to Enable the Continuous deployment trigger. Notice that when the trigger is enabled there is an option to only trigger for specific branches. I have used this option pretty often to only trigger a release when a build is done on master. When done click the X to close the dialog and then Save the release. I also renamed the release to Web App 1 & 2 to make it clearer what the release is doing.

Now pop over to the Pipelines and run a build. After the build is complete go back to the Release and you will see that a new release was triggered, and deployed in this case based on how the rest of the Release was set up.

Wrapping Up

In this post, we created our first Release, which doesn’t actually do anything, and configured it to deploy automatically on the build of our application. I use a similar setup to automatically deploy to a QA environment. This setup can open up a ton of scenarios.

Azure DevOps Releases: Auto Create New Release After Pipeline Build Read More »

Azure DevOps Pipelines: PowerShell Task

This is going to be a quick post that shows the use of the PowerShell task in a Pipeline. Nothing in the post is really specific to the Azure DevOps Project we have been using over the last few weeks, but just in case you’re totally new to Azure DevOps and/or this series you can use the following posts to get started.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML
Azure DevOps Pipelines: Naming and Tagging
Azure DevOps Pipelines: Manual Tagging
Azure DevOps Pipelines: Depends On with Conditionals in YAML

PowerShell Task

The PowerShell task will allow you to do pretty much anything. If there isn’t an existing DevOps task that fits your needs more than likely you can find a way to use the PowerShell task to accomplish what you need within the context of the computer the task is running on and even external computers that depending on your networking and security setup. The following is a sample task I added to a Pipeline that will output all the environment variables to the logs. This is an inline script, but you can also run scripts from files. Also, note that this works on both Windows and Linux agents.

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: 'Get-ChildItem -Path Env:\'

While this script isn’t super useful for a production Pipeline I often use it when setting up a Pipeline to get a good feel for what is available variable wise. Also, keep in mind that depending on the trigger of the run these variables can be different. For example, if a run was triggered by a pull request you will have a number of pull request related variables. The following is the output of this command on my test project that was triggered via a pull request and therefore contains a bunch of SYSTEM_PULLREQUEST_x variables with information about the pull request. The agent was running Linux.

Name                           Value
----                           -----
AGENT_ACCEPTTEEEULA            True
AGENT_BUILDDIRECTORY           /home/vsts/work/1
AGENT_DISABLELOGPLUGIN_TESTFI… true
AGENT_DISABLELOGPLUGIN_TESTRE… true
AGENT_HOMEDIRECTORY            /home/vsts/agents/2.165.2
AGENT_ID                       9
AGENT_JOBNAME                  Build WebApp1
AGENT_JOBSTATUS                Succeeded
AGENT_MACHINENAME              fv-az563
AGENT_NAME                     Hosted Agent
AGENT_OS                       Linux
AGENT_OSARCHITECTURE           X64
AGENT_READONLYVARIABLES        true
AGENT_RETAINDEFAULTENCODING    false
AGENT_ROOTDIRECTORY            /home/vsts/work
AGENT_TEMPDIRECTORY            /home/vsts/work/_temp
AGENT_TOOLSDIRECTORY           /opt/hostedtoolcache
AGENT_VERSION                  2.165.2
AGENT_WORKFOLDER               /home/vsts/work
agent.jobstatus                Succeeded
ANDROID_HOME                   /usr/local/lib/android/sdk
ANDROID_SDK_ROOT               /usr/local/lib/android/sdk
ANT_HOME                       /usr/share/ant
AZURE_EXTENSION_DIR            /opt/az/azcliextensions
AZURE_HTTP_USER_AGENT          VSTS_08ccc6b2-4e5e-4621-8f5b-3fe0de2efa22_build…
BOOST_ROOT_1_69_0              /usr/local/share/boost/1.69.0
BOOST_ROOT_1_72_0              /usr/local/share/boost/1.72.0
BUILD_ARTIFACTSTAGINGDIRECTORY /home/vsts/work/1/a
BUILD_BINARIESDIRECTORY        /home/vsts/work/1/b
BUILD_BUILDID                  73
BUILD_BUILDNUMBER              merge_20200422.1
BUILD_BUILDURI                 vstfs:///Build/Build/73
BUILD_CONTAINERID              3453972
BUILD_DEFINITIONNAME           Playground
BUILD_DEFINITIONVERSION        4
BUILD_QUEUEDBY                 Microsoft.VisualStudio.Services.TFS
BUILD_QUEUEDBYID               00000002-0000-8888-8000-000000000000
BUILD_REASON                   PullRequest
BUILD_REPOSITORY_CLEAN         False
BUILD_REPOSITORY_GIT_SUBMODUL… False
BUILD_REPOSITORY_ID            ff7a6325-1129-42e3-b095-6a39ef6a6bd3
BUILD_REPOSITORY_LOCALPATH     /home/vsts/work/1/s
BUILD_REPOSITORY_NAME          Playground
BUILD_REPOSITORY_PROVIDER      TfsGit
BUILD_REPOSITORY_URI           https://[email protected]/ericlanders…
BUILD_REQUESTEDFOR             Eric Anderson
BUILD_REQUESTEDFOREMAIL        [email protected]
BUILD_REQUESTEDFORID           45247cb1-8f49-4c03-a4c5-b03ac3286c99
BUILD_SOURCEBRANCH             refs/pull/10/merge
BUILD_SOURCEBRANCHNAME         merge
BUILD_SOURCESDIRECTORY         /home/vsts/work/1/s
BUILD_SOURCEVERSION            3e2b77c27f31a4c729a5f195b49d2e108500399d
BUILD_SOURCEVERSIONAUTHOR      Eric Anderson
BUILD_SOURCEVERSIONMESSAGE     Merge pull request 10 from docChanges into mast…
BUILD_STAGINGDIRECTORY         /home/vsts/work/1/a
BUILDCONFIGURATION             Release
BUILDWEBAPP2                   false
CHROME_BIN                     /usr/bin/google-chrome
CHROMEWEBDRIVER                /usr/local/share/chrome_driver
COMMON_TESTRESULTSDIRECTORY    /home/vsts/work/1/TestResults
CONDA                          /usr/share/miniconda
DEBIAN_FRONTEND                noninteractive
DOTNET_SKIP_FIRST_TIME_EXPERI… 1
ENDPOINT_URL_SYSTEMVSSCONNECT… https://dev.azure.com/ericlanderson/
GECKOWEBDRIVER                 /usr/local/share/gecko_driver
GIT_TERMINAL_PROMPT            0
GOROOT                         /usr/local/go1.14
GOROOT_1_11_X64                /usr/local/go1.11
GOROOT_1_12_X64                /usr/local/go1.12
GOROOT_1_13_X64                /usr/local/go1.13
GOROOT_1_14_X64                /usr/local/go1.14
GRADLE_HOME                    /usr/share/gradle
HOME                           /home/vsts
ImageOS                        ubuntu18
ImageVersion                   20200406.2
INPUT_ARGUMENTS                
INVOCATION_ID                  3e6abb812a484ab39fadc9e8721258ee
JAVA_HOME                      /usr/lib/jvm/zulu-8-azure-amd64
JAVA_HOME_11_X64               /usr/lib/jvm/zulu-11-azure-amd64
JAVA_HOME_12_X64               /usr/lib/jvm/zulu-12-azure-amd64
JAVA_HOME_7_X64                /usr/lib/jvm/zulu-7-azure-amd64
JAVA_HOME_8_X64                /usr/lib/jvm/zulu-8-azure-amd64
JOURNAL_STREAM                 9:30085
LANG                           C.UTF-8
LEIN_HOME                      /usr/local/lib/lein
LEIN_JAR                       /usr/local/lib/lein/self-installs/leiningen-2.9…
M2_HOME                        /usr/share/apache-maven-3.6.3
MSDEPLOY_HTTP_USER_AGENT       VSTS_08ccc6b2-4e5e-4621-8f5b-3fe0de2efa22_build…
PATH                           /opt/microsoft/powershell/7:/usr/share/rust/.ca…
PIPELINE_WORKSPACE             /home/vsts/work/1
POWERSHELL_DISTRIBUTION_CHANN… Azure-DevOps-ubuntu18
PSModulePath                   /home/vsts/.local/share/powershell/Modules:/usr…
RUNNER_TOOLSDIRECTORY          /opt/hostedtoolcache
SELENIUM_JAR_PATH              /usr/share/java/selenium-server-standalone.jar
SWIFT_PATH                     /usr/share/swift/usr/bin
SYSTEM                         build
SYSTEM_ARTIFACTSDIRECTORY      /home/vsts/work/1/a
SYSTEM_COLLECTIONID            08ccc6b2-4e5e-4621-8f5b-3fe0de2efa22
SYSTEM_COLLECTIONURI           https://dev.azure.com/ericlanderson/
SYSTEM_CULTURE                 en-US
SYSTEM_DEFAULTWORKINGDIRECTORY /home/vsts/work/1/s
SYSTEM_DEFINITIONID            5
SYSTEM_DEFINITIONNAME          Playground
SYSTEM_ENABLEACCESSTOKEN       SecretVariable
SYSTEM_HOSTTYPE                build
SYSTEM_ISSCHEDULED             False
SYSTEM_JOBATTEMPT              1
SYSTEM_JOBDISPLAYNAME          Build WebApp1
SYSTEM_JOBID                   98395c9e-7365-5c3f-03de-ec42b09a8a98
SYSTEM_JOBIDENTIFIER           WebApp1.__default
SYSTEM_JOBNAME                 __default
SYSTEM_JOBPARALLELISMTAG       Private
SYSTEM_JOBPOSITIONINPHASE      1
SYSTEM_PHASEATTEMPT            1
SYSTEM_PHASEDISPLAYNAME        Build WebApp1
SYSTEM_PHASEID                 a142d6c6-ff80-5cff-8292-5044e2c5b0ef
SYSTEM_PHASENAME               WebApp1
SYSTEM_PIPELINESTARTTIME       2020-04-22 06:11:44-05:00
SYSTEM_PLANID                  726fda14-a3a2-45b1-b745-bef8cf17bdaa
SYSTEM_PULLREQUEST_ISFORK      False
SYSTEM_PULLREQUEST_PULLREQUES… 10
SYSTEM_PULLREQUEST_PULLREQUES… 1
SYSTEM_PULLREQUEST_SOURCEBRAN… refs/heads/docChanges
SYSTEM_PULLREQUEST_SOURCECOMM… ba11cb768bc75ae65ff6b7ac6afb8a2950063f07
SYSTEM_PULLREQUEST_SOURCEREPO… https://[email protected]/ericlanders…
SYSTEM_PULLREQUEST_TARGETBRAN… refs/heads/master
SYSTEM_SERVERTYPE              Hosted
SYSTEM_STAGEATTEMPT            1
SYSTEM_STAGEDISPLAYNAME        __default
SYSTEM_STAGEID                 96ac2280-8cb4-5df5-99de-dd2da759617d
SYSTEM_STAGENAME               __default
SYSTEM_TASKDEFINITIONSURI      https://dev.azure.com/ericlanderson/
SYSTEM_TASKDISPLAYNAME         PowerShell
SYSTEM_TASKINSTANCEID          6417fa85-e8cf-55f9-817e-d698bd79d6f7
SYSTEM_TASKINSTANCENAME        PowerShell
SYSTEM_TEAMFOUNDATIONCOLLECTI… https://dev.azure.com/ericlanderson/
SYSTEM_TEAMFOUNDATIONSERVERURI https://dev.azure.com/ericlanderson/
SYSTEM_TEAMPROJECT             Playground
SYSTEM_TEAMPROJECTID           7550ca2f-9ffe-45b7-abd5-c4e92a4a5f4e
SYSTEM_TIMELINEID              726fda14-a3a2-45b1-b745-bef8cf17bdaa
SYSTEM_TOTALJOBSINPHASE        1
SYSTEM_WORKFOLDER              /home/vsts/work
TASK_DISPLAYNAME               PowerShell
TF_BUILD                       True
USER                           vsts
VCPKG_INSTALLATION_ROOT        /usr/local/share/vcpkg
VSTS_AGENT_PERFLOG             /home/vsts/perflog
VSTS_PROCESS_LOOKUP_ID         vsts_54420f58-c41f-4a43-8ce8-bbbac5023620

I don’t know about you but being able to see what paths the built-in path variables actually map to helps me a lot especially when files need to be moved around.

Wrapping Up

As stated above you can do just about anything with the PowerShell task. I have used it for everything from reading a JSON file to building a VM for QA. If you hadn’t used this task before I hope this post helped you get started and opened your eyes to the huge range of things you can do with the PowerShell task.

Azure DevOps Pipelines: PowerShell Task Read More »

Azure DevOps Repos: Bypass Branch Policies

Last week we covered adding branch policies to a branch in an Azure DevOps Repo and this week we are going to deal with what happens when you need to break the policies you set up for some reason.

Setting Bypass Security

There are a couple of ways to set up bypassing depending on how broadly you want to give someone rights to bypass. We are going to start with the narrower option which is allowing bypass for an individual branch (sadly security options are currently available at the folder level like policies are). Starting from the list of branches for your repo mouse over the branch you want to set security for and click the three dots for the menu and select Branch security.

On the dialog that shows find the user, you want to change for security for, Eric Anderson in this example. After selecting a user their specific setting will be loaded to the right. On Bypass policies when completing pull requests change the option to Allow.

The second option for setting bypass security is at the repo level. From your Project settings under Repos select Repositories and then fine the Branches node under the project you want to set the policy for.

From here it is the same as the branch level. Find the user, you want to change for security for and set Bypass policies when completing pull requests to Allow.

Bypassing Policies on a Pull Request

Now that we have our security setup we are going to walk through what the bypass process looks like. Here we have a pull request that is missing approval by a reviewer.

Now let us say we don’t have a reviewer available for some reason and we need to complete this PR without review. Use the dropdown on the Set auto-complete button and click Complete.

When the PR completion dialog show you will notice a section at the top with a red background that lists out the policies that haven’t been met. With your new-found security, you will also have a section for Policy override options. To proceed and bypass the policies check the Override branch policies and enable merge checkbox, enter your reason for overriding, and click the Override and complete button.

Do note that the fact a PR was overridden is visible on the list of completed PRs and the reason will show when mousing over the bypassed indicator as well as in the details of the PR.

Wrapping Up

Branch policies are great and will help you make sure the code that makes it in your branches are high quality and don’t break your builds. Hopefully, you won’t need to bypass your policies often, but now you know-how without having to temporarily remove the policies or getting people used to blindly approving changes.

Azure DevOps Repos: Bypass Branch Policies Read More »

Azure DevOps Repos: Branch Policies

For the last few weeks I have been doing a series of posts about Azure DevOps Pipelines and I hit a post I wanted to do that didn’t make sense without introducing the Branch Policies feature of Azure Repos. This post is going to assume you already have an Azure DevOps Project with some code in it. If not you can check out my post on Getting Started with Azure DevOps.

Repo Introduction

The repo used here is the same one used in the Pipelines posts linked above and contains two .NET Core 3.1 web applications. The repo also contains three branches (master, releases/1.0, and releases/1.1).

It is worth noting that putting a forward slash in a branch name displays as a folder in the UI as you can see with releases in the screenshot above.

Editing Policies

Mouse over either a specific branch or a folder and it will show the three dots for the menu. Click the dots and then select Branch policies. For this example, we are putting policies on the master branch.

This will bring you to the page that allows you to view and edit the policies on the selected branch or folder.

The descriptions do a good job of explaining what policies do what so I’m not going to bore you with repeating them. The official docs on branch policies also go into a lot more detail. If you are not working along I highly recommend using Require a minimum number of reviews and Check for comment resolution. Build validation I would recommend no matter your team size as it keeps you away from the possibility of having that one magical machine that is the only one your build will work on. The following screenshot is with the first two recommend policies set. The settings shown for the require a minimum number of reviews are based on the fact that my project only has one contributor. After your done make sure and click the Save changes button.

Build Validation Policies

There is a bit more to the build validation policy which is why I’m covering it in a different section. On the Branch policies screen click the Add build policy button.

In the edit build policy screen, the only required change is selecting the Build pipeline to make available when a PR that is targeting the branch that is policy is for. Here we are using the automatic Trigger so any time we push to our remote branch when it has an open pull request it will run the select build pipeline. Policy requirement controls if a successful build is required before the pull request can complete or not. For a since person project Build expiration isn’t a big deal, but if your working with a team it can be helpful. Click Save when you are done.

Back on the branch policies screen, you will see the new requirement listed. You can also add as many build validations as you need.

Wrapping Up

Hopefully this quick little into to branch policies will help your team improve the quality of the code that makes it into your branches. I know to require another person or two to review your code before you can check-in sounds like it will slow you down if you are new to the concept but in reality, it helps catch issues before they make it to QA and production which saves time and money in the long run.

Azure DevOps Repos: Branch Policies Read More »

Azure DevOps Pipelines: Depends On with Conditionals in YAML

A few weeks ago we covered Conditionals in YAML to show how to conditionally run tasks and jobs as well as how to make a job dependent on another job. This post is going to cover combing conditional and job dependencies. If you are new to this series you can use the following posts to catch up.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML
Azure DevOps Pipelines: Naming and Tagging
Azure DevOps Pipelines: Manual Tagging

Existing Job

As a reminder, our sample pipeline has 4 jobs. The WebApp1 job always runs, the WebApp2 job is run or skipped based on a pipeline variable, the DependentJob depends on WebApp1 and WebApp2, and finally, the TagSources job is dependent on all the previous jobs. We are going to be tweaking the TagSources job in this post. The following is the YAML for the setup of the TagSources job without its tasks.

- job: TagSources
  displayName: 'Tag Sources'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2
  - DependentJob

With this setup WebApp1, WebApp2, and DependentJob all have to report successful or the TagSources job will be skipped. The following screenshot shows a pipeline run with the variable to build WebApp2 set to false.

As you can see the TagSources job was skipped because one of its dependent jobs was skipped.

Dependencies with Conditions

Let us say for our pipeline we want the TagSources job to run as long as all jobs were successful or if WebApp1 was successful and the WebApp2 and DependentJob jobs were skipped. To do this we are going to add a condition element and manually check the results of the dependencies as you can see in the following.

- job: TagSources
  displayName: 'Tag Sources'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2
  - DependentJob
  condition: |
    and
    (
      eq(dependencies.WebApp1.result, 'Succeeded'),
      in(dependencies.WebApp2.result, 'Succeeded', 'Skipped'),
      in(dependencies.DependentJob.result, 'Succeeded', 'Skipped')
    )

And you can see in the results the TagSources job ran even with the two skipped jobs.

Wrapping Up

If there is a simpler way to accomplish what we did above I would love to hear about it. If I remember correctly I found the above in a GitHub issue, but I don’t have the link. I’m not sure how many of you will have Pipelines that will need this, but hopefully, this will save someone some research time.

Azure DevOps Pipelines: Depends On with Conditionals in YAML Read More »

Azure DevOps Pipelines: Manual Tagging

In this week’s post, we are going to cover manually tagging instead of using the tagging feature built into Azure DevOps. This post will be using a sample Azure DevOps project built over the last few weeks of posts. If you want to see how this project has gotten to this point see the following posts.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML
Azure DevOps Pipelines: Naming and Tagging

 

 

Why?

Tags give you information about the state of your repo when an event happens, a build for our case. This can be super useful especially when you need a place to branch for a hotfix, for example. The automatic tagging covered in my Azure DevOps Pipelines: Naming and Tagging post is the way to go if it works and it has for about 95% of my projects. I have a couple of projects where the automatic tagging would not work for some reason. Since I couldn’t work out why Azure DevOps wouldn’t tag I ended up having to add a job to the Pipeline to perform the tagging.

Permissions

To perform this manual tagging the account used in our Pipeline will need to be giving contribute permission to our repo so that it will be able to push the tag. Use the gear in the lower left to open the Project Setting page.

Select the Repositories option and then click on the specific repo you would like to change the settings for, Playground is the repo we are using in the example. Note that this setting can also be changed using the top-level Git repositories option if you want to change the permission for all of your repos.

On the Security tab under Users looks for the user than has Build Service in the name and select it. When the user is selected it will show their permissions to the right of the users. Find the Contribute option and change its value to Allow.

YAML Changes

As a quick reminder, the YAML for this project currently has 3 jobs. Two web application builds (WebApp1, WebApp2) and the third job is there to show how to use job dependencies (DependentJob). I’m going to skip showing the YAML for these existing jobs, but that if you need the full existing YAML it can be found in posts linked at the top.

Since we already have multiple jobs in our Pipeline we are going to add the tagging code as a new job. This will easily allow us to only tag when all the other jobs have run successfully. The following is the full YAML for the new job.

- job: TagSources
  displayName: 'Tag Sources'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2
  - DependentJob
 
  steps:
  - checkout: self
    persistCredentials: true
    clean: true
    fetchDepth: 1

  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: |
        $env:GIT_REDIRECT_STDERR` = '2>&1'
        $tag = "manual_$(Build.BuildNumber)".replace(' ', '_')
        git tag $tag
        Write-Host "Successfully created tag $tag" 

        git push --tags
         Write-Host "Successfully pushed tag $tag"     

      failOnStderr: false

First off you see that this job depends on our existing three jobs to complete successfully before this new job will run.

dependsOn: 
- WebApp1 
- WebApp2 
- DependentJob

Starting in the Steps section you will see a checkout step that is normally handled automatically by the job, but in this case, we need to use the persistCredential option so the job will still be authed and allow us to push to our git repo. The self option is used to signify the current repo/branch. See the official Checkout docs for more information.

- checkout: self
  persistCredentials: true
  clean: true
  fetchDepth: 1

The final bit of the job is the PowerShell task that performs the actual tagging and push. I found most of this on stackoverflow and/or in a GitHub issue, but it has been a while so I don’t have the links handy. The StdErr stuff was to work around some git output that didn’t affect the tagging but was causing the job to be marked as failed. Other than that it is using the standard git commands to tag and push.

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $env:GIT_REDIRECT_STDERR` = '2>&1'
      $tag = "manual_$(Build.BuildNumber)".replace(' ', '_')
      git tag $tag
      Write-Host "Successfully created tag $tag" 

      git push --tags
       Write-Host "Successfully pushed tag $tag"     

    failOnStderr: false

Wrapping Up

I honestly hope none of you have to use this. It was a huge pain to work out. It is also to work around some sort of issue with Azure DevOps built-in tagging support. To be fair I can see where in more complex Pipelines you might need a level of flexibility that the built-in tagging couldn’t provide and this would be your only option.

Azure DevOps Pipelines: Manual Tagging Read More »

Azure DevOps Pipelines: Naming and Tagging

In this week’s post, we are going to cover changing the naming of Pipeline runs to provide more information as well as tagging our source when a pipeline is run. This post will be using a sample Azure DevOps project built over the last few weeks of posts. If you want to see how this project has progressed check out the following posts.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos
Azure DevOps Pipelines: Conditionals in YAML

Naming

By default, Pipeline runs are naming using the current date with a number for how many times the Pipeline has run for the day. For example, the fourth build on March, 17th, 2020 would start with the name 20200317.4 plus the description of the last commit. If you have the need you can change this naming scheme by using a name element in your YAML. As with the rest of the YAML related things you have all the same information available as the rest of the Pipeline to use in building whatever name might be helpful for your situation. For our example, we are going to add the branch name to the front of the run date and count by adding the following name element to the top of our YAML file.

name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r)

resources:      
  repositories: 
  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

The above would result in master_20200317.4 using the same example as above. The following screenshot shows the actual results from this change in the sample Pipeline.

Tagging

Tagging source code when running a Pipeline is a helpful way to know exactly what was included when a Pipeline is run. Here we are going to walk through using Azure DevOps to automatically tag on successful builds. From the Pipeline, you want to tag click the Edit button as you would if you were going to edit the Pipeline’s YAML. Then click the three dots and select Triggers.

Now click on the YAML tab, then Get sources, under Tag sources we are going to select On success so tags will only happen if the build completes successfully. Also, notice the Tag format which allows you to change how the tag is named. When done make sure and Save your changes.

After running a build with the above changes head over to the Repos area of the project. From Files click on History and from there you can see the tag on the last commit that was included in the build, which is displayed here as master_20200325.1.

Wrapping Up

Using clear naming for your builds can give you a lot of information at a glance, but it does take some thought to make sure information your including is helpful. Tagging is also super helpful when viewing history to know what went out with what release, and of course, they can also be used for branching. Come back next week for a look at how to manually tag when Azure DevOps automatic tagging doesn’t work for whatever reason.

Azure DevOps Pipelines: Naming and Tagging Read More »