Troubleshooting Azure Pipeline Errors: A Crash Course on Error Handling in Azure Pipelines in 2023

Cloud computing can be attributed as one of the biggest stakeholders of the massive innovation in the digital world over the past decade.  Today, businesses use an average of 137 paid distinct SaaS apps while they use 3x more free SaaS apps, a figure that has continued to increase significantly over the past few years.

(Image Credits: Statista)

The SaaS market is estimated to be valued at $716.52 billion by 2028, with a CAGR of 27.5% during the forecast period 2021-2028. On the other hand, an average smartphone user among 6.5 billion users has 35 apps installed while using 9 apps on a daily basis. These figures reflect the mass utilization of cloud technology, and at the same time, they are indicative of the mammoth workforce and infrastructure required to keep such wide-scale systems running. Undoubtedly CI/CD plays a huge role in ensuring that the seamless user experience doesn’t turn obsolete and that businesses continue to innovate.

With a 21% market share, Microsoft Azure is among the leading cloud services providers. Combining the complexity of IT infrastructures with the technical sophistication it has to offer, running into pipeline errors.  As the Talent500 blog caters to a huge number of Azure DevOps engineers, our tech experts decided to curate this article on handling Azure Pipeline Errors.

Let’s get started.

Understanding Azure Pipelines And Errors Faced

To get a better understanding of the errors, let us first begin with Azure Pipelines. This service is used for automating the processes like building, compiling, deploying, and testing codes.  It helps minimize manual intervention, which in turn reduces errors as it enables DevOps engineers to configure CI/CD for various computational environments through several stages.

How to Find Out What Went Wrong When an Activity Fails in Azure Data Factory

Let us consider the case of three activities named Set variable1, Set variable2, and Set variable3 configured as conditional paths. To get the error details via stored procedure or email. 

For Parallel Pipelines In Combination With A Wait/Dummy Activity in Azure DataFactory:

To detect the failure, use the below syntax:

@concat(activity(‘Set variable1).Error?.message,’|’activity(‘Set variable2).Error?.message,’|’activity(‘Set variable3).Error?.message)

Here, the ‘?’ symbol is added to the respective properties to avoid any issues caused by null reference errors when an activity is completed successfully. 

Let us consider a scenario where the second activity fails. In that case, you will get the below error message:

Output

{

  “name”: “Error”,

  “value”: “|The variable ‘var1’ of type ‘String’ cannot be initialized or updated with value of type ‘Integer’. The variable ‘var1’ only supports values of types ‘String’./|”

}

However, in the case of series configuration for activities, using the syntax will result in the below error with Invalid Template code, type: User configuration issue:

The expression ‘concat(activity(‘Set

variable1).Error?.message, ‘|’,activity(‘Set

variable2′).Error?.message, ‘|’,activity(‘Set

variable3′).Error?.message)’ cannot be evaluated because

property “Error” cannot be selected.

Here, you need to use the ‘?’ symbol for the activity names also since the third activity, in this case, won’t be fired at all:

@concat(activity(‘Set variable1).Error?.message,’|’activity(‘Set variable2)?.Error?.message,’|’activity(‘Set variable3)?.Error?.message)

It is noteworthy that we used ‘?’ from the second activity onwards since the first activity is invoked by default. 

Here’s how the output error will appear in this case:

{

“name”: “Error”,

“value”: “[The variable ‘var1’ of type ‘String’ cannot be initialized or

updated with value of type ‘Integer’. The variable ‘var1’ only supports

values of types ‘String’.l

}

For more details on determining pipeline failure in Azure DataFactory, click here.

Error Handling In Azure Pipeline

In order to determine the reason behind Azure pipeline failure, you must follow the below process:

To begin with, most Azure pipeline errors fall into either of these three categories:

1. The pipeline won’t trigger
2. Pipeline queues but never gets an agent
3. The pipeline fails to complete

For the first case where the pipeline won’t trigger at all, you can try the following solutions:

• Verify if you are trying to fire pr triggers while using Azure Repos. In case you are doing so, note that pr triggers don’t work with Azure Repos.
• Check the “Override the YAML pull request trigger from here” in case your  trigger and pr triggers don’t fire.
• YAML PR/CI triggers may not fire in cases where the include cause doesn’t match the details of your commit or if you mention them in the exclude clause.
• There may be discrepancies in the time zone conversion (between your time zone and UTC.) The Scheduled firing is executed according to the local time zone of the computer accessing the Azure DevOps portal.
• If your UI-defined scheduled triggers overlap with the YAML scheduled triggers, only the UI-defined scheduled triggers will be fired. Simply eliminate the UI-defined scheduled triggers via pipeline settings UI and make a push to fire YAML scheduled triggers.

In the second scenario, where pipeline queues but fails to get an agent, follow the below procedures:

• Check if you have exhausted your parallel job quota or your free limits.
• Check if you have exhausted your concurrency limits.
• Check if your job is awaiting approval.
• All agents are being utilized at the moment; you need to add more agents to the pool.
• Check if there is a mismatch between the demands and the capacity of allocated agents.
• Check if you are facing service degradation on Azure DevOps Service Status Portal.

Now let us consider the cases where the pipeline fails to complete:

• If you are facing job time out as a free account user on your private repository, which happens after 60 minutes, you may upgrade to either a Microsoft hosted agent or self-hosted agent.
• You might try issuing instructions to MSBuild to use only a single process in case of intermittent/inconsistent MSBuild failures.
• In case you are using a Python application, you may need to use a post-deployment script in the App Service deployment task to install the necessary dependencies:

D:\home\python364x64\python.exe -m pip install -r requirements.txt

• Enable Storage Explorer to deploy static content to a static website from Azure DevOps when using Azure Pipelines. Here, examples of such content include.css and.js files.
• If you encounter any of these messages:

Access is denied.

Access to the path […] is denied.

Can’t move […] to […]

The process cannot access the file […] because it is being used by another process.

It is likely that you are encountering errors with the file or folder in use. In such cases, follow the below process:

1. You can collect a record of file events within a particular directory using Process Monitor, Process Explorer, or Handle.
2. You can compel MSBuild into running only one process at a time by including the /m:1 parameter in your build tasks, as MSBuild and Visual Studio Build tasks run MSBuild with the /m switch, causing problems such as multiple process file access issues. Also, you must pass the /nodeReuse:false argument. 
3. You may make an antivirus exception for the “work folder” and agent directory that you have set up.

• In cases where you encounter a process that stops responding, it might be either waiting for input, and you may run an agent from the command line on the interactive logged-in session. If this is the case, the process will prompt a dialog box for input. If that doesn’t work, you might want to analyze the dump. Also, in cases where you are working on WiX projects with custom MSBuild loggers enabled, you must add the argument /p:RunWixToolsOutOfProc=true.
• If you are running pipelines on multiple platforms and facing problems with line endings, you must configure Git to prefer LF by adding the below line to the .gitattributes file in the root of your repository:

* text eol=lf

• If you are using a Bash script in your pipeline along with the ##vso command, disable set -x before setting a variable:
  
  set +x

echo ##vso[task.setvariable variable=MY_VAR]my_value

set -x

• Your agent might be configured to run as a service which means that it won’t run within an interactive logged-on session, thereby imposing limitations on UI interaction. This is encountered as running a command on a local machine is different from a build/release running on an agent.

• If a build fails due to the inability to discover the location of required tools, libraries, headers and other necessary files, you can use the scripts below in a separate YAML pipeline in a temporary location to analyze the agent’s layout:

# Script for Linux and macOS

pool: { vmImage: ubuntu-latest } # or whatever pool you use

steps:

– checkout: none

– bash: |

    SEARCH_PATH=$PATH  # or any colon-delimited list of paths

    IFS=’:’ read -r -a PathDirs <<< “$SEARCH_PATH”

    echo “##[debug] Found directories”

    for element in “${PathDirs[@]}”; do

        echo “$element”

    done;

    echo;

    echo;  

    echo “##[debug] Found files”

    for element in “${PathDirs[@]}”; do

        find “$element” -type f

    done

# Script for Windows

pool: { vmImage: windows-2019 } # or whatever pool you use

steps:

– checkout: none

– powershell: |

    $SEARCH_PATH=$Env:Path

    Write-Host “##[debug] Found directories”

    ForEach ($Dir in $SEARCH_PATH -split “;”) {

      Write-Host “$Dir”

    }

    Write-Host “”

    Write-Host “”

    Write-Host “##[debug] Found files”

    ForEach ($Dir in $SEARCH_PATH -split “;”) {

      Get-ChildItem $Dir -File -ErrorAction Continue | ForEach-Object -Process {

        Write-Host $_.FullName

      }

    }

• If none of these work, you may analyze the logs of your completed build/release.

Wrap Up

We hope that this mini-guide helps you handle Azure DevOps pipeline errors by diagnosing the issue and looking for the right solutions.With an average compensation of $105,017 per annum, making a career in Azure DevOps is better suited than many other software profiles. 

If you are looking to find the right Azure DevOps job for yourself, join Talent500 today!