Setup Azure DevOps Self Hosted Agent On-Premise & Troubleshooting Guidelines

Azure DevOps supports the deployment of the builds from release pipelines to the on-premise environments using self-hosted pipeline agents. This article describes creating an Azure DevOps self-hosted agent in an on-premise Windows machine and some standard troubleshooting steps in case any issue.  

 Process  

Microsoft made creating Azure DevOps self-hosted agents easy in an on-premise environment. This section describes how to do it.  

• Create an Agent pool and download the agent  
•  Follow the below steps to create an agent pool in Azure DevOps and download the agent.
•  Open Azure DevOps portal and navigate to the organization where you want to add the agent.  
•  Click on Organization settings.  
•  Click on Agent pools under the pipelines.  
•  Click on Add pool.  
•  Select Pool type as Self-hosted and provide a suitable name (say MyAgentPool). And click Create.  
•  Click on the newly created Agent pool (MyAgentPool) and the New agent button.  
•  Click the Download button to download the agent pool zip file under the Windows tab. This is the server where you want to create an on-premise Agent pool.  

Configure Azure DevOps agent  

• Once you have downloaded the zip file, follow the below instruction to configure the Azure DevOps agent in an on-premise machine.  
•  Navigate to the organization in the DevOps portal and click on the user settings icon at the top-right corner of the page. Click on Personal access tokens from the drop-down menu. Click on New Token. Select Agent Pools (read, manage) for the scope and ensure all the other boxes are unchecked. Once the token is created, copy and store the token for future use.  
• The on-premise Azure DevOps agent will use this token to communicate with Azure DevOps organization.  
•  Open PowerShell ISE on the Windows machine and execute the below command.  
•  cd C:\ #base location for the agent folder.  

You can set it as per your needs.  

mkdir agent  

cd agent  

Add-Type -AssemblyName System.IO.Compression.FileSystem ; [System.IO.Compression.ZipFile]::ExtractToDirectory("$HOME\Downloads\vsts-agent-win-x64-2.217.2.zip", "$PWD") 

•  Navigate to the folder path C:\agent.  
•  Double-click on config.cmd file. This will open a cmd prompt with the below questionnaire.
• Enter server URL > https://dev.azure.com/<organisation name>/  
•  Enter authentication type (press enter for PAT) > <Press enter>  
•  Enter personal access token > <Enter access token generated in step 1> If the connection is successful, it will ask you to register an agent.  
•  Enter agent pool (press enter for default)> <Name of the agent pool created earlier (MyAgentPool)>  
•  Enter agent name (press enter for <servername>) > <Provide any suitable name for the agent. The default is Server name>  
•  It will connect and test the agent connection. If it is successful,  
•  Enter work folder (press enter for _work) > <press enter>  
•  Enter run agent as service? (Y/N) (press enter for N) > (Enter Y if you want to run this as a service)  
•  Enter configure autologon and run agent on startup? (Y/N) (press enter for N) > (Enter Y if you want to start the service on startup)  

Start and Verify the Azure DevOps self-hosted agent.  

• Navigate to the folder path C:\agent.  
•  Double-click on the run.cmd file. This will open a cmd prompt and connect to the server. After the successful connection, it will start listening for jobs.  
•  You can return to the Agent pools in Azure DevOps organization and see your agent status online.  
• Azure DevOps self-hosted agent is now ready to use in Release pipelines for on-premise deployments.  

 Troubleshooting the agent for connectivity issue  

Sometime self-hosted build agent lost connection to the DevOps Services. We may notice error while running build/release pipelines. To provide some context, the azure pipelines agent is composed of two processes: agent.listener and agent.worker (one of these per step in the job). The listener is responsible for reporting that workers are still making progress. If the agent.listener is unable to communicate with the server for 10 minutes (we attempt to communicate every minute), we assume something has Gone Wrong and abandon the job.  

 We can perform several troubleshooting steps based on the error we face. Below are some common steps I recommend to follow.   

Examples of error: -

 ERR Agent] System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.  

•  Re-Start the running Agent service – We can go to task management or services section and kill the running service and restart it from installation folder.   
•  Re-Installation of Agent - We can stop the running services and remove the contents from installation folder (e.g config.cmd). Follow the steps listed here - Deploy an Azure Pipelines agent on Windows - Azure Pipelines | Microsoft Learn 

Note: We can also try installing the agent again in a separate folder from the old agent. The issue might be related to that the old agent installed with the URL [org.name].visualstudio.com I guess there is now some SSL certificate issue with the old URL. We can try new agent with the dev.azure.com/[org.name] URL and this may solve the issues.  

•  TLS2 Configuration – Look this article (Azure DevOps TLS 1.2 transition readiness checker) and verify TLS configuration - Source and script - microsoft/azure-devops-tls12: Azure DevOps TLS 1.2 transition readiness checker (github.com) 
•  Check if anti-virus programs identifying it as a threat  
•  Check if local proxies acting up in various ways  
•  Physical machine running out of memory or disk space (quite common).  
•  Check if the machine rebooting unexpectedly for any scheduled maintenance window e.g. OS updates etc. 
• Sometime work payload being run at a way higher priority than the listener (thus "starving" the listener out).  
• Unit tests shutting down network adapters (quite common).  
• Having too many agents at normal priority on the same machine so they starve each other out, etc.  
• If you think you're seeing an issue that cannot be explained by any of the above (and nothing jumps out at you from the _diag logs folder), please file an issue at Azure DevOps Support | Microsoft Azure 
• Run a self-hosted agent behind a web proxy – To enable the agent to run behind a web proxy, pass --proxyurl, --proxyusername and --proxypassword. You can refer this document for more information - Run the agent behind a web proxy - Azure Pipelines | Microsoft Learn 
• Stop going Agent going into idle and sleep state– Sometime agents goes into sleep state if there is no activity on agent and remains idle for specific period of time. We can configure the agent machine not to go into sleep mode.  

If everything seems to be perfectly alright with your agent and none of the steps mentioned in the Pipeline troubleshooting guide help, please report it on Developer Community where the Azure DevOps Team and DevOps community are actively answering questions. 

Continue Part 2 - Setup Azure DevOps Self Hosted Agent On-Premise & Troubleshooting Guidelines (microsoft.com)