Blog Post

Azure AI Foundry Blog
2 MIN READ

Autonomous Visual Studio Code Desktop Automation using Computer Use Agent & PyAutoGUI

srikantan's avatar
srikantan
Icon for Microsoft rankMicrosoft
Jul 20, 2025

This article outlines an approach to autonomous code generation using the Computer Use Agent (CUA) model and PyAutoGUI. It automates VS Code desktop interactions with GitHub Copilot in Agent mode.

Project Overview

The system replicates a developer's workflow by autonomously launching VS Code, configuring the environment, and generating code via GitHub Copilot Agent mode. This automation is ideal for scenarios where:

  1. Codebase is not hosted on GitHub – GitHub Agents on github.com are not applicable.
  2. GitHub Codespaces cannot access the code – VS Code in the browser is ruled out.
  3. Desktop automation is required – Must work with VS Code desktop client on a local computer.

Key Innovation: CUA Model + PyAutoGUI Integration

The project demonstrates the synergy between CUA model and PyAutoGUI:

  • PyAutoGUI: Executes desktop actions (launching apps, typing, clicking).
  • CUA Model: Analyzes screenshots and determines next actions.
  • Deterministic Outcomes: Enables autonomous detection and correction at each step.

Complete Autonomous Workflow

Phase 1: Environment Setup

  1. Launch VS Code desktop application.
  2. Navigate to the project workspace directory.
  3. Maximize VS Code window.
  4. Open integrated PowerShell terminal.

Phase 2: Intelligent Package Management

  1. Read requirements.txt.
  2. Execute pip install -r requirements.txt.
  3. Monitor progress:
    • Capture screenshots.
    • CUA analyzes terminal output.
    • Detects "in_progress" vs "complete" states.
    • PyAutoGUI waits for CUA confirmation.

Phase 3: GitHub Copilot Integration

  1. Open GitHub Copilot chat panel (Ctrl+Shift+I).
  2. Switch to Agent mode.
  3. Submit developer prompt.
  4. Monitor status:
    • Capture screenshots every 3 seconds.
    • CUA checks "Keep" button status.
    • Accept generated code when ready.

CUA Model Decision Points

Package Installation Monitoring

{
  "installation_status": "complete" | "in_progress"
}
  • In Progress: Detects progress bars, downloads.
  • Complete: Identifies empty prompt ready for input.

Code Generation Monitoring

{
  "button": "enabled" | "disabled"
}
  • Disabled: Code generation ongoing.
  • Enabled: Ready for acceptance

Automation Flow in Action

The following are the sequence of steps that are automated in the flow

  1. Launch VS Code → Open project workspace.
  2. Open terminal → Execute pip install.
  3. CUA monitors → Detects completion.
  4. Open Copilot → Activate Agent mode.
  5. Submit prompt → Begin code generation.
  6. Monitor "Keep" button → Detect enabled state.
  7. Accept code → Automation complete.

Real-World Applications

Development Scenarios

  • Clone repositories.
  • Create branches.
  • Generate code.
  • Run tests and validations.

Regression Testing Use Cases

  • Routine code generation.
  • Test case creation.
  • Documentation updates.
  • Refactoring tasks.

Note: Requires a dedicated computer to run VS Code desktop. Use responsibly with safeguards and monitoring.

Refer to the GitHub Repo here. Refer to the video of this sample in action below:

Limitations:
With pyautogui, the App on which the automation needs to be done has to be on the foreground. It cannot choose a particular app from amongst other apps on the Desktop to monitor the visual state. There are other options to consider for that, which I will cover in another article later

Updated Jul 20, 2025
Version 1.0
No CommentsBe the first to comment