Tech corner - 15. December 2025

Azure DevOps: Private AI code reviews

In today's fast-paced development world, automated code reviews are no longer a luxury but a necessity. Tools like CodeRabbit and others offer fantastic AI-powered insights, streamlining the review process and enhancing code quality. However, for many organizations working with Azure DevOps, sharing proprietary code with third-party cloud services is a non-starter due to stringent data privacy and security requirements. This often leaves teams searching for in-house solutions.

Our team faced this exact dilemma. We needed intelligent code review capabilities, but our code could not leave our Azure environment. Furthermore, we had already invested in and deployed our own large language models (LLMs) on Azure OpenAI, including a specialized GPT-5.1 model. The ideal solution would leverage these existing internal deployments, ensuring our code remained within our trusted Azure ecosystem.

This led us to discover PR-Agent, an open-source project originally created by Qodo. While Qodo offers its own advanced paid cloud service, they generously maintain PR-Agent as an open-source tool. Crucially for us, PR-Agent supports Azure DevOps and allows integration with various LLM providers, including Azure OpenAI. This made it the perfect candidate for our data-private, Azure-native automated code review setup.

This post will guide you through setting up an Azure Pipeline that automatically performs code reviews on every Pull Request using PR-Agent and your Azure OpenAI deployments.

Understanding PR-Agent: Your Open-source AI reviewer

PR-Agent is a versatile AI assistant for pull requests that can:

/review - Scan code changes and generate comprehensive feedback.
/describe - Automatically create PR descriptions and titles.
/improve - Suggest actionable code improvements directly in the PR.
/ask "..." - Answer questions about the PR's code or changes.

Its flexibility in integrating with different Git providers and LLMs makes it a powerful tool for custom AI-driven workflows.

Setting up your automated code review pipeline on Azure DevOps

Let's dive into the practical steps to get this running in your Azure environment.

1. Prerequisites

Before you begin, ensure you have:

An Azure DevOps Personal Access Token (PAT) with Code (Read & Write) and Pull Request Threads (Read & Write) permissions. This allows PR-Agent to read PR content and post comments/labels.
An Azure OpenAI resource with a deployed model (e.g., GPT-4o, GPT-5, GPT-5.1... deployment).
Your Azure OpenAI API Key, Endpoint URL (e.g., https://your-resource-name.openai.azure.com/), and Deployment Name (the name you gave your deployed model, not the model type).

2. Create an Azure DevOps variable group

To securely store your secrets, create a variable group in Azure DevOps:

Navigate to Pipelines -> Library.
Click + Variable group.
Name it pr_agent.
Add the following variables, marking them as secret where appropriate:
azure_devops_pat: Your Azure DevOps PAT.
AZURE_OPENAI_KEY: Your Azure OpenAI API Key.
AZURE_OPENAI_ENDPOINT: Your Azure OpenAI Endpoint URL.
AZURE_OPENAI_DEPLOYMENT_NAME: The deployment name of your Azure OpenAI model.
Ensure you grant the necessary pipeline permissions to this variable group under "Pipeline permissions".

3. Configure your Azure pipeline YAML (azure-pipelines.yml)

Create an azure-pipelines.yml file at the root of your repository (or update an existing one). Remember, for Azure Repos Git, pr: triggers in YAML are ignored, so we'll rely on Branch Policies for triggering.

yaml

># Opt out of CI triggers
>trigger: none
>

>stages:
>- stage: pr_agent
>displayName: 'PR Agent Stage'
>jobs:
>- job: pr_agent_job
>displayName: 'PR Agent Job'
>pool:
>vmImage: 'ubuntu-latest' # Or your preferred agent image
>container:
>image: codiumai/pr-agent:latest # Using the pre-built Docker image
>options: --entrypoint "" # Override default entrypoint to run custom commands
>variables:
>- group: pr_agent # Link to your variable group
>steps:
>- script: |
>echo "Running PR Agent with Azure OpenAI"
>

># Extract repository name from System.PullRequest.SourceRepositoryURI
># This is crucial for pipelines shared across multiple repositories
>REPO_NAME=$(echo "$(System.PullRequest.SourceRepositoryURI)" | sed 's|.*/_git/||' | sed 's/\.git$//')
>echo "Extracted REPO_NAME: $REPO_NAME"
>

># Construct the correct PR URL
>PR_URL="${SYSTEM_COLLECTIONURI}${SYSTEM_TEAMPROJECT}/_git/${REPO_NAME}/pullrequest/${SYSTEM_PULLREQUEST_PULLREQUESTID}"
>echo "Final PR_URL: $PR_URL"
>

># Extract organization URL
>ORG_URL=$(echo "$(System.CollectionUri)" | sed 's/\/$//')
>echo "Organization URL: $ORG_URL"
>

># --- PR Agent Configuration (Environment Variables) ---
>export azure_devops__org="$ORG_URL"
>export config__git_provider="azure"
>

># Azure OpenAI Specific Configuration
>export config__model="dnts-gpt5dot1-datazone-dev" # Your deployed model name
>export config__custom_model_max_tokens="128000" # IMPORTANT: Set based on your model
>export config__custom_reasoning_model="true" # IMPORTANT: Set for reasoning models to avoid issues with temperature
>export config__reasoning_effort="high" # OPTIONAL: You can use it if your are using reasoning model to give you better results
>export openai__api_type="azure"
>export openai__api_base="$(AZURE_OPENAI_ENDPOINT)"
>export openai__api_version="2025-04-01-preview" # Use API version supported by your model
>export openai__deployment_id="$(AZURE_OPENAI_DEPLOYMENT_NAME)"
>export openai__key="$(AZURE_OPENAI_KEY)"
>

># Custom Reviewer Instructions
>export pr_reviewer__extra_instructions="Focus on:
>- **Security vulnerabilities**: Identify potential exploits, injection attacks, and data exposure.
>- **Performance bottlenecks**: Analyze algorithmic complexity, suggest optimizations.
>- **Code maintainability**: Assess readability, adherence to coding standards, naming conventions.
>- **Error handling**: Review exception handling, proper error propagation, edge case coverage.
>- **Documentation**: Evaluate code comments, API documentation, and clarity of function descriptions.
>Provide specific, actionable feedback with code examples where applicable."
>

># Other Reviewer Settings (Optional)
>export pr_reviewer__require_security_review="true"
>export pr_reviewer__require_tests_review="true"
>export pr_reviewer__num_max_findings="10"
>export pr_reviewer__enable_review_labels_security="true"
># --- End PR Agent Configuration ---
>

># Run automated review
>echo "Starting PR Agent review for $PR_URL..."
>pr-agent --pr_url="$PR_URL" review
>env:
># Pass secrets from the variable group to the container environment
>azure_devops__pat: $(azure_devops_pat)
>AZURE_OPENAI_KEY: $(AZURE_OPENAI_KEY)
>AZURE_OPENAI_ENDPOINT: $(AZURE_OPENAI_ENDPOINT)
>AZURE_OPENAI_DEPLOYMENT_NAME: $(AZURE_OPENAI_DEPLOYMENT_NAME)
>displayName: 'Run AI Code Review with Azure OpenAI'
>

>

4. Configure branch policies for PR triggering

This is the critical step for Azure Repos Git to ensure your pipeline triggers on every PR creation:

In your Azure DevOps project, navigate to Project Settings -> Repositories -> Branches.
Select the target branch (e.g., main, develop) for which you want automated reviews.
Open the Branch Policies for that branch.
Under the Build Validation section, click + Add policy.
From the dropdown, select the pipeline you just created (azure-pipelines.yml).
Set it as Required if you want to block PR merges until the review passes.
Save the policy.

Now, every time a Pull Request targets this branch, your pipeline will automatically run!

5. Customizing the review experience

The beauty of PR-Agent lies in its configurability. We've already included some key customizations in the pipeline YAML:

config__custom_model_max_tokens="128000": This was essential for our GPT-5.1 deployment, which has a large context window. If your model supports a different max token limit, adjust this value accordingly to prevent Model is not defined in MAX_TOKENS errors.
config__custom_reasoning_model="true": Most reasoning models do not support chat-style inputs (system and user messages) or temperature settings. To bypass chat templates and temperature controls, set this property has been set. For our specific GPT-5.1 deployment, we encountered a BadRequestError: Unsupported value: 'temperature' if we left is as default (0.2), so this has resolved the issue.
pr_reviewer__extra_instructions: This environment variable allows you to provide detailed instructions to the AI reviewer. As shown in the YAML, we've guided it to focus on security, performance, maintainability, error handling, and documentation, ensuring the feedback aligns with our team's priorities. You can tailor these instructions to your project's specific needs and coding standards.
Other Reviewer Settings: Variables like pr_reviewer__require_security_review="true" and pr_reviewer__require_tests_review="true" enable specific review sections, further customizing the AI's output.

Review in action

Once your pipeline is set up and a PR is created, PR-Agent will post its review directly in your Azure DevOps Pull Request.:

The Pipeline Run: Showing the PR Agent job successfully completing.

Detailed Review Comment: A screenshot of the comprehensive feedback, including code suggestions, security insights, and other observations, all tailored by your custom instructions.

Conclusion

By leveraging PR-Agent and our Azure OpenAI deployments, we've successfully implemented a robust, automated code review system within our secure Azure environment. This approach allows us to maintain strict data privacy, utilize our existing AI infrastructure, and significantly improve code quality and developer efficiency. While setting up custom solutions requires a bit more configuration than off-the-shelf services, the control, security, and tailored results make it a worthwhile investment for any organization with similar requirements.

Happy reviewing!

Author

Vladimir Hrušovský

I am a software architect and AI enthusiast passionate about building intelligent applications. With extensive experience in React, NestJS, and web & desktop development, I specialize in creating scalable and efficient solutions. I lead AI-driven initiatives at Hotovo, focusing on automation, LLM integration, and AI-assisted development. Beyond coding, I enjoy exploring AI’s impact on productivity, experimenting with music, and spending time with my family.