Story (Required)

As a developer trying to get contextual AI feedback on specific code changes I want LLM analysis results posted as GitHub annotations on the PR diff so that I can see AI insights inline with the code that triggered them, similar to how linters highlight issues

This feature enables LLM analysis results to appear as inline annotations on the PR's file diff view, highlighting specific lines or sections of code. This improves developer experience by placing AI insights directly in context, similar to how ESLint, CodeQL, and other code quality tools provide feedback.

Background (Required)

Currently, LLM analysis only supports pr-comment as an output destination. While comments provide general feedback, they:

• Don't point to specific lines of code that need attention
• Require developers to manually correlate feedback with their changes
• Lack the visual prominence of inline annotations in the Files Changed view
• Cannot leverage GitHub's annotation features (severity levels, file/line targeting)

GitHub annotations are inline code review comments that appear directly on the diff, making feedback immediately actionable and contextually relevant.

Related: Current LLM analysis implementation at docs/content/docs/guide/llm-analysis.md

Out of scope

• Automatic line-level mapping of arbitrary LLM feedback to specific code lines (initial implementation requires LLM to provide line numbers)
• Support for other Git providers' inline comment systems (GitLab discussions, Bitbucket inline comments)
• Resolving or updating annotations after they're created
• Batch annotation creation optimization (annotations created individually in this story)
• Advanced annotation features like suggestion blocks or fix-it actions

Approach (Required)

High-level technical approach:

Add github-annotation as a new valid value for the output field in AnalysisRole configuration

Parse LLM responses to extract annotation information

*File path

* Line number or line range

*Annotation message/content

* Severity level (notice, warning, failure)

Create GitHub review comments at specified locations using the GitHub API

Support structured LLM responses that specify where to place annotations

Handle cases where LLM doesn't provide location information (fall back to general PR comment)

Support multiple outputs simultaneously (e.g., output: "pr-comment,github-annotation")

Ensure annotations are created on the latest commit SHA in the PR

The feature requires LLM prompts to be designed to return location information for effective annotation placement.

Dependencies

• Structured JSON response format support for LLM analysis (required to reliably parse file paths, line numbers, and severity levels)
• Existing LLM analysis infrastructure and GitHub provider implementation
• GitHub API access with permissions to create review comments
• Repository CRD must support github-annotation as a valid output option

Acceptance Criteria (Mandatory)

Given a Repository with an LLM role configured with output: "github-annotation", When LLM analysis completes with location data, Then inline annotations are created on the PR diff at the specified locations

Given an LLM response with file path, line number, and message, When creating annotations, Then the annotation appears on the correct file and line in the PR's Files Changed view

Given an LLM response with severity information, When creating annotations, Then the annotation is created with the appropriate GitHub severity level (notice/warning/failure)

Given multiple annotation targets in a single LLM response, When processing the output, Then each annotation is created separately at its specified location

Given an LLM response without location information, When attempting to create annotations, Then the system falls back to creating a general PR comment and logs a warning

Given a role with multiple outputs (e.g., output: "github-annotation,pr-comment"), When analysis completes, Then both inline annotations and a summary comment are created

Given insufficient GitHub permissions or API errors, When attempting to create annotations, Then the system logs an error and falls back gracefully without blocking the pipeline

Edge cases to consider:

• LLM providing line numbers that don't exist in the diff (out of range)
• Annotations targeting files not modified in the PR
• GitHub API rate limits when creating many annotations
• Handling line number shifts when PR is updated with new commits
• Annotations on binary files or very large files
• LLM responses that mix general feedback with line-specific feedback