FixItFlow: Automated Troubleshooting Guide Generation from Cloud Incidents

# FixItFlow: Automated Troubleshooting Guide Generation from Cloud Incidents
Source: [https://arxiv.org/html/2607.13035](https://arxiv.org/html/2607.13035)
,Jaskaran Singh WaliaMicrosoft Research,Drishti GoelUniversity of Illinois Urbana\-ChampaignandSupriyo GhoshInception

###### Abstract\.

Cloud services experience frequent incidents that require rapid diagnosis and resolution\. Troubleshooting guides help engineers respond consistently, but creating them manually is labor\-intensive, resulting in incomplete coverage and outdated documentation\. We present FixItFlow, an automated system that generates troubleshooting guides from historical incident data using large language models\. The system extracts diagnostic patterns from engineer actions, synthesizes structured guides with verified commands, and enforces strict validation to prevent fabricated content\. In our evaluation with 26 engineers, generated guides achieved 61\.5% positive ratings for clarity and demonstrated a 2\.3x reduction in mitigation time for incidents with associated guides\. These results indicate that automated guide generation can improve incident response while reducing documentation burden on engineering teams\.

## 1\.Introduction

Troubleshooting Guides \(TSGs\) are step\-by\-step documents that help engineers diagnose and resolve incidents in cloud systems\. In large cloud environments with thousands of interconnected services, even minor incidents can significantly affect users and create pressure on on\-call engineers who must resolve problems quickly\. TSGs standardize incident response, reduce cognitive load during stressful situations, and preserve institutional knowledge across team members\(Jianget al\.,[2020](https://arxiv.org/html/2607.13035#bib.bib6)\)\. At Microsoft alone, engineers maintain over 50,000 TSGs used by more than 60,000 practitioners every month\.\(Shettyet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib5)\)

However, creating and maintaining high\-quality TSGs remains a manual process with several shortcomings\. First, coverage is inconsistent: some common problems lack guides while others become outdated\. Second, quality varies widely across teams, producing lengthy documents that are difficult to follow during emergencies\. An internal study revealed the most common TSG problems: missing information \(32\.24%\), broken links \(13\.32%\), and incorrect instructions \(11\.21%\)\. Together, these issues account for over half of all user complaints about TSG quality\.FixItFlowaddresses this gap by automatically generating new TSGs from historical incident data\. The system analyzes past incidents to identify patterns, then produces structured troubleshooting guides that include specific diagnostic steps and solutions tailored to each team’s tools and environment\. A validation system ensures that all technical information originates from real engineer actions, preventing the generation of fabricated or incorrect instructions\. This paper makes four contributions: \(i\) an automated pipeline that generates troubleshooting guides \(TSGs\) directly from raw incident data without relying on pre\-existing documentation; \(ii\) preprocessing techniques that filter and normalize incident artifacts to extract actionable troubleshooting signals; \(iii\) schema constraints and validation mechanisms that enforce completeness and factual grounding during TSG generation to reduce hallucination; and \(iv\) an empirical evaluation with domain experts demonstrating real\-world utility and adoption potential\.

## 2\.Background

Operating large\-scale cloud services involves managing thousands of components under recurrent failures, where production incidents remain inevitable despite reliability practices\(Ghoshet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib1); Ganatraet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib2)\)\. These incidents carry business risk and motivate comprehensive incident management through monitoring, alerting, detection, triage, diagnosis, and mitigation\(Ghoshet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib1); Chenget al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib3)\)\. In practice, delayed detection, misrouted ownership, and cross\-service dependencies complicate root\-cause analysis\(Ghoshet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib1); Ganatraet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib2); Chenet al\.,[2024](https://arxiv.org/html/2607.13035#bib.bib18); Chenget al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib3)\)\. Engineers face incomplete situational knowledge, time pressure, false alarms, and complex failure modes, while alert volume and heterogeneous traces strain responders\(Heet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib4); Ganatraet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib2); Chenget al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib3)\)\. This has accelerated interest in AIOps and intelligent incident management, emphasizing historical knowledge reuse and automated assistance\(Chenget al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib3); Ahmedet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib8); Jianget al\.,[2024](https://arxiv.org/html/2607.13035#bib.bib9)\)\. Practitioners widely rely on Troubleshooting Guides \(TSGs\) that codify diagnostic and mitigation steps, which speed consistent response and reuse historical failure patterns\(Jianget al\.,[2020](https://arxiv.org/html/2607.13035#bib.bib6)\)\. However, quality varies due to incompleteness, outdated commands, and unclear organization\(Bondelet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib7)\), with guides often insufficient during severe incidents\(Ghoshet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib1)\)\. Improving TSGs at scale is labor intensive, making automation increasingly desirable\(Jianget al\.,[2020](https://arxiv.org/html/2607.13035#bib.bib6)\)\.

Automation efforts include retrieval methods that recommend existing guides via textual similarity\(Jianget al\.,[2020](https://arxiv.org/html/2607.13035#bib.bib6)\), AutoTSG which converts guides into executable workflows\(Shettyet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib5)\), and mining approaches that extract patterns from tickets and post\-incident reviews\(Saha and Hoi,[2022](https://arxiv.org/html/2607.13035#bib.bib17)\)\. Industry systems pursue auto\-drafting and augmentation of operational knowledge\(Jinet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib10); Ahmedet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib8)\), though generating coherent TSGs from heterogeneous incident streams remains challenging\(Bondelet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib7)\)\. Large Language Models \(LLMs\) offer promise through unified handling of unstructured operational data, with applications across the incident lifecycle including root cause recommendation, outage summarization, and incident routing\(Chenget al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib3); Gaoet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib13); Ahmedet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib8); Zhanget al\.,[2024](https://arxiv.org/html/2607.13035#bib.bib12); Jinet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib10); Jianget al\.,[2024](https://arxiv.org/html/2607.13035#bib.bib9); Liuet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib11)\)\. Effective practice couples LLMs with retrieval augmentation, chain\-of\-thought prompting, and targeted fine\-tuning\(Gaoet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib13); Weiet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib14); Ahmedet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib8); Liuet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib11)\), with LLMs capable of generating procedural text and reasoning over tool\-using agents\(Yaoet al\.,[2022](https://arxiv.org/html/2607.13035#bib.bib15); Chenet al\.,[2023](https://arxiv.org/html/2607.13035#bib.bib16)\)\. In this work we address these gaps by generating new TSGs with a schema and validation layer that enforces grounding, link integrity, cross\-section consistency, and anti\-hallucination before generation\.

## 3\.Motivation

To investigate the importance of automating TSG synthesis, we conduct an empirical study to examine the impact of TSGs on Time to Mitigate \(TTM\) and uncover critical gaps in TSG linkage across incidents\. While it may seem intuitive that well\-documented TSGs can accelerate incident resolution, our analysis of the real\-world production systems at Microsoft can offer statistical support to this intuition and further motivate our approach to develop a scalable, automatic TSG\-synthesis framework\.

### 3\.1\.TSGs and Time\-to\-Mitigate \(TTM\)

In the lifecycle of an incident, the Time to Mitigate \(TTM\) refers to the duration from detection of the incident to its resolution\. A well\-structured Troubleshooting Guide plays a pivotal role in optimizing this process by providing engineers with precise and systematic instructions to triage, diagnose, and mitigate issues in a time\-efficient manner\. To ensure that TTM serves as an accurate representation of engineering effort and operational efficiency, we focus specifically on Severity 1 and Severity 2 incidents, as these are immediately escalated to the on\-call engineers upon detection\. This targeted selection allows for a more reliable study of the correlation, thus excluding lower\-severity cases and incidents that are automatically mitigated\.

### 3\.2\.TSG Coverage Across Incidents

Documenting and maintaining TSGs is a manual, time\-intensive, and continuous process that demands substantial engineering effort\. Teams in the early stages of TSG adoption often start from scratch, spending considerable time mining historical incidents and organizing their collective tribal knowledge into structured guides\. This process not only requires careful synthesis of recurring mitigation patterns, but also the ability to articulate implicit diagnostic workflows that experienced engineers follow intuitively\. In addition, keeping TSGs up\-to\-date is an ongoing challenge\. As services evolve and new edge\-case incidents emerge, teams must continuously refine and expand their documentation to ensure relevance and accuracy\. Our analysis shows that although many incidents do have associated TSGs, there is still room to broaden their reach especially in scenarios where rapid resolution is critical\.

## 4\.FixItFlow: Methodology

FixItFlow transforms raw incident data into structured troubleshooting guides through a four\-stage pipeline: data extraction, preprocessing, semantic analysis, and guide synthesis\. Each stage implements specific technical innovations to ensure quality, scalability, and accuracy\.

Algorithm 1FixItFlow Core Processing Algorithm0:Configuration parameters, Previous run timestamp

0:Validated troubleshooting guide

1:Extract:Fetch new incidents using delta query

2:Clean:Apply three\-pass sanitization algorithm

3:Batch:Partition incidents for parallel processing

4:foreach batchdo

5:Process:Extract summaries with anti\-hallucination protocol

6:endfor

7:Aggregate:Combine summaries within token limit

8:Synthesize:Generate three\-tier TSG structure

9:Validate:Verify all technical content against sources

10:returnValidated TSG with metadata

### 4\.1\.Smart Data Collection

#### 4\.1\.1\.Incremental Data Processing

FixItFlow performs incremental incident ingestion rather than full reprocessing\. Each run records a checkpoint timestamp and processes only incidents that occurred after this time\. Eligible incidents must be resolved and contain at least six engineer comments to ensure sufficient diagnostic signal\. This avoids redundant computation on previously processed incidents and significantly reduces processing and infrastructure cost\.

#### 4\.1\.2\.Flexible Data Collection Strategy

Incident collection is configurable via team ownership and alert or monitoring source filters\. For each incident, the system aggregates three data components: \(i\) scenario metadata \(e\.g\., severity, occurrence time, recurrence frequency\), \(ii\) diagnosis threads capturing full engineer investigation discussions, and \(iii\) mitigation summaries when available\. By restricting ingestion to incidents with substantial human troubleshooting activity, the system excludes auto\-resolved or low\-signal cases\.

### 4\.2\.Data Cleaning and Processing

#### 4\.2\.1\.Three\-Stage Cleaning Process

Before analyzing incident data, FixItFlow cleans it through a three\-step process to remove noise and focus on valuable information:

Step 1: Format Cleaning\.Removes HTML code, formatting markup, and other visual elements while preserving important technical content such as code blocks and commands\.

Step 2: Content Filtering\.Eliminates automated system messages, bot\-generated notifications, and very short comments \(fewer than 25 characters\) that do not contain useful troubleshooting information\.

Step 3: Duplicate Removal\.Identifies and removes duplicate incidents while combining related comment threads to create complete incident stories\.

This cleaning process reduces the data volume by 35 to 40 percent while preserving all engineer\-written technical content, making the system more efficient without losing important troubleshooting information\.

### 4\.3\.Learning from Engineer Actions

#### 4\.3\.1\.AI System Design

FixItFlow frames the language model as an expert Site Reliability Engineer through structured prompt constraints\. The system consists of four components: \(i\) a role specification enforcing expert\-level technical reasoning, \(ii\) predefined content categories covering 13 classes of troubleshooting actions, \(iii\) filters that exclude automated outputs and retain only human engineer contributions, and \(iv\) validation constraints requiring that all extracted information be grounded in the incident data\.

#### 4\.3\.2\.Preventing Fabricated Information

FixItFlow enforces strict grounding to prevent hallucination\. Every command, query, or procedural step included in the generated Troubleshooting Guide \(TSG\) must be an exact character\-level match to content present in engineer comments\. The system does not permit synthesis, paraphrasing, or completion of partial commands\. As a result, all extracted steps are directly traceable to original engineer actions, eliminating fabricated instructions and ensuring operational safety\.

### 4\.4\.Efficient Processing at Scale

FixItFlow intelligently manages the amount of text it can process at once\. AI systems have limits on how much text they can handle, so when there is too much incident data, the system prioritizes shorter comments first to maximize the number of different incidents covered, stays within the 100,000 token limit \(roughly equivalent to 75,000 words\), and ensures the maximum number of distinct troubleshooting scenarios are processed\.

### 4\.5\.Creating the Final Troubleshooting Guides

Each generated TSG follows a fixed three\-stage structure aligned with standard incident response workflows\. The Symptom section captures observable failure signals, detection methods, and impact\. The Diagnosis section lists ordered investigation steps, including the exact tools and commands used by engineers\. The Mitigation section documents the resolution procedure, verification steps, and any recurrence\-prevention actions\. Guides are generated to be complete, actionable, and strictly grounded in verified engineer activity\.

## 5\.Prompt\-Driven LLM Architecture in FixItFlow

### 5\.1\.Hierarchical Prompt Architecture

#### 5\.1\.1\.Incident Summarization Prompts

The primary prompt for incident summarization establishes the LLM as a Senior Site Reliability Engineer and implements comprehensive extraction rules\. The prompt structure includes:

Summarization Prompt \- Role Definition\# ENHANCED EXTRACTION FOR ACTIONABLE TROUBLESHOOTING CONTENTPRIMARY OBJECTIVE:Extractcomprehensiveandactionabletechnical information that engineers used during incident troubleshooting\. Focus on detailed engineer actions, investigations, and decision\-making processes that would help future engineers resolve similar incidents\.CRITICAL BALANCE:•BE COMPREHENSIVE:Extract detailed troubleshooting workflows•BE ACCURATE:Only include content explicitly mentioned by engineers•BE ACTIONABLE:Focus on information that provides clear guidance

This prompt establishes a clear operational persona and defines the extraction scope with three key constraints: comprehensiveness, accuracy, and actionability\.

#### 5\.1\.2\.Anti\-Hallucination Protocol Implementation

The central innovation of this work is the integration of a zero\-hallucination verification protocol directly within the prompting framework\. This mechanism enforces strict correspondence between the model’s outputs and verified source data, thereby preventing the generation of synthetic or unverified technical content\.

In essence, this transforms the LLM from a probabilistic text generator into a precision\-controlled retrieval system\. The model retains its semantic understanding and contextual reasoning capabilities while eliminating the risk of producing plausible but factually invalid technical statements\. This approach bridges the reliability gap between human\-authored engineering documentation and automated language model outputs\. :

Anti\-Hallucination ConstraintsCOMMAND/QUERY VERIFICATION \(ZERO TOLERANCE FOR INVENTION\):•MANDATORY VERIFICATION:Before includinganycommand/query, theexact textmust be found in comments of previous incidents\.•LITERAL EXTRACTION:Commands must be copiedCHARACTER\-BY\-CHARACTER\.•NO INFERENCE:If engineers describe actions without showing commands, describe the action but includeno commands\.•ACCEPTABLE:"Engineer restarted service \(command not provided\)"\[CORRECT\]•FORBIDDEN:Creating restart commands based on descriptions \[INCORRECT\]

### 5\.2\.TSG Generation Prompt Architecture

#### 5\.2\.1\.Structured Output Specification

The TSG generation prompt implements an output format that mirrors established incident response steps:

Each section follows detailed guidelines for content depth and technical accuracy:

##### Symptom Section Requirements\.

The prompt specifies two to three detailed paragraphs covering comprehensive error descriptions, impact analysis, timeline context, service dependencies, and monitoring indicators\. The prompt emphasizes actionable goals: ”Provide enough detail so future engineers can immediately understand the full scope and nature of the problem\.”

##### Diagnosis Section Requirements\.

The most technically demanding section requires three to four detailed paragraphs covering investigation strategy, diagnostic workflow, tool usage, and data analysis\. The prompt includes specific formatting requirements for technical content\.

##### Mitigation Section Requirements\.

The final section demands two to three detailed paragraphs covering decision\-making strategy, intervention implementation, risk coordination, execution monitoring, verification steps, and prevention measures\. The prompt emphasizes end\-to\-end playbook creation for future engineers\.

#### 5\.2\.2\.Technical Content Extraction Rules

The prompt implements detailed extraction rules with 13 specific categories of engineer activities to capture\.:

Content Extraction Categories•Detailed Investigation Steps:Full descriptions of engineer approach•Literal Commands/Queries:Exact technical commands as written•Diagnostic Reasoning:Engineer explanations of actions taken•Tool Usage:Specific tools, dashboards, and systems accessed•Error Analysis:How engineers interpreted error messages and logs•Service Dependencies:Issue tracing through dependent services•Timeline Analysis:When engineers performed actions and sequence•Escalation Details:Who was contacted, when, and why•Configuration Reviews:What configurations engineers examined•Workarounds and Fixes:Temporary and permanent solutions applied•Validation Steps:How engineers confirmed fixes worked•Root Cause Analysis:Engineer conclusions about underlying causes•Prevention Measures:Future prevention steps recommended

### 5\.3\.Template Standardization and Modularity

All FixItFlow prompts follow a standardized five\-component template: \(i\) a role specification that constrains the LLM to operate as a senior SRE, \(ii\) a task definition with explicit output constraints, \(iii\) context boundaries limiting inputs to relevant incident data, \(iv\) output format constraints enforcing structured multi\-section responses, and \(v\) validation rules specifying correctness and completeness criteria\.

Technical Content Formatting RulesTechnical Evidence:Copy all commands/queriesVERBATIMwith context:•kusto↪\\hookrightarrowfor actual Kusto/KQL queries written by engineers•shell↪\\hookrightarrowfor shell/PowerShell commands run by engineers•text↪\\hookrightarrowfor command outputs, query results, error messages•json↪\\hookrightarrowfor JSON data structures and configurations

Validation Checklist Implementation\# VALIDATION CHECKLIST FOR COMPREHENSIVE ACTIONABLE CONTENT:Before including ANY incident, verify:\- Complete Troubleshooting Story: 3\-section narrative provided?\- Detailed Engineer Context: 2\-4 paragraphs per section?\- Actionable Information: Enough detail for future engineers?\- Technical Verification: Every command copy\-pasted from source?\- NO INVENTED COMMANDS: Avoided creating any commands?\- KUSTO QUERY COMPLETENESS: All queries found and included?\- KUSTO QUERY ACCURACY: All queries copied CHARACTER\-BY\-CHARACTER?\- Educational Value: Teaches valuable troubleshooting techniques?

### 5\.4\.Performance Characteristics and Optimization

#### 5\.4\.1\.Multi\-Stage Processing Pipeline

FixItFlow’s workflow is a four\-stage pipeline that incrementally transforms raw incident data into a fully structured TSG:

##### Stage 1: Comment Classification\.

Incident comments are categorized using few\-shot prompting to identify relevant technical content\. This stage applies concurrency management and batching to handle high\-volume data efficiently while preserving classification accuracy\.

##### Stage 2: Incident Summarization\.

Classified comments are processed in parallel to generate detailed summaries, including problem descriptions, diagnostic reasoning, and mitigation steps\. Parallelization reduces processing latency while individual prompt design ensures high\-quality extraction\.

##### Stage 3: Scenario Aggregation\.

Summaries are analyzed to identify recurring failure patterns and clusters of similar incidents\. LLM\-guided similarity assessments capture meaningful operational patterns beyond superficial textual resemblance\.

##### Stage 4: TSG Synthesis\.

Aggregated scenarios are transformed into final troubleshooting guides\. This stage combines sophisticated prompt engineering, structured formatting requirements, command verification rules, and anti\-hallucination protocols to produce actionable, engineer\-verified content\.

#### 5\.4\.2\.Concurrency and Rate Management

The workflow employs adaptive concurrency controls to maximize throughput while adhering to API constraints\. Semaphore\-based rate limiting and dynamic batching prevent system overload and ensure stable execution\. Exponential backoff with jitter mitigates thundering herd effects under high\-concurrency conditions, preserving overall workflow stability\.

Table 1\.Per\-item satisfaction summary \(sorted by Top\-2 Box\)\.ItemNMeanTop\-2 Box \(%\)95% CI Low95% CI HighClarity and Understandability: is the document easy to read and follow?263\.4661\.542\.577\.6Logical Coherence and Flow: does the sequence of steps maintain a logical flow?263\.0842\.325\.561\.1Factual Accuracy: are the problem, root causes, and mitigation steps mentioned in the document factually correct?262\.9242\.325\.561\.1Document Completeness: does the document capture all the ideal TSG details to resolve incidents?262\.6223\.111\.042\.1How satisfied are you with the TSG synthesis?262\.5819\.28\.537\.9Would you like to adopt this new TSG for this monitor reported incidents \(with minor manual edits\)?130\.230\.00\.022\.8Table 2\.Key metrics from the TSG survey \(n=26\)\.MetricValueRespondents \(rows\)26\.0Likert Items \(count\)6\.0Utility Index \(mean\)2\.74Utility Index \(median\)2\.8Utility Index \(Top\-2 Box %\)23\.1NPS\-100\.0NPS n26\.0
#### 5\.4\.3\.Integrated Quality Assurance

The generation pipeline incorporates multiple layers of quality control:

##### Context Preservation

Token management and truncation strategies maintain critical context while preventing LLM input limits from being exceeded\.

##### Hallucination Prevention

Strict verification protocols require every command, query, or technical detail to be explicitly documented in the original incident data, preventing any inferred or fabricated content\.

##### Output Validation

Generated content undergoes checklist\-based validation for completeness, accuracy, and actionable utility, ensuring that TSGs provide a reliable roadmap for future incident resolution\.

Through this architecture, FixItFlow establishes a reproducible, high\-fidelity workflow for transforming unstructured incident data into operationally actionable troubleshooting guides, advancing the state of automated technical documentation generation\.

## 6\.Evaluation

To evaluate both the quality and adoptability of the TSGs generated by FixItFlow, we followed a two\-step evaluation approach that included collecting feedback from engineers through survey forms and interviews\. In the initial phase of our evaluation, we sent a form to multiple on\-call engineers, from which 26 completed the entire form\. We present the findings from the survey in Table[1](https://arxiv.org/html/2607.13035#S5.T1)\.

From the results in Table[2](https://arxiv.org/html/2607.13035#S5.T2), we can draw several important conclusions\. FixItFlow consistently produces higher\-quality TSGs compared to using basic GPT\-4o without any special instructions or processing\. This confirms that our careful design of prompts, incident summarization, and data cleaning steps makes a real difference\.

### 6\.1\.Interview Feedback

Engineer 1Feedback:Found the TSG useful, but had concerns about adoption\. Suggested aligning headings with OCE/Team templates\.Status:AcceptedNext Steps:Follow up with the technical lead on code integration\.

Engineer 2Feedback:Preferred v0\.2 TSG, citing missing mitigation steps\. Requested Communication Manager’s TSG improvements\.Status:In pipeline

### 6\.2\.Key Results from Engineer Feedback

An evaluation with 26 practicing engineers indicates that FixItFlow produces usable automated troubleshooting guides\. Readability and clarity achieved a mean score of 3\.46/5, with 61\.5% of participants rating the generated TSGs as easy to read and follow, suggesting suitability for time\-constrained incident response scenarios\.

The system also demonstrates acceptable technical quality\. Logical organization and factual correctness each received positive ratings from 42\.3% of participants, indicating that most guides follow a coherent diagnostic sequence and contain technically accurate content\. While these results highlight areas for improvement, they support the feasibility of reliable, automated TSG generation grounded in verified engineer actions\.

The evaluation confirms that FixItFlow successfully addresses a key challenge in creating understandable technical documentation automatically, though continued refinement will help improve adoption rates\.

## 7\.Conclusion

In this work, we introduce FixItFlow, an automated pipeline for generating Troubleshooting Guides from cloud incident data\. FixItFlow reduces manual effort, improves guide coverage and consistency, and speeds up incident resolution through incremental ingestion, structured cleaning, multi\-stage LLM reasoning, and schema\-based validation\. In our evaluation on production incidents, TSG\-linked cases showed approximately 2\.3 times faster mitigation\. FixItFlow improves coverage and reduces authoring effort, while feedback from engineers highlights the value of automated guide generation and the need for continued refinement to better align with team templates and incident management tools\.

## References

- T\. Ahmed, S\. Ghosh, C\. Bansal, T\. Zimmermann, X\. Zhang, and S\. Rajmohan \(2023\)Recommending root\-cause and mitigation steps for cloud incidents using large language models\.InProceedings of the 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14\-20, 2023,pp\. 1737–1749\.External Links:[Link](https://doi.org/10.1109/ICSE48619.2023.00149),[Document](https://dx.doi.org/10.1109/ICSE48619.2023.00149)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- G\. Bondel, A\. K\. Cerit, and F\. Matthes \(2022\)Challenges of API documentation from a provider perspective and best practices for examples in public web API documentation\.InProceedings of the 24th International Conference on Enterprise Information Systems, ICEIS 2022, Online Streaming, April 25\-27, 2022,pp\. 359–367\.External Links:[Link](https://doi.org/10.5220/0011089700003179),[Document](https://dx.doi.org/10.5220/0011089700003179)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- W\. Chen, X\. Ma, X\. Wang, and W\. W\. Cohen \(2023\)Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks\.External Links:2211\.12588,[Link](https://arxiv.org/abs/2211.12588),[Document](https://dx.doi.org/10.48550/arXiv.2211.12588)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- Y\. Chen, H\. Xie, M\. Ma, Y\. Kang, X\. Gao, L\. Shi, Y\. Cao, X\. Gao, H\. Fan, M\. Wen, J\. Zeng, S\. Ghosh, X\. Zhang, C\. Zhang, Q\. Lin, S\. Rajmohan, D\. Zhang, and T\. Xu \(2024\)Automatic root cause analysis via large language models for cloud incidents\.InProceedings of the Nineteenth European Conference on Computer Systems, EuroSys 2024, Athens, Greece, April 22\-25, 2024,pp\. 634–651\.External Links:[Link](https://doi.org/10.1145/3627703.3629553),[Document](https://dx.doi.org/10.1145/3627703.3629553)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1)\.
- L\. Cheng, Y\. Wu, P\. Chen, K\. Sui, T\. Xu, and D\. Pei \(2023\)AI for IT operations on cloud platforms: reviews, opportunities and challenges\.External Links:2304\.04661,[Link](https://arxiv.org/abs/2304.04661),[Document](https://dx.doi.org/10.48550/arXiv.2304.04661)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- V\. Ganatra, A\. Parayil, S\. Ghosh, Y\. Kang, M\. Ma, C\. Bansal, S\. Nath, and J\. Mace \(2023\)Detection is better than cure: A cloud incidents perspective\.InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3\-9, 2023,pp\. 838–850\.External Links:[Link](https://doi.org/10.1145/3611643.3613898),[Document](https://dx.doi.org/10.1145/3611643.3613898)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1)\.
- Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. Wang \(2023\)Retrieval\-augmented generation for large language models: A survey\.External Links:2312\.10997,[Link](https://arxiv.org/abs/2312.10997),[Document](https://dx.doi.org/10.48550/arXiv.2312.10997)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- S\. Ghosh, M\. Shetty, C\. Bansal, and S\. Nath \(2022\)How to fight production incidents? an empirical study on a large\-scale cloud service\.InProceedings of the ACM Symposium on Cloud Computing, SoCC 2022, San Francisco, CA, USA, November 7\-11, 2022,External Links:[Link](https://doi.org/10.1145/3542929.3563482),[Document](https://dx.doi.org/10.1145/3542929.3563482)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1)\.
- S\. He, X\. Zhang, P\. He, Y\. Xu, L\. Li, Y\. Kang, M\. Ma, Y\. Wei, Y\. Dang, S\. Rajmohan, and Q\. Lin \(2022\)An empirical study of log analysis at microsoft\.InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14\-18, 2022,pp\. 1490–1501\.External Links:[Link](https://doi.org/10.1145/3540250.3558963),[Document](https://dx.doi.org/10.1145/3540250.3558963)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1)\.
- B\. Jiang, Z\. Zhu, M\. Ma, Z\. Ren, Z\. Wang, H\. Jiang, C\. Zhi, Y\. Ma, L\. Zheng, Y\. Dang, C\. B\. Hong, K\. Wadhwani, D\. Arora, R\. Agarwal, R\. Bhagwan, C\. Zhang, P\. Huang, P\. Zhao, Q\. Lin, S\. Rajmohan, and D\. Zhang \(2024\)Xpert: empowering incident management with query recommendations via large language models\.InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14\-20, 2024,pp\. 179:1–179:13\.External Links:[Link](https://doi.org/10.1145/3597503.3639081),[Document](https://dx.doi.org/10.1145/3597503.3639081)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- J\. Jiang, W\. Lu, J\. Chen, Q\. Lin, P\. Zhao, Y\. Kang, H\. Zhang, Y\. Xiong, F\. Gao, Z\. Xu, Y\. Dang, and D\. Zhang \(2020\)How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems\.InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, Virtual Event, USA, November 8\-13, 2020,pp\. 1410–1420\.External Links:[Link](https://doi.org/10.1145/3368089.3417054),[Document](https://dx.doi.org/10.1145/3368089.3417054)Cited by:[§1](https://arxiv.org/html/2607.13035#S1.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- P\. Jin, S\. Zhang, M\. Ma, H\. Li, Y\. Kang, L\. Li, Y\. Liu, B\. Qiao, C\. Zhang, P\. Zhao, S\. He, F\. Sarro, Y\. Dang, S\. Rajmohan, Q\. Lin, and D\. Zhang \(2023\)Assess and summarize: improve outage understanding with large language models\.InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3\-9, 2023,pp\. 714–726\.External Links:[Link](https://doi.org/10.1145/3611643.3613891),[Document](https://dx.doi.org/10.1145/3611643.3613891)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- Z\. Liu, Y\. Ye, Y\. Liu, Z\. Ren, Y\. Li, L\. Zheng, B\. Qiao, C\. Zhang, Z\. Li, P\. Zhao, S\. He, Y\. Kang, L\. Li, X\. Zhang, Q\. Lin, Y\. Dang, S\. Rajmohan, and D\. Zhang \(2023\)IPACK: incident\-aware duplicate ticket aggregation for cloud systems\.InProceedings of the 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14\-20, 2023,pp\. 2285–2296\.External Links:[Link](https://doi.org/10.1109/ICSE48619.2023.00193),[Document](https://dx.doi.org/10.1109/ICSE48619.2023.00193)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- J\. Saha and S\. C\. H\. Hoi \(2022\)Mining root cause knowledge for AIOps\.External Links:2204\.11598,[Link](https://arxiv.org/abs/2204.11598),[Document](https://dx.doi.org/10.48550/arXiv.2204.11598)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- M\. Shetty, C\. Bansal, S\. P\. Upadhyayula, A\. Radhakrishna, and A\. Gupta \(2022\)AutoTSG: learning and synthesis for incident troubleshooting\.InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14\-18, 2022,pp\. 1477–1488\.External Links:[Link](https://doi.org/10.1145/3540250.3558958),[Document](https://dx.doi.org/10.1145/3540250.3558958)Cited by:[§1](https://arxiv.org/html/2607.13035#S1.p1.1),[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903),[Document](https://dx.doi.org/10.48550/arXiv.2201.11903)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629),[Document](https://dx.doi.org/10.48550/arXiv.2210.03629)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.
- X\. Zhang, H\. Li, S\. Saha, T\. Culhane, P\. Choksi, C\. Bansal, and S\. Rajmohan \(2024\)Automated root causing of cloud incidents using in\-context learning with GPT\-4\.InCompanion Proceedings of the 32nd ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE Companion 2024, Porto de Galinhas, Brazil, July 15\-19, 2024,pp\. 570–574\.External Links:[Link](https://doi.org/10.1145/3663529.3663846),[Document](https://dx.doi.org/10.1145/3663529.3663846)Cited by:[§2](https://arxiv.org/html/2607.13035#S2.p2.1)\.