Generative AI in VS Code Issues: What Developers Actually Complain About

Generative AI in VS Code Issues: What Developers Actually Complain About

Introduction

Generative AI didn’t just land in dev teams—it landed inside their daily workflows, and you can see that shift happening in the real places developers talk: GitHub Issues. A new longitudinal study (based on research from https://arxiv.org/abs/2609.04680) digs into how developers in the Visual Studio Code (VS Code) community discuss AI over time—looking at everything from setup problems to “agent mode” behavior.

The headline finding is simple (and kind of surprising): in the VS Code issue tracker, developers mostly talk about the operational stuff—agent management, configuration, reliability, authentication, quotas, and billing. Meanwhile, the risks that dominate survey conversations—like hallucinations or licensing—show up far less often than you might expect. And as the years go by, their discussion moves from AI as code-completion to AI as conversational assistants and increasingly autonomous “agents.”

This research is based on a large-scale analysis of 43,806 candidate GitHub Issues created between January 2021 and June 2026 (with the collection run on 9 June 2026, so that last month is partial). After keyword-and-semantic filtering, the researchers build a “cleaner” corpus of 25,227 AI-related issues and then map the discussion into evolving themes using topic modeling (not just simple keyword counts).

Why This Matters

Right now, most generative AI guidance tells you what AI can do or what could go wrong in theory. But the VS Code issue tracker suggests a different priority list: what teams struggle with after rollout—keeping the tooling working, wired into the environment, and usable under real constraints—is what repeatedly generates discussion.

Why this is significant now: AI features have moved from optional add-ons to core workflow components. Once AI is integrated into the editor, it becomes part of your infrastructure. That’s exactly when “it works on my machine” becomes “it works in my workspace if the auth token refreshes correctly, billing doesn’t throttle, and the agent respects context limits.” The paper’s timeline (especially the big increase in 2025) tracks this move from experimentation to dependency.

A scenario you can apply today: imagine you’re rolling out Copilot Chat (or agent-like features) to a team. The first wave of tickets won’t be about “hallucinated facts” first—they’ll be about authentication failures, “agent won’t respond,” quota/billing confusion, or misconfigured context. This study gives you evidence that those operational themes are likely to dominate issue volume. That means your rollout checklist should be operational by default: access patterns, billing alerts, fallback behavior when tools fail, and clear instructions for configuration.

How it builds on earlier research: prior work often focuses on productivity gains or conceptual risks (hallucinations, security weaknesses, licensing, code quality). This study complements that by showing what developers actually raise in a mainstream OSS venue as adoption accelerates. It’s not that those risks don’t matter—it’s that the issue tracker reflects what causes friction in the workflow week after week. Surveys measure perceived risk; GitHub Issues measure “what broke (or didn’t work) while shipping software.”

How the Researchers Mined 5 Years of AI-Related GitHub Issues

The study’s method is designed to be both broad (so it doesn’t miss AI-related issues) and careful (so it doesn’t drown in false positives).

Candidate collection: keyword-heavy, but capped (and deduped)

The researchers started with GitHub Search API queries against microsoft/vscode using twelve keywords (case-insensitive), including ChatGPT, GPT, Copilot, Claude, Cursor, Gemini, LLM, generative AI, gen AI, AI-generated, AI-assisted, and AI assistant.

A big practical constraint: the Search API caps results at 1,000 per query and limits boolean logic. So they ran queries one keyword per month, from Jan 2021 to Jun 2026, and split windows recursively when the cap was hit. Final collection happened 9 June 2026, meaning June 2026 is only issues created 1–9 June.

They also excluded PRs, retrieved title/body/comments for each issue, and kept a record only if at least one keyword appeared in the issue title, body, or any comment. The result: 43,806 candidate issues.

One extra detail: they excluded the server-side query term GPT due to substring false positives (like “Egypt”), applying GPT as a client-side post-filter instead.

Preprocessing: remove noise so the model can see meaning

Each document became a text unit made from:
- issue title
- issue body
- all comments

They then cleaned the text by removing code fragments, HTML tags, URLs, and other non-text artifacts. After lemmatization and stop-word removal, documents with fewer than 5 tokens were dropped, bringing the set from 43,806 down to 43,761 documents—this is the keyword corpus.

The key improvement: semantic filtering to remove incidental matches

Keyword matching alone brought in a lot of “AI term appears incidentally” noise. In their samples, they report that 44.7% of sampled issues co-mentioned AI terms incidentally—precision is fundamentally capped for lexical matching.

So they built a semantic relevance classifier:
- sentence embeddings: MiniLM
- classifier: logistic regression
- training data: 300 manually labeled issues
- evaluation: five-fold cross-validation

The classifier hit precision = 0.79 and recall = 0.84. Applying it to the keyword corpus produced the filtered corpus of 25,227 AI-related issues—57.6% of the keyword corpus.

That filtered set is what they use for theme validation and robustness checks, while the full keyword corpus is kept for topic modeling so classifier mistakes don’t silently remove whole themes before discovery.

Turning Issue Text Into Themes: BERTopic vs LDA (and why it matters)

Once they had a reasonable corpus, the next question was: how do you group messy issue text into coherent “topics” that can be tracked over time?

Topic modeling pipeline using BERTopic

Their primary model is BERTopic, configured with:
- embeddings: all-MiniLM-L6-v2
- dimensionality reduction: UMAP
- clustering: HDBSCAN (minimum topic size 50)
- topic representation: class-based TF-IDF using unigram–bigram
- outliers: reassigned to the nearest topic by embedding similarity

BERTopic tends to work well when text is short and full of jargon-like phrasing—exactly the kind of content you see in issue titles and comments.

Comparison: LDA baseline with tuned topic count

They also trained an optimized LDA model following guidance from earlier work, selecting topic count using coherence (CvC_v) over k=5–30, with the top coherence peaking around k=5.

Here’s how the topic-quality metrics compared in their setup:

They note that LDA’s diversity is higher partly because it outputs far fewer topics—so word reuse is rarer by construction.

Quality checks: metrics + human judgment + reliability test

They validated topics in three ways:
1. quantitative measures: CvC_v, CNPMIC_NPMI, and topic diversity
2. manual inspection of sampled issues from both large and small topics
3. intra-rater reliability: re-labeling 300 sampled issues blindly and computing Cohen’s κ

Results:
- Of the 25 largest topics, 18 (72%) were judged coherent.
- Across 125 topics, 50 (40.1% of documents) were “AI-substantive” (actually about generative AI), 26 were editor-incidental, and the rest was noise/unclear.
- Intra-rater test–retest reliability: κ = 0.73 (raw agreement 0.81), which is “substantial self-consistency.”

They also explicitly caution that keyword agreement alone is not a validation strategy (it’s low because keyword surface forms often don’t match the topic content).

What Developers Talk About: Operational Themes Beat Conceptual Risks

Now for the “so what”: what topics actually dominated the VS Code community’s AI conversations?

Two buckets: real AI discussions vs editor incidents

From the topic inventory, they grouped topics into two categories:
1. Genuinely AI-related topics (what the researchers interpret as “real” generative AI discussion)
2. Ordinary editor topics that were only present because AI keywords appeared incidentally

They excluded the second category from interpretation.

The AI-related topics: “running it” is the story

The most prominent AI-related topics include (examples from the study’s validated topic list):
- Agent file editing and sub-agent selection
- Agent and chat session management
- Coding-agent execution failures
- MCP server configuration
- File-context handling
- Chat responsiveness
- Installation and verification errors
- Quota and billing
- Authentication for Copilot services

If you’ve worked with developer tools in production, this will feel familiar: once a tool is integrated, people complain less about “does AI exist?” and more about “does it behave correctly in my environment?”

Theme distribution after semantic filtering

In their validated sample (post-filtering), the theme distribution is strongly operational:

• Bug fixing: 49.1%
• Productivity/workflow: 20.7%
• Security: 8.9%

Meanwhile, concepts that frequently dominate survey research show up much less:
- hallucination: 2.4%
- licensing: 0.6%
- code generation: 1.2%
- code review: absent (not reported as present)

This doesn’t mean developers don’t care about hallucinations or licensing—it suggests that in this specific venue (VS Code GitHub Issues), day-to-day friction is mostly operational.

Feature requests are workflow-first, too

They also note that benefits reported in practice come mainly as feature requests that improve workflows, rather than discussions focused on theoretical advantages.

If you want the “interpretation lens” of this paper, it’s this: VS Code issue discussions are closer to DevOps and integration reality than to risk brainstorming.

The original paper (https://arxiv.org/abs/2609.04680) makes the point directly: GitHub Issues provide a workflow-oriented perspective that complements survey-based studies.

How AI Discussion Evolved (2021 → 2026): Completion → Chat → Agents

The longitudinal part is where the study becomes really interesting.

The issue volume spike mirrors product integration

The annual AI-related issue count is heavily skewed toward recent years:
- 2021: 1,171
- 2022: 1,640
- 2023: 2,718
- 2024: 3,168
- 2025: 20,278
- Jan–early Jun 2026: 14,786 (partial month)

The researchers connect this inflection to VS Code integrating capabilities like Copilot Chat and agent-mode features directly into the editor—moving AI from peripheral add-on to mainstream default.

Tracking change properly: shares over raw counts

To understand “what’s gaining attention” vs “just getting more tickets because the world is louder now,” they analyze monthly topic shares, not raw counts. Corpus volume grows more than 10× across the period, so shares are more meaningful for trend detection.

Trend testing: Mann–Kendall + Sen’s slope

For topics with enough data, they use:
- Mann–Kendall test for trend direction and significance (p<0.05)
- Sen’s slope estimator for trend magnitude

Of 125 topics, 122 had sufficient data for trend analysis.

What increased: chat/agent operations and ecosystem plumbing

On the keyword corpus:
- 84 topics increased significantly
- 17 decreased significantly (p<0.05)

The increasing topics are mainly AI-assistant operational themes that emerged after 2023, then grew strongly once Copilot Chat and agent-mode features appeared:
- agent and chat session management
- file-context handling
- installation/verification
- inline chat

What decreased: editor basics lose relative share

The decreasing topics mostly correspond to traditional editor functions:
- cursor behavior
- terminal usage
- Jupyter notebooks
- keybindings

The researchers interpret this as composition effects: AI-related discussions expanded faster than the editor categories, so relative shares shrink even if absolute attention persists.

Robustness check: re-cluster on the filtered corpus

They re-fit the topic model on the filtered corpus (higher coherence: Cv = 0.648, CNPMI = 0.220) for robustness.

This robustness result matters:
- 78 topics increased significantly
- none decreased significantly in the re-clustered filtered corpus

That tells them the “decreases” are likely artifacts of changing topic mix, not real withdrawals of interest.

The three-phase evolution (the storyline)

They summarize the temporal shift as three phases:
1. 2021–2022: Copilot as code completion
2. 2023–2024: expansion with Copilot Chat
3. 2025 onward: dominated by agent-based development and ecosystem integration

In other words, developers didn’t just adopt AI—they adopted AI workflows, treating it as configurable editor infrastructure rather than a standalone productivity gimmick.

What This Means for Teams Building or Rolling Out AI in Editors

This paper offers practical implications even if you never touch topic modeling.

1) If you deploy AI features, plan for operational support

Because bug-fixing and operational issues dominate (49.1% bug fixing; lots of auth/billing/quota themes), teams should budget for:
- authentication troubleshooting playbooks
- quota/billing visibility and alerts
- configuration guides that match common environments
- clear behavior when the agent can’t execute or lacks context

2) Write better error messages and integration docs than you think you need

A lot of issue tracker traffic is basically “the tool didn’t behave as expected.” You can reduce that with:
- actionable error messages (what happened, what to do next)
- “known issues” dashboards
- consistent guidance for setup and verification

3) Don’t assume the “big AI risks” will be your first tickets

Surveys might emphasize hallucinations and licensing. In this VS Code issue dataset, those are rare (hallucination 2.4%, licensing 0.6%). That suggests you should treat operational rollout risks as immediate triage items, while still building longer-term safeguards.

4) Use GitHub Issues as an early warning system

Surveys are slow; issue trackers react quickly. If you’re responsible for AI-enabled developer tooling, you can use issue topic trends as a proxy for where integration friction is shifting:
- early spikes in completion features vs later spikes in agent session management
- the moment “chat responsiveness” becomes a recurring theme
- billing/quota topics surfacing after broader rollout

Key Takeaways

• Developers in VS Code mostly discuss operational realities, not conceptual AI risks.
• In the validated AI-related issue themes, bug fixing dominates (49.1%), followed by workflow/productivity (20.7%) and security (8.9%).
• Hallucination (2.4%) and licensing (0.6%) appear far less than prior survey-based discussions might suggest.
• The study covers 43,806 candidate issues (Jan 2021–Jun 2026) and filters down to 25,227 AI-related issues using a semantic classifier (precision 0.79, recall 0.84).
• Discussion evolves in phases: code completion → chat assistants → agent-based development (strongly reflected by 2025 growth after VS Code integrated these features).
• For real rollouts, prioritize auth, quotas/billing, configuration, reliability, and agent/session management—because that’s what generates the most sustained issue-tracker attention.
• GitHub Issues offer a workflow-oriented complement to survey research, giving you a practical view of what breaks (and what people need) as AI becomes integrated infrastructure.

If you want, I can also turn these findings into a “rollout checklist” for teams deploying Copilot-like features (auth, billing visibility, fallback modes, and support routing) based directly on the operational themes highlighted in the study.