From Labels to Deep Learning

The initial version of the feature, which launched in May 2019, surfaced issue recommendations based on labels that project maintainers applied manually. GitHub’s engineering team conducted an analysis and manual curation to compile a list of roughly 300 label names used across popular open source repositories. These labels were all synonyms for either “good first issue” or “documentation” — examples include “beginner friendly,” “easy bug fix,” and “low-hanging-fruit.”

The label-based approach had a significant limitation: only about 40% of recommended repositories had issues that could actually be surfaced this way. It also put the burden of triaging and labeling issues squarely on project maintainers — a time-consuming task that many repositories skipped.

The AI-Powered Recommender System

GitHub subsequently shipped an updated version incorporating a deep learning algorithm. According to GitHub senior machine learning engineer Tiferet Gazit, the AI system surfaces issues in approximately 70% of repositories recommended to users — a substantial improvement over the label-only baseline. GitHub described this as the first deep-learning-enabled product to launch on GitHub.com.

The new system is largely automatic, but building it required constructing an annotated training dataset of hundreds of thousands of samples.

How the Training Data Was Built

GitHub assembled the training set by starting with issues that carried any of the approximately 300 labels from the curated list. This was supplemented with additional sets of issues that were considered likely to be beginner-friendly based on behavioral signals, including:

• Issues closed by a user who had never previously contributed to that repository.
• Issues whose resolution touched only a few lines of code in a single file.

After assembling this raw dataset, near-duplicate issues were detected and removed. Training, validation, and test sets were then separated across repositories rather than within them, a deliberate choice to prevent data leakage from repositories with similar content.

The model was trained using only preprocessed and denoised issue titles and bodies. Restricting input to these fields means the model can evaluate a newly opened issue as soon as it is created, without needing additional context such as comments or commit history.

How the Production System Works

In production, the recommender pipeline works as follows:

• Any issue for which the AI algorithm predicts a probability above a defined threshold is flagged for recommendation, with a confidence score equal to its predicted probability.
• Open issues from non-archived public repositories that carry at least one label from the curated list are assigned a confidence score based on label relevance. Labels that are synonyms of “good first issue” receive higher confidence scores than labels that are synonyms of “documentation.”
• At the repository level, all detected issues are ranked primarily by confidence score. Label-based detections are generally given higher confidence than purely ML-based detections.
• An age penalty is applied to issues, so older, potentially stale issues rank lower than recent ones.

Data acquisition, model training, and inference pipelines run on a daily schedule using automated workflows, ensuring that recommendations remain current as new issues are opened and old ones are resolved.

What’s Planned Next

GitHub outlined several directions for future development of the feature:

• Better repository signals: Incorporating additional signals to improve the quality of repository-level recommendations.
• Maintainer controls: A mechanism for project maintainers and triagers to approve or remove AI-generated recommendations from their repositories — addressing concerns about automated labels appearing without maintainer consent.
• Personalized issue suggestions: Extending the system to offer personalized recommendations for contributors who have already made contributions to a project, guiding them toward logical next issues to tackle.

Why This Matters for Open Source Contribution

The label-based approach to surfacing beginner issues was always dependent on maintainer effort, and many repositories — especially smaller or less active ones — never maintained curated labels consistently. By automating detection through machine learning, GitHub shifts the responsibility away from maintainers and makes the ecosystem more accessible to first-time contributors regardless of how well a project is triaged.

For DevOps teams and platform engineers who maintain internal or open source tooling on GitHub, the planned maintainer controls will be worth watching: the ability to approve or suppress AI-generated labels gives teams fine-grained oversight of how their repositories appear to prospective contributors.

Original reporting by VentureBeat. Technical details attributed to GitHub senior machine learning engineer Tiferet Gazit.