How GitHub Uses Machine Learning to Surface ‘Good First Issues’ for Open Source Contributors

Original reporting credit: InfoQ. Original GitHub feature announcement: February 2020.

Background: What Are ‘Good First Issues’?

Contributing to open source for the first time can be intimidating. GitHub’s Good First Issues feature addresses this by surfacing issues that are well-suited to newcomers — tasks that are small in scope, clearly described, and unlikely to require deep familiarity with a codebase. GitHub first launched this feature in May 2019, relying entirely on a hand-curated list of issue labels applied by project maintainers.

In early 2020, GitHub shipped a significantly updated version that layers a machine learning (ML) model on top of that curated approach. The combined system increased the percentage of recommended repositories that surface at least one easy issue from 40 percent to 70 percent, while also reducing the manual labeling burden on maintainers.

Why Weak Supervision Instead of a Traditional Supervised Model?

Training a supervised ML model in the classical sense requires a large, carefully labeled dataset — a time-consuming process that typically demands significant human effort. GitHub sidestepped this bottleneck by adopting a weakly supervised learning approach, where labels are inferred automatically rather than annotated manually.

The process works as follows:

• Automatic label inference: Labels are inferred for hundreds of thousands of candidate issues drawn from repositories across GitHub.
• Curated label matching: Issues are matched against a list of approximately 300 curated labels commonly used by maintainers to denote beginner-friendly tasks (for example, good first issue, beginner, easy).
• Pull request signals: Issues closed by a pull request submitted by a new contributor are used as positive signals, since this behaviorally suggests the issue was approachable.
• Diff size heuristic: Issues closed by pull requests containing only small diffs touching a single file are treated as additional positive indicators of simplicity.
• Deduplication and splitting: Duplicate issues are removed, and the dataset is split into training, validation, and test sets — with care taken to split across repositories to prevent data leakage.

Model Architecture and Training

GitHub’s model uses preprocessed issue titles and bodies as input features, represented using one-hot encoding. The model itself is implemented with TensorFlow.

Several standard techniques are applied during training to improve generalization and prevent overfitting:

• Regularization to constrain model complexity.
• Text data augmentation to increase the effective size and diversity of the training set.
• Early stopping to halt training when validation performance stops improving, avoiding overfitting.

How Issues Are Ranked and Surfaced

Once trained, the model runs inference on all open issues in a repository. The ranking and recommendation pipeline works in several stages:

1. Classifier scoring: The model assigns a probability score to each open issue. Issues that exceed a defined threshold are added to a candidate recommendation bucket.
2. Label-based injection: Open issues that carry any label from the curated list are also added to the same bucket — and are assigned higher scores than ML-only candidates, reflecting the explicit intent of the maintainer.
3. Age penalty: Issues are ranked by their final score, with a penalty applied based on issue age. This helps surface fresher, more actionable issues rather than long-stale ones.

This hybrid approach — blending ML-inferred candidates with maintainer-curated labels — ensures both breadth of discovery and precision of intent.

Infrastructure: Offline Inference with Argo Workflows

At the time of the announcement, GitHub runs model training and inference offline rather than in real time. The ML pipeline is scheduled and orchestrated using Argo Workflows, a Kubernetes-native workflow engine well-suited to data processing and ML pipelines. This architecture allows GitHub to periodically refresh recommendations without requiring low-latency online inference infrastructure.

Measured Impact

Compared to the original May 2019 release, the ML-enhanced version delivered a meaningful improvement:

• Repositories with at least one surfaced easy issue increased from 40% to 70% of recommended repositories.
• Maintainers saw a reduced burden from manual issue triaging and labeling, as the ML model can identify approachable issues without requiring an explicit label.

What’s Next

GitHub outlined two areas of planned improvement at the time of the announcement:

• Iterating on training data and the model to improve recommendation quality over time.
• A maintainer interface that would allow project owners to explicitly enable or disable ML-based recommendations for their repositories, giving them more control over what contributors see.

Key Takeaways for DevOps and Platform Teams

GitHub’s approach to Good First Issues is a practical case study in applying ML to a real-world developer experience problem. Several aspects are worth noting for teams building similar systems:

• Weak supervision scales label generation without requiring a large manual annotation effort — a useful pattern when ground truth is expensive to collect.
• Behavioral signals (who closed an issue, how large the diff was) can serve as reliable proxies for human judgment at scale.
• Hybrid systems that blend ML scores with explicit human signals often outperform pure ML approaches in precision and user trust.
• Offline inference pipelines managed by workflow engines like Argo are a pragmatic starting point before investing in real-time serving infrastructure.