Source: devclass.com — Original announcement by GitHub machine learning engineer Kavita Ganesan.
Why Automatic Language Detection Matters on GitHub
At first glance, programming language detection might seem trivial — surely developers know what language they’re writing in. But GitHub’s scale introduces a different class of problem. When code is pushed to a repository, the platform needs to identify the language involved for several critical downstream functions:
- Search indexing: Language-aware search surfaces more relevant results.
- Security vulnerability alerting: Advisories and scanning tools are language-specific.
- Syntax highlighting: Correct highlighting depends on correct language identification.
- Repository composition display: The language breakdown shown on a repository’s main page must be accurate.
Why File Extensions Alone Are Not Enough
GitHub’s existing language detection tool, Linguist, relies heavily on filenames and extensions. According to Ganesan, this approach has real limitations:
- Some languages are associated with multiple file extensions (e.g.,
.hcan be C, C++, or Objective-C). - Some extensions are genuinely ambiguous and shared across multiple languages.
- Many files, particularly scripts, configuration files, and code snippets embedded in Markdown, carry no extension at all.
- Code snippets appearing inside READMEs, GitHub Issues, and pull requests are entirely outside the scope of filename-based detection.
That last point — language detection within prose documents like Issues and pull requests — is a significant gap that Linguist cannot address, and it was a primary motivation for building a new tool.
Introducing OctaLinuga: A Neural Network Approach
GitHub engineers built a new prototype called OctaLinuga (reportedly a playful Latin-inspired name suggesting “eight tongues”). The tool is implemented in Python using Keras and TensorFlow.
Model Architecture
OctaLinuga is described as a two-layer artificial neural network. Its output is a 51-dimensional probability vector: one dimension for each of the top 50 languages on GitHub, plus one additional dimension representing the probability that the input is not written in any of those 50 languages. The predicted language is the one with the highest probability score.
Training Data
The model was trained to recognise the top 50 programming languages on GitHub. Training data was sourced primarily from Rosetta Code, a community-maintained repository of programming tasks implemented in many different languages. For languages that were underrepresented in Rosetta Code, additional sources were used to supplement the dataset.
Extending to New Languages
GitHub describes adding support for new languages beyond the initial 50 as “fairly straightforward”: collect a sufficient corpus of files in the new language and run them through the training pipeline. This suggests the architecture is designed with extensibility in mind, though no specific data-size thresholds were published.
Current Status and Limitations
At the time of the announcement, OctaLinuga was at an advanced prototyping stage. GitHub characterised the classification engine as robust and reliable within its supported language set, while acknowledging that it does not yet cover all languages present on the platform.
Detection of code snippets embedded in prose — the motivating use case — was described as achievable with “a small modification to the machine learning engine,” and Ganesan noted it “wouldn’t be too far-fetched to take the model to the stage where it can reliably detect and classify embedded languages.”
Open Source Prospects
GitHub indicated it was actively considering open-sourcing OctaLinuga and expressed interest in community feedback. Given that GitHub’s existing Linguist tool is already open source and widely used by third-party platforms, open-sourcing OctaLinuga would be a natural continuation of that approach.
Key Takeaways for DevOps and Platform Engineering Teams
- Language detection is a non-trivial infrastructure problem at repository-platform scale.
- Neural network classifiers can handle ambiguous cases that rule-based tools like extension matching cannot.
- Training on a curated, multi-source corpus (Rosetta Code plus supplemental data) is a practical starting point for code classification models.
- A probability-vector output (rather than a hard label) gives consuming systems meaningful confidence information to act on.
- Extending such a model to new languages requires a labelled corpus, not architectural changes — a useful design principle for similar internal tooling.