In late September 2019, the engineering team at GitHub, in partnership with Weights & Biases, released the CodeSearchNet Challenge evaluation environment and leaderboard alongside the CodeSearchNet Corpus — a large open-source dataset aimed at advancing the state of semantic code search. The release also included several baseline models and a data preprocessing pipeline to help researchers and data scientists get started quickly.
What Is Semantic Code Search?
Semantic code search is the task of retrieving relevant code snippets given a natural language query. Unlike traditional keyword search, it must bridge the gap between how developers describe intent in plain English and how that logic is expressed in source code. Standard information retrieval techniques are largely ineffective here because the vocabulary used in search queries and the vocabulary found in code rarely overlap. Compounding this, there was no substantial annotated dataset specifically designed for evaluating code search quality — a gap this project directly addresses.
The CodeSearchNet Corpus
The CodeSearchNet Corpus contains approximately 6 million functions extracted from open-source code across six programming languages:
- Go
- Java
- Python
- JavaScript
- PHP
- Ruby
A subset of approximately 2 million functions includes automatically generated query-like natural language descriptions derived from associated documentation. This pairing of code with natural language makes the corpus suitable for training and evaluating semantic search models. Each record also includes metadata indicating the original source location of the code.
How the Corpus Was Collected
The team sourced code exclusively from publicly available, non-forked GitHub repositories. To ensure quality, they used libraries.io to identify projects that are depended upon by at least one other project — a proxy for real-world relevance. Projects were then ranked by popularity using star and fork counts. Any repository without a permissive license that allowed redistribution of its contents was excluded.
For parsing and tokenizing functions across all six languages, the team used Tree-sitter, an incremental parser generator library capable of handling multiple programming languages. Only functions that had associated documentation were included in the training split for the CodeSearchNet Challenge, ensuring that natural language–code pairs were available for supervised learning.
The team released their data preprocessing pipeline publicly so others can replicate or extend the dataset collection process, lowering the barrier to applying machine learning techniques to source code at scale.
The CodeSearchNet Challenge
The CodeSearchNet Challenge is a formal evaluation benchmark consisting of 99 natural language queries and approximately 4,000 expert relevance annotations. The goal is to measure how well a model can retrieve relevant code from the CodeSearchNet Corpus for each query.
How the Queries Were Selected
Query selection followed a multi-step curation process:
- Bing search logs: The team started with common code-related search queries that had high click-through rates on Bing.
- StaQC dataset: These were supplemented with queries drawn from StaQC, a dataset of Stack Overflow question-code pairs.
- Manual filtering: Queries that were purely technical keywords (e.g., API names or function signatures) were removed, leaving 99 natural language queries that describe developer intent.
How Relevance Annotations Were Collected
To generate annotation data, the team ran a standard Elasticsearch installation and their baseline models against the corpus, collecting the top 10 results per query. Data scientists, programmers, and machine learning researchers then manually annotated these results for relevance to each query. This human-labeled dataset forms the ground truth used for leaderboard evaluation.
Evaluation Protocol
To participate in the CodeSearchNet Challenge, a model must return a ranked list of results from the CodeSearchNet Corpus for each of the 99 predefined natural language queries. Results are then scored against the expert relevance annotations. The leaderboard, integrated with Weights & Biases, allows researchers to track and compare model performance transparently.
Why This Matters for the Developer Tools Ecosystem
CodeSearchNet fills a well-known gap in the code intelligence research community: the absence of a large, high-quality, multi-language benchmark for evaluating semantic code search. By releasing both the corpus and the evaluation harness, GitHub and Weights & Biases enable reproducible research and direct comparison across approaches — from classical information retrieval baselines to neural embedding models. The Tree-sitter-based preprocessing pipeline also gives practitioners a reusable starting point for building their own code-aware ML systems.
Original reporting via Packt Hub. The CodeSearchNet paper and dataset were published by the GitHub Next research team.