A new report, the 2021 State of Secrets Sprawl on GitHub, published today by GitGuardian, a provider of a tool for monitoring usage of application secrets, suggests developers are not especially good at keeping those secrets safe.

Based on an analysis of every single commit made to GitHub, the report finds there has been a 20% year-over-year increase in the number of secrets – such as application programming interface (API) keys, private keys, certificates, usernames and passwords – discovered on a public repository.

Developers, as they build an application, make secrets easily accessible as plain text for the sake of convenience. However, when those applications are deployed in a production environment, developers often forget to delete the secrets they stored in GitHub.

GitGuardian CEO Jeremey Thomas noted that cybercriminals are now actively scanning repositories looking for secrets that would enable them to compromise applications.

As the number of application development projects being launched increased, so too did the number of application secrets being stored in repositories like GitHub. The GitGuardian report notes 60 million repositories were created in the last year, representing a 35% increase over the previous year. GitGuardian claims it detected more than 5,000 application secrets per day. Most of those secrets have been detected in repositories that belong to developers (85%), while the other 15% belonged to organizations. The top four countries where secrets were disclosed were India, Brazil, the United States and Nigeria.

GitGuardian also notes it sent out 937,539 pro bono alerts to 558,086 developers last year that informed developers that application secrets were exposed. The mean time for GitGuardian’s algorithm to detect a leak is four seconds. The average mean time to response of a developer when informed of the issues is about 25 minutes, the report noted.

As a best practice, GitGuardian recommends not storing unencrypted secrets in git repositories, and that unencrypted secrets should never be shared via messaging systems like Slack. DevOps teams should also restrict API access and permissions.

As more attention is paid to how software supply chains are managed in the wake of some recent, high-profile breaches, there is no doubt the number of audits conducted will increase. Many of those auditors will specifically be trying to ascertain how application secrets are being managed. Developers may have to get used to a new, higher level of scrutiny, especially as DevSecOps practices become more widely adopted.

In the meantime, DevOps teams may want to review their dependency on public repositories. There are plenty of private repository options that provide most of the same source code sharing capabilities as a public repository. That may mean someone on the DevOps team is required to manage the private repository, but, given the fact that many regulatory bodies will soon be looking to make an example of organizations that don’t secure application secrets, chances are good those costs will be trivial compared to a fine that would be levied.

A new experiment by a Polish security researcher offers a fresh perspective on the well understood but still all too common problem of developers accidentally publishing secrets to code repositories.

Andrzej Dyjak recently ran an experiment to see how long it took before a secret committed to a public repository (such as API or cryptographic keys) was exploited.

An AWS key generated using the Thinkst Canary digital tripwire service was first compromised after 11 minutes when posted to GitHub, before getting repeatedly attacked thereafter.

A similar dummy secret went 62 minutes before suffering its first and only compromise on GitLab.

Dyjak used GitGuardian, a Git secrets scanning technology, during his experiment.

In the case of GitHub, Dyjak received an alert within seven minutes, and therefore before the first compromise, of possible secret leakage.

Time to pwnage
The time it takes for accidentally committed secrets on public repositories to being compromised has been the subject of previous, more extensive studies by North Carolina State University (PDF), IBM Research (PDF), and a GitHub honeypot exercise put together by security researcher Bob Diachenko.

Dyjak said that although people have focused on the honeypot part of his experiment, he was more interested in a comparison between the secret detection features of GitHub and GitLab.

“GitHub won – they caught the secret, alerted me, informed the provider, and on top of all that they also alerted me about vulnerable dependencies

“GitLab lost – it does offer similar SAST [static application security testing] features but they require setup of Auto DevOps which is a bottleneck.”

The Daily Swig approached both GitHub and GitLab for comment on this story, which we’ll update as and when more information comes to hand.

Dyjak deliberately leaked two secret API keys, attached to AWS and Slack, respectively.

“While AWS was compromised in both cases (GH and GL),” Dyjak reports, “Slack was not compromised in neither.”

This finding provides anecdotal evidence that adversaries are actively and selectively hunting for leaked AWS secrets rather than any inadvertently exposed tokens or credentials within code repositories.