Leveraging Custom GitLeaks TOML with Secret Magpie to Maximise Secret Detection
Relying on the built in rules for secret scanning tools only gets you so far. Here’s how to tailor your secret detection to maximise value!
Our Open Source secret scanning tool, Secret Magpie, is a powerful frontend to tools such as GitLeaks and Truffle Hog, allowing you to scan for secrets and generate either machine readable or human friendly output in the form of CSV, JSON or an interactive HTML file. As part of Secret Magpie, we allow the user to provide their own GitLeaks TOML configuration file so the user can tailor the detection rules they want to use to the specific project they’re working on, or specific secret formats they might use within a project.
Tools such as GitLeaks and TruffleHog almost always take a more conservative approach to their default rule sets in order to avoid overwhelming users with huge numbers of false positives. Unfortunately, this erring on the side of caution means that these tools out of the box can miss a lot of things, including patterns matching secrets you’re actively working with for a project, such as certain Cloudflare API keys. For instance, one might prototype some Python code using the Cloudflare library and hardcode API keys in to the code.
If we take a look at the format of one of these Cloudflare API keys (YWuTNpOM7iJ_pMg57mamZeF-KeeE3qlO9GDdlzkx), we can see that it’s the sort of pattern that would yield a lot of false positives if there was a default rule for it. If we take a look at Gitleak’s rule for how it detects Cloudflare API keys, we can see they expect the identifier “cloudflare” to precede the 40 characters of the API key. However, there are many situations when developing where you might not have such an arrangement within your code, such as practices like hardcoding API keys whilst prototyping code. In this case, if you were relying on these tools to detect commits that contained these sensitive keys, you would be out of luck!
Thankfully, we can provide a TOML file to Gitleaks that can define custom rules for Gitleaks to use, and our tool, Secret Magpie allows you to pass these TOML files through to Gitleaks to use.
Let’s start by writing a simple Gitleaks TOML file encoding a rule for detecting these Cloudflare API keys.
[extend]
useDefault = true
[[rules]]
id = "custom-cloudflare-api-key"
description = "Specialized Cloudflare API key detection rule"
regex = '''([a-zA-Z0-9_-]{40})'''
We can run Secret Magpie against a set of local repositories now with this custom Gitleaks configuration file with the following command.
docker run -it -v "./output:/output" -v "./gitleaks.toml:/gitleaks.toml" -v "./repositories:/repositories" -it punksecurity/secret-magpie --gl-config /gitleaks.toml filesystem --path /repositories
Note: You need at least secret-magpie version 1.3.2 for this feature to fully work.
If there any are instances of this pattern within our codebase that wouldn’t have been matched by Gitleaks’ built-in rules, we can now be sure they will be detected and reported rather than just ignored.
This tailoring of secret detection tools is extremely important, as developers we can become very reliant on automated solutions to keep us safe, but if the DevSecOps pipeline is completely devoid of human involvement, it ends up being all for nothing.
