LLM Witch-Hunt

AI code auditing

A CLI tool that scans codebases to detect AI-generated code patterns, LLM API calls, and potential API keys. Built for developers who want visibility into how much AI is actually in their projects. Available on NPM and GitHub.

Detects AI markers like comments, author tags, and variable names across 17+ file types
Identifies API calls to OpenAI, Anthropic, Google, Cohere, Hugging Face, and Azure OpenAI
Finds potential API keys with automatic masking in output for security
Integrates with CI/CD pipelines using --fail-on-detect for automated gates

Analyze

Finding the right problem

I kept seeing teams ship AI-generated code without anyone tracking what was actually in the codebase. I wanted a tool to audit my own projects first, then realized the pattern applied more broadly.

AI-generated code was appearing in production without documentation or review processes
API keys for LLM providers were being committed accidentally alongside the code they generated
Existing security scanners focused on vulnerabilities, not AI provenance or API usage patterns
No lightweight tool existed for answering "how much of this codebase came from ChatGPT?"

The question "what percentage of our code is AI-generated?" had no easy answer.

Create

Building and shipping

I built a multi-detector architecture that separates AI pattern detection, API call detection, and key detection into independent modules. The MVP focused on comprehensive scanning with sensible defaults.

Separate AIDetector and APIDetector classes allow each detection type to evolve independently
JSON reporting generates machine-readable output for integration with other tools and dashboards
Severity levels distinguish critical issues like exposed keys from informational findings like AI comments
The --fail-on-detect flag enables CI/CD integration as an automated gate before merge

API key detection was the hardest problem: finding keys without leaking them in the tool's own output.

Evaluate

Learning and iterating

Users wanted the tool to do more than detect. I said no to features that would change the tool's core purpose from auditing to modification.

I rejected auto-fix functionality that would remove AI code automatically because deletion is too destructive
Using AI to detect AI felt ironic and added complexity without proportional value
Slack and Teams notifications were scope creep that belonged in wrapper scripts, not the core tool
The target user turned out to be solo developers auditing their own code, not enterprise compliance teams

Audit tools should surface information, not make decisions for you.