Data capture and classification methodology

Signal Scout captures and processes data via GitHub dependency and topic indexing (e.g. a langgraph or crewai dependency, or an ai-agents/mcp topic tag) to identify potential agentic applications. These are then enriched and labeled into one of nine categories: agent, ai_enabled, framework, mcp_server, agent_adjacent, not_agent, template, education, resource_list, through deterministic rules and LLM classification.

The corpus is re-observed on a schedule rather than scanned once. Each pass records what a repo looks like at that moment, and nothing is overwritten, so the record accumulates what changed: a dependency added or dropped, a repo renamed, a repo deleted or made private. A repo that leaves the AI ecosystem stays in the corpus, because the leaving is the data.

• agent: integrates AI with a named agent SDK (i.e. LangGraph, CrewAI, Claude Agent SDK). The SDK dependency is what separates this from ai_enabled because they are often used for autonomous, multi-step behavior. Note that while these SDKs indicate likely autonomous behaviour, a known limitation of our classification is that we don't determine whether that behavior was actually used.
• ai_enabled: integrates AI, like an agent, but with no named agent SDK behind it, so it's a plain openai/anthropic API call. Most of these are straightforward, single-purpose LLM callers, but a few may have hand-rolled a genuine agentic loop without depending on a named SDK - which we don't detect, so those are classified as ai_enabled instead of agent, which is a known limitation.
• framework: helps other people build things that do something with AI, an SDK, toolkit, or platform, rather than being an AI application itself.
• mcp_server: implements a Model Context Protocol server.
• agent_adjacent: weak or ambiguous signal, displayed as "weak match."
• not_agent: no meaningful AI or agent connection found, displayed as "no match."
• template: starter or scaffold meant to be cloned, not a working app.
• education: a tutorial, course, or learning-path repo.
• resource_list: a curated list of links, "awesome-*" style.

01Build the candidate repo list

We create a list of candidate GitHub repos from two indexes.

• deps.dev: We search Google's public dependency database which indexes the runtime dependency for every published package. We filter for agent SDK declarations such as langgraph or @anthropic-ai/claude-agent-sdk and map it back to its GitHub repo. This step only sees published packages, noting that most agent applications are never published to a package registry and are missing from this step. The filter lists behind this search, indicators/deps_tier1.yaml and indicators/deps_tier2.yaml, are manually curated: tier 1 packages count as agent-building evidence on their own, a single hit is enough, while tier 2 packages need corroboration from a second hit before they count, since they are often just as likely to be a dependency of a general-purpose, non-agent application.
• GitHub topic search: Topics are tags added to repos by their owners. We query GitHub's REST search endpoint, search/repositories, for a manually defined set of tags such as ai-agents, mcp, llm-agent. The list of topics is hardcoded in pipeline/harvest_topics.py. Tags like agent and developer-tools were left out to minimize pulling in too many unrelated results. This is a wider net than deps.dev, noisier, and pulls in the frameworks themselves, tutorials, and thousands of near-identical starter clones, which is addressed in the next stage.

That funnel produces hundreds of thousands of candidate GitHub repos, and that number is then reduced to whatever is actually enriched and scored. The size of both numbers changes every run, as deps.dev and topic search surface more or fewer candidates each time.

The drop happens in two steps. First, the raw hits from deps.dev and topic search get deduplicated into a single queue, data/raw/enrich_queue.txt. Second, each of those candidates gets a real GitHub GraphQL lookup, and empty results (the repo has been renamed, deleted, or made private since it was first seeded) get removed. What is left after that is saved as data/observations/enriched*.jsonl; all the following steps are seeded from this source.

02Enrich each repo

For every repo on the candidate list additional data is captured in batches (pipeline/enrich_graphql.py). Two kinds of data come back.

• Metadata: star count, fork count, description, primary language, license, owner type, up to 20 topics, and the fork, archived, and template flags.
• File presence: a fixed list of named paths the indicator library cares about, checked against the default branch at enrichment time. The security and hygiene paths are listed in section 06; the rest are named by tool rather than by path.

The batcher watches GitHub's own GraphQL cost budget and sleeps to avoid rate limits. When GitHub returns nothing for a repo in the batch, no error, just a null node, that repo is counted as missing and never written to the output. This is the mechanism behind the drop described in section 01.

Enriched output is saved to data/observations/enriched.<shard>.jsonl, one JSON record per repo. This is the source data for the following steps.

Re-observing the same repos over time

Everything above describes one pass. One pass answers what a repo looks like now, which anyone with a scraper can also answer. It cannot answer what changed. That needs a second look at the same repo.

So enrichment repeats on a schedule. Each run is a wave, identified by its date, and runs in two phases (pipeline/observe_wave.py):

• Metadata pass: every repo in the corpus. Cheap enough to run over all of them, and it collects the counts that move (stars, forks, watchers, open issues and PRs, releases) plus the default branch's current commit id.
• Content pass: only repos whose default branch moved since the last observation, plus repos never captured before. Most repos are idle in any given week, so re-reading every manifest every time would be waste.

Two rules make the history usable. Nothing is ever overwritten, so an observation is a permanent record of a moment rather than a current-state row that gets updated. And a repo that stops matching our indicators stays in the corpus anyway, since a dependency being dropped is exactly the event worth having.

The single pass also threw away two things this one keeps. A repo that returns nothing is no longer just skipped: it is recorded as gone, deleted or made private, alongside the last full copy we hold of it. A repo that comes back under a different name is recorded as renamed, with both names. Neither can be recovered later, since the repo is no longer there to look at.

Classification is not part of a wave. Categories, scores, and precision are derived from stored observations, so they can be recomputed at any time, including for past waves after the rules change. Capture is the part that has to happen on time.

3aCategorization: deterministic rules

The rules engine (pipeline/detect_score.py) assigns a category deterministically, using fixed rules, no LLM involved. The full decision sequence, in order:

None of steps 1 through 7 look at what a repo depends on at all, only its name, description, and flags. Step 8 is the only one that looks at dependencies, and it only runs if nothing earlier already claimed the repo. So a repo can have an obvious dependency on langgraph and that dependency never gets examined, if the repo was already claimed by an earlier, non-dependency check: being on the hardcoded infrastructure list (step 2), or having a name ending in -framework (step 5).

Step 8 itself is not a single check, not every dependency carries the same weight, so it breaks into three sub-checks, run in order (see the table above for 8a/8b/8c).

An 8a hit does not map to a category by which package it is. Every 8a package contributes the same generic signal.

8b has no branching sequence of its own, only checked at all when 8a found nothing, and then it's a flat threshold count:

3bCategorization: LLM classification

A second, independent attempt at the same question, done probabilistically instead of by fixed rule. No new GitHub calls, it re-reads the same enrichment data 3a reads, just formatted differently and handed to a model instead of a lookup table.

Every non-fork repo gets condensed into a short text fact card (pipeline/build_fact_cards.py): description (truncated to 280 characters), up to 15 topics, up to 25 dependency names (the same manifest parser 3a uses), which of a dozen or so notable files are present (CLAUDE.md, SECURITY.md, CI workflows, and similar), star count, creation year, and the fork, archived, and template flags. Forks are skipped entirely, there is no packaging judgment to make on a fork.

Each card includes a README excerpt, cleaned of markup and cut to the first 1,000 characters (that length was tested and provided a good balance of accuracy and token cost). Cleaning removes HTML-wrapped logos and badge rows from READMEs.

Cards are sent in batches to a model against a single fixed system prompt, with structured output enforcing one of 8 categories: agent, framework, mcp_server, agent_adjacent, not_agent, template, education, resource_list. The prompt states a definitional priority between them (framework beats agent when a repo is infrastructure for building agents, not an agent itself, mcp_server beats framework when a repo's own identity centers on being an MCP server). For every repo the model returns a category plus one sentence of reasoning naming the specific evidence that drove the call.

Each repo is sent to an LLM regardless of determination from step 3a. Output is saved per repo (data/derived/llm_classifications.csv), including category and reasoning.

Treating repo content as untrusted

README text is considered untrusted and at risk of prompt injection, so we do some preprocessing, none of which are guarantees:

• Cleaning (pipeline/sanitize_readme.py). Strips HTML, comments, markdown image and link syntax, and invisible or direction-reversing Unicode, the characters that let text hide from a human reviewing the same file on GitHub.
• Flagging. Known injection phrasing ("ignore previous instructions" and similar) is matched and logged to data/derived/readme_injection_flags.txt for human review. It does not warn the model or block anything, and it's noisy: plenty of legitimate READMEs mention "system prompt" in an ordinary technical sense.
• Delimiting. README text sits inside explicit markers, and the prompt states that content within them is a repo describing itself, never an instruction.
• Enforcement. Every category the model returns must match a repo that was actually in that batch, or the row is rejected and logged.

Because of the limitations of the preprocessing, the LLM has no tools, no connectors, and no code execution, and its output is schema-constrained to one of the eight categories plus a sentence of text, to limit the impact of anything preprocessing misses.

The full taxonomy

Both classifiers choose from the same category list, with two exceptions.

04What this cannot tell you

• Misses unpublished applications. deps.dev only sees packages actually published to a registry. Most agent applications never are, so they are systematically under-represented unless they also happened to get topic-tagged.
• No template dedup yet. Vibe-coding platforms produce thousands of near-identical scaffold clones. A fingerprint-based dedup was designed, tree shape plus manifest hash, but never built, so growth numbers are not yet protected against clone inflation.
• About 35% of deps.dev to GitHub links are dead. Renamed, deleted, or now-private repos, simply absent from the corpus with no way to recover them. Up from 27% at the first scan.

05Four things we track about every repo

Kept deliberately separate, so a todo app built with one AI-assisted edit never counts the same as a research agent. Only the first and last drive the published category.

• What it does: ships an agent, itself
• Built by agents: built by a coding agent, from commit and branch fingerprints
• Built using AI coding tools: CLAUDE.md, .mcp.json, and similar are present
• Is it an MCP server: serves agents

What it does

Ships an agent, itself. Drives the published category.

Built by agents

Detected from commit and branch fingerprints. A separate stat, the "1 in 4 pushes" figure, not part of the category.

Built using AI coding tools

Presence of files like CLAUDE.md or .mcp.json. Captured, not yet a headline, and not allowed to influence "what it does."

Is it an MCP server

Serves agents. Drives the published category, alongside "what it does."

Terms

Indicator

One named, versioned rule, e.g. "depends on langgraph." The full set lives in indicators/*.yaml, versioned as a whole, and its categories and counts are in section 06. The library moves faster than the published numbers do, so counts on this site were produced by an older version than the catalogue in section 06 describes. They are not directly comparable, and the version each number came from is what makes it comparable to anything else.

Observation versus derived

What was actually seen (a file exists, a dependency is declared) is stored once and never edited. Category, score, and precision are derived from those observations and can be recomputed any time without re-scanning.

Fact card

A short plain-text summary of one repo's observations, built for the LLM classifier to read: description, dependencies, files present, stars, a cleaned README excerpt.

Validation sample

Not an ML holdout, since nothing here is trained. It guards against reusing a repo a human already looked at while writing the rules. The label is Claude reading the repo fresh, with more complete information than the classifier had (full README, real manifest, one at a time). It is trusted for being less lossy and unbiased by the classifier's own guess, not for being independently verified.

Precision

Of the repos the classifier put in a category, the share that actually belong there, per the validation sample. Says nothing about recall, how many real agents were missed entirely.

06What we track, by category

Every indicator sits at exactly one place in a fixed taxonomy, and the build fails on an indicator that has no place in it, so a category cannot be added without deciding where it belongs. Counts below are indicators, not repos.

Package-level detail, the tier assignments and the negative indicators are maintained internally rather than published here. What that detail changes is which repos match; it does not change the method above, the measured precision in section 03, or the limits below, which are the parts worth checking us on.

Why the inference layers undercount

These rows are floors rather than adoption figures, and one reason for that has just been removed while another has not.

Fixed on 2026-09-05: the corpus had never been seeded for them. Repos enter two ways, a GitHub topic search and a registry query for what depends on the packages we track, and both had been set up before the inference, hosting and GPU packages were added. So a repo whose only AI signal was a Groq or RunPod dependency could not be counted however good the detection was. Nine topics were added and the registry query re-run over 72 packages instead of 54. The effect was confined exactly to the categories predicted: GPU rental moved from 18 repos to 104, hosted inference from 2,894 to 5,007, model hosting from 348 to 602, while every other category moved by under 10%.

Not fixed, and not fixable: the API surface hides the rest. Hosted inference providers are mostly OpenAI-compatible. A repo calling Groq or Together typically does it through the openai package with the base URL pointed elsewhere, and a manifest cannot see a base URL, so the missing volume lands in the openai count instead. Comparing two inference providers is safer than quoting either alone, since both are undercounted the same way. The same applies to Azure OpenAI, whose dedicated client has not been published since 2024.

GPU rental stays the clearest case. Even after seeding it properly, 104 repos out of 250,930 carry one of these packages, because renting a machine happens through a dashboard, a CLI, or SSH, and none of that appears in a repo. A low number there is low visibility, not low adoption.

The evidence behind the security numbers

Every security and hygiene figure on this site rests on file presence, checked at enrichment time, and the exact paths are published so the figures can be checked. The SECURITY.md statistic counts repos where that file is absent from the default branch at the moment of observation, nothing more.

The paths behind the coding-agent and MCP categories are not listed here; the tools they detect are named in the catalogue above. Which exact file a given tool writes changes what matches, not what the categories mean or how they were validated.

Coding agents that write commits

18 signals covering 9 tools: Claude Code, Cursor, GitHub Copilot and its SWE agent, Devin, Jules, OpenAI Codex, Aider, and Lovable. These tools leave different traces, so there are four detection methods: bot actor logins (claude[bot]), branch prefixes (cursor/), commit message trailers, and commit author emails. Trailer and email indicators carry an era boundary, since several vendors stopped writing them in late 2025.

What we do not track yet

These gaps shape what the data can answer. No indicators exist yet for observability and evaluation tooling, guardrails and AI security libraries, vector stores and agent memory, structured output libraries, self-hosted inference servers, sandboxed execution, voice and realtime agents, or browser automation. To the rules engine, a repo using only these looks like no AI signal at all.

07How precision is measured

Every number under "precision" on the site follows the same recipe.

1. Pull a random sample of repos the classifier assigned to a category, roughly 100 to 125 total, spread across categories.
2. Make sure that sample is disjoint from anything used while writing or tuning the rules. A repo looked at while building the tiering logic does not count, since it is no longer a fair test.
3. A blind labeler, in this pass Claude reading each repo fresh with no access to the classifier's answer, assigns a true label.
4. Precision equals correct labels divided by total labels, per category, reported with a 95% confidence interval (Wilson score, which holds up better than a naive interval on small samples).

LLM classifier (3b) validation sample

Rules engine (3a) validation sample, agent / ai_enabled split