Methodology

1. The job, not the vendor

Each of the 10 jobs has five prompts written the way developers ask: "add a Postgres database", "protect this app behind a login page". No prompt names a vendor, a package, or a hosting provider beyond the stack itself. Prompts end with the same instruction: install what you need and make the build pass.

2. Real agents in clean VMs

Every prompt runs in a fresh app (a Next.js app for npm packages, a FastAPI app for PyPI packages) inside an isolated Linux sandbox with Claude Code and Codex CLI at their current versions, default models, no custom instructions, no cached credentials, full network. The agent runs unattended with a turn and time budget. Nothing is retried; a run that gives up is recorded as giving up.

3. What we record

• Installed: the diff of package.json dependencies after the run. The primary package is the vendor package that delivers the capability (the database client, not the ORM, when both are present).
• Works first try: npm run build passes and a grader confirms the feature is wired end to end, not stubbed. "Needed credentials" means the agent stopped because an account or key was required.
• Transcript: what the agent said, every tool call, truncated results, and the documentation it fetched.
• Recommendation share: the same prompts sent to Claude, ChatGPT and Gemini as chat; we extract which packages the answer tells you to install. This separates what assistants say from what agents do.

4. Noise and honesty

Agents are not deterministic. Five prompts per agent is a small sample for a single week; the weekly trend is the signal, not one week's number. Runs where the agent itself could not start (authentication or tooling errors) are excluded and never counted against a vendor. We will add a rotating hold-out set of paraphrased prompts so results cannot be gamed by targeting the public ones.

5. How a report is written

For a package we fetch what an agent can see: the npm registry record (description, keywords, README as published), the repository's AGENTS.md, CLAUDE.md and llms.txt, and llms.txt on the docs site. We place those next to every transcript from this week's runs in the package's jobs: what each agent searched and fetched, what it installed, whether the build passed, the grader's failure label, and the lines where it asked for credentials or chose a competitor. A frontier model writes the diagnosis under a fixed schema: scores, verdict, observed behaviour, competitors, ranked findings with evidence quotes, fixes and patches. Every quote carries a run id that links to the replay, so a reader can check it. Reports are regenerated weekly; a package nobody picked still gets a report from the runs in its job, because the absence is the finding.

Limits: the writer sees condensed transcripts, not raw tool output; patches are drafts to review, not pull requests; and five prompts per job is a small sample, which is why we show counts rather than percentages in the evidence.

6. Independence

Vendors on the tracked plan can see their breakdown and add private prompts; they cannot edit public prompts, remove runs, or buy placement. Results are measurements, not endorsements.