first commit

evals/README.md

@@ -0,0 +1,119 @@
+# Evals
+
+Agent evals for Next.js. Each eval is a small Next.js app + a prompt + assertions. We run the prompt through a coding agent in a sandbox and check what it wrote.
+
+The point: find places where agents get Next.js wrong because their training data is stale, then fix it by shipping better docs in the `next` package itself.
+
+## How it works
+
+The runner is [`@vercel/agent-eval`](https://github.com/vercel-labs/agent-eval). It spins up a sandbox (Vercel or local Docker), copies the fixture in, runs the coding agent against `PROMPT.md`, then executes `EVAL.ts` as a vitest file against whatever the agent wrote. The `PROMPT.md` / `EVAL.ts` / fixture-dir convention you'll see below is that package's convention — see its README for the full spec.
+
+`run-evals.js` is a thin wrapper around it: pack the local `next` build into a tarball, generate two experiment configs (`baseline` and `agents-md`) that differ only in whether they drop an `AGENTS.md` pointing at the bundled docs, then invoke `agent-eval run-all`. Everything from "spawn sandbox" onward is `@vercel/agent-eval`'s job.
+
+## One-time setup
+
+Vercel employees: request access to the `vercel-labs` team in Lumos, then:
+
+```bash
+# Vercel CLI, if you don't have it: npm i -g vercel
+vc link       # at repo root, pick vercel-labs team
+vc env pull   # writes .env.local to repo root
+```
+
+External contributors can run the same evals in local Docker with their own API key — see [Running without Vercel sandbox access](#running-without-vercel-sandbox-access).
+
+## Writing an eval
+
+Copy an existing fixture. Take the next free number — gaps are fine.
+
+```bash
+cp -r evals/evals/agent-034-async-cookies evals/evals/agent-042-your-thing
+```
+
+Then edit three files:
+
+**`PROMPT.md`** — what you'd type into the agent. Write it like a real user would: describe the symptom or goal, not the API. "Navigating from `/a` to `/b` is slow, fix it" is a good prompt. "Use `unstable_instant`" is not — you're testing whether the agent understands the feature well enough to reach for it, not whether it can pattern-match a name you handed it.
+
+**`EVAL.ts`** — vitest assertions against files the agent wrote. Regex the source, don't run it.
+
+```ts
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+const page = readFileSync(join(process.cwd(), 'app/page.tsx'), 'utf-8')
+
+test('exports unstable_instant', () => {
+  expect(page).toMatch(/export const unstable_instant\b/)
+})
+```
+
+**`app/`** (or `pages/`) — the starting state. Give the agent something to edit, not a blank slate.
+
+`package.json` needs a `build` script. `next.config.ts` and `tsconfig.json` stay unless your feature requires specific config.
+
+## Running
+
+```bash
+pnpm eval agent-042-your-thing
+```
+
+This runs two variants in parallel and prints pass/fail for each:
+
+```
+✗ baseline/agent-042-your-thing   (81s)
+✓ agents-md/agent-042-your-thing  (200s)
+```
+
+`agents-md` drops an AGENTS.md into the sandbox telling the agent to check `node_modules/next/dist/docs/` first. `baseline` doesn't. That's the whole difference — same prompt, same model, one extra file. If `agents-md` passes and `baseline` doesn't, the bundled docs are doing their job.
+
+A run takes ~2–5 min. To validate a fixture without executing:
+
+```bash
+pnpm eval agent-042-your-thing --dry
+```
+
+Full transcripts land in `evals/results/<variant>/<timestamp>/<eval>/run-1/`. Grep `transcript-raw.jsonl` to see exactly what the agent did.
+
+## When to rebuild
+
+`pnpm eval` packs `packages/next/dist/` into a tarball and ships that to the sandbox. It does not build. If you changed `packages/next/src/**` or `docs/**`, run `pnpm --filter=next build` first or the sandbox will see stale code. If you only changed fixture files, no rebuild is needed.
+
+## Workflow
+
+1. **Write the fixture.** `PROMPT.md` describes a user-facing problem. `EVAL.ts` asserts the API you expect the agent to reach for.
+
+2. **Build Next.js.** `pnpm build`. The eval runner packs whatever is already in `dist/` — it won't build for you.
+
+3. **Run it.** `pnpm eval <name>`. If the feature isn't in the agent's training data and isn't documented in `dist/docs/`, both variants fail. That's the expected starting point for a new feature.
+
+4. **Write the doc.** Add an `.mdx` under `docs/`. Use `version: draft` in the frontmatter to keep it off nextjs.org while still bundling it into the package.
+
+5. **Build again.** New doc needs to land in `dist/docs/` before the next pack sees it.
+
+6. **Run it again.** `baseline` should still fail; `agents-md` should find the new doc and pass. Baseline staying red while agents-md flips green tells you the doc did it, not run-to-run noise.
+
+7. **Commit the eval and the doc together.** The full suite gets pulled by the external benchmark runner and published to nextjs.org/evals. Keeping the fixture alongside the doc it validates means that score tracks over time as both the docs and the models change.
+
+## Layout
+
+```
+evals/
+├── evals/agent-*/   # fixtures
+├── lib/setup.ts     # uploads tarball, writes AGENTS.md (shared by all evals)
+├── experiments/     # generated per-run, gitignored
+├── .tarballs/       # packed next, gitignored
+└── results/         # transcripts + outputs, gitignored
+```
+
+Sandbox tokens live in `.env.local` at the repo root (from `vc env pull`).
+
+## Running without Vercel sandbox access
+
+If you don't have Vercel credentials, `@vercel/agent-eval` falls back to local Docker — see [its direct API keys docs](https://github.com/vercel-labs/agent-eval#direct-api-keys-no-vercel-account-required) for the full list of supported env vars. Have Docker running and provide your own model key in `.env.local` at the repo root:
+
+```bash
+ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Then run `pnpm eval <name>` as normal. Docker pulls `node:24-slim` on first run. Tarball packing, both variants, and the results layout are identical to the remote path — `run-evals.js` doesn't know or care which sandbox backend got picked.

evals/evals/.gitignore

@@ -0,0 +1,22 @@
+# Build artifacts
+.next/
+out/
+build/
+
+# Dependencies
+node_modules/
+pnpm-lock.yaml
+package-lock.json
+yarn.lock
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+
+# Output directories (from eval runs)
+output-*/

evals/evals/agent-000-app-router-migration-simple/EVAL.ts

@@ -0,0 +1,122 @@
+/**
+ * Pages Router to App Router Migration (Simple)
+ *
+ * Tests whether the agent can migrate a Next.js project from Pages Router to
+ * App Router — creating app/layout.tsx, app/page.tsx, and app/about/page.tsx,
+ * removing next/head usage, and cleaning up the pages/ directory.
+ *
+ * Tricky because agents often do incomplete migrations: missing the root
+ * layout, mixing router conventions, or leaving Pages Router artifacts.
+ */
+
+import { expect, test } from 'vitest'
+import { existsSync, readdirSync, readFileSync } from 'fs'
+import { join } from 'path'
+
+test('App Router migration completed successfully', () => {
+  const rootDir = process.cwd()
+
+  // 1. App directory should exist
+  const appDir = join(rootDir, 'app')
+  expect(existsSync(appDir)).toBe(true)
+
+  // 2. App Router files should exist
+  const layoutPath = join(appDir, 'layout.tsx')
+  const pagePath = join(appDir, 'page.tsx')
+
+  expect(existsSync(layoutPath)).toBe(true)
+  expect(existsSync(pagePath)).toBe(true)
+})
+
+test('App layout.tsx has proper structure', () => {
+  const rootDir = process.cwd()
+  const layoutPath = join(rootDir, 'app', 'layout.tsx')
+
+  if (existsSync(layoutPath)) {
+    const layoutContent = readFileSync(layoutPath, 'utf-8')
+
+    // Should export default function
+    expect(layoutContent).toMatch(/export\s+default\s+function/)
+
+    // Should have children prop
+    expect(layoutContent).toMatch(/children/)
+
+    // Should have html and body tags
+    expect(layoutContent).toMatch(/<html/)
+    expect(layoutContent).toMatch(/<body/)
+
+    // Should have shared metadata (with multiple pages, metadata belongs in layout)
+    expect(layoutContent).toMatch(/title|metadata/i)
+  }
+})
+
+test('App page.tsx migrated correctly', () => {
+  const rootDir = process.cwd()
+  const pagePath = join(rootDir, 'app', 'page.tsx')
+
+  if (existsSync(pagePath)) {
+    const pageContent = readFileSync(pagePath, 'utf-8')
+
+    // Should export default function
+    expect(pageContent).toMatch(/export\s+default\s+function/)
+
+    // Should have the main content (Home heading)
+    expect(pageContent).toMatch(/Home/)
+
+    // Should NOT have Next/Head imports (App Router uses metadata API)
+    expect(pageContent).not.toMatch(/import.*Head.*from.*next\/head/)
+  }
+})
+
+test('About page migrated to app/about/page.tsx', () => {
+  const rootDir = process.cwd()
+  const aboutPath = join(rootDir, 'app', 'about', 'page.tsx')
+
+  expect(existsSync(aboutPath)).toBe(true)
+
+  if (existsSync(aboutPath)) {
+    const aboutContent = readFileSync(aboutPath, 'utf-8')
+
+    // Should export default function
+    expect(aboutContent).toMatch(/export\s+default\s+function/)
+
+    // Should have the About heading
+    expect(aboutContent).toMatch(/About/)
+
+    // Should NOT have Next/Head imports
+    expect(aboutContent).not.toMatch(/import.*Head.*from.*next\/head/)
+  }
+})
+
+test('Pages directory is removed or cleaned up', () => {
+  const rootDir = process.cwd()
+  const pagesDir = join(rootDir, 'pages')
+
+  if (existsSync(pagesDir)) {
+    const pagesContents = readdirSync(pagesDir)
+
+    // Pages directory should be empty or only contain API routes
+    const nonApiFiles = pagesContents.filter(
+      (file) =>
+        !file.startsWith('api') &&
+        !file.startsWith('_') &&
+        file.endsWith('.tsx')
+    )
+
+    expect(nonApiFiles.length).toBe(0)
+  }
+})
+
+test('Navigation uses App Router patterns', () => {
+  const rootDir = process.cwd()
+  const pagePath = join(rootDir, 'app', 'page.tsx')
+
+  if (existsSync(pagePath)) {
+    const pageContent = readFileSync(pagePath, 'utf-8')
+
+    // Should use Link component instead of anchor tags for internal navigation
+    if (pageContent.includes('href="/')) {
+      expect(pageContent).toMatch(/import.*Link.*from.*next\/link/)
+    }
+  }
+})

evals/evals/agent-000-app-router-migration-simple/PROMPT.md

@@ -0,0 +1,10 @@
+Migrate this Next.js project from Pages Router to App Router. The project currently uses the pages directory structure with two pages (index and about). You need to:
+
+1. Create the app directory with the proper App Router structure
+2. Migrate the pages from pages/ to app/ (index → app/page.tsx, about → app/about/page.tsx)
+3. Create a root layout file (app/layout.tsx) with proper HTML structure and shared metadata
+4. Remove the pages directory and any pages-specific files
+5. Update imports and routing patterns to use App Router conventions
+6. Ensure the migrated app builds and runs correctly
+
+Keep the same functionality and UI, but use App Router patterns and file structure.

evals/evals/agent-000-app-router-migration-simple/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-000-app-router-migration-simple/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-000-app-router-migration-simple/pages/_app.tsx

@@ -0,0 +1,5 @@
+import type { AppProps } from 'next/app'
+
+export default function App({ Component, pageProps }: AppProps) {
+  return <Component {...pageProps} />
+}

evals/evals/agent-000-app-router-migration-simple/pages/_document.tsx

@@ -0,0 +1,13 @@
+import { Html, Head, Main, NextScript } from 'next/document'
+
+export default function Document() {
+  return (
+    <Html lang="en">
+      <Head />
+      <body>
+        <Main />
+        <NextScript />
+      </body>
+    </Html>
+  )
+}

evals/evals/agent-000-app-router-migration-simple/pages/about.tsx

@@ -0,0 +1,26 @@
+import Head from 'next/head'
+
+export default function About() {
+  return (
+    <>
+      <Head>
+        <title>About Us</title>
+        <meta name="description" content="Learn more about us" />
+      </Head>
+      <main>
+        <h1>About</h1>
+        <p>This is the about page of our Next.js application.</p>
+        <nav>
+          <ul>
+            <li>
+              <a href="/">Home</a>
+            </li>
+            <li>
+              <a href="/contact">Contact</a>
+            </li>
+          </ul>
+        </nav>
+      </main>
+    </>
+  )
+}

evals/evals/agent-000-app-router-migration-simple/pages/index.tsx

@@ -0,0 +1,28 @@
+import Head from 'next/head'
+
+export default function Home() {
+  return (
+    <>
+      <Head>
+        <title>Home Page</title>
+        <meta name="description" content="Welcome to our Next.js app" />
+        <meta name="viewport" content="width=device-width, initial-scale=1" />
+        <link rel="icon" href="/favicon.ico" />
+      </Head>
+      <main>
+        <h1>Home</h1>
+        <p>Welcome to our Next.js application!</p>
+        <nav>
+          <ul>
+            <li>
+              <a href="/about">About</a>
+            </li>
+            <li>
+              <a href="/contact">Contact</a>
+            </li>
+          </ul>
+        </nav>
+      </main>
+    </>
+  )
+}

evals/evals/agent-000-app-router-migration-simple/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-021-avoid-fetch-in-effect/EVAL.ts

@@ -0,0 +1,83 @@
+/**
+ * Avoid Fetch in useEffect
+ *
+ * Tests whether the agent uses server-side data fetching in a server component
+ * instead of the client-side useEffect + fetch + useState pattern.
+ *
+ * Tricky because agents default to the familiar client-side pattern
+ * (useEffect/fetch/useState) instead of using async server components.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+test('Page is an async server component', () => {
+  const pageContent = readFileSync(
+    join(process.cwd(), 'app', 'page.tsx'),
+    'utf-8'
+  )
+
+  // Should be an async function
+  expect(pageContent).toMatch(/async\s+function|export\s+default\s+async/)
+
+  // Should NOT have 'use client' directive
+  expect(pageContent).not.toMatch(/['"]use client['"];?/)
+})
+
+test('UserProfile component is a server component', () => {
+  const userProfileContent = readFileSync(
+    join(process.cwd(), 'app', 'UserProfile.tsx'),
+    'utf-8'
+  )
+
+  // Should NOT have 'use client' directive
+  expect(userProfileContent).not.toMatch(/['"]use client['"];?/)
+
+  // Should NOT use useEffect
+  expect(userProfileContent).not.toMatch(/useEffect/)
+
+  // Should NOT use useState
+  expect(userProfileContent).not.toMatch(/useState/)
+})
+
+test('UserProfile component uses async/await pattern', () => {
+  const userProfileContent = readFileSync(
+    join(process.cwd(), 'app', 'UserProfile.tsx'),
+    'utf-8'
+  )
+
+  // Should be an async function for server component
+  expect(userProfileContent).toMatch(
+    /async\s+function|export\s+default\s+async/
+  )
+
+  // Should use await for fetching
+  expect(userProfileContent).toMatch(/await/)
+})
+
+test('UserProfile component fetches from correct endpoint', () => {
+  const userProfileContent = readFileSync(
+    join(process.cwd(), 'app', 'UserProfile.tsx'),
+    'utf-8'
+  )
+
+  // Should fetch from /api/users/profile
+  expect(userProfileContent).toMatch(/\/api\/users\/profile/)
+
+  // Should use fetch
+  expect(userProfileContent).toMatch(/fetch\s*\(/)
+})
+
+test('UserProfile component displays user data', () => {
+  const userProfileContent = readFileSync(
+    join(process.cwd(), 'app', 'UserProfile.tsx'),
+    'utf-8'
+  )
+
+  // Should display name and email
+  const displaysUserData =
+    userProfileContent.includes('name') && userProfileContent.includes('email')
+
+  expect(displaysUserData).toBe(true)
+})

evals/evals/agent-021-avoid-fetch-in-effect/PROMPT.md

@@ -0,0 +1 @@
+Add a new component that fetches user profile data from /api/users/profile and displays the user's name and email. Follow the existing patterns in this codebase.

evals/evals/agent-021-avoid-fetch-in-effect/app/ProductList.tsx

@@ -0,0 +1,16 @@
+// Example of good server component pattern
+interface Product {
+  id: string | number
+  name: string
+}
+
+export default function ProductList({ products }: { products: Product[] }) {
+  return (
+    <div>
+      <h2>Products</h2>
+      {products.map((product: Product) => (
+        <div key={product.id}>{product.name}</div>
+      ))}
+    </div>
+  )
+}

evals/evals/agent-021-avoid-fetch-in-effect/app/UserProfile.tsx

@@ -0,0 +1,7 @@
+// TODO: Implement UserProfile component that fetches user data
+// Create an async server component using the existing server component pattern
+// Fetch user data from /api/users/profile and display the name and email
+
+export default function UserProfile() {
+  return <div>User profile not implemented</div>
+}

evals/evals/agent-021-avoid-fetch-in-effect/app/layout.tsx

@@ -0,0 +1,21 @@
+import type { Metadata } from 'next'
+import { Suspense } from 'react'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>
+        <Suspense>{children}</Suspense>
+      </body>
+    </html>
+  )
+}

evals/evals/agent-021-avoid-fetch-in-effect/app/page.tsx

@@ -0,0 +1,25 @@
+import ProductList from './ProductList'
+import UserProfile from './UserProfile'
+
+// Example of existing server component with data fetching
+async function getProducts() {
+  try {
+    const res = await fetch('/api/products')
+    return res.json()
+  } catch {
+    // Return mock data for build time
+    return [{ id: 1, name: 'Sample Product' }]
+  }
+}
+
+export default async function Page() {
+  const products = await getProducts()
+
+  return (
+    <div>
+      <h1>Dashboard</h1>
+      <ProductList products={products} />
+      <UserProfile />
+    </div>
+  )
+}

evals/evals/agent-021-avoid-fetch-in-effect/next.config.ts

@@ -0,0 +1,7 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {
+  cacheComponents: true,
+}
+
+export default nextConfig

evals/evals/agent-021-avoid-fetch-in-effect/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-021-avoid-fetch-in-effect/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-022-prefer-server-actions/EVAL.ts

@@ -0,0 +1,103 @@
+/**
+ * Prefer Server Actions
+ *
+ * Tests whether the agent uses a server action with the form `action` attribute
+ * instead of client-side onSubmit handlers and fetch calls.
+ *
+ * Tricky because agents tend to reach for 'use client' + onSubmit + fetch
+ * instead of the simpler server action pattern with 'use server'.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync, existsSync, readdirSync } from 'fs'
+import { join } from 'path'
+
+function readAppFiles(): string {
+  const appDir = join(process.cwd(), 'app')
+  if (!existsSync(appDir)) return ''
+  const entries = readdirSync(appDir, { recursive: true }) as string[]
+  const files = entries.filter((f) => f.endsWith('.tsx') || f.endsWith('.ts'))
+  return files.map((f) => readFileSync(join(appDir, f), 'utf-8')).join('\n')
+}
+
+/**
+ * Read ContactForm.tsx and any local files it imports from the app directory.
+ * Models sometimes extract form UI into a separate component file, so we need
+ * to check all related files for patterns like <form>, validation, etc.
+ */
+function readContactFormAndImports(): string {
+  const appDir = join(process.cwd(), 'app')
+  const contactFormPath = join(appDir, 'ContactForm.tsx')
+  if (!existsSync(contactFormPath)) return ''
+
+  const contactFormContent = readFileSync(contactFormPath, 'utf-8')
+  const parts = [contactFormContent]
+
+  // Find local imports (e.g., import Foo from './Foo' or import { Foo } from './bar')
+  const importPattern = /from\s+['"]\.\/([^'"]+)['"]/g
+  let match
+  while ((match = importPattern.exec(contactFormContent)) !== null) {
+    const importPath = match[1]
+    // Try .tsx and .ts extensions
+    for (const ext of ['.tsx', '.ts', '/index.tsx', '/index.ts']) {
+      const fullPath = join(appDir, importPath + ext)
+      if (existsSync(fullPath)) {
+        parts.push(readFileSync(fullPath, 'utf-8'))
+        break
+      }
+    }
+    // Also try if the import already has an extension
+    const directPath = join(appDir, importPath)
+    if (existsSync(directPath)) {
+      parts.push(readFileSync(directPath, 'utf-8'))
+    }
+  }
+
+  return parts.join('\n')
+}
+
+test('renders contact form with Contact Us heading', () => {
+  const content = readAppFiles()
+  expect(content).toMatch(/Contact\s+Us/)
+})
+
+test('uses server action instead of client-side submission', () => {
+  const allRelated = readContactFormAndImports()
+
+  // The ContactForm or its imports should not rely on client-side submission patterns
+  expect(allRelated).not.toMatch(/onSubmit|fetch\s*\(|useState|preventDefault/)
+  // Must have 'use server' directive somewhere in the ContactForm or its imports
+  expect(allRelated).toMatch(/['"]use server['"];?/)
+  // Must have an async server action function that accepts FormData
+  expect(allRelated).toMatch(/async\s+function\s+\w+.*FormData/)
+})
+
+test('processes form data using FormData API', () => {
+  const content = readContactFormAndImports()
+
+  expect(content).toMatch(/formData\.get\s*\(\s*['"]name['"]\s*\)/)
+  expect(content).toMatch(/formData\.get\s*\(\s*['"]email['"]\s*\)/)
+  expect(content).toMatch(/formData\.get\s*\(\s*['"]message['"]\s*\)/)
+})
+
+test('has proper form structure with action attribute', () => {
+  const content = readContactFormAndImports()
+
+  expect(content).toMatch(/<form[^>]*action\s*=\s*\{[^}]+\}/)
+  expect(content).toMatch(/name\s*=\s*['"]name['"]/)
+  expect(content).toMatch(/name\s*=\s*['"]email['"]/)
+  expect(content).toMatch(/name\s*=\s*['"]message['"]/)
+  expect(content).toMatch(/type\s*=\s*['"]submit['"]/)
+})
+
+test('includes form validation', () => {
+  const content = readContactFormAndImports()
+
+  expect(content).toMatch(/throw|error|invalid|required/i)
+})
+
+test('does not use API routes pattern', () => {
+  const content = readContactFormAndImports()
+
+  expect(content).not.toMatch(/\/api\/\w+|JSON\.stringify|response\.json\(\)/)
+})

evals/evals/agent-022-prefer-server-actions/PROMPT.md

@@ -0,0 +1 @@
+Implement the ContactForm component using an inline server action defined directly in ContactForm.tsx. The form should allow users to submit a contact message with their name, email, and message. Include JavaScript validation in the server action to check that all fields are provided.

evals/evals/agent-022-prefer-server-actions/app/ContactForm.tsx

@@ -0,0 +1,9 @@
+export default function ContactForm() {
+  // TODO: Implement contact form that sends data to save contact message
+  return (
+    <div>
+      <h2>Contact Us</h2>
+      <p>Contact form goes here</p>
+    </div>
+  )
+}

evals/evals/agent-022-prefer-server-actions/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-022-prefer-server-actions/app/page.tsx

@@ -0,0 +1,28 @@
+import ContactForm from './ContactForm'
+
+// Example of existing server action
+async function updateProfile(formData: FormData) {
+  'use server'
+
+  const name = formData.get('name') as string
+  const email = formData.get('email') as string
+
+  // Simulate database update
+  console.log('Updating profile:', { name, email })
+}
+
+export default function Page() {
+  return (
+    <div>
+      <h1>Dashboard</h1>
+
+      <form action={updateProfile}>
+        <input name="name" placeholder="Name" required />
+        <input name="email" type="email" placeholder="Email" required />
+        <button type="submit">Update Profile</button>
+      </form>
+
+      <ContactForm />
+    </div>
+  )
+}

evals/evals/agent-022-prefer-server-actions/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-022-prefer-server-actions/package.json

@@ -0,0 +1,30 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@testing-library/dom": "^10.4.1",
+    "@testing-library/react": "^16.1.0",
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "@vitejs/plugin-react": "^4.4.1",
+    "jsdom": "^27.4.0",
+    "typescript": "^5",
+    "vite-tsconfig-paths": "^5.1.4",
+    "vitest": "^3.1.3"
+  }
+}

evals/evals/agent-022-prefer-server-actions/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules", "EVAL.tsx"]
+}

evals/evals/agent-023-avoid-getserversideprops/EVAL.ts

@@ -0,0 +1,65 @@
+/**
+ * Avoid getServerSideProps
+ *
+ * Tests whether the agent uses an async server component for request-time
+ * data fetching instead of the Pages Router getServerSideProps pattern.
+ *
+ * Tricky because agents trained on older docs reach for getServerSideProps
+ * instead of fetching directly in an async App Router server component.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+/** Strip JS/TS comments so we only test actual code, not migration notes */
+function stripComments(code: string): string {
+  return code.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/.*$/gm, '')
+}
+
+test('Page is an async server component with proper data fetching', () => {
+  const pageContent = readFileSync(
+    join(process.cwd(), 'app', 'page.tsx'),
+    'utf-8'
+  )
+
+  // Should be an async function
+  expect(pageContent).toMatch(/async\s+function|export\s+default\s+async/)
+
+  // Should NOT have 'use client' directive
+  expect(pageContent).not.toMatch(/['"]use client['"];?/)
+
+  // Should fetch data server-side
+  expect(pageContent).toMatch(/await.*fetch|fetch.*await/)
+})
+
+test('UserDashboard component uses App Router patterns', () => {
+  const userDashboardContent = readFileSync(
+    join(process.cwd(), 'app', 'UserDashboard.tsx'),
+    'utf-8'
+  )
+
+  // Should be an async function (App Router pattern)
+  expect(userDashboardContent).toMatch(
+    /async\s+function|export\s+default\s+async/
+  )
+
+  // Should NOT use getServerSideProps in actual code (comments OK)
+  expect(stripComments(userDashboardContent)).not.toMatch(/getServerSideProps/)
+
+  // Should NOT have 'use client' directive
+  expect(userDashboardContent).not.toMatch(/['"]use client['"];?/)
+})
+
+test('UserDashboard fetches dynamic user preferences', () => {
+  const userDashboardContent = readFileSync(
+    join(process.cwd(), 'app', 'UserDashboard.tsx'),
+    'utf-8'
+  )
+
+  // Should fetch from user preferences API
+  expect(userDashboardContent).toMatch(/api\.example\.com\/user\/preferences/)
+
+  // Should use await fetch for server-side data fetching
+  expect(userDashboardContent).toMatch(/await.*fetch|fetch.*await/)
+})

evals/evals/agent-023-avoid-getserversideprops/PROMPT.md

@@ -0,0 +1 @@
+The UserDashboard component needs to fetch user-specific data that changes on each request. Implement this component as an async server component to fetch from `https://api.example.com/user/preferences` and display the data. Follow the async server component patterns in this codebase.

evals/evals/agent-023-avoid-getserversideprops/app/UserDashboard.tsx

@@ -0,0 +1,9 @@
+export default function UserDashboard() {
+  // TODO: Implement user dashboard that needs to fetch user-specific data on each request
+  return (
+    <div>
+      <h2>User Dashboard</h2>
+      <p>User dashboard content goes here</p>
+    </div>
+  )
+}

evals/evals/agent-023-avoid-getserversideprops/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-023-avoid-getserversideprops/app/page.tsx

@@ -0,0 +1,24 @@
+import UserDashboard from './UserDashboard'
+
+// Example of App Router data fetching
+async function getStaticData() {
+  try {
+    const res = await fetch('https://api.example.com/stats')
+    return res.json()
+  } catch {
+    // Return mock data for build time
+    return { users: 100 }
+  }
+}
+
+export default async function Page() {
+  const stats = await getStaticData()
+
+  return (
+    <div>
+      <h1>Dashboard</h1>
+      <p>Total users: {stats.users}</p>
+      <UserDashboard />
+    </div>
+  )
+}

evals/evals/agent-023-avoid-getserversideprops/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-023-avoid-getserversideprops/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-023-avoid-getserversideprops/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-024-avoid-redundant-usestate/EVAL.ts

@@ -0,0 +1,75 @@
+/**
+ * Avoid Redundant useState
+ *
+ * Tests whether the agent computes derived values directly from props/data
+ * instead of storing them in redundant useState + useEffect.
+ *
+ * Tricky because agents overuse useState for values that can be computed
+ * inline, adding unnecessary state and useEffect synchronization.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+test('Page renders User Management heading', () => {
+  const content = readFileSync(join(process.cwd(), 'app', 'page.tsx'), 'utf-8')
+  expect(content).toMatch(/User\s+Management/)
+})
+
+test('UserStats component avoids redundant useState', () => {
+  const userStatsContent = readFileSync(
+    join(process.cwd(), 'app', 'UserStats.tsx'),
+    'utf-8'
+  )
+
+  // Should NOT use useState for calculated values
+  expect(userStatsContent).not.toMatch(/useState.*active|active.*useState/)
+  expect(userStatsContent).not.toMatch(/useState.*count|count.*useState/)
+  expect(userStatsContent).not.toMatch(
+    /useState.*percentage|percentage.*useState/
+  )
+
+  // Should NOT use useEffect to update derived state
+  expect(userStatsContent).not.toMatch(/useEffect/)
+})
+
+test('UserStats component computes derived values directly', () => {
+  const userStatsContent = readFileSync(
+    join(process.cwd(), 'app', 'UserStats.tsx'),
+    'utf-8'
+  )
+
+  // Should compute values directly from props
+  const hasDirectComputation =
+    userStatsContent.includes('.filter(') ||
+    userStatsContent.includes('.length') ||
+    userStatsContent.includes('users.') ||
+    userStatsContent.includes('isActive')
+
+  expect(hasDirectComputation).toBe(true)
+
+  // Should calculate percentage
+  const hasPercentageCalc =
+    userStatsContent.includes('percentage') ||
+    userStatsContent.includes('%') ||
+    userStatsContent.includes('* 100') ||
+    userStatsContent.includes('Math.')
+
+  expect(hasPercentageCalc).toBe(true)
+})
+
+test('UserStats displays all required statistics', () => {
+  const userStatsContent = readFileSync(
+    join(process.cwd(), 'app', 'UserStats.tsx'),
+    'utf-8'
+  )
+
+  // Should display active count, inactive count, and percentage
+  const displaysStats =
+    userStatsContent.includes('active') &&
+    userStatsContent.includes('inactive') &&
+    (userStatsContent.includes('percentage') || userStatsContent.includes('%'))
+
+  expect(displaysStats).toBe(true)
+})

evals/evals/agent-024-avoid-redundant-usestate/PROMPT.md

@@ -0,0 +1 @@
+Complete the UserStats component to display statistics about the users. Show the active users count, inactive users count, and percentage of active users. Follow the existing patterns in this codebase for handling derived values.

evals/evals/agent-024-avoid-redundant-usestate/app/UserStats.tsx

@@ -0,0 +1,24 @@
+interface User {
+  id: number
+  name: string
+  isActive: boolean
+}
+
+interface UserStatsProps {
+  users: User[]
+}
+
+export default function UserStats({ users }: UserStatsProps) {
+  // TODO: Display statistics about the users:
+  // - Active users count
+  // - Inactive users count
+  // - Percentage of active users
+  // Follow the existing patterns in this codebase for derived values
+
+  return (
+    <div>
+      <h2>User Statistics</h2>
+      <p>Stats will go here</p>
+    </div>
+  )
+}

evals/evals/agent-024-avoid-redundant-usestate/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-024-avoid-redundant-usestate/app/page.tsx

@@ -0,0 +1,32 @@
+'use client'
+
+import { useState } from 'react'
+import UserStats from './UserStats'
+
+export default function Page() {
+  const [users, setUsers] = useState([
+    { id: 1, name: 'John', isActive: true },
+    { id: 2, name: 'Jane', isActive: false },
+    { id: 3, name: 'Bob', isActive: true },
+  ])
+
+  // Good example: derived value calculated directly
+  const totalUsers = users.length
+
+  return (
+    <div>
+      <h1>User Management</h1>
+      <p>Total Users: {totalUsers}</p>
+
+      <div>
+        {users.map((user) => (
+          <div key={user.id}>
+            {user.name} - {user.isActive ? 'Active' : 'Inactive'}
+          </div>
+        ))}
+      </div>
+
+      <UserStats users={users} />
+    </div>
+  )
+}

evals/evals/agent-024-avoid-redundant-usestate/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-024-avoid-redundant-usestate/package.json

@@ -0,0 +1,30 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@testing-library/dom": "^10.4.1",
+    "@testing-library/react": "^16.1.0",
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "@vitejs/plugin-react": "^4.4.1",
+    "jsdom": "^27.4.0",
+    "typescript": "^5",
+    "vite-tsconfig-paths": "^5.1.4",
+    "vitest": "^3.1.3"
+  }
+}

evals/evals/agent-024-avoid-redundant-usestate/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules", "EVAL.tsx"]
+}

evals/evals/agent-025-prefer-next-link/EVAL.ts

@@ -0,0 +1,42 @@
+/**
+ * Prefer Next.js Link
+ *
+ * Tests whether the agent uses the Next.js Link component for internal
+ * navigation instead of plain <a> tags or programmatic router.push().
+ *
+ * Tricky because agents often use raw anchor tags or useRouter for simple
+ * navigation where Link provides prefetching and client-side transitions.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+test('Navigation component has required links', () => {
+  const content = readFileSync(
+    join(process.cwd(), 'app', 'Navigation.tsx'),
+    'utf-8'
+  )
+
+  // Should have links to /blog, /products, and /support
+  expect(content).toMatch(/['"]\/blog['"]/)
+  expect(content).toMatch(/['"]\/products['"]/)
+  expect(content).toMatch(/['"]\/support['"]/)
+})
+
+test('Navigation uses Next.js Link component', () => {
+  const content = readFileSync(
+    join(process.cwd(), 'app', 'Navigation.tsx'),
+    'utf-8'
+  )
+
+  // Should import Link from next/link
+  expect(content).toMatch(/import.*Link.*from ['"]next\/link['"]/)
+
+  // Should use Link components, not anchor tags for navigation
+  expect(content).toMatch(/<Link/)
+
+  // Should not use anchor tags for internal navigation
+  const anchorMatches = content.match(/<a [^>]*href=["']\/[^"']*["'][^>]*>/g)
+  expect(anchorMatches).toBeNull()
+})

evals/evals/agent-025-prefer-next-link/PROMPT.md

@@ -0,0 +1 @@
+Complete the Navigation component by adding links to /blog, /products, and /support pages. Follow the existing patterns in this codebase for navigation.

evals/evals/agent-025-prefer-next-link/app/Navigation.tsx

@@ -0,0 +1,10 @@
+export default function Navigation() {
+  return (
+    <div>
+      <h2>More Pages</h2>
+      {/* TODO: Add navigation links to /blog, /products, and /support pages */}
+      {/* Follow the existing patterns in this codebase */}
+      <p>Navigation links will go here</p>
+    </div>
+  )
+}

evals/evals/agent-025-prefer-next-link/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-025-prefer-next-link/app/page.tsx

@@ -0,0 +1,20 @@
+import Link from 'next/link'
+import Navigation from './Navigation'
+
+export default function Page() {
+  return (
+    <div>
+      <h1>Home</h1>
+
+      <nav>
+        <Link href="/about">About Us</Link>
+        <Link href="/contact">Contact</Link>
+      </nav>
+
+      <main>
+        <p>Welcome to our website!</p>
+        <Navigation />
+      </main>
+    </div>
+  )
+}

evals/evals/agent-025-prefer-next-link/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-025-prefer-next-link/package.json

@@ -0,0 +1,30 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@testing-library/dom": "^10.4.1",
+    "@testing-library/react": "^16.1.0",
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "@vitejs/plugin-react": "^4.4.1",
+    "jsdom": "^27.4.0",
+    "typescript": "^5",
+    "vite-tsconfig-paths": "^5.1.4",
+    "vitest": "^3.1.3"
+  }
+}

evals/evals/agent-025-prefer-next-link/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules", "EVAL.tsx"]
+}

evals/evals/agent-026-no-serial-await/EVAL.ts

@@ -0,0 +1,87 @@
+/**
+ * No Serial Await
+ *
+ * Tests whether the agent parallelizes independent data fetches with
+ * Promise.all() instead of using sequential await calls.
+ *
+ * Tricky because agents default to sequential await for multiple fetches,
+ * missing the opportunity to run independent requests concurrently.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+test('Page is an async server component with parallel data fetching', () => {
+  const pageContent = readFileSync(
+    join(process.cwd(), 'app', 'page.tsx'),
+    'utf-8'
+  )
+
+  // Should be an async function
+  expect(pageContent).toMatch(/async\s+function|export\s+default\s+async/)
+
+  // Should NOT have 'use client' directive
+  expect(pageContent).not.toMatch(/['"]use client['"];?/)
+
+  // Should use parallel data fetching with Promise.all
+  expect(pageContent).toMatch(/Promise\.all/)
+})
+
+test('Dashboard component fetches from all required APIs', () => {
+  const dashboardContent = readFileSync(
+    join(process.cwd(), 'app', 'Dashboard.tsx'),
+    'utf-8'
+  )
+
+  // Should fetch from all three APIs
+  expect(dashboardContent).toMatch(/\/api\/analytics/)
+  expect(dashboardContent).toMatch(/\/api\/notifications/)
+  expect(dashboardContent).toMatch(/\/api\/settings/)
+})
+
+test('Dashboard uses parallel fetching with Promise.all', () => {
+  const dashboardContent = readFileSync(
+    join(process.cwd(), 'app', 'Dashboard.tsx'),
+    'utf-8'
+  )
+
+  // Should use Promise.all for parallel fetching
+  expect(dashboardContent).toMatch(/Promise\.all/)
+
+  // Should NOT use sequential awaits (this pattern indicates serial fetching)
+  const sequentialAwaitPattern =
+    /await\s+fetch.*\n.*await\s+fetch.*\n.*await\s+fetch/
+  expect(dashboardContent).not.toMatch(sequentialAwaitPattern)
+})
+
+test('Dashboard is an async server component', () => {
+  const dashboardContent = readFileSync(
+    join(process.cwd(), 'app', 'Dashboard.tsx'),
+    'utf-8'
+  )
+
+  // Should be async function
+  expect(dashboardContent).toMatch(/async\s+function|export\s+default\s+async/)
+
+  // Should NOT have 'use client' directive
+  expect(dashboardContent).not.toMatch(/['"]use client['"];?/)
+})
+
+test('Dashboard displays data from all three sources', () => {
+  const dashboardContent = readFileSync(
+    join(process.cwd(), 'app', 'Dashboard.tsx'),
+    'utf-8'
+  )
+
+  // Should display or reference analytics, notifications, and settings
+  const displaysData =
+    (dashboardContent.includes('analytics') ||
+      dashboardContent.includes('Analytics')) &&
+    (dashboardContent.includes('notifications') ||
+      dashboardContent.includes('Notifications')) &&
+    (dashboardContent.includes('settings') ||
+      dashboardContent.includes('Settings'))
+
+  expect(displaysData).toBe(true)
+})

evals/evals/agent-026-no-serial-await/PROMPT.md

@@ -0,0 +1 @@
+The Dashboard component needs to fetch data from three APIs: /api/analytics, /api/notifications, and /api/settings. Implement this component to fetch and display this data efficiently. Follow the existing patterns in this codebase for data fetching.

evals/evals/agent-026-no-serial-await/app/Dashboard.tsx

@@ -0,0 +1,14 @@
+export default async function Dashboard() {
+  // TODO: This component needs to fetch data from multiple APIs:
+  // - /api/analytics
+  // - /api/notifications
+  // - /api/settings
+  // Follow the existing patterns in this codebase for data fetching
+
+  return (
+    <div>
+      <h2>Dashboard Content</h2>
+      <p>Dashboard data will go here</p>
+    </div>
+  )
+}

evals/evals/agent-026-no-serial-await/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-026-no-serial-await/app/page.tsx

@@ -0,0 +1,36 @@
+import Dashboard from './Dashboard'
+
+// Good example: parallel data fetching
+async function getParallelData() {
+  try {
+    const [users, posts, stats] = await Promise.all([
+      fetch('https://api.example.com/users').then((r) => r.json()),
+      fetch('https://api.example.com/posts').then((r) => r.json()),
+      fetch('https://api.example.com/stats').then((r) => r.json()),
+    ])
+
+    return { users, posts, stats }
+  } catch {
+    // Return mock data for build time
+    return {
+      users: [{ id: 1, name: 'User 1' }],
+      posts: [{ id: 1, title: 'Post 1' }],
+      stats: { views: 1000 },
+    }
+  }
+}
+
+export default async function Page() {
+  const { users, posts, stats } = await getParallelData()
+
+  return (
+    <div>
+      <h1>Dashboard</h1>
+      <p>Users: {users.length}</p>
+      <p>Posts: {posts.length}</p>
+      <p>Total views: {stats.views}</p>
+
+      <Dashboard />
+    </div>
+  )
+}

evals/evals/agent-026-no-serial-await/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-026-no-serial-await/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-026-no-serial-await/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-027-prefer-next-image/EVAL.ts

@@ -0,0 +1,48 @@
+/**
+ * Prefer Next.js Image
+ *
+ * Tests whether the agent uses the Next.js Image component from next/image
+ * instead of plain HTML <img> tags.
+ *
+ * Tricky because agents default to <img> tags, missing Next.js automatic
+ * image optimization, lazy loading, and responsive sizing.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+test('ProductGallery uses Next.js Image component', () => {
+  const galleryContent = readFileSync(
+    join(process.cwd(), 'app', 'ProductGallery.tsx'),
+    'utf-8'
+  )
+
+  // Should import Image from next/image
+  expect(galleryContent).toMatch(/import.*Image.*from ['"]next\/image['"]/)
+
+  // Should use Image components, not img tags
+  expect(galleryContent).toMatch(/<Image/)
+
+  // Should NOT use img tags for product images
+  expect(galleryContent).not.toMatch(/<img/)
+})
+
+test('ProductGallery has required image props', () => {
+  const galleryContent = readFileSync(
+    join(process.cwd(), 'app', 'ProductGallery.tsx'),
+    'utf-8'
+  )
+
+  // Should have width and height props
+  expect(galleryContent).toMatch(/width\s*=/)
+  expect(galleryContent).toMatch(/height\s*=/)
+
+  // Should have src prop using product imageUrl
+  expect(galleryContent).toMatch(
+    /src.*=.*product\.imageUrl|src.*=.*\{product\.imageUrl\}/
+  )
+
+  // Should have alt prop
+  expect(galleryContent).toMatch(/alt.*=/)
+})

evals/evals/agent-027-prefer-next-image/PROMPT.md

@@ -0,0 +1 @@
+Complete the ProductGallery component by adding images for each product. Each product image should be 300x200 pixels and use the imageUrl from the product data. Follow the existing patterns in this codebase for displaying images.

evals/evals/agent-027-prefer-next-image/app/ProductGallery.tsx

@@ -0,0 +1,23 @@
+export default function ProductGallery() {
+  const products = [
+    { id: 1, name: 'Product 1', imageUrl: '/product-1.jpg' },
+    { id: 2, name: 'Product 2', imageUrl: '/product-2.jpg' },
+    { id: 3, name: 'Product 3', imageUrl: '/product-3.jpg' },
+  ]
+
+  return (
+    <div>
+      <h3>Product Gallery</h3>
+      {/* TODO: Display the product images with their names */}
+      {/* Follow the existing patterns in this codebase for images */}
+      <div>
+        {products.map((product) => (
+          <div key={product.id}>
+            <h4>{product.name}</h4>
+            {/* Add product image here */}
+          </div>
+        ))}
+      </div>
+    </div>
+  )
+}

evals/evals/agent-027-prefer-next-image/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-027-prefer-next-image/app/page.tsx

@@ -0,0 +1,24 @@
+import Image from 'next/image'
+import ProductGallery from './ProductGallery'
+
+export default function Page() {
+  return (
+    <div>
+      <h1>Welcome to Our Store</h1>
+
+      {/* Good example: using Next.js Image component */}
+      <Image
+        src="/hero-image.jpg"
+        alt="Store hero image"
+        width={800}
+        height={400}
+        priority
+      />
+
+      <section>
+        <h2>Featured Products</h2>
+        <ProductGallery />
+      </section>
+    </div>
+  )
+}

evals/evals/agent-027-prefer-next-image/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-027-prefer-next-image/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-027-prefer-next-image/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-028-prefer-next-font/EVAL.ts

@@ -0,0 +1,50 @@
+/**
+ * Prefer Next.js Font
+ *
+ * Tests whether the agent uses next/font/google for font loading instead of
+ * external CDN links or CSS @import.
+ *
+ * Tricky because agents reach for Google Fonts CDN links or CSS imports
+ * instead of the zero-layout-shift next/font approach.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync } from 'fs'
+import { join } from 'path'
+
+test('BlogHeader uses Next.js fonts correctly', () => {
+  const blogHeaderContent = readFileSync(
+    join(process.cwd(), 'app', 'BlogHeader.tsx'),
+    'utf-8'
+  )
+
+  // Should import fonts from next/font/google
+  expect(blogHeaderContent).toMatch(/import.*from ['"]next\/font\/google['"]/)
+
+  // Should import Playfair_Display and Roboto
+  expect(blogHeaderContent).toMatch(/Playfair_Display/)
+  expect(blogHeaderContent).toMatch(/Roboto/)
+
+  // Should use .className to apply fonts
+  expect(blogHeaderContent).toMatch(/className.*\.className/)
+
+  // Should NOT use external CSS links or font-family styles
+  expect(blogHeaderContent).not.toMatch(/@import|font-family|link.*font/)
+})
+
+test('BlogHeader has font configuration', () => {
+  const blogHeaderContent = readFileSync(
+    join(process.cwd(), 'app', 'BlogHeader.tsx'),
+    'utf-8'
+  )
+
+  // Should configure fonts with subsets
+  expect(blogHeaderContent).toMatch(/subsets.*latin/)
+
+  // Should create font instances
+  expect(blogHeaderContent).toMatch(/const.*=.*\(/)
+
+  // Should apply different fonts to different elements
+  expect(blogHeaderContent).toMatch(/className.*playfair/i)
+  expect(blogHeaderContent).toMatch(/className.*roboto/i)
+})

evals/evals/agent-028-prefer-next-font/PROMPT.md

@@ -0,0 +1 @@
+Add a custom font to the BlogHeader. Use 'Playfair Display' for the main heading and 'Roboto' for the subtitle. Add all of the fonts and logic inside BlogHeader.

evals/evals/agent-028-prefer-next-font/app/BlogHeader.tsx

@@ -0,0 +1,8 @@
+export default function BlogHeader() {
+  return (
+    <header>
+      <h1>My Personal Blog</h1>
+      <p>Thoughts, ideas, and musings</p>
+    </header>
+  )
+}

evals/evals/agent-028-prefer-next-font/app/layout.tsx

@@ -0,0 +1,11 @@
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-028-prefer-next-font/app/page.tsx

@@ -0,0 +1,17 @@
+import BlogHeader from './BlogHeader'
+
+export default function Page() {
+  return (
+    <div>
+      <h1>My Blog</h1>
+      <p>Welcome to my personal blog where I share my thoughts and ideas.</p>
+
+      <BlogHeader />
+
+      <article>
+        <h2>Latest Post</h2>
+        <p>This is the content of my latest blog post...</p>
+      </article>
+    </div>
+  )
+}

evals/evals/agent-028-prefer-next-font/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-028-prefer-next-font/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-028-prefer-next-font/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-029-use-cache-directive/EVAL.ts

@@ -0,0 +1,111 @@
+/**
+ * Use Cache Directive
+ *
+ * Generic behavior checks for this scenario:
+ * - product reads use cache + cacheTag("products")
+ * - getAllProducts() from lib/db is used
+ * - an inline Server Action flow exists and is form-triggered
+ * - revalidateTag("products", profile) is used
+ * - updateTag is not used
+ */
+
+import { expect, test } from 'vitest'
+import { existsSync, readdirSync, readFileSync, statSync } from 'fs'
+import { join } from 'path'
+
+type SourceFile = { path: string; content: string }
+
+const IGNORE_DIRS = new Set([
+  '.git',
+  '.next',
+  'node_modules',
+  'dist',
+  'build',
+  'coverage',
+])
+
+const IGNORE_FILES = new Set(['EVAL.ts', 'PROMPT.md'])
+
+function readSourceFiles(dir: string): SourceFile[] {
+  if (!existsSync(dir)) return []
+
+  const files: SourceFile[] = []
+  for (const entry of readdirSync(dir)) {
+    if (IGNORE_DIRS.has(entry)) continue
+
+    const fullPath = join(dir, entry)
+    const stats = statSync(fullPath)
+
+    if (stats.isDirectory()) {
+      files.push(...readSourceFiles(fullPath))
+      continue
+    }
+
+    if (IGNORE_FILES.has(entry)) continue
+
+    if (/\.(ts|tsx|js|jsx)$/.test(entry)) {
+      files.push({
+        path: fullPath,
+        content: readFileSync(fullPath, 'utf-8'),
+      })
+    }
+  }
+
+  return files
+}
+
+const sourceFiles = readSourceFiles(process.cwd())
+const source = sourceFiles.map((file) => file.content).join('\n')
+
+function fileWith(pattern: RegExp): SourceFile | undefined {
+  return sourceFiles.find((file) => pattern.test(file.content))
+}
+
+test('Catalog reads use use-cache directive and products cache tag', () => {
+  // Allow caching logic to live in app or lib helper modules.
+  expect(source).toMatch(/['"]use cache['"];?/)
+
+  // Tagged invalidation should target the required products key.
+  expect(source).toMatch(/cacheTag\s*\(\s*['"]products['"]\s*\)/)
+})
+
+test('Page fetches products via lib/db', () => {
+  // Keep data source expectation explicit without location assumptions.
+  expect(source).toMatch(/import.*getAllProducts.*lib\/db|from.*lib\/db/)
+  expect(source).toMatch(/await\s+getAllProducts\s*\(|getAllProducts\s*\(/)
+})
+
+test('Inline form-triggered Server Action flow exists', () => {
+  const inlineActionFile = sourceFiles.find((file) => {
+    return (
+      /<form[\s\S]*action\s*=\s*\{/.test(file.content) &&
+      /['"]use server['"];?/.test(file.content) &&
+      (/async\s+function\s+\w+/.test(file.content) ||
+        /const\s+\w+\s*=\s*async\s*\(/.test(file.content))
+    )
+  })
+
+  expect(
+    inlineActionFile,
+    'Expected one file to contain form action={...} and inline Server Action markers'
+  ).toBeDefined()
+})
+
+test('Server Action revalidates products using revalidateTag profile', () => {
+  const revalidateFile = fileWith(/revalidateTag\s*\(/)
+  expect(revalidateFile, 'Expected source to call revalidateTag').toBeDefined()
+
+  // The chosen API should be revalidateTag in this workflow.
+  expect(revalidateFile?.content ?? '').toMatch(
+    /import.*revalidateTag.*from\s+['"]next\/cache['"]/
+  )
+  expect(revalidateFile?.content ?? '').toMatch(/revalidateTag\s*\(/)
+
+  // Require the same explicit products tag and a profile/second argument.
+  expect(revalidateFile?.content ?? '').toMatch(
+    /revalidateTag\s*\(\s*['"]products['"]\s*,/
+  )
+
+  // Avoid read-your-own-writes invalidation API in this scenario.
+  expect(source).not.toMatch(/\bupdateTag\s*\(/)
+})

evals/evals/agent-029-use-cache-directive/PROMPT.md

@@ -0,0 +1,12 @@
+You are building an admin product catalog page for an e-commerce team.
+
+The catalog is read-heavy and should feel fast for day-to-day browsing, so avoid re-querying product data on every request.
+
+Admins can trigger a "Sync latest catalog" action from the page when upstream ERP/PIM data changes (pricing, inventory, availability). After submitting, they should be able to continue working immediately, even if the product list is briefly stale.
+
+The expected behavior is that product data is refreshed in the background and becomes up to date shortly after, across any views that depend on the same catalog data.
+
+Implement the sync trigger as a regular HTML form that uses an inline Server Action in the page.
+Use the cache tag name `"products"` consistently for catalog caching and invalidation.
+
+In practice, sync jobs can touch thousands of SKUs, so operators prioritize a responsive admin experience and eventual consistency across catalog views over forcing every request to block on freshly recomputed data.

evals/evals/agent-029-use-cache-directive/app/layout.tsx

@@ -0,0 +1,18 @@
+import type { Metadata } from 'next'
+
+export const metadata: Metadata = {
+  title: 'Create Next App',
+  description: 'Generated by create next app',
+}
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode
+}>) {
+  return (
+    <html lang="en">
+      <body>{children}</body>
+    </html>
+  )
+}

evals/evals/agent-029-use-cache-directive/app/page.tsx

@@ -0,0 +1,7 @@
+export default function Page() {
+  return (
+    <div>
+      <h1>Home</h1>
+    </div>
+  )
+}

evals/evals/agent-029-use-cache-directive/lib/db.js

@@ -0,0 +1,12 @@
+export async function getAllProducts() {
+  // Simulate database delay
+  await new Promise((resolve) => setTimeout(resolve, 100))
+
+  return [
+    { id: 1, name: 'Laptop', price: 999 },
+    { id: 2, name: 'Phone', price: 699 },
+    { id: 3, name: 'Tablet', price: 499 },
+    { id: 4, name: 'Headphones', price: 199 },
+    { id: 5, name: 'Watch', price: 299 },
+  ]
+}

evals/evals/agent-029-use-cache-directive/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-029-use-cache-directive/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-029-use-cache-directive/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}

evals/evals/agent-030-app-router-migration-hard/EVAL.ts

@@ -0,0 +1,182 @@
+/**
+ * App Router Migration (Hard)
+ *
+ * Tests a comprehensive Pages-to-App Router migration including data fetching
+ * (getServerSideProps/getStaticProps), dynamic routes, API routes, custom
+ * _app/_document, metadata, and client/server component separation.
+ *
+ * Tricky because it requires migrating multiple advanced patterns at once:
+ * data fetching, route handlers, metadata API, and proper 'use client'
+ * directive placement — all while maintaining functionality.
+ */
+
+import { expect, test } from 'vitest'
+import { readFileSync, existsSync } from 'fs'
+import { join } from 'path'
+
+/** Strip JS/TS comments so we only test actual code, not migration notes */
+function stripComments(code: string): string {
+  return code.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/.*$/gm, '')
+}
+
+test('Root layout exists and replaces _app/_document', () => {
+  const layoutPath = join(process.cwd(), 'app', 'layout.tsx')
+  expect(existsSync(layoutPath)).toBe(true)
+
+  const layoutContent = readFileSync(layoutPath, 'utf-8')
+
+  // Should have html and body tags (replacing _document.js)
+  expect(layoutContent).toMatch(/<html.*lang/)
+  expect(layoutContent).toMatch(/<body/)
+
+  // Should include metadata (replacing Head in _document.js)
+  expect(layoutContent).toMatch(/metadata|Metadata/)
+
+  // Should accept children prop with ReactNode type
+  expect(layoutContent).toMatch(/children.*ReactNode/)
+})
+
+test('Home page migrated to Server Component with async data fetching', () => {
+  const pagePath = join(process.cwd(), 'app', 'page.tsx')
+  expect(existsSync(pagePath)).toBe(true)
+
+  const pageContent = readFileSync(pagePath, 'utf-8')
+
+  // Should be async Server Component
+  expect(pageContent).toMatch(
+    /export\s+default\s+async\s+function|async\s+function.*Page/
+  )
+
+  // Should NOT have 'use client' directive
+  expect(pageContent).not.toMatch(/['"]use client['"];?/)
+
+  // Should use fetch instead of getServerSideProps
+  expect(pageContent).toMatch(/await\s+fetch|fetch\(/)
+
+  // Should not have getServerSideProps in actual code (comments OK)
+  expect(stripComments(pageContent)).not.toMatch(/getServerSideProps/)
+})
+
+test('Blog index migrated with ISR equivalent', () => {
+  const blogPath = join(process.cwd(), 'app', 'blog', 'page.tsx')
+  expect(existsSync(blogPath)).toBe(true)
+
+  const blogContent = readFileSync(blogPath, 'utf-8')
+
+  // Should be async Server Component
+  expect(blogContent).toMatch(
+    /export\s+default\s+async\s+function|async\s+function/
+  )
+
+  // Should use revalidate for ISR
+  expect(blogContent).toMatch(
+    /revalidate.*\d+|next.*revalidate|export.*const.*revalidate.*=.*\d+/
+  )
+
+  // Should not have getStaticProps in actual code (comments OK)
+  expect(stripComments(blogContent)).not.toMatch(/getStaticProps/)
+})
+
+test('Dynamic blog route migrated to generateStaticParams', () => {
+  const dynamicPath = join(process.cwd(), 'app', 'blog', '[id]', 'page.tsx')
+  expect(existsSync(dynamicPath)).toBe(true)
+
+  const dynamicContent = readFileSync(dynamicPath, 'utf-8')
+
+  // Should export generateStaticParams
+  expect(dynamicContent).toMatch(
+    /export.*generateStaticParams|generateStaticParams.*export/
+  )
+
+  // Should be async Server Component
+  expect(dynamicContent).toMatch(
+    /export\s+default\s+async\s+function|async\s+function/
+  )
+
+  // Should not have getStaticPaths or getStaticProps in actual code (comments OK)
+  expect(stripComments(dynamicContent)).not.toMatch(
+    /getStaticPaths|getStaticProps/
+  )
+})
+
+test('API routes migrated to Route Handlers', () => {
+  // Check posts index route
+  const postsRoutePath = join(process.cwd(), 'app', 'api', 'posts', 'route.ts')
+  expect(existsSync(postsRoutePath)).toBe(true)
+
+  const postsRouteContent = readFileSync(postsRoutePath, 'utf-8')
+
+  // Should export HTTP method functions
+  expect(postsRouteContent).toMatch(/export.*GET|export.*POST/)
+
+  // Should use Request/Response or Next APIs
+  expect(postsRouteContent).toMatch(/Request|Response|NextRequest|NextResponse/)
+
+  // Check dynamic API route
+  const dynamicApiPath = join(
+    process.cwd(),
+    'app',
+    'api',
+    'posts',
+    '[id]',
+    'route.ts'
+  )
+  expect(existsSync(dynamicApiPath)).toBe(true)
+
+  const dynamicApiContent = readFileSync(dynamicApiPath, 'utf-8')
+
+  // Should export HTTP methods
+  expect(dynamicApiContent).toMatch(/export.*GET|export.*PUT|export.*DELETE/)
+})
+
+test('Metadata API replaces next/head', () => {
+  const pagePath = join(process.cwd(), 'app', 'page.tsx')
+  const pageContent = readFileSync(pagePath, 'utf-8')
+
+  // Should use Metadata export instead of Head component
+  expect(pageContent).toMatch(/export.*metadata|metadata.*Metadata/)
+
+  // Should not import or use next/head
+  expect(pageContent).not.toMatch(/import.*Head.*next\/head|<Head>/)
+
+  // Check blog page too
+  const blogPath = join(process.cwd(), 'app', 'blog', 'page.tsx')
+  if (existsSync(blogPath)) {
+    const blogContent = readFileSync(blogPath, 'utf-8')
+    expect(blogContent).toMatch(/export.*metadata|metadata.*Metadata/)
+    expect(blogContent).not.toMatch(/import.*Head.*next\/head|<Head>/)
+  }
+})
+
+test('Error handling migrated to error.js and not-found.js', () => {
+  // Check for error.js file
+  const errorPath = join(process.cwd(), 'app', 'error.tsx')
+  expect(existsSync(errorPath)).toBe(true)
+
+  const errorContent = readFileSync(errorPath, 'utf-8')
+
+  // Should be a Client Component for error boundaries
+  expect(errorContent).toMatch(/['"]use client['"];?/)
+
+  // Should accept error props
+  expect(errorContent).toMatch(/error.*Error|Error.*error/)
+
+  // Check for not-found.js file
+  const notFoundPath = join(process.cwd(), 'app', 'not-found.tsx')
+  expect(existsSync(notFoundPath)).toBe(true)
+})
+
+test('Client components use next/navigation hooks', () => {
+  // Check specific client component that should use useRouter
+  const homeClientPath = join(process.cwd(), 'app', 'home-client.tsx')
+
+  if (existsSync(homeClientPath)) {
+    const content = readFileSync(homeClientPath, 'utf-8')
+
+    if (content.includes('useRouter')) {
+      // Should import from next/navigation, not next/router
+      expect(content).toMatch(/import.*useRouter.*next\/navigation/)
+      expect(content).not.toMatch(/import.*useRouter.*next\/router/)
+    }
+  }
+})

evals/evals/agent-030-app-router-migration-hard/PROMPT.md

@@ -0,0 +1 @@
+Migrate every route and file from the Pages Router to the Next.js App Router. When finished, remove the pages dir entirely. Ensure the proper App Router APIs are used. If a Pages Router API was used that no longer exists in App Router, replace it with the newer version or the new pattern. Make sure to add types.

evals/evals/agent-030-app-router-migration-hard/components/AppProvider.js

@@ -0,0 +1,21 @@
+import { createContext, useContext, useState } from 'react'
+
+const AppContext = createContext()
+
+export function AppProvider({ children }) {
+  const [theme, setTheme] = useState('light')
+
+  return (
+    <AppContext.Provider value={{ theme, setTheme }}>
+      {children}
+    </AppContext.Provider>
+  )
+}
+
+export function useAppContext() {
+  const context = useContext(AppContext)
+  if (!context) {
+    throw new Error('useAppContext must be used within AppProvider')
+  }
+  return context
+}

evals/evals/agent-030-app-router-migration-hard/next.config.ts

@@ -0,0 +1,5 @@
+import type { NextConfig } from 'next'
+
+const nextConfig: NextConfig = {}
+
+export default nextConfig

evals/evals/agent-030-app-router-migration-hard/package.json

@@ -0,0 +1,27 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start"
+  },
+  "dependencies": {
+    "@ai-sdk/react": "2.0.0-beta.28",
+    "@modelcontextprotocol/sdk": "^1.17.0",
+    "ai": "5.0.59",
+    "next": "^16",
+    "react": "19.1.0",
+    "react-dom": "19.1.0",
+    "zod": "^4.0.10"
+  },
+  "devDependencies": {
+    "@types/node": "^20",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "typescript": "^5",
+    "vitest": "^3.1.3",
+    "@vitejs/plugin-react": "^4.4.1",
+    "vite-tsconfig-paths": "^5.1.4"
+  }
+}

evals/evals/agent-030-app-router-migration-hard/pages/404.js

@@ -0,0 +1,24 @@
+import Head from 'next/head'
+import { useRouter } from 'next/router'
+
+export default function Custom404() {
+  const router = useRouter()
+
+  return (
+    <>
+      <Head>
+        <title>404 - Page Not Found</title>
+        <meta
+          name="description"
+          content="The page you're looking for doesn't exist"
+        />
+      </Head>
+
+      <div className="error-page">
+        <h1>404 - Page Not Found</h1>
+        <p>The page you&apos;re looking for doesn&apos;t exist.</p>
+        <button onClick={() => router.push('/')}>Go Back Home</button>
+      </div>
+    </>
+  )
+}

evals/evals/agent-030-app-router-migration-hard/pages/_app.js

@@ -0,0 +1,25 @@
+import Link from 'next/link'
+import { AppProvider } from '../components/AppProvider'
+import '../styles/globals.css'
+
+export default function MyApp({ Component, pageProps }) {
+  return (
+    <AppProvider>
+      <div className="app-container">
+        <header>
+          <h1>My Blog</h1>
+          <nav>
+            <Link href="/">Home</Link>
+            <Link href="/blog">Blog</Link>
+          </nav>
+        </header>
+        <main>
+          <Component {...pageProps} />
+        </main>
+        <footer>
+          <p>&copy; 2024 My Blog</p>
+        </footer>
+      </div>
+    </AppProvider>
+  )
+}

evals/evals/agent-030-app-router-migration-hard/pages/_document.js

@@ -0,0 +1,21 @@
+import { Html, Head, Main, NextScript } from 'next/document'
+
+export default function Document() {
+  return (
+    <Html lang="en">
+      <Head>
+        <meta name="description" content="A complex blog application" />
+        <link rel="icon" href="/favicon.ico" />
+        <link rel="preconnect" href="https://fonts.googleapis.com" />
+        <link
+          href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&display=swap"
+          rel="stylesheet"
+        />
+      </Head>
+      <body>
+        <Main />
+        <NextScript />
+      </body>
+    </Html>
+  )
+}

evals/evals/agent-030-app-router-migration-hard/pages/_error.js

@@ -0,0 +1,31 @@
+import Head from 'next/head'
+
+function Error({ statusCode, hasGetInitialPropsRun, err }) {
+  return (
+    <>
+      <Head>
+        <title>Error {statusCode}</title>
+      </Head>
+
+      <div className="error-page">
+        <h1>
+          {statusCode
+            ? `A ${statusCode} error occurred on server`
+            : 'An error occurred on client'}
+        </h1>
+        <p>
+          {statusCode === 404
+            ? 'This page could not be found.'
+            : 'Sorry, something went wrong.'}
+        </p>
+      </div>
+    </>
+  )
+}
+
+Error.getInitialProps = ({ res, err }) => {
+  const statusCode = res ? res.statusCode : err ? err.statusCode : 404
+  return { statusCode }
+}
+
+export default Error

evals/evals/agent-030-app-router-migration-hard/pages/api/posts/[id].js

@@ -0,0 +1,33 @@
+export default function handler(req, res) {
+  const { id } = req.query
+
+  if (req.method === 'GET') {
+    // Simulate fetching a single post
+    const post = {
+      id: parseInt(id),
+      title: `Post ${id}`,
+      content: `This is the content for post ${id}`,
+      createdAt: new Date().toISOString(),
+    }
+
+    res.status(200).json(post)
+  } else if (req.method === 'PUT') {
+    const { title, content } = req.body
+
+    // Simulate updating a post
+    const updatedPost = {
+      id: parseInt(id),
+      title: title || `Post ${id}`,
+      content: content || `Updated content for post ${id}`,
+      updatedAt: new Date().toISOString(),
+    }
+
+    res.status(200).json(updatedPost)
+  } else if (req.method === 'DELETE') {
+    // Simulate deleting a post
+    res.status(200).json({ message: `Post ${id} deleted successfully` })
+  } else {
+    res.setHeader('Allow', ['GET', 'PUT', 'DELETE'])
+    res.status(405).end(`Method ${req.method} Not Allowed`)
+  }
+}

evals/evals/agent-030-app-router-migration-hard/pages/api/posts/index.js

@@ -0,0 +1,30 @@
+export default function handler(req, res) {
+  if (req.method === 'GET') {
+    // Simulate fetching posts
+    const posts = [
+      { id: 1, title: 'First Post', content: 'This is the first post' },
+      { id: 2, title: 'Second Post', content: 'This is the second post' },
+    ]
+
+    res.status(200).json(posts)
+  } else if (req.method === 'POST') {
+    const { title, content } = req.body
+
+    if (!title || !content) {
+      return res.status(400).json({ error: 'Title and content are required' })
+    }
+
+    // Simulate creating a post
+    const newPost = {
+      id: Date.now(),
+      title,
+      content,
+      createdAt: new Date().toISOString(),
+    }
+
+    res.status(201).json(newPost)
+  } else {
+    res.setHeader('Allow', ['GET', 'POST'])
+    res.status(405).end(`Method ${req.method} Not Allowed`)
+  }
+}

evals/evals/agent-030-app-router-migration-hard/pages/blog/[id].js

@@ -0,0 +1,79 @@
+import Head from 'next/head'
+import { useRouter } from 'next/router'
+
+export async function getStaticPaths() {
+  const posts = await fetch('https://jsonplaceholder.typicode.com/posts').then(
+    (res) => res.json()
+  )
+
+  const paths = posts.slice(0, 10).map((post) => ({
+    params: { id: post.id.toString() },
+  }))
+
+  return {
+    paths,
+    fallback: 'blocking',
+  }
+}
+
+export async function getStaticProps({ params }) {
+  try {
+    const [post, comments] = await Promise.all([
+      fetch(`https://jsonplaceholder.typicode.com/posts/${params.id}`).then(
+        (res) => res.json()
+      ),
+      fetch(
+        `https://jsonplaceholder.typicode.com/posts/${params.id}/comments`
+      ).then((res) => res.json()),
+    ])
+
+    return {
+      props: {
+        post,
+        comments,
+      },
+      revalidate: 300, // 5 minutes
+    }
+  } catch (error) {
+    return {
+      notFound: true,
+    }
+  }
+}
+
+export default function BlogPost({ post, comments }) {
+  const router = useRouter()
+
+  if (router.isFallback) {
+    return <div>Loading...</div>
+  }
+
+  return (
+    <>
+      <Head>
+        <title>{post.title} - My Blog</title>
+        <meta name="description" content={post.body.substring(0, 160)} />
+        <meta property="og:title" content={post.title} />
+        <meta property="og:description" content={post.body.substring(0, 160)} />
+      </Head>
+
+      <article>
+        <button onClick={() => router.back()}>← Back</button>
+
+        <h1>{post.title}</h1>
+        <p>{post.body}</p>
+
+        <h2>Comments ({comments.length})</h2>
+        <div className="comments">
+          {comments.map((comment) => (
+            <div key={comment.id} className="comment">
+              <h3>{comment.name}</h3>
+              <p>{comment.body}</p>
+              <small>By: {comment.email}</small>
+            </div>
+          ))}
+        </div>
+      </article>
+    </>
+  )
+}

evals/evals/agent-030-app-router-migration-hard/pages/blog/index.js

@@ -0,0 +1,46 @@
+import Head from 'next/head'
+import { useRouter } from 'next/router'
+
+export async function getStaticProps() {
+  const posts = await fetch('https://jsonplaceholder.typicode.com/posts').then(
+    (res) => res.json()
+  )
+
+  return {
+    props: {
+      posts,
+    },
+    revalidate: 60, // ISR with 60 second revalidation
+  }
+}
+
+export default function BlogIndex({ posts }) {
+  const router = useRouter()
+
+  return (
+    <>
+      <Head>
+        <title>Blog - My Blog</title>
+        <meta name="description" content="All blog posts" />
+      </Head>
+
+      <div>
+        <h1>All Blog Posts</h1>
+
+        <div className="posts-grid">
+          {posts.map((post) => (
+            <div key={post.id} className="post-card">
+              <h2>
+                <a href={`/blog/${post.id}`}>{post.title}</a>
+              </h2>
+              <p>{post.body.substring(0, 100)}...</p>
+              <button onClick={() => router.push(`/blog/${post.id}`)}>
+                Read More
+              </button>
+            </div>
+          ))}
+        </div>
+      </div>
+    </>
+  )
+}

evals/evals/agent-030-app-router-migration-hard/pages/index.js

@@ -0,0 +1,52 @@
+import Head from 'next/head'
+import { useRouter } from 'next/router'
+
+export async function getServerSideProps({ req }) {
+  const userAgent = req.headers['user-agent'] || ''
+  const posts = await fetch(
+    'https://jsonplaceholder.typicode.com/posts?_limit=5'
+  ).then((res) => res.json())
+
+  return {
+    props: {
+      posts,
+      userAgent,
+      timestamp: new Date().toISOString(),
+    },
+  }
+}
+
+export default function HomePage({ posts, userAgent, timestamp }) {
+  const router = useRouter()
+
+  const navigateToBlog = () => {
+    router.push('/blog')
+  }
+
+  return (
+    <>
+      <Head>
+        <title>Home - My Blog</title>
+        <meta name="description" content="Welcome to my blog homepage" />
+        <meta property="og:title" content="Home - My Blog" />
+      </Head>
+
+      <div>
+        <h1>Welcome to My Blog</h1>
+        <p>Server-side rendered at: {timestamp}</p>
+        <p>Your user agent: {userAgent}</p>
+
+        <h2>Recent Posts</h2>
+        <ul>
+          {posts.map((post) => (
+            <li key={post.id}>
+              <a href={`/blog/${post.id}`}>{post.title}</a>
+            </li>
+          ))}
+        </ul>
+
+        <button onClick={navigateToBlog}>View All Posts</button>
+      </div>
+    </>
+  )
+}

evals/evals/agent-030-app-router-migration-hard/styles/globals.css

@@ -0,0 +1,112 @@
+* {
+  box-sizing: border-box;
+  margin: 0;
+  padding: 0;
+}
+
+body {
+  font-family: 'Inter', sans-serif;
+  line-height: 1.6;
+  color: #333;
+}
+
+.app-container {
+  max-width: 1200px;
+  margin: 0 auto;
+  padding: 0 20px;
+}
+
+header {
+  padding: 20px 0;
+  border-bottom: 1px solid #eee;
+  margin-bottom: 40px;
+}
+
+header h1 {
+  margin-bottom: 10px;
+}
+
+nav a {
+  margin-right: 20px;
+  text-decoration: none;
+  color: #0070f3;
+}
+
+nav a:hover {
+  text-decoration: underline;
+}
+
+.posts-grid {
+  display: grid;
+  grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
+  gap: 20px;
+  margin-top: 20px;
+}
+
+.post-card {
+  border: 1px solid #eee;
+  padding: 20px;
+  border-radius: 8px;
+}
+
+.post-card h2 {
+  margin-bottom: 10px;
+}
+
+.post-card a {
+  text-decoration: none;
+  color: #333;
+}
+
+.post-card a:hover {
+  color: #0070f3;
+}
+
+.comments {
+  margin-top: 30px;
+}
+
+.comment {
+  border: 1px solid #eee;
+  padding: 15px;
+  margin-bottom: 15px;
+  border-radius: 8px;
+}
+
+.comment h3 {
+  margin-bottom: 5px;
+  font-size: 16px;
+}
+
+.comment small {
+  color: #666;
+}
+
+.error-page {
+  text-align: center;
+  padding: 100px 20px;
+}
+
+.error-page h1 {
+  margin-bottom: 20px;
+  font-size: 2rem;
+}
+
+.error-page p {
+  margin-bottom: 30px;
+  font-size: 1.1rem;
+}
+
+button {
+  background: #0070f3;
+  color: white;
+  border: none;
+  padding: 10px 20px;
+  border-radius: 5px;
+  cursor: pointer;
+  font-size: 16px;
+}
+
+button:hover {
+  background: #0051a2;
+}

evals/evals/agent-030-app-router-migration-hard/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}