first commit
Some checks failed
Test examples / Test Examples (20) (push) Has been cancelled
Test examples / Test Examples (22) (push) Has been cancelled
Lock Threads / action (push) Has been cancelled
Trigger Release / start (push) Has been cancelled
Stale issue handler / stale (push) Has been cancelled
Update Font Data / create-pull-request (push) Has been cancelled
build-and-deploy / deploy-target (push) Has been cancelled
build-and-deploy / build (push) Has been cancelled
build-and-deploy / stable - aarch64-unknown-linux-musl - node@16 (push) Has been cancelled
build-and-deploy / stable - x86_64-unknown-linux-musl - node@16 (push) Has been cancelled
build-and-deploy / stable - aarch64-unknown-linux-gnu - node@16 (push) Has been cancelled
build-and-deploy / stable - x86_64-unknown-linux-gnu - node@16 (push) Has been cancelled
build-and-deploy / stable - aarch64-pc-windows-msvc - node@16 (push) Has been cancelled
build-and-deploy / stable - x86_64-pc-windows-msvc - node@16 (push) Has been cancelled
build-and-deploy / stable - aarch64-apple-darwin - node@16 (push) Has been cancelled
build-and-deploy / stable - x86_64-apple-darwin - node@16 (push) Has been cancelled
build-and-deploy / build-wasm (nodejs) (push) Has been cancelled
build-and-deploy / build-wasm (web) (push) Has been cancelled
build-and-deploy / Deploy preview tarball (push) Has been cancelled
build-and-deploy / Potentially publish release (push) Has been cancelled
build-and-deploy / publish-turbopack-npm-packages (push) Has been cancelled
build-and-deploy / Deploy examples (push) Has been cancelled
build-and-deploy / thank you, build (push) Has been cancelled
build-and-deploy / Upload Turbopack Bytesize metrics to Datadog (push) Has been cancelled
Rspack Next.js development integration tests / Rspack integration tests (push) Has been cancelled
Rspack Next.js production integration tests / Rspack integration tests (push) Has been cancelled
Turbopack Next.js development integration tests / Next.js integration tests (push) Has been cancelled
Turbopack Next.js production integration tests / Next.js integration tests (push) Has been cancelled
Update Rspack test manifest / Update and upload Rspack development test manifest (push) Has been cancelled
Update Rspack test manifest / Update and upload Rspack production test manifest (push) Has been cancelled
Upload bundler test manifests to areweturboyet.com / Upload test results (push) Has been cancelled
Update React / create-pull-request (push) Has been cancelled
test-e2e-project-reset-cron / reset-test-project (push) Has been cancelled
Notify about the top 15 issues/PRs/feature requests (most reacted) in the last 90 days / run (push) Has been cancelled

This commit is contained in:
Arian Tron
2026-03-10 19:37:31 +03:30
commit 61f56f997c
27684 changed files with 2784175 additions and 0 deletions

View File

@@ -0,0 +1,31 @@
# syntax=docker.io/docker/dockerfile:1
FROM ubuntu:22.04
LABEL com.github.actions.name="Next.js PR Stats"
LABEL com.github.actions.description="Compares stats of a PR with the main branch"
LABEL repository="https://github.com/vercel/next-stats-action"
RUN apt update && apt upgrade -y
RUN apt install unzip wget curl nano htop screen build-essential pkg-config libssl-dev git build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libreadline-dev libffi-dev python3 moreutils jq iproute2 openssh-server sudo whois dnsutils apache2-utils -y
RUN ln $(which python3) /usr/bin/python
RUN curl -sfLS https://install-node.vercel.app/v20.9.0 | bash -s -- -f
RUN npm i -g corepack@0.31
RUN corepack enable
WORKDIR /next-stats
# Install node_modules
COPY package.json ./
RUN pnpm install --production
# caching optimization
COPY . .
RUN git config --global user.email 'stats@localhost' && \
git config --global user.name 'next stats'
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

View File

@@ -0,0 +1,93 @@
# Next.js Stats GitHub Action
> Downloads and runs project with provided configs gathering stats to compare branches
See it in action at Next.js https://github.com/vercel/next.js
## Getting Started
1. Add a `.stats-app` folder to your project with a [`stats-config.js`](#stats-config) and any files to run against for example a test app that is to be built
2. Add the action to your [workflow](https://help.github.com/en/articles/configuring-a-workflow)
3. Enjoy the stats
## Stats Config
```TypeScript
const StatsConfig = {
// the Heading to show at the top of stats comments
commentHeading: 'Stats from current PR' | undefined,
commentReleaseHeading: 'Stats from current release' | undefined,
// the command to build your project if not done on post install
initialBuildCommand: undefined | string,
skipInitialInstall: undefined | boolean,
// the command to build the app (app source should be in `.stats-app`)
appBuildCommand: string,
appStartCommand: string | undefined,
// the main branch to compare against (what PRs will be merging into)
mainBranch: 'canary',
// the main repository path (relative to https://github.com/)
mainRepo: 'vercel/next.js',
// whether to attempt auto merging the main branch into PR before running stats
autoMergeMain: boolean | undefined,
// an array of configs for each run
configs: [
{ // first run's config
// title of the run
title: 'fastMode stats',
// whether to diff the outputted files (default: onOutputChange)
diff: 'onOutputChange' | false | undefined,
// config files to add before running diff (if `undefined` uses `configFiles`)
diffConfigFiles: [] | undefined,
// renames to apply to make file names deterministic
renames: [
{
srcGlob: 'main-*.js',
dest: 'main.js'
}
],
// config files to add before running (removed before successive runs)
configFiles: [
{
path: './next.config.js',
content: 'module.exports = { fastMode: true }'
}
],
// an array of file groups to diff/track
filesToTrack: [
{
name: 'Pages',
globs: [
'build/pages/**/*.js'
]
}
],
// an array of URLs to fetch while `appStartCommand` is running
// will be output to fetched-pages/${pathname}.html
pagesToFetch: [
'https://localhost:$PORT/page-1'
]
},
{ // second run's config
title: 'slowMode stats',
diff: false,
configFiles: [
{
path: './next.config.js',
content: 'module.exports = { slowMode: true }'
}
],
filesToTrack: [
{
name: 'Main Bundles',
globs: [
'build/runtime/webpack-*.js',
'build/runtime/main-*.js',
]
}
]
},
]
}
module.exports = StatsConfig
```

View File

@@ -0,0 +1,10 @@
name: 'Next.js PR Stats'
description: 'Compare PR stats with canary'
inputs:
bundler:
description: 'Bundler to benchmark (webpack|turbopack|both)'
default: 'both'
required: false
runs:
using: 'docker'
image: 'Dockerfile'

View File

@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -eu # stop on error
export HOME=/root
node /next-stats/src/index.js

View File

@@ -0,0 +1,4 @@
# Ignore everything in this directory
*
# Except this file
!.gitignore

View File

@@ -0,0 +1,25 @@
{
"private": true,
"main": "src/index.js",
"dependencies": {
"@vercel/kv": "^1.0.1",
"async-sema": "^3.1.0",
"execa": "2.0.3",
"get-port": "^5.0.0",
"glob": "^7.1.4",
"gzip-size": "^5.1.1",
"minimatch": "^3.0.4",
"node-fetch": "^2.6.0",
"prettier": "^2.8.4",
"pretty-bytes": "^5.3.0",
"pretty-ms": "^5.0.0",
"semver": "7.3.4"
},
"devDependencies": {
"typescript": "5.1.6"
},
"engines": {
"node": ">=20.9.0"
},
"packageManager": "pnpm@9.6.0"
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,127 @@
#!/usr/bin/env node
/**
* Aggregates results from sharded stats runs and posts combined comment
*
* Usage: node aggregate-results.js <results-dir>
*
* Expects JSON files named pr-stats-*.json in the results directory
*/
const path = require('path')
const fs = require('fs/promises')
const { existsSync } = require('fs')
const addComment = require('./add-comment')
const logger = require('./util/logger')
async function main() {
const resultsDir = process.argv[2] || process.cwd()
logger(`Aggregating results from: ${resultsDir}`)
// Find all pr-stats-*.json files
const files = await fs.readdir(resultsDir)
const statsFiles = files.filter(
(f) => f.startsWith('pr-stats-') && f.endsWith('.json')
)
if (statsFiles.length === 0) {
// This can happen for docs-only changes where stats jobs are skipped
logger('No pr-stats-*.json files found - this may be a docs-only change')
process.exit(0)
}
logger(`Found ${statsFiles.length} results files: ${statsFiles.join(', ')}`)
// Load all results
const allData = []
for (const file of statsFiles) {
const filePath = path.join(resultsDir, file)
try {
const content = await fs.readFile(filePath, 'utf8')
const data = JSON.parse(content)
allData.push(data)
logger(`Loaded ${file} successfully`)
} catch (err) {
logger(`Warning: Failed to load ${file}: ${err.message}`)
}
}
if (allData.length === 0) {
logger('No valid results files could be loaded')
process.exit(1)
}
// Use the first file's actionInfo and statsConfig
const { actionInfo, statsConfig } = allData[0]
// Re-inject the GitHub token from env (it's excluded from JSON serialization for security)
actionInfo.githubToken = process.env.PR_STATS_COMMENT_TOKEN
// Merge results from all files
// Each file has results array with {title, mainRepoStats, diffRepoStats, diffs}
// We need to merge stats objects by combining their keys
const mergedResults = []
// Assume all files have the same number of configs with same titles
const numConfigs = allData[0].results.length
for (let i = 0; i < numConfigs; i++) {
const title = allData[0].results[i].title
const mergedMainRepoStats = {}
const mergedDiffRepoStats = {}
let mergedDiffs = null
for (const data of allData) {
const result = data.results[i]
// Merge mainRepoStats
if (result.mainRepoStats) {
for (const [key, value] of Object.entries(result.mainRepoStats)) {
if (!mergedMainRepoStats[key]) {
mergedMainRepoStats[key] = {}
}
Object.assign(mergedMainRepoStats[key], value)
}
}
// Merge diffRepoStats
if (result.diffRepoStats) {
for (const [key, value] of Object.entries(result.diffRepoStats)) {
if (!mergedDiffRepoStats[key]) {
mergedDiffRepoStats[key] = {}
}
Object.assign(mergedDiffRepoStats[key], value)
}
}
// Merge diffs (just combine all diff objects)
if (result.diffs) {
if (!mergedDiffs) {
mergedDiffs = {}
}
Object.assign(mergedDiffs, result.diffs)
}
}
mergedResults.push({
title,
mainRepoStats: mergedMainRepoStats,
diffRepoStats: mergedDiffRepoStats,
diffs: mergedDiffs,
})
}
logger(
`Merged ${allData.length} result sets into ${mergedResults.length} configs`
)
// Post the combined comment
await addComment(mergedResults, actionInfo, statsConfig)
logger('Aggregation complete')
}
main().catch((err) => {
console.error('Error aggregating results:', err)
process.exit(1)
})

View File

@@ -0,0 +1,26 @@
const path = require('path')
const os = require('os')
const fs = require('fs')
const benchTitle = 'Page Load Tests'
const workDir = fs.mkdtempSync(path.join(os.tmpdir(), 'next-stats'))
const mainRepoDir = path.join(workDir, 'main-repo')
const diffRepoDir = path.join(workDir, 'diff-repo')
const statsAppDir = path.join(workDir, 'stats-app')
const diffingDir = path.join(workDir, 'diff')
const allowedConfigLocations = [
'./',
'.stats-app',
'test/.stats-app',
'.github/.stats-app',
]
module.exports = {
benchTitle,
workDir,
diffingDir,
mainRepoDir,
diffRepoDir,
statsAppDir,
allowedConfigLocations,
}

View File

@@ -0,0 +1,192 @@
const path = require('path')
const fs = require('fs/promises')
const { existsSync } = require('fs')
const exec = require('./util/exec')
const logger = require('./util/logger')
const runConfigs = require('./run')
const addComment = require('./add-comment')
const actionInfo = require('./prepare/action-info')()
const { mainRepoDir, diffRepoDir } = require('./constants')
const loadStatsConfig = require('./prepare/load-stats-config')
const { cloneRepo, mergeBranch, getCommitId, linkPackages, getLastStable } =
require('./prepare/repo-setup')(actionInfo)
const allowedActions = new Set(['synchronize', 'opened'])
// Get bundler filter from action input (set by GitHub Actions as INPUT_BUNDLER)
const bundlerInput = (process.env.INPUT_BUNDLER || 'both').toLowerCase()
const isShardedRun = bundlerInput !== 'both'
if (isShardedRun) {
logger(`Running in sharded mode for bundler: ${bundlerInput}`)
}
if (!allowedActions.has(actionInfo.actionName) && !actionInfo.isRelease) {
logger(
`Not running for ${actionInfo.actionName} event action on repo: ${actionInfo.prRepo} and ref ${actionInfo.prRef}`
)
process.exit(0)
}
;(async () => {
try {
if (existsSync(path.join(__dirname, '../SKIP_NEXT_STATS.txt'))) {
console.log(
'SKIP_NEXT_STATS.txt file present, exiting stats generation..'
)
process.exit(0)
}
const { stdout: gitName } = await exec(
'git config user.name && git config user.email'
)
console.log('git author result:', gitName)
// clone PR/newer repository/ref first to get settings
if (!actionInfo.skipClone) {
await cloneRepo(actionInfo.prRepo, diffRepoDir, actionInfo.prRef)
}
if (actionInfo.isRelease) {
process.env.STATS_IS_RELEASE = 'true'
}
// load stats config from allowed locations
const { statsConfig, relativeStatsAppDir } = loadStatsConfig()
if (actionInfo.isLocal && actionInfo.prRef === statsConfig.mainBranch) {
throw new Error(
`'GITHUB_REF' can not be the same as mainBranch in 'stats-config.js'.\n` +
`This will result in comparing against the same branch`
)
}
if (actionInfo.isLocal) {
// make sure to use local repo location instead of the
// one provided in statsConfig
statsConfig.mainRepo = actionInfo.prRepo
}
/* eslint-disable-next-line */
actionInfo.commitId = await getCommitId(diffRepoDir)
let mainNextSwcVersion
if (!actionInfo.skipClone) {
let mainRef = statsConfig.mainBranch
if (actionInfo.isRelease) {
logger(`Release detected, using last stable tag: "${actionInfo.prRef}"`)
const lastStableTag = await getLastStable(diffRepoDir, actionInfo.prRef)
mainRef = lastStableTag
mainNextSwcVersion = lastStableTag
if (!lastStableTag) throw new Error('failed to get last stable tag')
logger(`using latestStable: "${lastStableTag}"`)
/* eslint-disable-next-line */
actionInfo.lastStableTag = lastStableTag
/* eslint-disable-next-line */
actionInfo.commitId = await getCommitId(diffRepoDir)
if (!actionInfo.customCommentEndpoint) {
/* eslint-disable-next-line */
actionInfo.commentEndpoint = `https://api.github.com/repos/${statsConfig.mainRepo}/commits/${actionInfo.commitId}/comments`
}
}
await cloneRepo(statsConfig.mainRepo, mainRepoDir, mainRef)
if (!actionInfo.isRelease && statsConfig.autoMergeMain) {
logger('Attempting auto merge of main branch')
await mergeBranch(statsConfig.mainBranch, mainRepoDir, diffRepoDir)
}
}
let mainRepoPkgPaths
let diffRepoPkgPaths
// run install/initialBuildCommand
const repoDirs = [mainRepoDir, diffRepoDir]
for (const dir of repoDirs) {
logger(`Running initial build for ${dir}`)
if (!actionInfo.skipClone) {
const usePnpm = existsSync(path.join(dir, 'pnpm-lock.yaml'))
if (!statsConfig.skipInitialInstall) {
await exec.spawnPromise(
`cd ${dir}${
usePnpm
? // --no-frozen-lockfile is used here to tolerate lockfile
// changes from merging latest changes
// --package-import-method=copy avoids EXDEV hardlink failures
// on self-hosted runners where pnpm store/workdirs cross devices.
` && pnpm install --no-frozen-lockfile --package-import-method=copy`
: ' && yarn install --network-timeout 1000000'
}`,
{ env: { PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: '1' } }
)
await exec.spawnPromise(
statsConfig.initialBuildCommand ||
`cd ${dir} && ${usePnpm ? 'pnpm build' : 'echo built'}`
)
}
}
await fs
.cp(
path.join(__dirname, '../native'),
path.join(dir, 'packages/next-swc/native'),
{ recursive: true, force: true }
)
.catch(console.error)
logger(`Linking packages in ${dir}`)
const isMainRepo = dir === mainRepoDir
const pkgPaths = await linkPackages({
repoDir: dir,
nextSwcVersion: isMainRepo ? mainNextSwcVersion : null,
})
if (isMainRepo) mainRepoPkgPaths = pkgPaths
else diffRepoPkgPaths = pkgPaths
}
// run the configs and collect results
const results = await runConfigs(statsConfig.configs, {
statsConfig,
mainRepoPkgPaths,
diffRepoPkgPaths,
relativeStatsAppDir,
bundlerFilter: isShardedRun ? bundlerInput : null,
})
if (isShardedRun) {
// In sharded mode, save results to JSON for later aggregation
const resultsPath = path.join(
process.env.GITHUB_WORKSPACE || process.cwd(),
`pr-stats-${bundlerInput}.json`
)
// Exclude sensitive fields (githubToken) before serializing to JSON
const { githubToken, ...safeActionInfo } = actionInfo
await fs.writeFile(
resultsPath,
JSON.stringify(
{ results, actionInfo: safeActionInfo, statsConfig },
null,
2
)
)
logger(`Saved results to ${resultsPath}`)
} else {
// In non-sharded mode, post comment directly
await addComment(results, actionInfo, statsConfig)
}
logger('finished')
process.exit(0)
} catch (err) {
console.error('Error occurred generating stats:')
console.error(err)
process.exit(1)
}
})()

View File

@@ -0,0 +1,111 @@
const path = require('path')
const logger = require('../util/logger')
const { execSync } = require('child_process')
const releaseTypes = new Set(['release', 'published'])
module.exports = function actionInfo() {
let {
ISSUE_ID,
SKIP_CLONE,
GITHUB_REF,
GITHUB_SHA,
LOCAL_STATS,
GIT_ROOT_DIR,
GITHUB_ACTION,
COMMENT_ENDPOINT,
GITHUB_REPOSITORY,
GITHUB_EVENT_PATH,
PR_STATS_COMMENT_TOKEN,
} = process.env
delete process.env.GITHUB_TOKEN
delete process.env.PR_STATS_COMMENT_TOKEN
// only use custom endpoint if we don't have a token
const commentEndpoint = !PR_STATS_COMMENT_TOKEN && COMMENT_ENDPOINT
if (LOCAL_STATS === 'true') {
const cwd = process.cwd()
const parentDir = path.join(cwd, '../..')
if (!GITHUB_REF) {
// get the current branch name
GITHUB_REF = execSync(`cd "${cwd}" && git rev-parse --abbrev-ref HEAD`)
.toString()
.trim()
}
if (!GIT_ROOT_DIR) {
GIT_ROOT_DIR = path.join(parentDir, '/')
}
if (!GITHUB_REPOSITORY) {
GITHUB_REPOSITORY = path.relative(parentDir, cwd)
}
if (!GITHUB_ACTION) {
GITHUB_ACTION = 'opened'
}
}
const info = {
commentEndpoint,
skipClone: SKIP_CLONE,
actionName: GITHUB_ACTION,
githubToken: PR_STATS_COMMENT_TOKEN,
customCommentEndpoint: !!commentEndpoint,
gitRoot: GIT_ROOT_DIR || 'https://github.com/',
prRepo: GITHUB_REPOSITORY,
prRef: GITHUB_REF,
isLocal: LOCAL_STATS,
commitId: null,
// Mirrors github.event.after || github.sha used by build_and_deploy.yml
// to ensure the tarball URL matches the deployed artifact.
githubHeadSha: GITHUB_SHA || null,
issueId: ISSUE_ID,
isRelease:
GITHUB_REPOSITORY === 'vercel/next.js' &&
(GITHUB_REF || '').includes('canary'),
}
if (info.isRelease) {
info.prRef = 'canary'
}
// get comment
if (GITHUB_EVENT_PATH) {
const event = require(GITHUB_EVENT_PATH)
info.actionName = event.action || info.actionName
if (event.after) {
info.githubHeadSha = event.after
}
if (releaseTypes.has(info.actionName)) {
info.isRelease = true
} else {
// since GITHUB_REPOSITORY and REF might not match the fork
// use event data to get repository and ref info
const prData = event['pull_request']
if (prData) {
info.prRepo = prData.head.repo.full_name
info.prRef = prData.head.ref
info.issueId = prData.number
if (!info.commentEndpoint) {
info.commentEndpoint = prData._links.comments || ''
}
// comment endpoint might be under `href`
if (typeof info.commentEndpoint === 'object') {
info.commentEndpoint = info.commentEndpoint.href
}
}
}
}
logger('Got actionInfo:')
logger.json({
...info,
githubToken: PR_STATS_COMMENT_TOKEN ? 'found' : 'missing',
})
return info
}

View File

@@ -0,0 +1,42 @@
const path = require('path')
const logger = require('../util/logger')
const { diffRepoDir, allowedConfigLocations } = require('../constants')
// load stats-config
function loadStatsConfig() {
let statsConfig
let relativeStatsAppDir
for (const configPath of allowedConfigLocations) {
try {
relativeStatsAppDir = configPath
statsConfig = require(
path.join(diffRepoDir, configPath, 'stats-config.js')
)
break
} catch (err) {
if (err.code !== 'MODULE_NOT_FOUND') {
console.error('Failed to load stats-config at', configPath, err)
}
/* */
}
}
if (!statsConfig) {
throw new Error(
`Failed to locate \`.stats-app\`, allowed locations are: ${allowedConfigLocations.join(
', '
)}`
)
}
logger(
'Got statsConfig at',
path.join(relativeStatsAppDir, 'stats-config.js'),
statsConfig,
'\n'
)
return { statsConfig, relativeStatsAppDir }
}
module.exports = loadStatsConfig

View File

@@ -0,0 +1,289 @@
const path = require('path')
const fs = require('fs')
const { existsSync } = require('fs')
const exec = require('../util/exec')
const logger = require('../util/logger')
const execa = require('execa')
const mockSpan = () => ({
traceAsyncFn: (fn) => fn(mockSpan()),
traceFn: (fn) => fn(mockSpan()),
traceChild: () => mockSpan(),
})
module.exports = (actionInfo) => {
return {
async cloneRepo(repoPath = '', dest = '', branch = '', depth = '20') {
await fs.promises.rm(dest, { recursive: true, force: true })
await exec(
`git clone ${actionInfo.gitRoot}${repoPath} --single-branch --branch ${branch} --depth=${depth} ${dest}`
)
},
async getLastStable() {
const res = await fetch(
`https://api.github.com/repos/vercel/next.js/releases/latest`,
{
headers: {
'X-GitHub-Api-Version': '2022-11-28',
},
}
)
if (!res.ok) {
throw new Error(
`Failed to get latest stable tag ${res.status}: ${await res.text()}`
)
}
const data = await res.json()
return data.tag_name
},
async getCommitId(repoDir = '') {
const { stdout } = await exec(`cd ${repoDir} && git rev-parse HEAD`)
return stdout.trim()
},
async resetToRef(ref = '', repoDir = '') {
await exec(`cd ${repoDir} && git reset --hard ${ref}`)
},
async mergeBranch(ref = '', origRepoDir = '', destRepoDir = '') {
await exec(`cd ${destRepoDir} && git remote add upstream ${origRepoDir}`)
await exec(`cd ${destRepoDir} && git fetch upstream`)
try {
await exec(`cd ${destRepoDir} && git merge upstream/${ref}`)
logger('Auto merge of main branch successful')
} catch (err) {
logger.error('Failed to auto merge main branch:', err)
if (err.stdout && err.stdout.includes('CONFLICT')) {
await exec(`cd ${destRepoDir} && git merge --abort`)
logger('aborted auto merge')
}
}
},
/**
* Runs `pnpm pack` on each package in the `packages` folder of the provided `repoDir`
* @param {{ repoDir: string, nextSwcVersion: null | string }} options Required options
* @returns {Promise<Map<string, string>>} List packages key is the package name, value is the path to the packed tar file.'
*/
async linkPackages({
repoDir,
nextSwcVersion: nextSwcVersionSpecified,
parentSpan,
}) {
if (!parentSpan) {
// Not all callers provide a parent span
parentSpan = mockSpan()
}
/** @type {Map<string, string>} */
const pkgPaths = new Map()
/** @type {Map<string, { packageJsonPath: string, packagePath: string, packageJson: any, packedPackageTarPath: string }>} */
const pkgDatas = new Map()
let packageFolders
try {
packageFolders = await parentSpan
.traceChild('read-packages-folder')
.traceAsyncFn(() =>
fs.promises.readdir(path.join(repoDir, 'packages'))
)
} catch (err) {
if (err.code === 'ENOENT') {
require('console').log('no packages to link')
return pkgPaths
}
throw err
}
parentSpan.traceChild('get-pkgdatas').traceFn(() => {
for (const packageFolder of packageFolders) {
const packagePath = path.join(repoDir, 'packages', packageFolder)
const packedPackageTarPath = path.join(
packagePath,
`${packageFolder}-packed.tgz`
)
const packageJsonPath = path.join(packagePath, 'package.json')
if (!existsSync(packageJsonPath)) {
require('console').log(`Skipping ${packageFolder}, no package.json`)
continue
}
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath))
const { name: packageName } = packageJson
pkgDatas.set(packageName, {
packageJsonPath,
packagePath,
packageJson,
packedPackageTarPath,
})
pkgPaths.set(packageName, packedPackageTarPath)
}
})
const nextSwcVersion =
nextSwcVersionSpecified ??
pkgDatas.get('@next/swc')?.packedPackageTarPath ??
null
await parentSpan
.traceChild('write-packagejson')
.traceAsyncFn(async () => {
for (const [
packageName,
{ packageJsonPath, packagePath, packageJson },
] of pkgDatas.entries()) {
// This loops through all items to get the packagedPkgPath of each item and add it to pkgData.dependencies
for (const [
packageName,
{ packedPackageTarPath },
] of pkgDatas.entries()) {
if (
!packageJson.dependencies ||
!packageJson.dependencies[packageName]
)
continue
// Edit the pkgData of the current item to point to the packed tgz
packageJson.dependencies[packageName] = packedPackageTarPath
}
// make sure native binaries are included in local linking
if (packageName === '@next/swc') {
packageJson.files ||= []
packageJson.files.push('native')
try {
const swcBinariesDirContents = (
await fs.promises.readdir(path.join(packagePath, 'native'))
).filter(
(file) => file !== '.gitignore' && file !== 'index.d.ts'
)
require('console').log(
'using swc binaries: ',
swcBinariesDirContents.join(', ')
)
} catch (err) {
if (err.code === 'ENOENT') {
require('console').log('swc binaries dir is missing!')
}
throw err
}
} else if (packageName === 'next') {
const nextSwcPkg = pkgDatas.get('@next/swc')
console.log('using swc dep', {
nextSwcVersion,
nextSwcPkg,
})
if (nextSwcVersion) {
Object.assign(packageJson.dependencies, {
// CI
'@next/swc-linux-x64-gnu': nextSwcVersion,
// Vercel issued laptops
'@next/swc-darwin-arm64': nextSwcVersion,
})
}
}
await fs.promises.writeFile(
packageJsonPath,
JSON.stringify(packageJson, null, 2),
'utf8'
)
}
})
await parentSpan
.traceChild('pnpm-packing')
.traceAsyncFn(async (packingSpan) => {
// wait to pack packages until after dependency paths have been updated
// to the correct versions
await Promise.all(
Array.from(pkgDatas.entries()).map(
async ([
packageName,
{ packagePath: pkgPath, packedPackageTarPath: packedPkgPath },
]) => {
return packingSpan
.traceChild('handle-package', { packageName })
.traceAsyncFn(async (handlePackageSpan) => {
/** @type {null | () => Promise<void>} */
let cleanup = null
if (packageName === '@next/swc') {
// next-swc uses a gitignore to prevent the committing of native builds but it doesn't
// use files in package.json because it publishes to individual packages based on architecture.
// When we used yarn to pack these packages the gitignore was ignored so the native builds were packed
// however npm does respect gitignore when packing so we need to remove it in this specific case
// to ensure the native builds are packed for use in gh actions and related scripts
const nativeGitignorePath = path.join(
pkgPath,
'native/.gitignore'
)
const renamedGitignorePath = path.join(
pkgPath,
'disabled-native-gitignore'
)
await handlePackageSpan
.traceChild('rename-gitignore')
.traceAsyncFn(() =>
fs.promises.rename(
nativeGitignorePath,
renamedGitignorePath
)
)
cleanup = async () => {
await fs.promises.rename(
renamedGitignorePath,
nativeGitignorePath
)
}
}
const options = {
cwd: pkgPath,
env: {
...process.env,
COREPACK_ENABLE_STRICT: '0',
},
}
let execResult
try {
execResult = await handlePackageSpan
.traceChild('pnpm-pack-try-1')
.traceAsyncFn(() => execa('pnpm', ['pack'], options))
} catch {
execResult = await handlePackageSpan
.traceChild('pnpm-pack-try-2')
.traceAsyncFn(() => execa('pnpm', ['pack'], options))
}
const { stdout } = execResult
const packedFileName = stdout.trim()
await handlePackageSpan
.traceChild('rename-packed-tar-and-cleanup')
.traceAsyncFn(() =>
Promise.all([
fs.promises.rename(
path.join(pkgPath, packedFileName),
packedPkgPath
),
cleanup?.(),
])
)
})
}
)
)
})
return pkgPaths
},
}
}

View File

@@ -0,0 +1,32 @@
const exec = require('../util/exec')
const parseField = (stdout = '', field = '') => {
return stdout.split(field).pop().trim().split(/\s/).shift().trim()
}
// benchmark an url
async function benchmarkUrl(
url = '',
options = {
reqTimeout: 60,
concurrency: 50,
numRequests: 2500,
}
) {
const { numRequests, concurrency, reqTimeout } = options
const { stdout } = await exec(
`ab -n ${numRequests} -c ${concurrency} -s ${reqTimeout} "${url}"`
)
const totalTime = parseFloat(parseField(stdout, 'Time taken for tests:'), 10)
const failedRequests = parseInt(parseField(stdout, 'Failed requests:'), 10)
const avgReqPerSec = parseFloat(parseField(stdout, 'Requests per second:'))
return {
totalTime,
avgReqPerSec,
failedRequests,
}
}
module.exports = benchmarkUrl

View File

@@ -0,0 +1,172 @@
const path = require('path')
const fs = require('fs/promises')
const { existsSync } = require('fs')
const exec = require('../util/exec')
const glob = require('../util/glob')
const logger = require('../util/logger')
const { statsAppDir, diffingDir } = require('../constants')
module.exports = async function collectDiffs(
filesToTrack = [],
initial = false
) {
if (initial) {
logger('Setting up directory for diffing')
// set-up diffing directory
await fs.rm(diffingDir, { recursive: true, force: true })
await fs.mkdir(diffingDir, { recursive: true })
await exec(`cd ${diffingDir} && git init`)
} else {
// remove any previous files in case they won't be overwritten
const toRemove = await glob('!(.git)', { cwd: diffingDir, dot: true })
await Promise.all(
toRemove.map((file) =>
fs.rm(path.join(diffingDir, file), { recursive: true, force: true })
)
)
}
const diffs = {}
await Promise.all(
filesToTrack.map(async (fileGroup) => {
const { globs } = fileGroup
const curFiles = []
const prettierExts = ['.js', '.html', '.css', '.json']
await Promise.all(
globs.map(async (pattern) => {
curFiles.push(
...(await glob(pattern, { cwd: statsAppDir })).filter((item) =>
prettierExts.includes(path.extname(item))
)
)
})
)
logger('Tracking the following files:')
logger(curFiles)
for (let file of curFiles) {
const absPath = path.join(statsAppDir, file)
const diffDest = path.join(diffingDir, file)
await fs.cp(absPath, diffDest, { recursive: true, force: true })
}
if (curFiles.length > 0) {
const prettierPath = path.join(
__dirname,
'../../node_modules/.bin/prettier'
)
await exec(
`cd "${process.env.LOCAL_STATS ? process.cwd() : diffingDir}" && ` +
`${prettierPath} --write --no-error-on-unmatched-pattern ${curFiles
.map((f) => path.join(diffingDir, f))
.join(' ')}`
)
}
})
)
await exec(`cd ${diffingDir} && git add .`, true)
if (initial) {
await exec(`cd ${diffingDir} && git commit -m 'initial commit'`)
} else {
let { stdout: renamedFiles } = await exec(
`cd ${diffingDir} && git diff --name-status HEAD`
)
renamedFiles = renamedFiles
.trim()
.split('\n')
.filter((line) => line.startsWith('R'))
diffs._renames = []
for (const line of renamedFiles) {
const [, prev, cur] = line.split('\t')
await fs.rename(path.join(diffingDir, cur), path.join(diffingDir, prev))
diffs._renames.push({
prev,
cur,
})
}
await exec(`cd ${diffingDir} && git add .`)
let { stdout: changedFiles } = await exec(
`cd ${diffingDir} && git diff --name-only HEAD`
)
changedFiles = changedFiles.trim().split('\n')
for (const file of changedFiles) {
const fileKey = path.basename(file)
const hasFile = existsSync(path.join(diffingDir, file))
if (!hasFile) {
diffs[fileKey] = 'deleted'
continue
}
try {
let { stdout } = await exec(
`cd ${diffingDir} && git diff --minimal HEAD ${file}`
)
stdout = (stdout.split(file).pop() || '').trim()
if (stdout.length > 0 && !isLikelyHashOrIDChange(stdout)) {
diffs[fileKey] = stdout
}
} catch (err) {
console.error(`Failed to diff ${file}: ${err.message}`)
diffs[fileKey] = `failed to diff`
}
}
}
return diffs
}
function isLikelyHashOrIDChange(diff) {
const lines = diff.split('\n')
let additions = []
let deletions = []
// Separate additions and deletions
for (const line of lines) {
if (line.startsWith('+')) {
additions.push(line.substring(1).split(/\b/))
} else if (line.startsWith('-')) {
deletions.push(line.substring(1).split(/\b/))
}
}
// If the number of additions and deletions is different, it's not a hash or ID change
if (additions.length !== deletions.length) {
return false
}
// Compare each addition with each deletion
for (let i = 0; i < additions.length; i++) {
const additionTokens = additions[i]
const deletionTokens = deletions[i]
// Identify differing tokens
const differingTokens = additionTokens.filter(
(token, index) => token !== deletionTokens[index]
)
// Analyze differing tokens
for (const token of differingTokens) {
const isLikelyHash = /^[a-f0-9]+$/.test(token)
const isLikelyID = /^[0-9]+$/.test(token)
// this is most likely noise because some path include the repo name, which can be main or diff
const isLikelyNoise = ['main', 'diff'].includes(token)
if (!isLikelyHash && !isLikelyID && !isLikelyNoise) {
return false
}
}
}
return true
}

View File

@@ -0,0 +1,582 @@
const path = require('path')
const net = require('net')
const fs = require('fs/promises')
const { existsSync } = require('fs')
const getPort = require('get-port')
const fetch = require('node-fetch')
const glob = require('../util/glob')
const gzipSize = require('gzip-size')
const logger = require('../util/logger')
const exec = require('../util/exec')
const { spawn } = require('../util/exec')
const { parse: urlParse } = require('url')
const benchmarkUrl = require('./benchmark-url')
const { statsAppDir, diffingDir, benchTitle } = require('../constants')
const { calcStats } = require('../util/stats')
// Number of iterations for timing benchmarks to get stable median
const BENCHMARK_ITERATIONS = process.env.BENCHMARK_ITERATIONS
? parseInt(process.env.BENCHMARK_ITERATIONS)
: 9
// Check if a port is accepting TCP connections
function checkPort(port, timeout = 100) {
return new Promise((resolve) => {
const socket = new net.Socket()
socket.setTimeout(timeout)
socket.once('connect', () => {
socket.destroy()
resolve(true)
})
socket.once('timeout', () => {
socket.destroy()
resolve(false)
})
socket.once('error', () => {
socket.destroy()
resolve(false)
})
socket.connect(port, 'localhost')
})
}
// Wait for port to start accepting TCP connections
async function waitForPort(port, timeoutMs = 60000) {
const start = Date.now()
while (Date.now() - start < timeoutMs) {
if (await checkPort(port)) {
return Date.now() - start
}
await new Promise((r) => setTimeout(r, 50))
}
return null
}
// Wait for HTTP server to respond
async function waitForHttp(port, timeoutMs = 60000) {
const start = Date.now()
while (Date.now() - start < timeoutMs) {
try {
const res = await fetch(`http://localhost:${port}/`, { timeout: 2000 })
if (res.ok) {
return Date.now() - start
}
} catch (e) {
// Server not ready yet
}
await new Promise((r) => setTimeout(r, 50))
}
return null
}
// Run a single dev server boot benchmark
async function benchmarkDevBoot(appDevCommand, curDir, port, cleanBuild) {
// Clean .next directory for cold start
if (cleanBuild) {
const nextDir = path.join(curDir, '.next')
await fs.rm(nextDir, { recursive: true, force: true })
}
const startTime = Date.now()
const devChild = spawn(appDevCommand, {
cwd: curDir,
env: {
PORT: port,
},
stdio: 'pipe',
})
let exited = false
devChild.on('exit', () => {
exited = true
})
let readyInTime = null
let stdoutBuffer = ''
// Capture output for debugging
devChild.stdout.on('data', (data) => {
stdoutBuffer += data.toString()
if (readyInTime === null && stdoutBuffer.includes('Ready in')) {
readyInTime = Date.now() - startTime
}
process.stdout.write(data)
})
devChild.stderr.on('data', (data) => {
process.stderr.write(data)
})
// Measure time to port listening (TCP level)
const listenTime = await waitForPort(port, 60000)
// Measure time to HTTP ready
let readyTime = null
if (listenTime !== null && !exited) {
readyTime = await waitForHttp(port, 60000)
}
devChild.kill()
// Wait for process to fully exit to avoid port conflicts on subsequent runs
if (!exited) {
await new Promise((resolve) => {
devChild.on('exit', resolve)
// Timeout after 5 seconds in case process doesn't exit cleanly
setTimeout(resolve, 5000)
})
}
return {
listenTime,
readyInTime,
readyTime,
}
}
// Run multiple iterations of dev boot benchmark and return median times
async function benchmarkDevBootWithIterations(
appDevCommand,
curDir,
port,
cleanBuild,
label
) {
const listenTimes = []
const readyInTimes = []
const readyTimes = []
for (let i = 0; i < BENCHMARK_ITERATIONS; i++) {
// For cold start benchmarks, clean before EVERY iteration to get true cold times
logger(` ${label} iteration ${i + 1}/${BENCHMARK_ITERATIONS}...`)
const result = await benchmarkDevBoot(
appDevCommand,
curDir,
port,
cleanBuild
)
if (result.listenTime !== null) {
listenTimes.push(result.listenTime)
logger(` Boot: ${result.listenTime}ms`)
}
if (result.readyInTime !== null) {
readyInTimes.push(result.readyInTime)
logger(` Ready in: ${result.readyInTime}ms`)
}
if (result.readyTime !== null) {
readyTimes.push(result.readyTime)
logger(` Ready: ${result.readyTime}ms`)
}
// Small delay between iterations to let system settle
await new Promise((r) => setTimeout(r, 500))
}
const listenStats = calcStats(listenTimes)
const readyInStats = calcStats(readyInTimes)
const readyStats = calcStats(readyTimes)
// Log detailed stats for debugging
if (listenStats) {
logger(
` ${label} Boot: median=${listenStats.median}ms, range=${listenStats.min}-${listenStats.max}ms, CV=${listenStats.cv}%`
)
}
if (readyInStats) {
logger(
` ${label} Ready in: median=${readyInStats.median}ms, range=${readyInStats.min}-${readyInStats.max}ms, CV=${readyInStats.cv}%`
)
}
if (readyStats) {
logger(
` ${label} Ready: median=${readyStats.median}ms, range=${readyStats.min}-${readyStats.max}ms, CV=${readyStats.cv}%`
)
}
return {
listenTime: listenStats?.median ?? null,
readyInTime: readyInStats?.median ?? null,
readyTime: readyStats?.median ?? null,
}
}
async function defaultGetRequiredFiles(nextAppDir, fileName) {
return [fileName]
}
module.exports = async function collectStats(
runConfig = {},
statsConfig = {},
fromDiff = false,
bundlerSuffix = null,
benchmarkOnly = false,
bundlerFilter = null
) {
const stats = {
[benchTitle]: {},
}
const orderedStats = {
[benchTitle]: {},
}
const curDir = fromDiff ? diffingDir : statsAppDir
// If bundlerSuffix is provided, we're collecting file sizes only (skip benchmarks)
// If benchmarkOnly is true, we're running benchmarks only (skip file sizes)
const collectFileSizes = !benchmarkOnly
const runBenchmarks = !bundlerSuffix
const hasPagesToFetch =
Array.isArray(runConfig.pagesToFetch) && runConfig.pagesToFetch.length > 0
const hasPagesToBench =
Array.isArray(runConfig.pagesToBench) && runConfig.pagesToBench.length > 0
// Run production start benchmark FIRST (before dev benchmark which cleans .next)
// Only run benchmarks when not collecting bundler-specific file sizes
// Skip production start in sharded mode (bundlerFilter set) as these metrics don't vary by bundler
if (
runBenchmarks &&
!fromDiff &&
!bundlerFilter &&
statsConfig.appStartCommand &&
(hasPagesToFetch || hasPagesToBench)
) {
const port = await getPort()
const readyTimes = []
// Helper to run a single production start and measure time
async function runProdStartTiming() {
const startTime = Date.now()
const child = spawn(statsConfig.appStartCommand, {
cwd: curDir,
env: {
PORT: port,
},
stdio: 'pipe',
})
let serverReadyResolve
let serverReadyResolved = false
const serverReadyPromise = new Promise((resolve) => {
serverReadyResolve = resolve
})
child.stdout.on('data', (data) => {
if (data.toString().includes('- Local:') && !serverReadyResolved) {
serverReadyResolved = true
serverReadyResolve()
}
})
child.on('exit', () => {
if (!serverReadyResolved) {
serverReadyResolve()
serverReadyResolved = true
}
})
await serverReadyPromise
const readyTime = Date.now() - startTime
child.kill()
await new Promise((r) => setTimeout(r, 300)) // Let port release
return readyTime
}
// Run multiple timing iterations for stable median
logger(
`=== Production Start Benchmark (${BENCHMARK_ITERATIONS} iterations) ===`
)
for (let i = 0; i < BENCHMARK_ITERATIONS; i++) {
logger(` Prod Start iteration ${i + 1}/${BENCHMARK_ITERATIONS}...`)
const readyTime = await runProdStartTiming()
readyTimes.push(readyTime)
logger(` Ready: ${readyTime}ms`)
}
const readyStats = calcStats(readyTimes)
if (readyStats) {
logger(
` Prod Start: median=${readyStats.median}ms, range=${readyStats.min}-${readyStats.max}ms, CV=${readyStats.cv}%`
)
if (!orderedStats['General']) {
orderedStats['General'] = {}
}
orderedStats['General']['nextStartReadyDuration'] = readyStats.median
} else {
logger(` Prod Start: Failed to collect timing data`)
}
// Now run one more time to do page fetching/benchmarking
logger('=== Production Start for Page Fetching ===')
const startTime = Date.now()
const child = spawn(statsConfig.appStartCommand, {
cwd: curDir,
env: {
PORT: port,
},
stdio: 'pipe',
})
let exitCode = null
let logStderr = true
let serverReadyResolve
let serverReadyResolved = false
const serverReadyPromise = new Promise((resolve) => {
serverReadyResolve = resolve
})
child.stdout.on('data', (data) => {
if (data.toString().includes('- Local:') && !serverReadyResolved) {
serverReadyResolved = true
serverReadyResolve()
}
process.stdout.write(data)
})
child.stderr.on('data', (data) => logStderr && process.stderr.write(data))
child.on('exit', (code) => {
if (!serverReadyResolved) {
serverReadyResolve()
serverReadyResolved = true
}
exitCode = code
})
await serverReadyPromise
if (exitCode !== null) {
throw new Error(
`Failed to run \`${statsConfig.appStartCommand}\` process exited with code ${exitCode}`
)
}
if (hasPagesToFetch) {
const fetchedPagesDir = path.join(curDir, 'fetched-pages')
await fs.mkdir(fetchedPagesDir, { recursive: true })
for (let url of runConfig.pagesToFetch) {
url = url.replace('$PORT', port)
const { pathname } = urlParse(url)
try {
const res = await fetch(url)
if (!res.ok) {
throw new Error(`Failed to fetch ${url} got status: ${res.status}`)
}
const responseText = (await res.text()).trim()
let fileName = pathname === '/' ? '/index' : pathname
if (fileName.endsWith('/')) fileName = fileName.slice(0, -1)
logger(
`Writing file to ${path.join(fetchedPagesDir, `${fileName}.html`)}`
)
await fs.writeFile(
path.join(fetchedPagesDir, `${fileName}.html`),
responseText,
'utf8'
)
} catch (err) {
logger.error(err)
}
}
}
if (hasPagesToBench) {
// disable stderr so we don't clobber logs while benchmarking
// any pages that create logs
logStderr = false
for (let url of runConfig.pagesToBench) {
url = url.replace('$PORT', port)
logger(`Benchmarking ${url}`)
const results = await benchmarkUrl(url, runConfig.benchOptions)
logger(`Finished benchmarking ${url}`)
const { pathname: key } = urlParse(url)
stats[benchTitle][`${key} failed reqs`] = results.failedRequests
stats[benchTitle][`${key} total time (seconds)`] = results.totalTime
stats[benchTitle][`${key} avg req/sec`] = results.avgReqPerSec
}
}
child.kill()
}
// Measure dev server boot time if configured
// Runs full matrix: (Turbopack + Webpack) x (Cold + Warm) x (Boot + Ready)
// Each timing uses median of BENCHMARK_ITERATIONS runs for stability
// NOTE: This runs AFTER the production start benchmark because it cleans the .next directory
// Only run benchmarks when not collecting bundler-specific file sizes
if (
runBenchmarks &&
!fromDiff &&
statsConfig.appDevCommand &&
statsConfig.measureDevBoot
) {
const devPort = await getPort()
if (!orderedStats['General']) {
orderedStats['General'] = {}
}
// Run benchmarks for selected bundler(s)
// Default is now turbopack, so we need --webpack for webpack
const allBundlers = [
{ name: 'Turbopack', flag: '', suffix: 'Turbo' },
{ name: 'Webpack', flag: '--webpack', suffix: 'Webpack' },
]
const bundlers = bundlerFilter
? allBundlers.filter((b) => b.name.toLowerCase() === bundlerFilter)
: allBundlers
for (const bundler of bundlers) {
logger(`\n=== ${bundler.name} Dev Server Benchmarks ===`)
// Build the command with the bundler flag
const devCommand = bundler.flag
? `${statsConfig.appDevCommand} ${bundler.flag}`
: statsConfig.appDevCommand
// 1. Cold start benchmark (clean .next directory, multiple iterations)
logger(
`=== ${bundler.name} Cold Start (${BENCHMARK_ITERATIONS} iterations) ===`
)
const coldResult = await benchmarkDevBootWithIterations(
devCommand,
curDir,
devPort,
true, // clean .next before each iteration
`${bundler.name} Cold`
)
if (coldResult.listenTime !== null) {
orderedStats['General'][`nextDevColdListenDuration${bundler.suffix}`] =
coldResult.listenTime
}
if (coldResult.readyInTime !== null) {
orderedStats['General'][`nextDevColdReadyInDuration${bundler.suffix}`] =
coldResult.readyInTime
}
if (coldResult.readyTime !== null) {
orderedStats['General'][`nextDevColdReadyDuration${bundler.suffix}`] =
coldResult.readyTime
}
// 2. Warm up bytecode cache by running server for ~10 seconds
if (coldResult.readyTime !== null) {
logger(`=== ${bundler.name} Warming up bytecode cache (10s) ===`)
const warmupChild = spawn(devCommand, {
cwd: curDir,
env: {
PORT: devPort,
},
stdio: 'pipe',
})
let warmupExited = false
warmupChild.on('exit', () => {
warmupExited = true
})
// Wait for server to be ready
await waitForHttp(devPort, 60000)
// Let it run for 10 seconds to warm bytecode cache
await new Promise((r) => setTimeout(r, 10000))
warmupChild.kill()
// Wait for warmup server to fully exit to avoid port conflicts
if (!warmupExited) {
await new Promise((resolve) => {
warmupChild.on('exit', resolve)
// Timeout after 5 seconds in case process doesn't exit cleanly
setTimeout(resolve, 5000)
})
}
// 3. Warm start benchmark (keep .next directory, multiple iterations)
logger(
`=== ${bundler.name} Warm Start (${BENCHMARK_ITERATIONS} iterations) ===`
)
const warmResult = await benchmarkDevBootWithIterations(
devCommand,
curDir,
devPort,
false, // keep build
`${bundler.name} Warm`
)
if (warmResult.listenTime !== null) {
orderedStats['General'][
`nextDevWarmListenDuration${bundler.suffix}`
] = warmResult.listenTime
}
if (warmResult.readyInTime !== null) {
orderedStats['General'][
`nextDevWarmReadyInDuration${bundler.suffix}`
] = warmResult.readyInTime
}
if (warmResult.readyTime !== null) {
orderedStats['General'][`nextDevWarmReadyDuration${bundler.suffix}`] =
warmResult.readyTime
}
}
}
logger('\n=== Dev Boot Benchmark Complete ===')
}
// Collect file sizes only when not in benchmark-only mode
if (collectFileSizes) {
for (const fileGroup of runConfig.filesToTrack) {
const {
getRequiredFiles = defaultGetRequiredFiles,
name,
globs,
} = fileGroup
const groupStats = {}
const curFiles = new Set()
for (const pattern of globs) {
const results = await glob(pattern, { cwd: curDir, nodir: true })
results.forEach((result) => curFiles.add(result))
}
for (const file of curFiles) {
const fileKey = path.basename(file)
try {
let parsedSizeSum = 0
let gzipSizeSum = 0
for (const requiredFile of await getRequiredFiles(curDir, file)) {
const absPath = path.join(curDir, requiredFile)
const fileInfo = await fs.stat(absPath)
parsedSizeSum += fileInfo.size
gzipSizeSum += await gzipSize.file(absPath)
}
groupStats[fileKey] = parsedSizeSum
groupStats[`${fileKey} gzip`] = gzipSizeSum
} catch (err) {
logger.error('Failed to get file stats', err)
}
}
stats[name] = groupStats
}
for (const fileGroup of runConfig.filesToTrack) {
const { name } = fileGroup
orderedStats[name] = stats[name]
}
}
if (stats[benchTitle]) {
orderedStats[benchTitle] = stats[benchTitle]
}
return orderedStats
}

View File

@@ -0,0 +1,25 @@
const path = require('path')
const fs = require('fs/promises')
// getDirSize recursively gets size of all files in a directory
async function getDirSize(dir, ctx = { size: 0 }) {
let subDirs = await fs.readdir(dir)
subDirs = subDirs.map((d) => path.join(dir, d))
await Promise.all(
subDirs.map(async (curDir) => {
// we use dev builds so the size isn't helpful to track
// here as it's not reflective of full releases
if (curDir.includes('@next/swc')) return
const fileStat = await fs.stat(curDir)
if (fileStat.isDirectory()) {
return getDirSize(curDir, ctx)
}
ctx.size += fileStat.size
})
)
return ctx.size
}
module.exports = getDirSize

View File

@@ -0,0 +1,304 @@
const path = require('path')
const fs = require('fs/promises')
const glob = require('../util/glob')
const exec = require('../util/exec')
const logger = require('../util/logger')
const getDirSize = require('./get-dir-size')
const collectStats = require('./collect-stats')
const collectDiffs = require('./collect-diffs')
const { statsAppDir, diffRepoDir } = require('../constants')
const { calcStats } = require('../util/stats')
// Number of iterations for build benchmarks to get stable median
const BUILD_BENCHMARK_ITERATIONS = 5
// Bundler configurations for dual-bundler benchmarking
const BUNDLERS = [
{ name: 'Webpack', flag: '--webpack', suffix: 'Webpack' },
{ name: 'Turbopack', flag: '', suffix: 'Turbo' },
]
async function runConfigs(
configs = [],
{
statsConfig,
relativeStatsAppDir,
mainRepoPkgPaths,
diffRepoPkgPaths,
bundlerFilter = null,
},
diffing = false
) {
// Filter bundlers based on input
const bundlersToRun = bundlerFilter
? BUNDLERS.filter((b) => b.name.toLowerCase() === bundlerFilter)
: BUNDLERS
if (bundlerFilter && bundlersToRun.length === 0) {
throw new Error(
`Invalid bundler filter: ${bundlerFilter}. Must be 'webpack' or 'turbopack'`
)
}
logger(
`Running benchmarks for bundlers: ${bundlersToRun.map((b) => b.name).join(', ')}`
)
const results = []
for (const config of configs) {
logger(`Running config: ${config.title}${diffing ? ' (diff)' : ''}`)
let mainRepoStats
let diffRepoStats
let diffs
for (const pkgPaths of [mainRepoPkgPaths, diffRepoPkgPaths]) {
let curStats = {
General: {
nodeModulesSize: null,
},
}
// if stats-config is in root of project we're analyzing
// the whole project so copy from each repo
const curStatsAppPath = path.join(diffRepoDir, relativeStatsAppDir)
// clean statsAppDir
await fs.rm(statsAppDir, { recursive: true, force: true })
await fs.cp(curStatsAppPath, statsAppDir, { recursive: true })
logger(`Copying ${curStatsAppPath} ${statsAppDir}`)
// apply config files
for (const configFile of config.configFiles || []) {
const filePath = path.join(statsAppDir, configFile.path)
await fs.writeFile(filePath, configFile.content, 'utf8')
}
// links local builds of the packages and installs dependencies
await linkPkgs(statsAppDir, pkgPaths)
if (!diffing) {
curStats.General.nodeModulesSize = await getDirSize(
path.join(statsAppDir, 'node_modules')
)
}
// Run builds for selected bundler(s) and collect stats separately
for (const bundler of bundlersToRun) {
logger(`\n=== ${bundler.name} Production Build ===`)
// Build base command without --webpack flag (we add it per bundler)
const baseBuildCommand = statsConfig.appBuildCommand.replace(
/ --webpack/g,
''
)
const buildCommand = bundler.flag
? `${baseBuildCommand} ${bundler.flag}`
: baseBuildCommand
// Run multiple fresh build iterations for stable timing
const freshBuildTimes = []
logger(` Fresh build (${BUILD_BENCHMARK_ITERATIONS} iterations)...`)
for (let i = 0; i < BUILD_BENCHMARK_ITERATIONS; i++) {
// Clean .next directory for fresh build
await fs.rm(path.join(statsAppDir, '.next'), {
recursive: true,
force: true,
})
const buildStart = Date.now()
console.log(await exec(`cd ${statsAppDir} && ${buildCommand}`, false))
const buildDuration = Date.now() - buildStart
freshBuildTimes.push(buildDuration)
logger(` Iteration ${i + 1}: ${buildDuration}ms`)
}
const freshStats = calcStats(freshBuildTimes)
logger(
` Fresh build: median=${freshStats.median}ms, range=${freshStats.min}-${freshStats.max}ms`
)
curStats.General[`buildDuration${bundler.suffix}`] = freshStats.median
// Run cached build iterations BEFORE renames (renames invalidate cache)
const cachedBuildTimes = []
logger(` Cached build (${BUILD_BENCHMARK_ITERATIONS} iterations)...`)
for (let i = 0; i < BUILD_BENCHMARK_ITERATIONS; i++) {
const buildStart = Date.now()
console.log(await exec(`cd ${statsAppDir} && ${buildCommand}`, false))
const buildDuration = Date.now() - buildStart
cachedBuildTimes.push(buildDuration)
logger(` Iteration ${i + 1}: ${buildDuration}ms`)
}
const cachedStats = calcStats(cachedBuildTimes)
logger(
` Cached build: median=${cachedStats.median}ms, range=${cachedStats.min}-${cachedStats.max}ms`
)
curStats.General[`buildDurationCached${bundler.suffix}`] =
cachedStats.median
// Apply renames to get deterministic output names (after cached builds)
for (const rename of config.renames) {
const renameResults = await glob(rename.srcGlob, { cwd: statsAppDir })
for (const result of renameResults) {
let dest = rename.removeHash
? result.replace(/(\.|-)[0-9a-f]{16}(\.|-)/g, '$1HASH$2')
: rename.dest
if (result === dest) continue
try {
await fs.rename(
path.join(statsAppDir, result),
path.join(statsAppDir, dest)
)
} catch (e) {
// File may not exist for this bundler
}
}
}
// Collect file stats for this bundler (after renames for deterministic names)
const collectedStats = await collectStats(
config,
statsConfig,
false,
bundler.suffix
)
for (const key of Object.keys(collectedStats)) {
// Prefix group names with bundler suffix (except General which is shared)
const groupKey = key === 'General' ? key : `${key} (${bundler.name})`
curStats[groupKey] = Object.assign(
{},
curStats[groupKey],
collectedStats[key]
)
}
}
// Run benchmarks for selected bundler(s) - dev boot and prod start
const benchmarkStats = await collectStats(
config,
statsConfig,
false,
null,
true,
bundlerFilter
)
for (const key of Object.keys(benchmarkStats)) {
curStats[key] = Object.assign({}, curStats[key], benchmarkStats[key])
}
const applyRenames = (renames, stats) => {
if (renames) {
for (const rename of renames) {
let { cur, prev } = rename
cur = path.basename(cur)
prev = path.basename(prev)
Object.keys(stats).forEach((group) => {
if (stats[group][cur]) {
stats[group][prev] = stats[group][cur]
stats[group][prev + ' gzip'] = stats[group][cur + ' gzip']
delete stats[group][cur]
delete stats[group][cur + ' gzip']
}
})
}
}
}
if (mainRepoStats) {
diffRepoStats = curStats
if (!diffing && config.diff !== false) {
for (const groupKey of Object.keys(curStats)) {
if (groupKey === 'General') continue
let changeDetected = config.diff === 'always'
const curDiffs = await collectDiffs(config.filesToTrack)
changeDetected = changeDetected || Object.keys(curDiffs).length > 0
applyRenames(curDiffs._renames, diffRepoStats)
delete curDiffs._renames
if (changeDetected) {
logger('Detected change, running diff')
diffs = await runConfigs(
[
{
...config,
configFiles: config.diffConfigFiles,
},
],
{
statsConfig,
mainRepoPkgPaths,
diffRepoPkgPaths,
relativeStatsAppDir,
bundlerFilter,
},
true
)
delete diffs._renames
break
}
}
}
if (diffing) {
// copy new files and get diff results
return collectDiffs(config.filesToTrack)
}
} else {
// set up diffing folder and copy initial files
await collectDiffs(config.filesToTrack, true)
/* eslint-disable-next-line */
mainRepoStats = curStats
}
}
logger(`Finished running: ${config.title}`)
results.push({
title: config.title,
mainRepoStats,
diffRepoStats,
diffs,
})
}
return results
}
async function linkPkgs(pkgDir = '', pkgPaths) {
await fs.rm(path.join(pkgDir, 'node_modules'), {
recursive: true,
force: true,
})
const pkgJsonPath = path.join(pkgDir, 'package.json')
const pkgData = require(pkgJsonPath)
if (!pkgData.dependencies && !pkgData.devDependencies) return
for (const pkg of pkgPaths.keys()) {
const pkgPath = pkgPaths.get(pkg)
if (pkgData.dependencies && pkgData.dependencies[pkg]) {
pkgData.dependencies[pkg] = pkgPath
} else if (pkgData.devDependencies && pkgData.devDependencies[pkg]) {
pkgData.devDependencies[pkg] = pkgPath
}
}
await fs.writeFile(pkgJsonPath, JSON.stringify(pkgData, null, 2), 'utf8')
await exec(
`cd ${pkgDir} && pnpm install --strict-peer-dependencies=false --package-import-method=copy`,
false
)
}
module.exports = runConfigs

View File

@@ -0,0 +1,51 @@
const logger = require('./logger')
const { promisify } = require('util')
const { exec: execOrig, spawn: spawnOrig } = require('child_process')
const execP = promisify(execOrig)
const env = {
...process.env,
GITHUB_TOKEN: '',
PR_STATS_COMMENT_TOKEN: '',
}
function exec(command, noLog = false, opts = {}) {
if (!noLog) logger(`exec: ${command}`)
return execP(command, {
...opts,
env: { ...env, ...opts.env },
})
}
exec.spawn = function spawn(command = '', opts = {}) {
logger(`spawn: ${command}`)
const child = spawnOrig('/bin/bash', ['-c', command], {
...opts,
env: {
...env,
...opts.env,
},
stdio: opts.stdio || 'inherit',
})
child.on('exit', (code, signal) => {
logger(`spawn exit (${code}, ${signal}): ${command}`)
})
return child
}
exec.spawnPromise = function spawnPromise(command = '', opts = {}) {
return new Promise((resolve, reject) => {
const child = exec.spawn(command, opts)
child.on('exit', (code, signal) => {
if (code || signal) {
return reject(
new Error(`bad exit code/signal code: ${code} signal: ${signal}`)
)
}
resolve()
})
})
}
module.exports = exec

View File

@@ -0,0 +1,3 @@
const globOrig = require('glob')
const { promisify } = require('util')
module.exports = promisify(globOrig)

View File

@@ -0,0 +1,17 @@
function logger(...args) {
console.log(...args)
}
logger.json = (obj) => {
logger('\n', JSON.stringify(obj, null, 2), '\n')
}
logger.error = (...args) => {
console.error(...args)
}
logger.warn = (...args) => {
console.warn(...args)
}
module.exports = logger

View File

@@ -0,0 +1,35 @@
/**
* Shared statistics utilities for benchmark measurements
*/
/**
* Calculate statistical summary for an array of numbers
* @param {number[]} arr - Array of numeric values
* @returns {Object|null} Stats object with median, min, max, mean, stddev, cv or null if empty
*/
function calcStats(arr) {
if (arr.length === 0) return null
const sorted = [...arr].sort((a, b) => a - b)
const mid = Math.floor(sorted.length / 2)
const median =
sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2
const min = sorted[0]
const max = sorted[sorted.length - 1]
const mean = arr.reduce((a, b) => a + b, 0) / arr.length
const variance =
arr.reduce((sum, val) => sum + Math.pow(val - mean, 2), 0) / arr.length
const stddev = Math.sqrt(variance)
const cv = mean > 0 ? (stddev / mean) * 100 : 0 // coefficient of variation as %
return {
median,
min,
max,
mean: Math.round(mean),
stddev: Math.round(stddev),
cv: Math.round(cv),
}
}
module.exports = { calcStats }

View File

@@ -0,0 +1,121 @@
#!/usr/bin/env node
/**
* Test the stats action comment formatting locally
*
* Usage: node test-local.js [--with-history]
*
* This generates a sample PR comment using mock data so you can
* quickly verify formatting changes without running the full action.
*
* Options:
* --with-history Simulate KV history data to test trend sparklines
*/
const addComment = require('./src/add-comment')
const withHistory = process.argv.includes('--with-history')
// Mock data simulating real benchmark results
const mockResults = [
{
title: 'Default Build',
mainRepoStats: {
General: {
nextDevColdListenDurationTurbo: 280,
nextDevColdReadyDurationTurbo: 450,
nextDevWarmListenDurationTurbo: 180,
nextDevWarmReadyDurationTurbo: 320,
nextDevColdListenDurationWebpack: 350,
nextDevColdReadyDurationWebpack: 1200,
nextDevWarmListenDurationWebpack: 250,
nextDevWarmReadyDurationWebpack: 800,
nextStartReadyDuration: 150,
buildDurationTurbo: 4500,
buildDurationCachedTurbo: 4200,
buildDurationWebpack: 14000,
buildDurationCachedWebpack: 13500,
nodeModulesSize: 250000000,
},
},
diffRepoStats: {
General: {
nextDevColdListenDurationTurbo: 290, // +10ms (insignificant)
nextDevColdReadyDurationTurbo: 520, // +70ms at 15% (significant regression)
nextDevWarmListenDurationTurbo: 175, // -5ms (insignificant)
nextDevWarmReadyDurationTurbo: 280, // -40ms at 12% (significant improvement)
nextDevColdListenDurationWebpack: 360,
nextDevColdReadyDurationWebpack: 1180,
nextDevWarmListenDurationWebpack: 245,
nextDevWarmReadyDurationWebpack: 790,
nextStartReadyDuration: 145,
buildDurationTurbo: 4400,
buildDurationCachedTurbo: 4250,
buildDurationWebpack: 13800,
buildDurationCachedWebpack: 13600,
nodeModulesSize: 251000000,
},
},
diffs: null,
},
]
const mockActionInfo = {
isRelease: false,
commitId: 'abc123',
issueId: 12345,
}
const mockStatsConfig = {
commentHeading: 'Stats from current PR',
}
// Run with LOCAL_STATS to output to file instead of posting
process.env.LOCAL_STATS = 'true'
// Mock the KV module to provide fake history if --with-history is passed
if (withHistory) {
// Generate mock history entries showing a trend
const mockHistory = []
for (let i = 0; i < 10; i++) {
mockHistory.push({
commitId: `commit-${i}`,
timestamp: new Date(Date.now() - i * 86400000).toISOString(),
metrics: {
nextDevColdReadyDurationTurbo: 400 + Math.random() * 100, // varies 400-500
nextDevWarmReadyDurationTurbo: 300 + Math.random() * 50,
buildDurationTurbo: 4000 + Math.random() * 1000,
},
})
}
// Inject mock KV client
process.env.KV_REST_API_URL = 'mock://kv'
process.env.KV_REST_API_TOKEN = 'mock-token'
// Patch the require to intercept @vercel/kv
const Module = require('module')
const originalRequire = Module.prototype.require
Module.prototype.require = function (id) {
if (id === '@vercel/kv') {
return {
createClient: () => ({
lrange: async () => mockHistory,
rpush: async () => {},
ltrim: async () => {},
}),
}
}
return originalRequire.apply(this, arguments)
}
console.log('Running with mock history data (10 entries)')
}
addComment(mockResults, mockActionInfo, mockStatsConfig)
.then(() =>
console.log(
'\nGenerated pr-stats.md - open it to see the comment' +
(withHistory ? ' (with trend sparklines)' : '')
)
)
.catch(console.error)