Improve Nextjs tracing #5505

Goal: Improve tracing in nextjs SDK - make parameterization more reliable, trace data fetchers, trace page requests for folks on Vercel

TL;DR Plan:

• get page path at build time
• use loader to inject into page as global variable
• use loader to wrap canonical functions
  • getStaticPaths - start transaction, add span
  • getStaticProps - start or continue transaction, add span
  • getServerSideProps - start transaction, add span
  • getInitialProps - needs investigation, can run on client
• use loader to wrap _app or _document
  • start or continue transaction, add span, finish transaction

Effects:

• Parameterized name is known when transaction is started
• Wrapping is at page level, not server level, so works on and off of Vercel (currently tracing for page requests only works off of Vercel)
• Spans for data-fetching functions (none now)
• Hopefully lets us eventually retire instrumentServer (brittle monkeypatching, only works off of vercel)

Open Questions:

• How to deal with domains when action happens in multiple functions at multiple times?
• What about background/pre-load/data-only requests?
• How to communicate why transactions may get marginally shorter?
• How to grab request data if no GSSP?

Tasks

⚠️ ... Required for making changes non-experimental

Prework

General prework necessary to improve the Next.js performance experience as a whole.

• ⚠️ Add loader to inject prefix for RewriteFrames (ref(nextjs): Use loader to set RewriteFrames helper value #5445)
• ⚠️ Create MVP for experimental loader to wrap getServerSideProps, getStaticProps and getStaticPaths during build-time (ref(nextjs): Wrap server-side data-fetching methods during build #5503)
• ⚠️ Actually, don't wrap getStaticPaths (fix(nextjs): Remove experimental wrapping of getStaticPaths #5561)
• ⚠️ Turn loader MVP into sustainable solution (ref(nextjs): Use proxy loader for wrapping all data-fetching functions #5602)

Transaction Name Parameterization

• ⚠️ Have spans and proper transaction names for getServerSideProps and getStaticProps (feat(nextjs): Add spans and route parameterization in data fetching wrappers #5564)
• ⚠️ Have spans and proper transaction names for getInitialProps for normal pages (not _app.js, _error.js, _document.js) (feat(nextjs): Create spans and route parameterization in server-side getInitialProps #5587)
• ⚠️ Have spans and proper transaction names for getInitialProps in _app.js, _error.js and _document.js (feat(nextjs): Instrument server-side getInitialProps of _app, _document and _error #5604)

Connected Traces

• ⚠️ Allow for connected traces when traces start on Next.js server (frontend changes) (feat(nextjs): Improve pageload transaction creation #5574)
• ⚠️ Extract traceparent data from incoming requests in data functions and attach it to transactions and send trace parent data and baggage from data fetching methods (Make sure to handle redirect and notFound responses from getServerSideProps (also getStaticProps?)) (feat(nextjs): Connect trace between data-fetching methods and pageload #5655)
• ⚠️ Find a solution for data fetchers + navigation transactions (feat(nextjs): Trace navigation transactions #5676)
• Somehow mark prefetch requests (add isPrefetchRequest: boolean tag to errors and do the one or more of the following to transactions 1. different name 2. data field 3. different op) (there is a purpose: prefetch header in those requests)
• Solve the case when navigating to a getServerSideProps page that throws: Transaction name is /_error but DSC contains route of getServerSideProps. (See TODO comment in src/performance/client.ts) ([nextjs] Fix transaction name getting lost when hitting _error page #5826)

Serverside Transaction improvements

Currently, for mysterious reasons, in some circumstances no server-side non-API-route transactions are started for Next.js apps. (This is over and above the known limitation of non-API-route tracing not working on Vercel.) The following changes will enable serverside transactions in both those mysterious situations and on Vercel.

• ⚠️ Have serverside transactions by starting transactions inside getInitialProps and getServerSideProps (feat(nextjs): Create transactions in getInitialProps and getServerSideProps #5593)
• ⚠️ Add request data to transactions started by data-fetching functions (ref(nextjs): Use integration to add request data to transaction events #5703)
• ⚠️ Extend RequestData integration to work for error events (ref(nextjs): UseRequestData integration for errors #5729)
• Automatically wrap getStaticProps (Wrapper exists but we don't have access to the request. Is this solvable? Do we even care, given that this generally runs in the background?)
• Add proper error monitoring in data fetchers (by extracting logic from withSentry into a wrapper/helper)
• ⚠️ Figure out how to block res.send similar to how we do it in withSentry (ref(nextjs): Use flush code from withSentry in all backend wrappers #5814)
• Figure out how to propagate scope between data-fetching functions in the same transaction (or document scope-propagation limitations)
• Delete instrumentServer.ts?
• ⚠️ Add "ok" status to transactions and spans and update to "internal_error" on error (feat(nextjs): Add status to data-fetcher spans #5777)
• What happens for pages with no data fetchers? Should _app have a different helper? (no name for transaction)
• ⚠️ Automatically wrap API routes (feat(nextjs): Auto-wrap API routes #5778)

Folow-up/Cleanup/Polishing

• ⚠️ Remove MVP loader code and remove its dependencies from package.json (chore(nextjs): Remove obsolete dataFetchers loader #5713)
• Use RequestData integration everywhere
• Check if new model works with CJS, if no, support people that use CJS
• ⚠️ Check Webpack 4 support of the loaders we created (our integration tests run under webpack 4 just fine, so I think we're good here)
• Bring withSentry into the fold (consolidate helpers)
• Nudge users to stop wrapping API routes if using auto-wrapping (once auto-wrapping includes API routes)
• Factor more common parts out of wrappers if possible

Documentation

• Document new RequestData integration and options
• Document auto-wrapping

Misc (stretch goals)

• Have spans for getInitialProps when run client-side
• Add instrumentation for how long server-side rendering takes
• Investigate how it would be possible to support custom servers
• Can common route-handling work be consolidated into helper functions across frameworks, so we're not always reinventing the wheel?

Steps for Beta

• Rename option and make it an actual option (feat(nextjs): Promote option to automatically wrap data fetchers and API routes to non-experimental #5793)
• Add option to wizard (feat(nextjs): Add option to auto-wrap data fetchers and API routes to Next.js config sentry-wizard#194, meta: Update changelog for 2.2.1 sentry-wizard#195)
• Update docs that explains auto wrapping and new option (Add docs for Next.js SDK option autoInstrumentServerFunctions sentry-docs#5542)
• Update Next.js examples (Remove references to withSentry function in with-sentry example vercel/next.js#41326)

Steps for GA (Planned date: October 5, 2022)

• Remove option from wizard (feat(nextjs): Remove option to auto-wrap data fetchers and API routes sentry-wizard#196)
• Flip default for option (feat(nextjs): Enable autoInstrumentServerFunctions per default #5919)
• Update docs - removing the section where we tell people to wrap routes with withSentry (Remove references to manually use withSentry from Next.js docs sentry-docs#5543)

Follow ups

Undone tasks from above have been sorted into:

• Clean up/consolidation of wrapper code (in nextjs, and across packages: Consolidate request-handling utils #5599)
• Expanded coverage of less-common/not-yet-handled cases ([nextjs] Handle more cases when auto-instrumenting tracing #5827)
• v8 deprecations - withSentry and possibly instrumentServer (JS SDK v8 Deprecations List #5194)
• Handling _error better ([nextjs] Fix transaction name getting lost when hitting _error page #5826)
• Further work on RequestData integration (RequestData integration prework, work, and postwork #5756)