Skip to content

fix(cpp): a call through a template parameter links to each class written as its argument - #3196

Open
colbymchenry wants to merge 1 commit into
mainfrom
claude/fx-template-instantiations
Open

colbymchenry wants to merge 1 commit into
mainfrom
claude/fx-template-instantiations

Conversation

@colbymchenry

Copy link
Copy Markdown
Owner

A call through a C++ template parameter now links to that member of each class the project writes as the template's argument, one edge per class.

Txn::PutEntity(…) in template <typename Txn> class SecondaryIndexMixin names a member of whatever class the template is given, so no lookup at the call can answer it. 1.6.2 reached three such calls on rocksdb by name coincidence. #2723's boundary fix took that rule away, correcting 4,219 edges, and it stays gone. What decides the target is written elsewhere: new SecondaryIndexMixin<WriteCommittedTxn>(…). With several instantiations each really runs its own, so a pick among them is a guess and all of them is the truth.

What it does

cppTemplateEdges is a new pass in SYNTH_PASSES, gated on C++, in a file of its own (src/resolution/cpp-template-instantiations.ts). It reads the failed references that go through a template parameter, finds what each template is written with, and gives each reference one edge per class. Edges are provenance: 'heuristic', metadata.synthesizedBy: 'cpp-template-argument', registeredAt the instantiation, and the hop reads, in codegraph_explore's Flow and the codegraph_node trail:

AddPrimaryEntry → PutEntity   [dynamic: template argument `WriteCommittedTxn` for `Txn` @utilities/transactions/pessimistic_transaction_db.cc:209]
ShardedCache.SetCapacity → LRUCacheShard.SetCapacity   [dynamic: template argument `LRUCacheShard` for `CacheShard` @cache/lru_cache.h:455]
ShardedCache.SetCapacity → ClockCacheShard.SetCapacity   [dynamic: template argument `ClockCacheShard` for `CacheShard` @cache/clock_cache.cc:1394]

The three entries

Entry Result on the real rocksdb index
lost/36 Txn::PutEntity SecondaryIndexMixin::AddPrimaryEntry → WriteCommittedTxn::PutEntity @pessimistic_transaction.h:291, via new SecondaryIndexMixin<WriteCommittedTxn> at pessimistic_transaction_db.cc:209
lost/35 Traits::ClearOutput GetFromBatchImpl → both local Traits: GetFromBatch::Traits::ClearOutput (:920) and GetEntityFromBatch::Traits::ClearOutput (:991)
lost/25 TypeParam::KeyGen Each of the four TYPED_TEST bodies → the six KeyGen member aliases of the 31 listed types that declare one (:213, :309, :319, :324, :332, :336), TypesAndSettings_Hash32_SmallKeyGen::KeyGen at :336 among them

An edge's line is the first call in the function, as with every synthesized edge: the merge keeps one edge per source and target.

How much it reaches

Not almost nothing: 197 references on rocksdb, 1,745 on folly, and real flows among them, ShardedCache → LRUCacheShard on rocksdb and parse_chrono_format → each handler on fmt.

Repo Refs through a parameter Linked Edges No class written Deduced calls only No such member Specialization Over cap
rocksdb 1,623 197 347 1,355 64 5 2 0
abseil 2,247 964 1,013 569 103 609 2 0
protobuf 2,852 405 969 315 2,094 38 0 0
folly 3,374 1,745 1,978 1,051 291 279 7 1
nlohmann/json 1,311 458 479 316 236 299 2 0
fmt 247 95 254 105 44 2 1 0
googletest 136 27 114 79 16 14 0 0

Why it abstained:

  • No class written: the parameter is only ever given int, std::string, a pointer and the like, or a class no lookup finds. 1,325 of rocksdb's are one shape, its vendored gtest's ValueArrayN conversions.
  • Deduced calls only: a function template whose calls deduce the parameter, where the function's own parameter isn't the whole type (std::vector<T>&) or the argument's type isn't known. Protobuf's 2,094 are mostly test helpers such as TestUtil::ExpectAllFieldsSet<TestAllTypes>, given generated classes the checkout doesn't hold.
  • No such member: the class is written and the member isn't its own or inherited as far as the index can tell. 334 of abseil's 609 are one class, ValueTable, whose base is InstantiateRawHashSet<…>::type, a member alias of a template.
  • Specialization: an instantiation a specialization takes runs the specialization's members, so the template's own reference has nothing to link.
  • Over cap: one reference, below.

The cap

40 classes per parameter, chosen from the census with no cap at all on seven repos. The sizes run 52, 38, 38, 37, 37, 31, 31, 25, 24, 22, 22, 21, 21, and the next is 19. 64 would cut nothing, 32 would cut 10 references of real families (protobuf's GetNewImpl, folly's Try), 24 would cut rocksdb's own 31-type ribbon list. 40 sits in the gap and cuts exactly one reference on all seven: folly::detail::OptionalStorage<Value>'s placement new at folly/Optional.h:149, 52 classes, a container. It gets no edges, not some. The cap is per parameter: a parameter under it is linked in full whatever a sibling parameter does.

Forms of an instantiation

From the census, 20,307 sites across the seven repos, each a parameter the pass followed:

Form Sites
Variable, member 8,977
Function template call, written or deduced 4,201
Argument of another template 2,060
Qualifier, Mixin<Foo>::Build() 1,644
Pointer or reference to it 948
Base clause 817
Temporary 595
using or typedef 572
gtest type list 434
new 35
template class Mixin<Foo>; 24

An instantiation inside another template is followed to that template's arguments, four templates at most, each parameter once, so a cycle ends.

What I changed from the design, and why

  • Value receivers. The design read T::x. Failed calls on a variable of the parameter's type outnumber those on every repo (rocksdb 172 against 80, folly 2,272 against 246), and they are the flows: shard.SetCapacity(…) on a CacheShard& shard, handler.on_year(…) on a Handler& handler. The pass types the receiver the way a call is typed, and a -> on a value is the argument's own operator->, so that is left alone.
  • Function templates are read from the call edges the graph already holds, not from text, so a file that doesn't call one is never read for it.
  • Deduced arguments are built where the parameter's whole type is T, const T&, T&& or T* and the argument has a written type. C++ deduces exactly that, so the result is exact. Anything else is counted and left.
  • Lookup knows what a file includes. CppNameSite.sees narrows cppClassWritten: nlohmann/json's tests/abi holds a vendored copy of the library, and two abseil headers each declare a ConstructorTest for separate translation units. The first version sent map tests to set classes (736 edges against the true 475) until this went in.
  • Specializations are read too, since the index names them <anonymous>: a full or partial specialization takes its instantiations away from the template, and its own members read their parameters from what it matched (Foo for T of Name<T*> given Name<Foo*>). A condition (enable_if_t<…>) can't be read from text, so it counts as taken, which costs an edge rather than adding a false one.
  • Registered right after the last C-family pass, so a pair cppEdges or cFnPtrEdges bridges keeps its edge.

Proof

Tests. __tests__/cpp-template-instantiations.test.ts: 106 pass, 87 fail with the pass off (the other 19 say what must not link). Each form, an outside argument, the cap on both sides, adding, changing, removing and deleting an instantiation in a third file against a fresh index, pool against sequential. Mutation: 62 guards broken one at a time, 60 killed, and the two left are a prefilter and a fast path that change no result. That run sharpened a dozen tests and took out three guards that proved redundant. A fourth it called redundant wasn't: json's A/B showed 39 edges into its vendored copy without it, since a lookup through an alias falls back on what it can't see. It went back with a fixture that fails without it.

A/B against main, compared in full: every node, every edge with its column, metadata and provenance, every unresolved ref.

Repo Nodes Edges, main → branch Added Everything else
rocksdb 74,930 342,058 → 342,405 347 identical
abseil 30,665 82,865 → 83,878 1,013 identical
protobuf 126,602 412,990 → 413,959 969 identical
folly 71,884 196,717 → 198,695 1,978 identical
nlohmann/json 14,499 36,616 → 37,095 479 identical
fmt 8,390 20,186 → 20,440 254 identical
googletest 10,567 25,765 → 25,879 114 identical
leveldb (control) 3,093 10,804 → 10,807 3 identical
gin (control) 2,667 8,073 → 8,073 0 identical
flask (control) 2,736 5,292 → 5,292 0 identical

Every added edge is cpp-template-argument. leveldb's control isn't byte-identical on purpose: its three instantiates edges (NoDestructor, SingletonEnv) are true.

Classes added, on the whole diff:

Class rocksdb abseil protobuf folly json fmt googletest leveldb
calls method → method 263 263 213 310 403 87 84 0
calls function → method 45 485 606 967 73 167 22 0
calls class → method 0 26 0 13 0 0 0 0
instantiates method → class 3 27 113 486 1 0 2 3
instantiates function → class 4 176 0 201 0 0 6 0
instantiates function → type alias 24 36 0 1 2 0 0 0
references method → method 8 0 37 0 0 0 0 0

Read against source: 40 on folly, 36 on protobuf, 30 on rocksdb, 30 on fmt, 26 on json, 70 on abseil and leveldb's 3, drawn across every source, kind and parameter: all real. Each repo's edge set is the one those samples came from, except seven folly pairs cppEdges already bridges, which keep its edge. Mechanical checks over all edges (the call line, the site, the member's owner) flagged only aliases and T::_Internal::New-style nested types, which read right. One note: protobuf's HasBits::empty lands on the member of a HasBits<1> specialization, the template's name being all the argument says.

Determinism. Pool, sequential and a repeat: identical edges in identical id order on rocksdb (347 of 342,405), fmt, abseil and json.

Sync. Remove an instantiation, change a typed list, add a third file, then compare every edge with a fresh index of the same tree: identical on rocksdb (all 342,294), abseil and fmt. A file holding only template class Mixin<Foo>; now triggers the pass (hasSynthesisPattern).

Cost on rocksdb, cold index, 10 interleaved rounds, measured as REG did:

main branch
CPU, mean 107.1 s 108.1 s +1.0%
CPU, median 106.9 s 108.2 s +1.2%
Peak RSS, median 2,564 MB 2,550 MB -0.5%
Wall, median 29.0 s 28.9 s -0.1%

CPU sits on the 1% line: paired, +1.1 s ± 0.6 of 107 s. The pass itself takes 0.57 s on a worker. Main got about a quarter faster under this work (#3182, #3183), which is what moved it from +0.7% to here, so the same pass now costs a bigger share. Peak memory comes at the end of cFnPtrEdges and moves between two modes about 90 MB apart from run to run. A rotated bench of a no-op pass and the full pass couldn't tell them apart (against main, no-op +17 MB ± 20, full -20 MB ± 27).

A file that doesn't write a template's name is walked once for its names and never parsed for it. That walk cost more than it looked: it went through the content cache and pushed out what other passes wanted, and a name kept from a file's text keeps the whole text alive (V8 sliced strings). ResolutionContext.scanFile (no cache, no stat, no UTF-8 decode) and detached fixed both, and took the pass alone from 2.8 s to 1.6 s cold and 0.75 s to 0.36 s warm. The walk of the failed refs reads a whole table, so it empties the connection's page cache after, which gives back about 33 MB on rocksdb. Caches are bounded at 4,096 files and 1,024 templates for projects larger than these.

Explore connects on rocksdb and fmt, output above. No extraction change, so no kernel parity run.

Suite on the exact commit, 4 workers, CODEGRAPH_KERNEL_EXPECT=1: 884 files, 10,976 passed, 0 failed, 87 skipped, 2 expected fail.

🤖 Generated with Claude Code

…tten as its argument

`Txn::PutEntity(…)` in `template <typename Txn> class SecondaryIndexMixin` names
a member of whatever class the template is given, so no lookup at the call can
answer it. 1.6.2 reached three such calls on rocksdb by name coincidence, and the
boundary fix that ended that rule left them failed. What decides the target is
written elsewhere, in `new SecondaryIndexMixin<WriteCommittedTxn>(…)`, and with
several instantiations each really runs its own, so every class gets an edge.

A new pass, cppTemplateEdges, reads the failed references that go through a
template parameter: `T::x`, `T::T/2`, what `T x;` constructs, an alias of a member
of the parameter, and a call on a value of its type. It finds what the template
is written with: base clauses, variables, members, `new`, aliases, explicit
instantiations, `f<Foo>()`, deduced `f(x)` where the argument's type is known
exactly, gtest's type lists, and arguments that are another template's parameter,
four templates at most. An argument is a project class by the lookup a declared
type gets, a class local to the function first, in a file that sees it; anything
else gives no edge and takes nothing from the rest. A specialization takes its
instantiations from the template. A parameter that reaches more than 40 classes
gets no edges: on seven repos that cut one reference, folly's OptionalStorage.

The hop names the argument, the parameter and where it's written, in explore's
Flow and the node trail. A walk over every file now has ResolutionContext.scanFile,
which stays out of the content cache and skips UTF-8 decoding.

Validation against main: all three rocksdb entries reached, 106 tests, 87 of them
red with the pass off. Added edges: rocksdb 347, abseil 1,013, protobuf 969, folly
1,978, json 479, fmt 254, googletest 114, leveldb 3, gin and flask 0. Nodes,
unresolved refs and every other edge are identical on all ten repos, with column
and metadata. Pool, sequential and repeat runs give the same edges in the same
order, and sync matches a fresh index on rocksdb, abseil and fmt. rocksdb index,
10 interleaved rounds: CPU 107.1 to 108.1 s mean, +1.0% (median +1.2%), peak RSS
-0.5%, wall -0.1%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant