Automatically annotates types across an entire codebase, even one too big to fit in a model's context window, by treating it as a dependency graph problem.
Python
CodeQL
NetworkX
vLLM
Slurm / HPC
Qwen3
Scaffolding over scale: pin an LLM to one small, well-posed task and let procedural code manage the context. Done right, a 0.6B model that runs on a laptop can annotate repositories no frontier model can even fit in its window.
Python and JavaScript dominate web backends (Django, Flask, FastAPI, Express) and are dynamically typed — type information isn't enforced at runtime. That invites type-confusion vulnerabilities, where a program processes an object of the wrong type and misbehaves in security-relevant ways. Reliable type annotations are the foundation for catching these statically, before they ship.
LLMs are a natural annotator. They read code semantically and generalize across languages. However, the quality of any downstream type-enforcement work is capped by the quality of that first annotation pass, so the real question is whether an LLM can annotate types accurately at repository scale, not just within one file. Security is what motivates this, but it sits downstream: deciding which functions are worth guarding with runtime checks is a judgment you can only make once the types exist. Accurate repo-wide annotation is the prerequisite, and it's what this project sets out to deliver.
A value's type in one file is often determined by a definition in another, whether it be an imported class, a function's return type, or a parameter set by its caller. A model that sees one file at a time guesses at these and gets them wrong.
The naive fix of "dump the whole repo into the prompt" fails on real codebases. They exceed even long-context models, a failure we hit repeatedly during baseline runs. A single function's transitive dependency tree can be a directed acyclic graph (DAG) of depth 4+ that alone would exceed an LLM's context window. The framework's job is to compute exactly the context each function needs, and nothing more.
The system splits the work between two kinds of reasoning. Static analysis (CodeQL) deterministically extracts the facts it can prove: who calls what, how arguments flow, which methods are used. The LLM handles the genuinely ambiguous part: inferring a semantic type from a function's intent, where no fixed rule applies. The LLM proposes; static analysis constrains and corrects. That division is the core design.
Concretely, typing is reframed as a graph problem. Functions become nodes; caller→callee relationships become edges. Resolving the graph in dependency order means every function is typed only after the things it depends on already are, so their inferred types become grounded context for it.
CodeQL converts the entire repository into a queryable database, then seven queries extract a rich evidence set: function definitions, the call graph, return-value usage, parameter method/attribute accesses, local-variable assignments, developer isinstance() constraints, and call-site argument types.
NetworkX assembles a graph where nodes are file_path::func_name (carrying file, name, def_line, return_hints) and edges are caller→callee, annotated with argument positions. External/stdlib calls become stub nodes with file="".
Real call graphs contain cycles (mutual recursion), so rather than deleting edges, strongly-connected components are condensed into single super-nodes via NetworkX, yielding a true DAG over the condensation without losing dependency information.
A reverse topological sort orders the DAG so that every function is typed only after the functions it calls (its callees) are already done, so each resolved callee type is available as context when its callers are typed.
A pipeline walks the sorted DAG and has the LLM type each function in turn, accumulating known types as it goes. Each inference reuses the already-resolved types of the functions it calls.
--language=python for another target reuses the same query structure..pyi stub, so the output mechanism ports too.Each per-function prompt is assembled from graph-derived evidence rather than raw file dumps:
The raw source of the single function being typed.
How callers pass arguments into this function, attached to DAG edges before annotation.
Every x = f(...) assignment emits a (callee, caller_file, caller_name, lhs_var, line) record, capturing how the function's output is consumed.
Types resolved for already-typed callees, accumulated as the walk proceeds.
The model returns only a JSON object: {"param": "type", ..., "return": "type"}.
.pyi stub. That keeps annotations usable for downstream taint analysis and type enforcement, and is what makes the approach generalize beyond Python.
The model's output is treated as a fallible proposal, not ground truth. A static-analysis validation layer overrides or discards annotations whenever CodeQL evidence contradicts them:
CodeQL points-to inferences (majority-voted across call sites) and developer-written isinstance() constraints take precedence over the model's guesses, static evidence wins.
If a parameter is annotated as a builtin but its observed method calls don't belong to that type (e.g. annotated list but .upper() is called), the annotation is dropped as inconsistent.
Annotations that are actually function names masquerading as types (a common LLM error that mypy rejects) are detected via CamelCase/snake_case heuristics and removed.
Targeted fixes encode domain rules: widen int→float when local evidence shows float literals or division; drop None returns on generator functions; discard bare object (the model signalling "I don't know").
Beyond annotations, runtime isinstance assertions are injected into the rewritten functions, turning inferred types into actual runtime checks. This serves as the bridge from annotation to type-confusion prevention.
An evidence threshold guards precision: a function is only annotated when there are at least two call-site arguments or explicit isinstance evidence, as a single speculative hint isn't enough. The guiding rule throughout is that an omitted annotation beats a wrong one, since a single bad cross-file type can cascade.
A representative function, before and after the pipeline runs. Inline type annotations are added from cross-file evidence, and runtime assertions are injected for the types that can actually be checked at runtime.
def load_config(path, parser):
raw = path.read_text()
cfg = parser(raw)
return cfg.get("settings")
def load_config(path: Path, parser: Callable[[str], dict]) -> dict: assert isinstance(path, Path) raw = path.read_text() cfg = parser(raw) return cfg.get("settings")
path and parser are inferred from how callers pass them and how they're used. An assertion is emitted for path but not parser: a Callable signature can't be verified by isinstance, so the validation layer annotates it without asserting, enforcing only what's safely checkable at runtime.Feeding the model explicit type evidence drove the biggest gain. Where the code itself reveals a type (an isinstance() guard, a string method like .upper()) that evidence is extracted and injected into the DAG context. On Qwen3-30B-A3B-Instruct this single design change cut cross-file type errors (mypy-detected, lower is better) by ~37%, from 86 down to 54, providing a direct demonstration that the right scaffolding, not a bigger model, is what moves the score.
Seeding root nodes didn't. Pre-filling ground-truth types at root nodes barely improved results, and would defeat the purpose of this project, since a user would then have to hand-type a large fraction of the repo. Unseeded operation was retained.
Coder models lost to instruct models. Qwen-3 Coder variants underperformed: trained for code completion rather than instruction-following, they were worse at emitting the rigid JSON schema the pipeline demands and at honoring the prompt's constraints. Standard instruct LLMs handled the structured-output task far better.
Evaluated on Typybench, a benchmark of 50 well-typed Python repositories, each provided in both typed and untyped form. Inference ran en masse on the Pittsburgh Supercomputing Center cluster (batched Slurm jobs serving the Qwen3 models through vLLM) which is what made evaluating across all 50 repositories at multiple model sizes tractable.
| Across all 50 repositories (Qwen3-8B) | TypeSim Ratio |
|---|---|
| Median | 0.84 |
| Mean | 0.78 |
| Repos scoring ≥ 0.80 | 29 / 50 |
| Repos scoring < 0.50 | 4 / 50 |
| Model | TypeSim Ratio | TypeCheck | Pred. Errors | Truth |
|---|---|---|---|---|
| Qwen3-0.6B | 0.840 | 0.284 | 42 | 4 |
| Qwen3-8B | 0.856 | 0.583 | 18 | 4 |
| Qwen3-30B-A3B | 0.872 | 0.602 | 18 | 4 |
Across all 50 repositories, the small models held up remarkably well: Qwen3-8B averaged a 0.781 TypeSim Ratio and Qwen3-0.6B 0.752 (a gap of just 0.029) meaning the 0.6B model retained about 96% of the TypeSim Ratio of a model more than ten times its size. Bigger models type complex code somewhat better, but a sub-billion-parameter model running in a 32K window stays well within a usable range. The scaffolding, more than the model size, is what carries the performance.
The per-repo distribution is the useful part: the method is reliably strong across most of the benchmark, with a small, sharp tail. The top performers are well-structured, primitive-heavy libraries: pip, Pillow, flask, poetry, and faceswap all clear 0.96. The weakest are exactly the codebases built on heavy generic and abstract typing, such as pydantic (0.33), fastapi (0.37), OpenBB (0.29), where a function's “type” is often a complex parameterized generic the system doesn't yet resolve. The method excels where types are concrete and degrades where they're abstract: a clean, predictable operating envelope.
An early puzzle: raw TypeSim trailed the Typybench paper's numbers badly, even though exact and overall TypeSim were close together, a sign the system was typing its targets correctly. Investigation revealed the cause wasn't capability but scope: the framework annotates function signatures (language-agnostic, the part that matters for cross-file flow), while Typybench's metric heavily rewards annotating bare local variables (x = 5 → x: int = 5), which the system deliberately didn't target.
Rather than chase the metric, the TypeSim Ratio was introduced to measure performance on what the system actually types. Then a post-hoc mypy pass would close most of the remaining gap while preserving language independence (the same trick works for JS/TS). Diagnosing this as a measurement-scope mismatch, not a model failure, was as valuable as any raw score.
Two claims, kept separate. First: the system works. It annotates entire repositories end-to-end (including ones no baseline could load) at TypeSim Ratios that track ground truth well. That was the hard engineering goal, and it was met.
Second, the benchmark comparison: on gptme, Claude Sonnet with the full repo in context produced 8 cross-file type errors to this framework's 18, and overall the system landed near GPT-4o. It did not beat the strongest baseline on raw error count—the secondary metric. But two factors reframe what that comparison actually measures:
gptme, the headline test repo, is a widely-used open-source project that was likely in Claude's and GPT-4o's training data. This is training-data contamination: a baseline may be recalling memorized annotations rather than inferring them, which undercuts the comparison. The fair test is unseen code, where neither side has memorized the answer. However, it is difficult to source this given the lack of knowledge on the training data of these frontier models.
The gap that remains traces to one identifiable, fixable mechanism. Functions are typed in reverse topological order (callees before the callers that depend on them) so each resolved type becomes context for the calls above it. If an early callee is typed wrong, that mistake is inherited by every caller in its chain. TypeCheck is an unforgiving metric: a couple of bad cross-file types collapse the score. It's a scaffolding-robustness problem, not a ceiling on the approach.
The decisive contribution, though, is one the baselines structurally cannot match: the system annotates repositories that exceed any frontier model's context window, with models as small as 0.6B, where the full-context approach simply hits a wall (and rate limits) on anything beyond a few small repos.
The design is language-agnostic but was validated only on Python. An attempt to ground the work in real vulnerabilities via PyVul (1,157 reported Python CVEs) stalled. Its type-confusion cases turned out to live in the C++ layer, rather than the Python code, of libraries such as TensorFlow. However, one genuine Python case (python-ldap) was found by searching advisories directly. And on measurement: mypy-integration bugs prevented a complete TypeCheck readout, so TypeSim (which correlated strongly with TypeCheck where both were available) served as the primary surrogate.
Each of those, and the error-propagation finding above, maps to a concrete next step. The following are directions, not yet results:
python-ldap lead) to measure detection on cases no model has memorized to ensure a fair measurement.The broader takeaway holds regardless: scaffold an LLM so it does one small thing well while procedural code manages context and orchestration. Good scaffolding can lift a tiny model to frontier-adjacent quality, but bad scaffolding can underperform a naive dump, and the line between them is where the engineering lives.
Source for this graduate-level project is kept private to comply with Carnegie Mellon academic-integrity policy; architectural walk-throughs are available on request.