navigaraMethodology
IssuesAI ROILive dataMethodologyContactWhite paper
MethodologyOpen sourceQ1 2025 – Q2 2026

How we measure
engineering performance.

A two-stage scoring engine: an LLM classifies the work, deterministic algorithms compute the weight. Reproducible, file-by-file, no black box.

Overview

What ETV measures and why it exists.

A model reads every diff the way a senior engineer would, and grades what the change was worth instead of how much was typed.

It scores each change against the repository around it: what calls what, what a table is read by, what breaks downstream. Those file scores roll up to engineers and organizations with nothing estimated on the way, so every number here walks back down to the diffs it came from.

That unit is ETV — Engineering Throughput Value. It is applied per file, per merged commit. A commit touching ten files contributes ten independent measurements. Five factors set the weight of each one:

  • Complexity — structural weight and cognitive load of the modified lines, computed per function scope.
  • Engagement — how much of the surrounding codebase the change had to reason about, so targeted edits in dense areas score higher than equivalent edits in trivial files.
  • Architecture — where the change lands in the feature graph; cross-feature surfaces and integration seams carry more weight than leaf-node edits.
  • Decay — reduces credit when a change isn't real cognitive work: mechanical refactors, self‑rewrites of yesterday's code, copy‑paste from elsewhere in the repo.
  • Multiplier — amplifies fixes when the bug was costly: old code, unfamiliar code, high‑churn areas.
per_file_score = f(complexity, engagement) × architecture × decay × multiplier

Each file lands in one of five buckets: Features, Maintenance, Tests, Docs, or Fixes. There is no single “performance score.” ETV is additive inside a bucket and deliberately not additive across them — two engineers with identical totals can be doing very different work, and the split shows it.

ETV is answerable from commit history alone. No PM tools, no surveys, no self‑report.

The Five Buckets

Features, Maintenance, Tests, Docs, Fixes — classified per file.

The classifier reads what the diff actually does, not what the commit message claims. Conventional Commit prefixes are a hint, not a rule. Classification is per file: a single commit can carry feature work in one file, a test in another and a fix in a third, each scored independently.

Features

feat

New functionality and net-new capabilities. Added endpoints, new modules, new product surface area.

Maintenance

chore · refactor · perf · style · build · ci

Production-code upkeep: refactors, cleanup, performance tuning, dependency updates, style, build and CI.

Tests

test

Test files and fixtures — unit, integration, end-to-end. A category of its own, never folded into production work.

Docs

docs

Documentation files, architecture notes and comment-only changes. Scored mechanically like anything else; typically low complexity and low engagement, so the contribution is small.

Fixes

fix

Work that corrects previous output — bug fixes, regressions, hotfixes. Each fix is traced back to the commit that introduced it.

A healthy codebase typically shows steady feature work, moderate maintenance, tests that track the feature work, and few fixes. A spike in fixes is a signal worth investigating.

Scoring Engine

Two stages: the AI decides what kind of work; the algorithms decide how much.

Two things happen to every merged commit, in order.

Stage 1 — AI analysis

The model reads the commit message, the full diff and the surrounding code context. For each changed file it classifies the work type, identifies the changed symbols (functions, classes, endpoints) and, for bug fixes, traces the issue back to the originating commit — recording the original author and timestamp. Results land in a knowledge graph connecting commits, files, symbols and issues.

Stage 2 — Mechanical scoring

Deterministic algorithms compute complexity and engagement over the same files and apply the architecture, decay and multiplier factors. No LLM is involved at this stage. Identical inputs produce identical scores on every run.

Calibration

Thresholds and coefficients inside the factors — dampener sensitivities, engagement bounds, fix-multiplier curves — are calibrated against a corpus of labelled commits and recalibrated periodically. They are identical across every organization; there is no per-customer model training. The formulas that consume them are fixed and auditable.

Complexity & Engagement

The two deterministic inputs behind every file score.

Context complexity

Computed per function scope over the added and modified lines, so the same number of changed lines can carry very different weight depending on what those lines do. The calculation works consistently across languages and paradigms: a React component blending JSX, JavaScript and CSS-in-JS is scored on the same basis as a plain Go file.

Engagement

How much of the existing codebase the change had to reason about, derived from two views:

  • Inside the file. For each modified function, Navigara identifies the existing lines the change actually interacts with — those sharing identifiers with the changed lines, and those flowing into or out of calls on the changed lines via data-flow analysis. Unrelated code in the same file is ignored.
  • Across the repository. When a change alters a function's inputs, outputs or externally-visible behaviour, the affected values are traced into callers and callees elsewhere in the repo — the same reasoning the engineer had to do to make the change safely.

Engagement reflects surface area the developer actually had to understand, not a raw count of references. Engagement from heavily-reused utilities is bounded, so one-line edits to universal helpers don't dominate the score.

Architecture

A feature graph inferred from code, used to weight changes by where they land.

Navigara builds a structural model of each repository — a feature graph. The graph is derived from code organization alone (no external metadata, no PM tooling) and informs how per-file scores are weighted.

Feature graph

The engine discovers distinct named features (e.g. auth, billing, checkout) and assigns each to a vertical layer — frontend, backend, or data. Edges between feature nodes capture inter-feature dependencies.

Commit → feature mapping

Each commit is mapped to one or more features via weighted path scoring (exact path match > directory containment > filename affinity). A single commit can touch multiple features and is split accordingly.

Architecture multiplier

Integration points and cross-layer seams receive higher multipliers than leaf-node edits. A change inside a deeply connected feature carries more weight than the same change inside a peripheral one.

Inputs: code structure only. No connected ticketing system, no external metadata.

Decay & Amplification

Where credit is reduced — and where it's amplified.

Several factors run before the per-file score is finalized, and all of them apply before aggregation. Three dampeners reduce credit when a change exists but doesn't represent genuine cognitive work; one multiplier amplifies fixes when the surrounding signals say the bug was costly.

Similarity dampener

Reduces credit when the change's structure closely matches patterns already in the codebase — mechanical refactors, boilerplate replication. Works on structural signatures of the change.

Blame decay

Discounts changes that overwrite very recent work by the same author. The signal fades over a short business-day window — rewriting your own code from yesterday is partial credit; revisiting it weeks later is scored normally.

Copy decay

Reduces credit when added lines are literally duplicated from elsewhere in the repo. Works on the text of the diff.

Fix multiplier

For Fixes only. Once a fix is traced back to the commit that introduced the bug, the score is amplified to reflect how much context a reader had to rebuild in order to fix it safely. It grows with how long the bug lived in the codebase, whether the fix touches another author's code, and how frequently the affected area has been modified since. A trivial self-fix on code written the same day barely moves the score; a fix in a high-churn area on code the fixer has never touched is amplified substantially.

Worked Example

One commit, six files, five different outcomes.

A single commit adds session refresh to an API. Each file is classified and scored on its own; the commit's totals are the sums by work type.

api/auth/session.ts

Feature · large

Adds refreshSession(). Substantial context complexity (new control flow, async error paths) and high engagement — its return value flows into call sites in other files that all had to stay compatible.

api/auth/login.ts

Feature · moderate

Wires the new function into the login handler. A small local edit, but engagement is lifted by cross-file data flow rather than the size of the diff.

api/payments/charge.ts

Fix · notable

A null-check bug introduced by a different engineer months earlier, in a file that has churned since. Small context complexity, but the fix multiplier amplifies it significantly.

api/auth/session.test.ts

Test · moderate

Unit tests covering the new paths. Moderate complexity across the added cases; engagement limited to the function under test.

docs/auth.md

Docs · small

Documents the refresh flow. Minimal context complexity, no meaningful engagement.

.github/workflows/test.yml

Maintenance · small

Bumps the Node version used in CI. Minimal context complexity, no meaningful engagement.

Merge Policy

Squash, merge or rebase — the totals converge.

Performance is defined over whatever commits exist on the default branch after landing. When a branch lands via merge commit, each constituent commit is scored individually and later commits are dampened where they overlap with earlier ones (similarity, blame decay, copy decay). When the same branch is squashed, the resulting commit carries the full scope in one shot, with no intermediate overlap to dampen.

The dampening factors are calibrated so these paths converge. Totals come out close either way, so you don't need to change how you merge to get meaningful numbers.

What's Excluded

Generated code, lockfiles, binaries — out before scoring.

The filter list runs before both AI analysis and mechanical scoring, and is identical across organizations. If a commit shows fewer scored files than files changed, the difference is filtered files — the knowledge graph records which, and why.

  • Generated code. Protocol Buffer output (.pb.go, _grpc.pb.go, .pb.ts), GraphQL codegen (.graphql.ts), OpenAPI specs, and machine-generated files matching *_generated.go, *.gen.go, zz_generated.*.
  • Dependency lockfiles. go.sum, package-lock.json, yarn.lock, pnpm-lock.yaml, Cargo.lock, Gemfile.lock, and similar.
  • Build artifacts. dist/, build/, .next/, vendor/, node_modules/, and bundled outputs carrying content hashes.
  • Minified files. Detected by content heuristics when average line length exceeds 300 characters.
  • Binary and media files. Images, fonts, PDFs, archives, compiled binaries.

Teams with unusual generated-code conventions or build layouts can extend the filter list from organization settings.

Languages

Full structural analysis for 13 languages; partial for the rest.

Full analysis

C · C++ · C# · Go · Java · JavaScript/TypeScript (incl. JSX/TSX) · Kotlin · PHP · Python · Ruby · Rust · Scala · Swift

Fuzzy code skeletal matching, data-flow analysis, architectural outline extraction, function-scope context complexity.

Partial analysis

HTML · CSS · SQL · Terraform · shell · YAML · Markdown

Work-type classification still runs and the change still contributes to the score; context complexity is approximated from line-level signals rather than structural analysis.

Context Boundary

The repository is the unit of context — deliberately.

Engagement is measured against the repo's other functions and call sites; context complexity is computed against local function scopes. Navigara does not perform cross-repository analysis: a change in repository A carries no engagement weight from repository B, even if the two are related services.

This is deliberate. The repository is the natural boundary of abstraction in most engineering organizations — it has a coherent build, review process and ownership model. Cross-repo call graphs exist in practice but are rarely stable enough to use as a measurement substrate.

The consequence: scores are not automatically comparable across repositories of very different size or language mix. When comparing teams that work in different repos, compare trends within each team rather than raw totals across teams.

Aggregation

How per-file scores become the headline number.

File-level scores sum per work type, and those totals roll up through three levels with nothing estimated or re-weighted on the way.

Per commit

Five per-file work-type sub-scores, summed within each type.

Per SWE, per quarter

Sum of that engineer's merged commits by work type over the quarter.

Per org, per quarter

Mean ETV across the organization's qualifying software engineers that quarter.

Cross-org aggregate

Developer-weighted mean. Every qualifying SWE contributes one observation per quarter, weighted equally regardless of organization size. A 30-person org and a 200-person org are pooled engineer-by-engineer, not org-by-org, so the headline is never dominated by the largest org.

Cohorts and intervals

Quarter-over-quarter comparisons use fixed-period cohorts: the intersection of qualifying software engineers across every quarter in the reporting window, each of whom must have merged at least one scored commit in every quarter. Contributors appearing in only some quarters remain visible in per-quarter aggregates but are excluded from comparative trend analysis. Confidence intervals are bootstrapped at 95% (resampling with replacement) rather than assuming a parametric distribution.

Figures in the report label the unit as “performance” for readability. The formal definition is ETV, and the five buckets stay separate underneath it.

Attribution

Who gets credit for a commit, and when it counts.

Primary credit goes to the git author of the merged commit, after email-alias resolution. Co-authors recorded in commit trailers are tracked in the knowledge graph but do not receive score credit. A contributor active in multiple connected repositories within one organization is counted once, with their output summed across repos.

Commits are attributed to the time window in which they were merged to the default branch, not when they were authored — so out-of-order merges land in the quarter they actually shipped. In AI-native teams work typically ships in days rather than weeks, so merge-date attribution closely tracks when the work happened.

Automation accounts (dependabot, renovate, github-actions) are excluded by default. Organizations can flag additional bots or service accounts; flagged contributors stay visible in the knowledge graph for audit, but their commits do not contribute to aggregated metrics. Non-engineering roles are excluded from both the numerators and the denominators of reported figures.

Supporting Roles

Why the team view is the primary lens.

Senior engineers spend significant time on code review, architecture, mentoring and technical decisions — work that rarely produces commits. Per-commit scoring does not capture these contributions directly, and aggregating commits to individuals will systematically underrate them.

A strong supporting engineer raises the output quality of everyone around them: fewer bugs (fewer fixes), cleaner architecture, faster onboarding. That impact shows up in the team's aggregate even when the individual's own commit-based score is low. Individual scores remain useful for understanding work distribution and spotting trends, but they are not a complete performance picture where supporting roles are present.

Limitations

What ETV does not measure.

ETV is descriptive, not normative. It tells you what shipped, not whether the right thing shipped.

  • Public default branches only. Private forks, unmerged work and commits in repositories that were not connected are not scored. A contributor who does most of their work outside the analyzed set will appear low.
  • Merged code only. Code review depth, incident response, planning, mentorship and pair-programming sessions that never produce a standalone commit carry no ETV.
  • Cross-repository comparisons are not straightforward. Engagement is repository-scoped and cross-repo data flow is not modelled, so repositories of very different size or language mix produce ETV on different effective scales. Cross-organization totals are not auto-normalized.
  • Co-authors are tracked but uncredited. The merged commit's git author receives the score; pair- and mob-programming need to be reconstructed externally. Author-rewriting tools — squash policies that discard original authorship, or AI coding assistants that replace the human author — shift credit accordingly.
  • Classification is probabilistic per file. Individual files can be misclassified at the edges; aggregates are stable despite those cases.
  • Causal claims require care. ETV moving up after a tooling change is consistent with the tooling helping. It is also consistent with several other explanations. ETV describes output, not business impact, and the report deliberately avoids causal language.
  • Business-goal alignment is out of scope here. ETV measures what shipped, not whether what shipped advanced a particular business objective. Navigara's Alignment concept extends the model toward that question by mapping engineering output to quarterly goals and OKRs, but it requires either a connected goal-tracking tool or manually entered goals. It is not part of this report.

Display Modes

Five buckets, or Features + KTLO.

Features / Maintenance / Tests / Docs / Fixes is the canonical view, and the one used throughout the report. For executive and cross-team reporting, the product also offers a two-bucket view: Maintenance, Tests, Docs and Fixes combined into Keep The Lights On (KTLO), with Features on its own — a clean split between new value and everything else.

Switching modes changes only how the data is displayed. The underlying five-category analysis is unchanged, and nothing is lost by switching back.

FAQ

Four questions the score usually raises.

Why did my big commit score low?

A large diff isn't automatically a high score. If most added lines are structurally similar to existing code, literally duplicated from elsewhere, or don't interact with much surrounding code, the base score stays modest. Bulk renames, regenerated migrations and generated bindings score low by design.

Why did a one-line change score high?

Because complexity and engagement can both be high in a single line. Rewriting one line inside a hot function called from dozens of places, or correcting an old bug in a high-churn area, produces a score larger than the line count suggests.

Do documentation changes count?

Yes. Documentation files and comment-only changes are classified as Docs and scored mechanically like anything else. They typically carry low complexity and low engagement, so the contribution is small.

Does reverting a commit score negatively?

No. A revert is classified on intent — reverting a broken feature tends toward Fixes, reverting a merge-timing mistake toward Maintenance — and scored like any other change. The original commit's score is not retroactively removed.

The full appendix is in the white paper.

Sample floor, fixed-panel sanity check, the 418-engineer constant-population result, OpenAI's four-quarter window, and the complete list of 66 repositories analyzed.

Read the white paperProduct docs
© 2026 NavigaraMethodology · open source