The Question That No Longer Works

In 2023, asking an engineer whether they used AI was a differentiator. In 2025, it is table stakes. Every candidate says yes. The self-report has become meaningless — not because engineers are being dishonest, but because "using AI" no longer has a shared definition.

One engineer uses Claude to autocomplete boilerplate. Another uses it to navigate a legacy codebase with security constraints that would take three days to understand manually. Both answer "yes" to the same interview question.

The hiring process that cannot distinguish between them will consistently make the wrong decision.

Why Self-Reported AI Capability Is Unreliable

When AI capability influences hiring outcomes — rate, level, project fit — candidates have every incentive to report upward. This is not deception. It is rational behavior in a system that asks the wrong question.

The deeper problem is structural. "I use AI every day" describes a habit, not a capability. It tells you nothing about:

• Whether the engineer can identify when AI output is wrong

• Whether they can feed project-specific context — legacy systems, compliance layers, security constraints — into AI and get relevant results

• Whether their judgment improves with AI assistance or simply speeds up their existing approach

Across 500+ engineering assessments in enterprise contexts, the single most consistent finding is this: self-reported AI usage has near-zero correlation with actual AI-augmented output quality in complex projects.

Three Tiers That Actually Matter

Based on those 500+ assessments, three meaningfully different capability levels emerge. They are not about which tools an engineer uses. They are about what the engineer does with the output.

Tier 1 — AI-Enabled

The engineer uses AI to move faster. Autocomplete, documentation generation, boilerplate reduction. Output volume increases. Code quality at the task level improves.

This is the commodity tier in 2025. Most engineers who say they "use AI" are here. It is real value — but it is not what differentiates candidates in enterprise project contexts where the work is complex, constraints are dense, and the cost of a wrong output compounds.

Tier 2 — AI-Integrated

The engineer knows when to trust the output and when not to. They override AI suggestions when context the model cannot see makes those suggestions wrong. They recognize hallucination patterns specific to the domain they work in.

This tier is where productive engineers in enterprise environments actually sit. The gap between Tier 1 and Tier 2 is not technical sophistication — it is judgment. It develops through projects where the cost of trusting wrong output was real.

Tier 3 — AI-Native

The engineer can feed project-specific context — legacy architecture, security requirements, compliance constraints, organizational history — into AI and get output that is relevant to the actual problem, not a generic approximation of it.

This is the rarest tier. It requires understanding both the domain deeply enough to know what context matters, and AI systems well enough to know how to structure that context for useful output. In enterprise AI projects, this is the tier that makes a team function differently — not just faster.

Observable Behavior Questions — Not Tool Questions

The assessment shift is from "what do you use" to "what do you do when it is wrong."

Questions that reveal Tier 1 vs. Tier 2:

• "Walk me through a task you completed with AI assistance this week. Where did you accept the output? Where did you change it, and why?"

• "Describe a situation where an AI suggestion would have been correct in general but wrong for your specific project. How did you identify that?"

Questions that reveal Tier 3:

• "In a project with a legacy system and compliance requirements, how do you structure the context you give to an AI tool? What do you include, what do you leave out?"

• "Can you show me how you would prompt an AI tool to help you work through a problem in this codebase?" (Live demonstration — the most reliable signal at this tier.)

The quality being assessed is the reasoning, not the answer. An engineer who can articulate why they overrode an AI suggestion — and what in the project context made that the right call — is demonstrating Tier 2 regardless of the tool they used.

Why Joint Assessment Works Better Than Sequential Interviews

Standard interview processes assess AI capability through two sequential conversations: a technical screen, then a delivery conversation. The problem is that candidates adjust between sessions. By the second conversation, they have calibrated their answers to what the technical assessor responded to.

Joint assessment — where a technical evaluator and a delivery evaluator are in the same session simultaneously — generates signals that neither assessor would surface alone. The technical evaluator probes the AI reasoning. The delivery evaluator watches how the candidate handles uncertainty in real time. The emergent signal from that interaction cannot be reconstructed from sequential notes.

In practice, the candidates who perform best in joint sessions are the ones who stop performing. They treat the uncertainty as a genuine problem to work through rather than a test to pass. That behavior is the clearest indicator of Tier 2 and above.

What This Means for Hiring Decisions

Tier placement should directly inform project fit and governance structure, not just rate.

Tier 1 engineers in execution roles with clear specifications and tight feedback loops will outperform expectations. The same engineers in ambiguous enterprise environments with AI-dependent deliverables will struggle — not because they lack skill, but because the role requires judgment that comes from a different tier.

Tier 3 engineers are rare enough that the hiring decision carries significant exit cost. Governance matters more here, not less — because the cost of losing a Tier 3 engineer mid-project is not just the replacement search. It is the loss of the context they were carrying.

Identifying tier accurately at the assessment stage changes what you govern after the hire, not just whether you make it.

The Assessment Shift in Practice

None of this requires a new interview framework from scratch. It requires three changes to existing processes:

1. Remove tool questions. Replace with behavior questions that reveal what the engineer does when AI is wrong.

2. Add at least one live demonstration element — even brief. Self-reported behavior and demonstrated behavior diverge significantly at every tier.

3. Assess jointly when possible. Sequential conversations let candidates manage information. Joint sessions surface the unmanaged version.

The question "do you use AI" stopped being useful some time ago. The question "what do you do when it is wrong" is where the actual signal lives.