The Problem With How Most Teams Screen for AI Capability

Every engineer on the market today says they use AI. That statement is now the baseline, not a differentiator.

The question hiring managers at IT consulting firms in Singapore are actually trying to answer is different: which engineers produce reliable output when the AI is wrong?

That distinction does not show up in CVs. It does not show up in portfolio links. It rarely shows up in a standard technical interview. After reviewing over 500 engineering assessments conducted across projects in Southeast Asia over six years, we found that most evaluation processes test the wrong thing — they test whether a candidate can use AI tools, not whether they can govern their own output when those tools fail.

Three Observable Tiers of AI-Augmented Engineers

The following framework is based on behavioral signals observed across 500+ assessments. The tiers are not about tool preference or familiarity. They are about what the engineer does when the AI produces a plausible but incorrect result.

Tier 1 — Output-dependent. Uses AI to generate code, documentation, and solutions. Reviews output visually. Accepts it if it looks correct. Cannot reliably identify errors that are syntactically valid but logically wrong. Signs: code quality degrades under time pressure; debugging process starts with re-prompting.

Tier 2 — Output-aware. Uses AI as a drafting tool, then validates against their own understanding. Can identify a category of error before running the code. Debugging process starts with reading. These engineers exist, but are harder to find than Tier 1 candidates who have learned to interview well.

Tier 3 — Output-governing. Uses AI to accelerate work they already understand how to do without it. Treats AI output as a first draft with known failure modes. Has a mental model of where the model is likely to be wrong for this specific problem. In our dataset, this group represents roughly 15–20% of candidates who describe themselves as "AI-proficient."

For a practical guide to spotting these tiers on a CV before the interview stage, see what 'AI-proficient' actually means on a CV.

The Evaluation Questions That Actually Reveal the Difference

Standard interview questions do not separate Tier 1 from Tier 3. The signals come from a different set of prompts.

Question 1: "Walk me through the last time an AI-generated solution failed in production or in testing. What did you catch? How did you catch it?"

A Tier 1 engineer will describe a situation where a teammate or a test caught the error. A Tier 3 engineer will describe the specific logic error, when they noticed it, and what their mental model predicted before they ran anything.

Question 2: "Describe a problem where you deliberately chose not to use AI. What made you make that call?"

Tier 1 engineers struggle with this question — not because they haven't had the experience, but because they do not have a conscious framework for when AI assistance degrades output quality. Tier 3 engineers answer immediately.

Question 3: "If I gave you a section of AI-generated code to review right now, what would you look for first?"

This is a live assessment question. Tier 1 candidates default to running it. Tier 3 candidates describe what categories of errors they would look for before touching a keyboard.

A full 45-minute interview structure using these three questions — with expected answer patterns per tier — is covered in the governance-focused interview framework.

A Starting Evaluation Framework

For hiring managers evaluating AI-augmented engineers today:

Step 1 — Separate AI familiarity from AI governance. Assume all candidates use AI tools. The screening question is not whether they use them. The screening question is whether they can govern their own output.

Step 2 — Use behavioral retrospective questions, not hypotheticals. Ask about specific past failures, not about what they would do in a hypothetical scenario. Tier 1 candidates answer hypotheticals well. They often struggle with specifics.

Step 3 — Assess the failure model, not just the output. In any live technical assessment, create one situation where the AI-generated answer is plausible but wrong. Observe whether the candidate catches it, how quickly, and what process they use. This single signal predicts Tier classification more reliably than any other single indicator in our dataset.

Why This Matters for the Next 18 Months

The market for AI-augmented engineering talent is moving faster than the evaluation infrastructure that governs it. The engineers who will be available at scale by late 2026 will be predominantly Tier 1 by our classification — capable, fast, and increasingly difficult to evaluate with traditional methods.

The consulting firms that build governance frameworks now — for evaluation, for onboarding, for ongoing visibility into project health — will have a structural advantage that does not replicate quickly. The advantage is not the framework itself. It is the data that accumulates inside it.

This article is based on assessments conducted across 30+ projects and 500+ engineers in Southeast Asia from 2019–2025.