Every engineer on the market today says they use AI. That statement is now the baseline, not a differentiator.

The question hiring managers at IT consulting firms are actually trying to answer is different: which engineers produce reliable output when the AI is wrong?

After 500+ technical assessments conducted across Southeast Asia over six years, we found that most evaluation processes test the wrong thing.

Why “I Use AI” Tells You Nothing

The problem is structural. In a market where every CV lists ChatGPT and Copilot as tools, the signal to noise ratio has collapsed. Hiring managers are left with three options: ignore AI claims, take them at face value, or build a framework to evaluate AI proficiency directly.

Almost no one does the third. This article shows how.

The Three Observable Tiers

These tiers are not self reported. They are observable through specific behaviours during technical discussions, code reviews, and structured interviews. Every engineer we assess falls into one of these three categories.

Tier 1 · Output dependent (65–70% of market)

Uses AI to generate code and documentation. Reviews output visually. Accepts it if it looks correct. Cannot reliably identify errors that are syntactically valid but logically wrong.

Observable signal: When asked about an AI failure, they describe a situation where a teammate or a test caught the error. The answer is never specific.

Tier 2 · Output aware (15–20% of market)

Uses AI as a drafting tool, then validates against their own understanding. Can identify a category of error before running the code. Debugging process starts with reading, not re prompting.

Observable signal: Can walk through a piece of AI generated code and explain why a specific section is correct or incorrect, not just what it does.

Tier 3 · Output governing (15–20% of market)

Uses AI to accelerate work they already understand how to do without it. Treats AI output as a first draft with known failure modes. Has a mental model of where the model is likely to be wrong for a specific problem.

Observable signal: Describes a specific failure mode before being asked. Names the tool, the context, and exactly what their mental model predicted before they ran anything.

82% of engineers who claim AI proficiency will not demonstrate Tier 3 behaviour under structured evaluation. That is not a criticism, it is a market reality that every IT leader hiring for AI augmented delivery needs to account for.

The 5 Questions That Reveal the Tier

These questions are diagnostic, not adversarial. The goal is not to catch the candidate, it is to observe how they reason about AI outputs under light pressure. A Tier 1 engineer will find them straightforward. A Tier 3 engineer will find them uncomfortable.

1. Tell me about a time an AI tool gave you incorrect output. What happened, and what did you do?

What you are evaluating: Tier 1 answers are specific, they name the tool, the context, the failure mode, and the correction method. Tier 3 answers claim this rarely happens, or describe a vague generic incident with no detail.

1. Walk me through a piece of AI generated code you shipped. What did you change before it went to production?

What you are evaluating: Tier 1 engineers always modified the output, optimised, refactored, validated edge cases. Tier 2 engineers shipped it largely unchanged. They describe what it does, not whether it is correct.

1. What is a task you deliberately do not use AI for, and why?

What you are evaluating: Engineers with genuine AI integration have conscious boundaries. They know where it adds noise. Tier 3 engineers struggle to answer, they have not thought about it.

1. How do you verify that an AI generated architecture recommendation fits your specific context?

What you are evaluating: This separates engineers who reason architecturally from those who cargo cult AI output. Tier 1 answers reference constraints, tradeoffs, and existing system properties. Tier 3 answers are circular.

1. If a junior engineer on your team was over relying on AI tools, what would you do?

What you are evaluating: Leadership dimension. Tier 1 engineers have an opinion grounded in experience. They have seen this pattern before and know its downstream failure modes.

What Our Assessment Data Shows

Across 500+ technical assessments conducted across Southeast Asia from 2019–2025, this is the distribution we observe among engineers claiming AI proficiency

• 65–70% Tier 1 (Output dependent): Accept AI output visually, cannot identify logical errors
• 15–20% Tier 2 (Output aware): Validate against own understanding, catch errors before running
• 15–20% Tier 3 (Output governing): Treat AI as a first draft with known, specific failure modes

For every 10 engineers who claim AI proficiency on their CV, approximately 1–2 can demonstrate Tier 3 behaviour under structured evaluation.

How Talex Applies This in Practice

Every engineer in the Talex pool has been through a structured vetting process that includes AI proficiency evaluation as a dedicated assessment dimension, not a checkbox, not a self reported score.

Our assessors use this framework alongside technical skill evaluation, communication assessment, and project history review. The result: when a client requests an AI augmented engineer, they receive a candidate who has demonstrated Tier 3 behaviour under structured conditions.

Of our current 500+ pool, 90% are Senior or Tech Lead level. All assessed against current AI proficiency standards.

“The evaluation framework Talex uses changed how we think about the hiring conversation entirely. We stopped asking whether engineers use AI and started asking how they govern it.”

Head of Delivery · IT Consulting Firm · Singapore · 18 month engagement

The Practical Summary

Hiring for AI capability in 2026 requires a framework, not a gut feeling. The three tier model gives a structured lens. The five questions give the diagnostic tool.

The data is clear: 82% of engineers who claim AI proficiency will not demonstrate Tier 3 behaviour under structured evaluation. Account for this before your next hire.

If wanting to skip the evaluation overhead entirely, the Talex vetting process has already done this work. Every engineer placed has been assessed against these criteria. First CV in approximately 2 days.

See pre vetted AI augmented engineers
Enter Playground
Browse the Talex talent pool, 500+ engineers, all assessed for AI proficiency.