Hiring Engineers for the Age of Agents

AI tools have and continue to fundamentally transform most roles, especially software engineering. Gone are the days of remembering whether Python’s zip returns a list or a generator, or putting together a team of engineers to build a new full-stack dashboard app for the finance team.

Internally at Applied Intuition, we aggressively retooled our entire organization to use AI coding agents. Everyone here (engineer or not) is using AI regularly, and dedicates weekly time specifically to optimizing our agentic workflows.

Amid all these internal changes, however, we weren’t conveying the same message to our candidates. Until recently, our engineering interview process was entirely no-AI coding and whiteboarding. This process is tried and tested in the industry, but it was clearly built for a different era. As a company working at the forefront of physical AI, we needed to evaluate engineers accordingly.

The Approach

Rethinking how we hire in a company of over 1,000 engineers (and growing!) is not something to be taken lightly. While there’s tons of literature on Leetcode-style interviewing, redesigning engineering interviewing in the wake of wide AI adoption is the wild west. A few things were top of mind as we pursued this revamp:

• We must continue to hire great engineers, and reject candidates who would not be happy working the way we do.
• AI fluency is important, but only one aspect of the job. As the day-to-day craft of coding has become significantly easier, decision-making, technical excellence, and cultural factors matter more than ever. Any approach needs to test these core skills - using AI is just the medium to enable it.
• The format change should allow us to capture more data and make better informed decisions in the final stages of hiring.
• Candidates should take away from the interview an idea of what it would be like to work at Applied Intuition.

Other companies have come up with a variety of ways to introduce AI to the interview process, but with the above guiding principles this is where we landed:

• Keep our technical phone screen as is. A 45-minute coding task with no AI. Ultimately engineers are still responsible for every line that goes into our codebase, so we need to know that they understand code independent of AI.
• Onsite gets the revamp. Once we’ve gotten a technical baseline for a candidate, we want to see how they actually work.

Our onsite now involves a 45-minute system design question where candidates talk through an open-ended problem – usually derived from something we’ve actually had to solve at Applied – and tell us how they’d engineer a solution. After that, the candidate gets two hours to build a working prototype of the system they just designed using any AI model they want, and any frameworks. Then they demo it in front of a panel of engineers, and walk through what they built, the decisions they made, and what they'd do differently.

The Data

To try out this new format, we ran over 30 real onsite interviews with candidates. We brought in the most vocally skeptical senior engineers from across the company to trial the new format, including the folks who wrote the last generation of interview questions. There was some healthy criticism at first, as with any big change, and with it some great feedback on how to make the new format more effective. But we’re also finding that the more of these interviews people do, the more they see the value.

We wanted to keep the candidate experience top of mind as this structure is new for both parties. So we collected interviewer feedback on the format alongside candidate scorecards and tracked pipeline data.

The initial data showed that this was a better interview format. Across dozens of responses from 15 unique interviewers who were involved in the initial 30-candidate trial, 86% thought it gave them a better read on candidates’ ability to work here, 85% said it was better than our old format, and 91% reported they were more confident in their candidate scorecard decision. And 99% recommend we continue with this format.

“This format was so refreshing and much better than recycled leetcode questions. It signalled to me that Applied is cutting edge. I also got a sense of the type of work we do here. This format played a major factor in why I joined.”

— Recent hire from the new format

We also asked interviewers to rate their confidence in each of the individual hiring signals we look for, on a scale of 1 to 5. Six of the seven signals scored between 4.0 and 4.7. For practical skills, tackling unfamiliar challenges, technical communication, logical thinking, and technical mastery, interviewers felt the format surfaced all of it clearly. Unsurprisingly, the one signal that scored significantly lower was code elegance and design, which is what we intend to continue evaluating with the phone screen.

What’s next

Now that we have signal from candidates and interviewers that this format works, we’re working on scaling it to the whole company.

It took years to build up the question library for our previous interview format, and now we needed to start over for AI interviews. We knew if we waited for engineers to come to us with a library of new AI-assisted questions, it’d take a backseat over all the day-to-day work we still have. So we decided to use the momentum we'd generated from the trial and go live.

Luckily, our engineers were more than up to the task. There’s no shortage of complex and difficult problems to solve in physical AI, and after we solve them, they make for really representative interview questions. In just a month since moving to this format, we have many new questions either approved or in development, and more being written every week. As the library grows and more interviewers learn how to run it, we’re learning a residual benefit of this process: it’s intuitive! Engineers are quick to train because the interview process mirrors a day at their job, and can be quickly assessed.

Through the interview trial, we realized rolling this out to hundreds of engineering interviewers would take a lot of convincing and standardization. So we compiled all the conversations, learnings, data, and common challenges into an internal application to assist with the rollout. With the help of Apps Platform, an internal tool we built that lets anyone at Applied spin up a fully working vibecoded app in under a minute, we built a portal to put the trial data right next to the training material:

And as we write more questions and train more interviewers, we’re always collecting feedback and adjusting the format. Since this is so new and every model release from the AI labs meaningfully changes what candidates are capable of in this format, we need to constantly re-evaluate how well the interview follows our guiding principles and course correct accordingly.

So, if you ever wanted to know what working at Applied Intuition might be like, check out our careers page!