Designing first-round voice screens candidates can trust

What is a first-round voice screen in Emoti?

It is a short structured interview run by an AI interviewer: a fixed list of questions, each with a time guide and a rubric, which a candidate takes by voice through a link at a time that suits them.

A recruiter builds the screen from these parts:

A new screen starts with two general questions to edit: one about a piece of work the candidate is proudest of, and one about what they want next and when they could start. The builder shows the total length and comments on it: under 10 minutes is short, up to 25 minutes is comfortable to finish in one sitting, and anything longer is "a lot to ask before a first conversation". A screen can't be published until every question has wording. Candidates are invited by a shared link or by email, with one reminder.

The format borrows from research on structured interviews, which finds that asking everyone the same job-related questions and rating answers against set standards predicts job performance better than unstructured conversation. A 2022 re-analysis ranked structured interviews first among common selection methods, with a mean validity of .42, and also found wide variation between studies (Sackett et al., 2022). That research is the reason for the format. It is not evidence that any particular AI screen works, and we have not validated ours.

How are candidates told what is happening?

Before the first question, the welcome page tells the candidate how long the screen takes, what the camera rule is, which language and voice they will hear, and who listens: an AI interviewer asks the questions, and people at the company review every answer before any decision.

The page also says the candidate can pause between questions, and its footer says the interview is run by Emoti. The default welcome message, which teams can rewrite, says the same thing in a warmer voice:

Thanks for applying. An AI interviewer will ask a few questions about your work. Answer as you would with a person, and take your time. Someone on our team reviews every answer before any decision is made.

The invitation email says an AI interviewer asks the questions, that a person reviews every answer, and that candidates can ask for an adjustment before they start. On the welcome page, "Start the screen" stays unavailable until the candidate ticks a box confirming that an AI interviewer asks the questions, that their answers are recorded and transcribed, and that people review them before any decision.

Some laws ask for more. Illinois's Artificial Intelligence Video Interview Act requires employers that use AI to analyze recorded video interviews for Illinois-based positions to tell applicants beforehand, explain how the AI works and what general types of characteristics it uses, and obtain consent before the interview. New York City's Local Law 144 requires employers using an automated employment decision tool to assess candidates who live in the city to give notice 10 business days before use, including instructions for requesting a reasonable accommodation. Those duties sit with the employer, and this paper is not legal advice.

Does everyone get the same questions?

Within one version of a screen, yes: the same questions, in the same order, with the same time guides. The differences are the language a candidate chooses and, where they are turned on, follow-up questions.

Follow-ups are a real tension. They can help a nervous or brief candidate give a fuller answer, and they also make interviews less alike. A major review of the research recommends using only planned, neutral probes that don't cue the desired answer, probing equally across applicants, and probing only when an answer is clearly deficient or off track, or the applicant is shy (Levashina et al., 2014). Emoti's follow-ups are switched on per question and used "when an answer is thin". In version 1.0 they are not written in advance, which falls short of that recommendation.

Screens can also change while they are live. Edits apply to anyone who starts the screen afterwards, so candidates who started earlier answered the earlier version. When a question is removed, the scorecards of people who answered it say "This question has since been removed from the screen." Teams comparing a group of candidates should avoid editing a screen partway through the group.

How are answers scored, and who decides?

Each answer is rated on a four-point scale (weak, mixed, good, strong) against that question's rubric, with a one-line summary, and the screen's overall score is the average rating rescaled from 0 to 100. A recruiter reads the scorecard and makes every decision.

The rescaling is simple: all strong answers give 100, all weak answers give 0, and five answers rated strong, strong, good, strong and good give 87. The overall score uses the same bands and words as fit scores elsewhere in Emoti: 80 and above strong, 60 to 79 good, 40 to 59 partial, below 40 weak. The scorecard labels the ratings "Rated by Emoti against your rubric" and marks the summary as written by Emoti.

What is rated is the content of the answer, against the rubric. The camera never affects a score, and nothing in the design infers emotion or personality from a face or a voice. That matters beyond good practice: the EU AI Act prohibits AI systems that infer emotions in the workplace (Article 5(1)(f)), and the European Commission's guidelines on prohibited practices treat hiring as part of the workplace for that rule.

Two weaknesses in the rubric design

• Anchors only at the ends. The builder asks what a strong answer and a weak answer look like, but not the two points between them. One study summarized by Levashina et al. (2014) found that ratings on a five-point scale with an anchor at every point were more resistant to disability bias than ratings on a scale anchored only at the highest and lowest points.
• The summary sits where the answer should. In version 1.0 the scorecard shows Emoti's rating and a one-line summary of each answer, but not the candidate's own words or recording beside it. A reviewer who wants to disagree with a rating needs the answer itself. Until it is shown, "people review every answer" depends on reviewers going beyond the summary.

The recruiter's decision

A completed screen waits as "To review" until a person chooses Advance or Reject, and either choice can be moved back to review. There is no pass mark and no automatic rejection. The candidate list shows people waiting for review first, highest score first, then people who have started or been invited, then decisions already made. Sorting by score helps a recruiter start somewhere, and it also invites anchoring on the number, which is one more reason the answer belongs on the scorecard.

Keeping a person on the decision also reflects the law in some places. In the EU, the GDPR gives people the right not to be subject to a decision based solely on automated processing that significantly affects them, with limited exceptions (Article 22).

How does the design handle accents, speech differences and nerves?

Partly. Candidates can choose a language, hear a question again, pause between questions and take the screen when they choose, and the rubric rates what was said rather than how it sounded. The design does not yet correct for speech recognition errors, offer a written or human alternative inside the flow, or let a candidate ask for more time.

The risks are real:

• Accents and dialects. Rating spoken answers depends on a transcript, and speech recognition makes more errors for some speakers than others. In a study of five commercial systems, the average word error rate was 0.35 for Black speakers and 0.19 for white speakers (Koenecke et al., 2020).
• Speech differences. People who stammer, have a voice disorder or speak through assistive technology may be misheard more often or need more time. We have not measured how Emoti's transcripts or ratings behave for them.
• Nerves. Across studies, candidates who report more interview anxiety tend to receive somewhat lower interview ratings, with an overall correlation of −.19 in one meta-analysis (Powell et al., 2018). Speaking to a machine may calm some candidates and unsettle others.

Employers in many places must make adjustments. In the US, the Americans with Disabilities Act counts not making reasonable accommodations to the known limitations of a qualified applicant with a disability as discrimination, unless the accommodation would impose an undue hardship. In Great Britain, the Equality Act 2010 places a duty on employers to make reasonable adjustments. The welcome page carries a link for this: a candidate can ask for more time, written answers instead of speaking, a conversation with a person, or something else, with a note for the team. We still recommend that every team says in its own welcome message how it handles those requests.

Limitations

• We have not validated Emoti's ratings against trained human raters or against later job performance.
• Ratings depend on transcripts, and transcripts are less accurate for some speakers.
• Rubrics anchor only the strong and weak ends of the scale.
• The scorecard shows summaries, not the answers themselves.
• Follow-ups are not scripted in advance, so they can differ between candidates.
• A live screen can be edited partway through a group of candidates.
• Adjustment requests are not yet collected into their own queue for recruiters.
• There is no setting for how long screen recordings are kept.

What we have not measured yet

We have not yet measured:

• Agreement between Emoti's ratings and trained raters using the same rubric.
• Whether ratings differ by accent, dialect, language or speech difference when the content of an answer is the same.
• Whether advance rates differ by group, using a measure such as the four-fifths guideline. That needs outcome data from employers.
• How candidates experience the screen, and whether completion rates differ by group.
• Whether follow-up questions make ratings more or less consistent.
• Whether screen scores predict later interview results or job performance.