Providing semantic feedback on uncompilable codes, for the first time – Our IAAI paper

Hurray! Our paper “Grading uncompilable programs” has been accepted at IAAI 2019! This is the first attempt to provide semantic feedback on uncompilable codes. This is our fourth paper on the topic as we have improved technology to grade and provide feedback on programming skills.

Automata, the world’s only AI-based evaluator of programming skills, is being used across the world to hire software engineers. In our earlier post, we discussed how a US-based company used the platform to improve hiring efficiency.

In our IAAI paper, we mention a study with a Chinese company. We find that the system helped in selecting around 26% more candidates for the interviews. These candidates had written a logically meaningful code, but the code did not compile. Out of these candidates, around 19% of candidates were hired. This is a big win as many worthy candidates who would have got missed out by traditional program grading systems were hired. (See details below)

Table 1: Distribution of scores for candidates with compilable and uncompilable programs. The table also includes the number of candidates selected for interview and the number of candidates who were hired.
Rubric Definition Compilable Uncompilable
1 Code unrelated to given task 3361 1979
2 Appropriate keywords and tokens are present 3264 2125
3 Right control structure exists with missing data dependency 2547 1440
4 Correct with inadvertent errors 2955 1017
5 Completely correct 3828 -
≥ 3 Selected for interview 9330 2457
- Hired 2986 565

Recently, the system has been used by a large IT company in India for large scale hiring of entry-level software engineers. The company hires 1000’s of software engineers every year and struggles with filling all open positions. By way of grading non-compilable codes, they have been able to improve hiring throughput by 30-40%.

Stay tuned, more to come… Programs + AI opens up a million opportunities!!!

- Rohit and Varun