Aspiring Minds’ AI work appears in Communications of ACM

Skill Evaluation, By Shashank Srikant, Rohit Takhar, Vishal Venugopal, Varun Aggarwal Communications of the ACM, November 2019, Vol. 62 No. 11, Pages 60-61

Upward of four million graduates enter the labor market every year in India alone. India boasts of a large services economy, wherein a single company hires thousands of new employees every year. Meanwhile, product companies and small and medium enterprises (SMEs) look for a few skilled people each. This requires cost-effective and scalable methods of hiring. Interviewing every applicant is not a feasible solution.

On the other hand, graduates from 30,000+ institutes of higher education spread across 20+ Indian states face a constant challenge in signaling their competence to potential employers. Companies, most of which are located in the top 20 biggest cities in the country, bias their search by relying on proxies like university name and the city a college is located in. Applying such crude filters results in meritorious students from various demographics being ignored. Further, these students have no mechanism to get feedback on how their skills compare to those required by the industry.

Having systems that can intelligently and scalably assess a wide variety of skills is essential to addressing this broader problem affecting every modern-day labor market. Aspiring Minds was formed 10 years ago to address this challenging problem. We have developed a scalable platform to deliver standardized assessments to test job skills. The platform tests more than two million students every year and is used by 5,000+ companies including 100+ Fortune 500 companies. Read more

Providing semantic feedback on uncompilable codes, for the first time – Our IAAI paper

Hurray! Our paper “Grading uncompilable programs” has been accepted at IAAI 2019! This is the first attempt to provide semantic feedback on uncompilable codes. This is our fourth paper on the topic as we have improved technology to grade and provide feedback on programming skills.

Automata, the world’s only AI-based evaluator of programming skills, is being used across the world to hire software engineers. In our earlier post, we discussed how a US-based company used the platform to improve hiring efficiency.

In our IAAI paper, we mention a study with a Chinese company. We find that the system helped in selecting around 26% more candidates for the interviews. These candidates had written a logically meaningful code, but the code did not compile. Out of these candidates, around 19% of candidates were hired. This is a big win as many worthy candidates who would have got missed out by traditional program grading systems were hired. (See details below)

Table 1: Distribution of scores for candidates with compilable and uncompilable programs. The table also includes the number of candidates selected for interview and the number of candidates who were hired.
Rubric Definition Compilable Uncompilable
1 Code unrelated to given task 3361 1979
2 Appropriate keywords and tokens are present 3264 2125
3 Right control structure exists with missing data dependency 2547 1440
4 Correct with inadvertent errors 2955 1017
5 Completely correct 3828 -
≥ 3 Selected for interview 9330 2457
- Hired 2986 565

Recently, the system has been used by a large IT company in India for large scale hiring of entry-level software engineers. The company hires 1000’s of software engineers every year and struggles with filling all open positions. By way of grading non-compilable codes, they have been able to improve hiring throughput by 30-40%.

Stay tuned, more to come… Programs + AI opens up a million opportunities!!!

- Rohit and Varun

Aspiring Minds and AI

Aspiring Minds has been doing machine learning, aka artificial intelligence, for 8 years now, much before it became a vogue. We solved original questions using AI, not copycatting the West – doing a lot of firsts in the world. Here is a quick recap of Aspiring Minds’ tryst with AI, together with how AI evolved in India.

Phase I- “ML in a niche”- We had hired two engineers to work on Machine learning projects in 2010. After one year, they came to my room and inquired about their future, since all their friends were doing software development. Hardly anyone knew about ML.

- 2012: Launched SVAR: An AI based spoken English evaluation product

Today, SVAR is used across the world including India, Philippines, China and Latin America. It automatically generates scores on pronunciation and fluency based on speech samples of a person.

Was this the first AI-based product from India that reached scale?

- 2012: Made one of our data set public and organized a Machine Learning Competition

The competition had entries from India, Brazil, Belgium and Pakistan. See the leader-board and winners hereThis was probably the first by an Indian company and among the first few in the world.

- 2013: Launched AUTOMATA: World’s first machine learning based programming assessment

Automata is used by companies across the world – some examples include Wipro, Cognizant, Baidu, ZTE and one of the largest ecommerce giants in the USA. It is backed by three publications and several patents.

- 2014: Published our first ML paper on grading programming skills automatically at KDD

The paper has quickly garnered 28 citations. This was followed by several other papers on automatic grading of spoken English, motor skills, soft-skills, published at KDD, ACL, Ubicomp, IJSE and others. We also did a first workshop on AI+assessments at KDD with international collaborators.

Aspiring Minds remains one of the very few Indian companies that publish in ML conferences

Aspiring Minds and AI-03

Phase II- Big Data Science Fascination- Everyone by now had started talking about Big Data and Data Science – a new name for machine learning! Most work in India was around data engineering and not deriving intelligence from data. MOOCs on AI exploded –everyone who took the course didn’t really learn.

- 2015: Organized the world’s first Data Science Camp for Kids

We organized a very successful hands-on data science camp for kids of standard 5th-8th. Kids performed the full flow of supervised learning. Since then this open source project has been replicated at Illinois, Seattle, Pune and Bangalore. It also led to a paper on the pedagogy of teaching machine learning to kids.

- 2015: Launched, the first effort ever to audit India’s ML activity and a resource repository for all MLers

ML India brings all ML efforts in India under a single roof. Read more about how does India fare in ML – the main motivation behind setting up this forum. The group has 1800+ members, hosted 27 machine learning meetups, lists 146 ML professionals, 55 companies, 28 data sets and 11 groups.

We also launched a new data set AMEO, at iKDD. Attracted users from Harvard Kennedy School, Dublin Institute of Technology, New York University, TCS, Sapient and Flytxt.

- 2016: Launched the World’s first automated motor skills test

Aptitude tests have been automated for ages. But motor skills test, a way to measure skills of blue collar workers have not. We used the power of tablets and machine learning to do it and show that it is predictive of the job performance of blue collar workers. Read here.

- 2016: US Skill Map and India Skill Map- Big Data Analysis

Automatically crawled the web to aggregate jobs of USA and India, to create the world’s first interactive Skill Demand Map. Check it out here.

Phase III- National Interest in AI, but with nascent understanding –Data science had by now died a silent death, only to be replaced by Artificial Intelligence. From the PMO, Finance Minister to Niti Aayog, today everyone is interested in AI. Yet, we have little novel methods or application of AI from India. We have little local expertise in AI – our research contribution is 1/15th of US and 1/8th of China.

- 2017: Machines started understanding codes that do not compile!

Automata, our program skill grading platform started scoring uncompilable codes, a first in the world! Our algorithm could read meaning of programs, which a compiler couldn’t and generate feedback for so many more students.

And the journey continues!!!

This has been possible by efforts of many in Aspiring Minds’ research team, most notably, Shashank Srikant, Rohit Takhar, Vishal Venugopal, Gursimran Singh, Bhanu Pratap Singh, Vinay Shashidhar, and Milan Sachdeva.

Phase IV: How can India lead in Artificial Intelligence? From doing research, we started thinking about research policy. My recent book, ‘Leading Science and Technology: India Next’ focuses primarily on the research ecosystem in India and highlights several areas where we should improve. It is supported by a white paper on how India should invigorate its Artificial Intelligence ecosystem. This is where we need to go next…

- Varun Aggarwal

AI powered coding platform ‘understands’ programs that do not compile!

If you ever took a coding job test on a machine, you will probably frown if you couldn’t make your code to compile. Your program might be almost right, but due to some silly bug, unidentified in a small time frame, you will get a ZERO.




Not any more! Aspiring Minds’ research team has created a technology which can detect how good the program’s algorithm is, even if it doesn’t compile.

How do we do it? First, we can fix some of the codes using artificial intelligence. By looking at patterns in good compilable codes, our algorithms minimally modify existing programs to make them compilable. By using this approach, we can compile 40% of uncompilable codes. Once compilable, our patented machine learning based algorithm can generate a grade which mimics human raters.

Fancy as it may seem, we had a harder problem to solve. What about the codes that do not compile? Using smart static analysis of codes, we are able to derive features, signatures of the logic of the program, from these codes automatically. With these features and a customized form of our machine learning algorithm, we can provide grades as accurately as you could think!

On a set of programs attempted for a job in a large e-commerce player in USA, we find that 46% codes were not compiling, but weren’t blank.

Our AI based algorithm found that 6% of these codes, for 596 students, had nearly correct logic. Another 29% candidates, with a little bit of guidance, would have reached the right logic. All these candidates deserved a shot with the company!

In another data set of a technology giant in China, we find that 27% candidates whose codes do not compile, have sound programming logic.

What is more? Our AI algorithm can provide feedback to all candidates whose code do not compile. To some, we can tell how to fix their programs and make them to compile. To all, we can give them feedback on their algorithmic approach, tips to reach the correct logic and provide feedback on the stylistic and maintainability issues in their code.

Disappointed with coding platforms which gives everyone a poor score and no feedback… We have corrected this for all times to come!

- Varun Aggarwal with Rohit Takhar
Learn more about Automata – our coding platform.

Scaling up machine learning to grade computer programs for 1000s of questions in multiple languages

Machine learning has helped solved many grading challenges – spoken english, essay grading, program grading and math problem grading to cite a few examples. However, there is a big impedance in using these methods in real world settings. This is because one needs to build an ML model for every question/prompt – for instance, in essay grading, a different model designed to grade an essay on ‘Socialism’ will be very different from one which can grade essays on ‘Theatre’. These models require a large number of expert rated samples and a fresh model building exercise each time. A real-world practical assessment works on 100s of questions which then translates to requiring 100s of graders and 100s of models. The approach doesn’t yield to be scalable, takes too much time and most of the times, is impractical.

In our KDD paper accepted today, we solve this challenge quite a bit for grading computer programs. In KDD 2014, we had presented the first machine learning approach to grade computer programs, but we had to build a model per problem. We have now invented a technique where we need no expert graded samples for a new problem and we don’t need to build any new models! As soon as we have around a few tens of ‘good’ codes for a problem (automatically identified using test case coverage and static analysis), our newly invented question-agnostic models automatically take charge. How will this help us? With this technology, our machine learning based models can scale, in an automated way, to grade 1000s of questions in multiple languages in a really short span of time. Within a couple of weeks of a new question being introduced into our question pool, the machine learning evaluation kicks in.

There were couple of innovations which led to this work, a semi-supervised approach to model building:

  • We can identify a subset of the ‘good’ set automatically. In the case of programs, the ‘good set’, codes which get a high grade, can be identified automatically using test cases. We exploit this to find other programs similar to these in a feature space that we define. To get a sense of this, think of a distance measure from programs identified as part of the ‘good set’. Such a ‘nearness’ feature would then correlate with grades across questions irrespective of whether it is a binary search problem or a tree traversal problem. Such features help us build generic models across questions.

  • We design a number of such features which are invariant to the question and correlate to the expert grade. These features are inspired by the grammar we proposed in our earlier work. For instance, one feature is how different is an unseen program from the set of keywords present in the ‘good set’; while another is the difference in the programs in the kind of computations they are doing. Using such features, we learn generic models for a set of problems using supervised learning. These generic models work super well for any new problem as soon as we get our set of good codes!

Check out this illustrative and easy-to-grasp video which demonstrates our latest innovation.


The table presents a snapshot of the results presented in the paper. As shown in the last two columns, the ‘question-independent’ machine learning model (ML Model) constantly outperforms the test suite based baseline (Baseline). The claim of ‘question-independence’ is corroborated by similar and encouraging results (depicted in last three rows) obtained on totally unseen questions, which were not used to train the model.

Question Set
ML Model
All questions
All questions
All questions
Unseen questions only
Unseen questions only
Unseen questions only

What does this all mean?

  • We can really scale ML based grading of computer programs. We can continue to add new problems and the models will automatically start working within a couple of weeks.
  • These set of innovations apply to a number of other problems where we can automatically identify a good set. For instance, in circuit solving problems, the ones with the correct final answer could be considered a good set; this can similarly be applied to mathematics problems or an automata design problem; problems where computer science techniques are mature to verify functional correctness of a solution. Machine learning can automatically then help grade other unseen responses using this information.

Hoping to see more and more ML applied to grading!


Work done with Gursimran Singh and Shashank Srikant