AM Research is a division of Aspiring Minds. Aspiring Minds aspires to build an assessment-driven job marketplace (a SAT/GRE for jobs) to drive accountability in higher education and meritocracy in labor markets. The products developed based on our research has impacted more than two million lives and the resulting data is a source of continuous new research.


A cocktail of assessment, HR, machine learning, data science, education, social impact with two teaspoons of common sense stirred in it.

The tree of program difficulty

What makes a programming problem hard?

Why are some programming problems solved by more number of students while others are not. The varying numbers we saw got us thinking on how the human brain responds to programming problems. This was also an important question for us to have an answer for when we designed an assessment or wanted guidance on pedagogy. Understanding what makes a programming problem hard would enable us to put questions into a programming assessment of a given difficulty where neither everyone would get a perfect score nor a zero and would also help us in creating equivalent testforms for the test.

We tried taking a jab at it by answering it empirically. We marked 23 programming problems on four different parameters on a 2 or 3 point scale — how hard is the data structure used in the implementation, what data structure is being returned from the target function, how hard it is to conceive the algorithm, the implementation of the algorithm and how hard is it to handle edge cases for the given problem. [See the attached PDF for more details on the metrics and the rubric followed]. There was some nuance involved in choosing these metrics – for instance, the algorithm to a problem could be hard to conceive if, say, it requires thinking through a dynamic programming approach, but its implementation can be fairly easy, involving a couple of loops. On the other hand, the algorithm to sort and then to merge a bunch of arrays can be simple in themselves but implementing such a requirement could be a hassle.

For these problems, we had responses from some 8000 CS undergraduates each. Each problem was delivered to a test-taker in a randomized testform. From this we pulled out how many people were able to write compilable code (this was as low as 3.6% :( to as high as 74% for different problems) and how many got all test cases right. We wanted to see how well we could predict this using our expert-driven difficulty metrics (our difficulties are relative and can change based on sample; for an absolute analysis we could have predicted the IRT parameters of the question — wanna try?)

So, what came out? Yes! we can predict. Here is the base correlations matrix. They are negative because a harder problem has a lower correct rate.

Correlations Data Structure Algorithm Implementation Edge-logic
Percent-pass all test cases -0.25 -0.42 -0.43 -0.05

program-difficulty-treeWe tried a first cut analysis on our data by building a regression tree with some simple cross-validation. We got a really cool, intuitive tree and a prediction accuracy of 0.81! This is our ‘Tree of Program Difficulty’ ;-) . So what do we learn?

The primary metric in predicting whether a good percentage of people are able to solve a problem right is the algorithmic difficulty. Problems for which the algorithm is easy to deduce (<1.5) immediately witness a high pass rate whereas those for which it is hard (>2.5) witness a very poor pass rate. For those that’re moderately hard algorithmically (between 1 and 2.5), the next criterion deciding the pass percentage is the difficulty in implementing the algorithm. If it’s easy to implement (<2), we see a high pass rate being predicted. For those that're moderately hard in implementation and algorithm, the difficulty of the data structures used in the problem then predicts the pass rate. If an advanced data structure is used, the rate falls to less than 6% and is around a moderate 11% otherwise.

So, what nodes do your problems fall on? Does it match our result? Tell us!

Thanks Ramakant for the nifty work with data!

-Shashank and Varun

March 2015

A re-beginning : Welcome to AM Research!

We finally have a place to feature the work which we began five years ago. Great effort, Tarun, to get this up and running.

We thought this was important since education technology and assessments are going through a revolution. We wish to add our two teaspoons of wisdom (did I actually say that!) to the ongoing battle against the conventional non-scalable and unscientific ways of training, assessing and skill matching. We look forward to making this as a means to collaborate with academics, the industry and anyone who feels positively about education technology.

Sector/Roles Employability(%)
BUSINESS FUNCTIONS
Sales and Business Development 15.88
Operations/Customer Service 14.23
Clerical/Secretarial Roles 35.95
ANALYTICS AND COMMUNICATION
Analyst 3.03
Corporate Communication/Content Development 2.20
IT AND ITeS INDUSTRY
IT Services 12.97
ITes and BPO 21.37
IT Operations 15.66
ACCOUNTING ROLES
Accounting 2.55
TEACHING
Teaching 15.23

Table 1: By using standardized assessments of job suitability, in a study of 60,000 Indian undergraduates, we find that a strikingly low proportion of them have skills required for the industry. All these students got detailed feedback from us to improve. The table shows the percentage of students that have the required skills for different jobs. (Refer: National Employability Report for Graduates, under Reports in Publications)

We think assessments will be the key to democratize learning and employment opportunity: it provides a benchmark for measuring success of training interventions, provides feedback to learners creating a ‘dialogue’ in the learning process and most importantly, helps link learning to tangible outcomes in terms of jobs and otherwise.

Let me state it simply: To scale learning and make employment markets meritocratic, we need to scale automated assessments. This is the space we dabble in!

If you are thirsty for data, refer to the table and figure in this post. It tells the story of the problem we are up against and trying to solve.

Figure 1: 2500 undergraduates were surveyed to find their employment outcomes one year after they got their undergraduate education. We categorized their colleges in three categories (tier 1-3) based on their overall performance in AMCAT, our employability test. We find that a candidate in a tier 3 college has 24% lower odds of getting a job and 26% lower salary when he/she has the same merit (AMCAT scores) as a tier 1 students. Similarly, a 1 point drop in college GPA (on a 10 pt scale) decreases job odds by 16% and salary by 9%. Neither of these two parameters are useful predictors of job success beyond AMCAT scores. This shows a clear bias in the employment ecosystem. (Refer ‘Who gets a job’ under Reports in Publications)

How do we solve it? Stay tuned to our subsequent job posts…

Varun

Tweets

I just uploaded “Optimize hiring for Call Center / BPOs” to #Vimeo: vimeo.com/338250881

I just uploaded “Aspiring Minds Overview - Optimizing hiring” to #Vimeo: vimeo.com/338249939

Thank you for connecting with #AspiringMinds at #SHRMTech19. It was great to share how we are helping leading compa… twitter.com/i/web/status/1…

I just uploaded “Motor skills video” to #Vimeo: vimeo.com/337974587

I just uploaded “Aspiring Minds' Automated English language test” to #Vimeo: vimeo.com/337949644

I just uploaded “Copy of Autoview - AI powered video interview” to #Vimeo: vimeo.com/337947238

I just uploaded “AUTOMATA_ Revolutionizing Computer Programming assessment using Machine Learning(1)” to #Vimeo: vimeo.com/337751605

Meet our experts at the #SHRM Tech APAC Conference 2019, Booth #85 to get more insights on how you can completely r… twitter.com/i/web/status/1…

Excited to be a part of the #SHRM Tech APAC Conference 2019, Come join us at Booth #85 and understand how you can a… twitter.com/i/web/status/1…

Our workforce is stuck in the complex web of being unemployed, underemployed and unemployable! @varaggarwal talks… twitter.com/i/web/status/1…

Since its 1st edition in 2010, National Employability Report for Engineers by @AspiringMinds has sparked national d… twitter.com/i/web/status/1…

Anirvan Mukherjee, director Human Resources at CGI India talks about the merits of Aspiring Minds' products and why… twitter.com/i/web/status/1…

Aspiring Minds' Advisor, Tarun Khanna talks to @TimesofIndia scientific illiteracy in India and how Aspiring Minds'… twitter.com/i/web/status/1…

Bengaluru chapter of the Digital HR Summit witnessed overwhelming response from the HR fraternity. Lots of insightf… twitter.com/i/web/status/1…

Siddharth Nayak talks about Aspiring Minds' #AI powered Codemeet interface that enables anytime, anywhere, seamless… twitter.com/i/web/status/1…

And it's a full house! @sdwivedy addresses our #HR fraternity as we Kickstart our Digital HR summit #Bangalore.… twitter.com/i/web/status/1…

We are about to start. Meanwhile the Smile App #ChallengeAccepted by the #HR fraternity gracing our event. The #ai-… twitter.com/i/web/status/1…

We are excited to meet the brightest minds of #HR at @AspiringmindsAM Digital HR Summit in Bangalore. Insightful co… twitter.com/i/web/status/1…

8 Days to go! #JoinUs on 24th April for the #AspiringMinds #DigitalHRSummit. Here's our impressive line-up of speak… twitter.com/i/web/status/1…

US institutes are leaders in research. Chinese colleges have made exponential progress while India slugs behind. Ho… twitter.com/i/web/status/1…

Over 2 hours of stimulating discussions attended by hundreds of HR professionals from@Genpact , @bt_uk ,… twitter.com/i/web/status/1…

Unembloyability numbers remain "stubborn". Huge gap in new-age skills reported for Indian Engineers. With such a sc… twitter.com/i/web/status/1…

Mr. Manuj Sethi talks about how knowing what not to do in a situation is a key skill.#AI powered Situational Judgem… twitter.com/i/web/status/1…

A man-machine interaction enable #HR community take better decisions. Watch what @varaggarwal has to say about… twitter.com/i/web/status/1…

‘Artificial Intelligence (AI) ’, ‘Immersive Experiences’ and ‘Digital Platforms’ are going to be the technology MEG… twitter.com/i/web/status/1…

The stage is set !! We are excited to meet the brightest minds of #HR at @AspiringmindsAM Digital HR Summit. Awaiti… twitter.com/i/web/status/1…

#AI #ML are rapidly revolutionizing the #HR landscape, from simulation-based assessments to automated video intervi… twitter.com/i/web/status/1…

#JoinUs tomorrow for the #AspiringMinds #DigitalHRSummit. Our impressive line-up of speakers, Free Registration for… twitter.com/i/web/status/1…

Businesses are hunting talent with AI, data engineering, mobile and cloud skills. Sadly 97% Indian engineers are un… twitter.com/i/web/status/1…

In IT sector, close to 19% engineers in the US write correct codes while the number drops to single digits (5%) for… twitter.com/i/web/status/1…

9 years and running, yet employability numbers remain at a stagnant low, as per Aspiring Minds National Employabili… twitter.com/i/web/status/1…

Announcing the #AspiringMinds #DigitalHRSummit, Bengaluru 2019.Join us for some interesting dialogues on how… twitter.com/i/web/status/1…

Use of tech and AI is the key to simplifying sub-tasks of recruitment to save his/her valuable time. Watch what… twitter.com/i/web/status/1…

Aspiring Minds released 5th edition of National employability report - engineers today. The report presents thought… twitter.com/i/web/status/1…

Aspiring Mind's #AI-powered #Video-interviewing platform, Autoview reduces hiring time by 80% with real time scorin… twitter.com/i/web/status/1…

It's a wrap for of an insightful evening of discussions with our fellow colleagues of the #HR fraternity.… twitter.com/i/web/status/1…

The best minds of #HR Industry discussed issues pertaining to #DigitalTransformation in recruitment and how we can… twitter.com/i/web/status/1…

It's a full house with an awesome audience. Some very stimulating discussions shaping up on the power of #AI and it… twitter.com/i/web/status/1…

Know more about how #AI is enabling #DigitalTransformation in recruitment. @varaggarwal explains the ways machine a… twitter.com/i/web/status/1…

Mr. Manuj Sethi explains how we can do lateral hiring for technical job profiles across locations. #AIinrecruitmenttwitter.com/i/web/status/1…

Recruitment can be automated across sectors to hire the best talent. @sdwivedy explains how we can make the best us… twitter.com/i/web/status/1…

Know more about the role of AI in learning and development. @varaggarwal @sdwivedy #AIinrecruitmenttwitter.com/i/web/status/1…

HR leaders of the country discuss #AI in recruitment and how we can leverage machines for streamlining our processe… twitter.com/i/web/status/1…

How is HR gearing up for AI led transformations in recruitment in 2019? We are geared up for ideating & networking… twitter.com/i/web/status/1…

‘Artificial Intelligence (AI) ’, ‘Immersive Experiences’ and ‘Digital Platforms’ are set to be the technology MEGA-… twitter.com/i/web/status/1…

The stage is set. We are excited to meet you all at @AspiringmindsAM Digital HR Summit. Awaiting some enriching con… twitter.com/i/web/status/1…