AM Research is a division of Aspiring Minds. Aspiring Minds aspires to build an assessment-driven job marketplace (a SAT/GRE for jobs) to drive accountability in higher education and meritocracy in labor markets. The products developed based on our research has impacted more than two million lives and the resulting data is a source of continuous new research.


A cocktail of assessment, HR, machine learning, data science, education, social impact with two teaspoons of common sense stirred in it.

The tree of program difficulty

What makes a programming problem hard?

Why are some programming problems solved by more number of students while others are not. The varying numbers we saw got us thinking on how the human brain responds to programming problems. This was also an important question for us to have an answer for when we designed an assessment or wanted guidance on pedagogy. Understanding what makes a programming problem hard would enable us to put questions into a programming assessment of a given difficulty where neither everyone would get a perfect score nor a zero and would also help us in creating equivalent testforms for the test.

We tried taking a jab at it by answering it empirically. We marked 23 programming problems on four different parameters on a 2 or 3 point scale — how hard is the data structure used in the implementation, what data structure is being returned from the target function, how hard it is to conceive the algorithm, the implementation of the algorithm and how hard is it to handle edge cases for the given problem. [See the attached PDF for more details on the metrics and the rubric followed]. There was some nuance involved in choosing these metrics – for instance, the algorithm to a problem could be hard to conceive if, say, it requires thinking through a dynamic programming approach, but its implementation can be fairly easy, involving a couple of loops. On the other hand, the algorithm to sort and then to merge a bunch of arrays can be simple in themselves but implementing such a requirement could be a hassle.

For these problems, we had responses from some 8000 CS undergraduates each. Each problem was delivered to a test-taker in a randomized testform. From this we pulled out how many people were able to write compilable code (this was as low as 3.6% :( to as high as 74% for different problems) and how many got all test cases right. We wanted to see how well we could predict this using our expert-driven difficulty metrics (our difficulties are relative and can change based on sample; for an absolute analysis we could have predicted the IRT parameters of the question — wanna try?)

So, what came out? Yes! we can predict. Here is the base correlations matrix. They are negative because a harder problem has a lower correct rate.

Correlations Data Structure Algorithm Implementation Edge-logic
Percent-pass all test cases -0.25 -0.42 -0.43 -0.05

program-difficulty-treeWe tried a first cut analysis on our data by building a regression tree with some simple cross-validation. We got a really cool, intuitive tree and a prediction accuracy of 0.81! This is our ‘Tree of Program Difficulty’ ;-) . So what do we learn?

The primary metric in predicting whether a good percentage of people are able to solve a problem right is the algorithmic difficulty. Problems for which the algorithm is easy to deduce (<1.5) immediately witness a high pass rate whereas those for which it is hard (>2.5) witness a very poor pass rate. For those that’re moderately hard algorithmically (between 1 and 2.5), the next criterion deciding the pass percentage is the difficulty in implementing the algorithm. If it’s easy to implement (<2), we see a high pass rate being predicted. For those that're moderately hard in implementation and algorithm, the difficulty of the data structures used in the problem then predicts the pass rate. If an advanced data structure is used, the rate falls to less than 6% and is around a moderate 11% otherwise.

So, what nodes do your problems fall on? Does it match our result? Tell us!

Thanks Ramakant for the nifty work with data!

-Shashank and Varun

March 2015

A re-beginning : Welcome to AM Research!

We finally have a place to feature the work which we began five years ago. Great effort, Tarun, to get this up and running.

We thought this was important since education technology and assessments are going through a revolution. We wish to add our two teaspoons of wisdom (did I actually say that!) to the ongoing battle against the conventional non-scalable and unscientific ways of training, assessing and skill matching. We look forward to making this as a means to collaborate with academics, the industry and anyone who feels positively about education technology.

Sector/Roles Employability(%)
BUSINESS FUNCTIONS
Sales and Business Development 15.88
Operations/Customer Service 14.23
Clerical/Secretarial Roles 35.95
ANALYTICS AND COMMUNICATION
Analyst 3.03
Corporate Communication/Content Development 2.20
IT AND ITeS INDUSTRY
IT Services 12.97
ITes and BPO 21.37
IT Operations 15.66
ACCOUNTING ROLES
Accounting 2.55
TEACHING
Teaching 15.23

Table 1: By using standardized assessments of job suitability, in a study of 60,000 Indian undergraduates, we find that a strikingly low proportion of them have skills required for the industry. All these students got detailed feedback from us to improve. The table shows the percentage of students that have the required skills for different jobs. (Refer: National Employability Report for Graduates, under Reports in Publications)

We think assessments will be the key to democratize learning and employment opportunity: it provides a benchmark for measuring success of training interventions, provides feedback to learners creating a ‘dialogue’ in the learning process and most importantly, helps link learning to tangible outcomes in terms of jobs and otherwise.

Let me state it simply: To scale learning and make employment markets meritocratic, we need to scale automated assessments. This is the space we dabble in!

If you are thirsty for data, refer to the table and figure in this post. It tells the story of the problem we are up against and trying to solve.

Figure 1: 2500 undergraduates were surveyed to find their employment outcomes one year after they got their undergraduate education. We categorized their colleges in three categories (tier 1-3) based on their overall performance in AMCAT, our employability test. We find that a candidate in a tier 3 college has 24% lower odds of getting a job and 26% lower salary when he/she has the same merit (AMCAT scores) as a tier 1 students. Similarly, a 1 point drop in college GPA (on a 10 pt scale) decreases job odds by 16% and salary by 9%. Neither of these two parameters are useful predictors of job success beyond AMCAT scores. This shows a clear bias in the employment ecosystem. (Refer ‘Who gets a job’ under Reports in Publications)

How do we solve it? Stay tuned to our subsequent job posts…

Varun

Tweets

A5 : Replying to @Ester_Matters Technology helps analyze factors that sometimes a human eye can miss out. Its bet… twitter.com/i/web/status/1…

A4 : #AI. With the power of #AI you can assess not only functional skills but emotional intelligence, body language… twitter.com/i/web/status/1…

A3 : Replying to @Ester_Matters Driving #changemanagement from top to bottom. Build a dedicated #HRDigital squad t… twitter.com/i/web/status/1…

A1: #Tracking and #Assessing the potential candidates with the power of #AI and right technology. Also bridging the… twitter.com/i/web/status/1…

Aspiring Minds is partnering with HDFC Bank to launch the Future Bankers Program, in association with Manipal Globa… twitter.com/i/web/status/1…

Its time for Shanghai Recruitment Technology Conference, 2019. Look out for an interesting session on "How AI Power… twitter.com/i/web/status/1…

Codemeet can transform your 360-degree hiring experience with live online tech interviews, integrated code editor,… twitter.com/i/web/status/1…

Recruitment Tech Conference 2019, Shenzhen was a hit. Now its time for the Shanghai conference. Really looking forw… twitter.com/i/web/status/1…

Are you still finding needle in a haystack. Fret no more. Now with Aspiring Mind's CODEMEET platform get… twitter.com/i/web/status/1…

I just uploaded “GMT20190626-092813_Webinar-on_1366x768” to #Vimeo: vimeo.com/344760539

I just uploaded “CodeMeet webinar recording” to #Vimeo: vimeo.com/344757085

Join our webinar today on "Simplify tech hiring with AI enabled coding interviews" and understand how you can attra… twitter.com/i/web/status/1…

What if your efforts to hire the best Tech #talent are just like an endless loop of processes and instructions? Joi… twitter.com/i/web/status/1…

Excited to be a part of @SHRM19 Annual Conference & Exposition. Do drop by our Booth #725 to check our AI-powered t… twitter.com/i/web/status/1…

Catch @varaggarwal and @himanshu0820 at ET Now as they discuss how Aspiring Minds had the first mover's advantage… twitter.com/i/web/status/1…

Have you met #SmartMeet? Transform how you recruit and interview with AI-powered live interviewing platform. Intera… twitter.com/i/web/status/1…

Register now for the Webinar: bit.ly/2WWBMZb and know how #AI enabled coding interview platform can give e… twitter.com/i/web/status/1…

Very excited to meet #HR leaders @ Recruitment Technology #Conference 2019 at Shenzhen, #China. Lots of interesting… twitter.com/i/web/status/1…

When every function is driving the Digital DNA mantra, Know "How #AI in #Assessments" can help you in transforming… twitter.com/i/web/status/1…

Join us at Recruitment Technology Conference 2019 Shenzhen, China as we help you to unlock the #Talent potential wi… twitter.com/i/web/status/1…

Himanshu Aggarwal,CEO & Sushant Dwivedy,SVP,Aspiring Minds conducted an engaging session on latest innovations brou… twitter.com/i/web/status/1…

We are honored to be selected as Most Innovative Technology Partner from India at the prestigious Morgan Stanley CT… twitter.com/i/web/status/1…

Thank you all for such a wonderful response at #FICCI Talent Acquisition Conference 2019. We really had a good time… twitter.com/i/web/status/1…

It's a full house at the @ficci_india Talent Acquisition Conference 2019. An insightful panel discussion along with… twitter.com/i/web/status/1…

Glimpse of stimulating discussion happening right now #FICCI Talent Acquisition Conference. Sushant Dwivedy from… twitter.com/i/web/status/1…

Let the show begin. We are all buckled up for the #FICCI Talent Acquisition Conference today. Do drop by at our ass… twitter.com/i/web/status/1…

From #Assessments to interviews & beyond, we are all excited to talk about the new age #Recruitment solutions tomor… twitter.com/i/web/status/1…

Join Karan Chatrath & Rajeev Ranjan on Wed,12th June 2019 @ 3:00 pm as they demonstrate how #AI/#ML can help you op… twitter.com/i/web/status/1…

Lets discuss & share ideas on a very interesting session, being moderated by Sushant Dwivedy, SVP, Enterprise Clien… twitter.com/i/web/status/1…

Visit us at #FICCI Talent Acquisition Conference 2019 on 30 May, 2019 at New Delhi to experience our AI-powered ass… twitter.com/i/web/status/1…

I just uploaded “Optimize hiring for Call Center / BPOs” to #Vimeo: vimeo.com/338250881

I just uploaded “Aspiring Minds Overview - Optimizing hiring” to #Vimeo: vimeo.com/338249939

Thank you for connecting with #AspiringMinds at #SHRMTech19. It was great to share how we are helping leading compa… twitter.com/i/web/status/1…

I just uploaded “Motor skills video” to #Vimeo: vimeo.com/337974587

I just uploaded “Aspiring Minds' Automated English language test” to #Vimeo: vimeo.com/337949644

I just uploaded “Copy of Autoview - AI powered video interview” to #Vimeo: vimeo.com/337947238

I just uploaded “AUTOMATA_ Revolutionizing Computer Programming assessment using Machine Learning(1)” to #Vimeo: vimeo.com/337751605

Meet our experts at the #SHRM Tech APAC Conference 2019, Booth #85 to get more insights on how you can completely r… twitter.com/i/web/status/1…

Excited to be a part of the #SHRM Tech APAC Conference 2019, Come join us at Booth #85 and understand how you can a… twitter.com/i/web/status/1…

Our workforce is stuck in the complex web of being unemployed, underemployed and unemployable! @varaggarwal talks… twitter.com/i/web/status/1…

Since its 1st edition in 2010, National Employability Report for Engineers by @AspiringMinds has sparked national d… twitter.com/i/web/status/1…

Anirvan Mukherjee, director Human Resources at CGI India talks about the merits of Aspiring Minds' products and why… twitter.com/i/web/status/1…

Aspiring Minds' Advisor, Tarun Khanna talks to @TimesofIndia scientific illiteracy in India and how Aspiring Minds'… twitter.com/i/web/status/1…

Bengaluru chapter of the Digital HR Summit witnessed overwhelming response from the HR fraternity. Lots of insightf… twitter.com/i/web/status/1…

Siddharth Nayak talks about Aspiring Minds' #AI powered Codemeet interface that enables anytime, anywhere, seamless… twitter.com/i/web/status/1…

And it's a full house! @sdwivedy addresses our #HR fraternity as we Kickstart our Digital HR summit #Bangalore.… twitter.com/i/web/status/1…

We are about to start. Meanwhile the Smile App #ChallengeAccepted by the #HR fraternity gracing our event. The #ai-… twitter.com/i/web/status/1…

We are excited to meet the brightest minds of #HR at @AspiringmindsAM Digital HR Summit in Bangalore. Insightful co… twitter.com/i/web/status/1…

8 Days to go! #JoinUs on 24th April for the #AspiringMinds #DigitalHRSummit. Here's our impressive line-up of speak… twitter.com/i/web/status/1…

US institutes are leaders in research. Chinese colleges have made exponential progress while India slugs behind. Ho… twitter.com/i/web/status/1…