HackerRank: a good - but strangely controversial - tool for evaluating candidates

We've recently started using HackerRank for screening candidates, and we're encountering mixed opinions about it from our candidates and from the community as a whole.  This article aims addresses these points - and suggests what could be done about them - from the perspective of an evaluator and a candidate.

Last week, I mis-typed "HackerRank for work" when looking to navigate to the login page via way of Chrome's search bar.  Although my first instinct was to correct the mistake and continue on my way, I took note of several results: "Beware of HackerRank"; "Don't work for companies that use HackerRank"; there was a common theme here.  The main points put across were:

  1. Understanding lower-level concepts such as asymptotic complexity is completely unnecessary when you're building a web application on an MVC framework; therefore, rejecting candidates on this basis is asinine.
  2. Candidates were not asked questions that evaluated their practical competence as a software engineer, as the questions were too focused on computer science fundamentals.
  3. A developer you're looking to employ needs to know more than how to "knock out" a coding exercise in a few minutes.
  4. Perhaps most of all, some experienced developers and superb candidates undertake a HackerRank test attain a depressingly low score.  This can be quite demoralising to the experienced and inexperienced alike, and this happening to a proven developer usually yields the conclusion that the test is wrong (or, at least, is focusing on the wrong areas).

My experience with HackerRank inclines me to generally disagree with these opinions (with the exception of point 4, to which I agree wholeheartedly) for reasons I will lay out throughout the course of this post; but the theme of a disparity between academic study and later career is one to which I am very sympathetic.  By this point in my career, I've interviewed more than I've been interviewed, and this has emerged as a common theme, manifested in a number of different forms.  As an example, A web developer who - at University - studied software engineering concepts with C++/Java now works on PHP/Wordpress websites, and believes that they make absolutely no use of what they learned through study.

I worked as a contractor throughout the entirety of my full-time degree.  This gave me the opportunity to feed my professional experience into my academic learning, and the academic concepts into my professional abilities.  While it's true that you don't *always* use in your career what you learn through academic study, I've never failed to find what I've learned useful (often in a surprising ways).  The blanket assertion that you simply don't need to possess, retain and exhibit at least some level of underlying knowledge of computer science fundamentals is problematic in the following ways:

  • It creates a gulf between software engineers who create games and software engineers who create websites; suggesting that the latter are incapable of the former.
  • There are many situations where a developer or engineer could solve [part of] a problem by drawing from this knowledge.
  • It can manifest as bitter negativity that is non-conducive to a productive, collorative working environment.  It's not unlike framework-averse developers claiming to the non-averse that OOP is nonsense, and procedural spaghetti coding is "the right way to do things" simply because that is their preference.

I have personally struggled - even without a tool like HackerRank - with interviews that forced me to draw on knowledge that had remained largely dormant for six or seven years, while I worked on established technology stacks that largely insulated me from such thinking.  Although demoralising, it was an eye-opener for me, and ultimately encouraged me to keep those skills sharp through open-source projects or online tests.  Although I originally signed up for HackerRank to grade candidate tests, I often find myself completing coding challenges every other day; in that respect, it has been invaluable to me.

So, taking each of the main points in turn:

Why know data structures when you have databases, ORM and frameworks?

Until "NoSQL" solutions experienced a sudden surge of popularity in the early 2010s, relational database management systems were the "go-to" solution for data storage.  NoSQL represented a complete change in thinking; going back to the basics (data structures, algorithms) and producing data storage solutions that - for certain applications - were significantly more performant.  This concept does not apply to lower-level systems alone; consider an expensive loop in a modern web application in which the 50th of 100 iterations renders the remaining 50 iterations pointless.  It might be immediately apparent to someone without the benefit of a degree education, but it's no less important that we evaluate a candidate's ability to optimise sensibly.

Candidates aren't practically evaluated on their ability as software engineers

Our original, somewhat out-of-the-box HackerRank test did somewhat justify this assertion, which makes me think that the writers of the aforementioned article had a similar experience (I completed the test while we were still evaluating it and attained a depressingly low score).  We recognised this, reviewed the questions and refactored them to include a mixture of multiple-choice, coding and "subjective questions" (questions which cannot be automatically marked) closer aligned to our workload, but not at the cost of evaluating fundamental skills.

An engineer needs to do more than knock out a piece of code in a few minutes

Correct, but sometimes it is necessary that they do so, and often under pressure.  The engineer's attitude to such an exercise is as important as their approach, regardless of their score.  And, again, it's not the only thing that can be evaluated about a candidate.

A low mark can shatter the confidence of the most experienced engineer

As I mentioned above, I have experienced this personally and can speak for how unpleasant it is.  That's why the content of a test should be subject to regular review, the candidate's approach always considered and constructive feedback always given.  If you're hiring developers to work on a RESTful web service platform and the test asks no questions about the HTTP protocol, then something needs to change.  At the same time, if a candidate is out of practice in areas evaluated by the test, then it is better for the candidate to learn from the experience and feed that into future applications rather than decry the tool used to evaluate them.

In summation...

HackerRank isn't the only way to evaluate a candidate, nor is it the best in the world.  However, it's not intended to be; it's goal is to increase the efficiency of our hiring process and for us, it has been very successful in that regard.  The tests themselves aren't set in stone, and the HackerRank team have been excellent in helping us refine them.  We've aligned our pass mark with this in mind, reviewed and redesigned our questions to better fit our usual workflow.  We take into account the candidate's feedback (HackerRank offers this feature to the candidates once they have completed the test) and their approach to coding exercises, and these feed into later interview stages.

It's doubtful that we're the only company using HackerRank who follow this approach.  A low mark doesn't make you a terrible engineer, but a negative attitude about that bad mark doesn't afford you the opportunity to learn.  If you reject a prospective software engineering job just because the company is using HackerRank, you may well regret it.