Testing the development skills of a programmer

The first problem with these online test is that they usually take the questions and answers from a database, true they might shuffle it around a bit but if you did 2 practice runs you might have seen all the questions that are in the database. Knowing this, how serious can you take the result of such test?

Furthermore, these online test are in general time based, e.g. 90 minutes for 65 questions or “problems” to solve that might be connected to a specific programming language but are not connected at all. E.g. a question in a PHP test could be about Traits, where one needs to select all of the correct implementations of Traits, the next question is about Cross-Site Scripting Attacks or XSS in PHP where a correct assessment needs to be done on a code snippet the next question might be about Object Inheritance.

As a PHP developer I know how to explain and solve the problems mentioned above, but switching around from one subject to another and then back again with a timer on the clock might make things a bit harder to solve.

I personally believe that software development is not a race, therefore testing a developer using an online questionnaire with multiple choice questions, true/false questions while jumping around subjects all with a countdown timer counting down is not the right way to test the skills of a software developer.

Furthermore, most developers today work with an integrated development environment (IDE) which in most cases helps the developer in programming. Some people may say that this is bad while others say it’s good. I personally believe it’s okay to rely on an IDE that does autocompletion and checks the code for mistakes. Programming languages get new functions and methods added while others get deprecated or removed. An IDE helps the programmer to speed up development, double check the produced code and more, therefore a developer might know the correct implementations of Traits in PHP when using the development tool.

An example: I once did a test for software making Windows Desktop applications, I can no longer remember whether it was for Delphi or .NET… does not matter. One of the timed questions was:

What property to set when you want to change the caption / title of a Windows Dialog box.
Select one or all that apply.

• Title
• Caption
• Header
• Label

At the moment I was doing the test I was developing in Visual Basic and Borland Delphi both having their own IDE and naming things different. Right at that spot, with a timer running I could not give you the correct answer because in one programming language the property was title while in the other language it was caption. If I had the IDE in front of me I could give you the answer in less than 30 seconds, either setting the property through the property editor or through code.

The right way to test a programmer

I personally believe that a good software developer is language agnostic. It usually takes a a few days for a good programmer to get used to the constructs of a (new) programming language. The basic principles around programming apply across all languages be it PHP, Python, Java, C, C++ or whatever.

Web developers especially need to be language agnostic, one moment they are developing stuff in JavaScript while the next moment they are doing things in Ruby or PHP and in the next moment they are programming in the CSS language using e.g. Less.

When looking for a programmer I would ask them questions around data structures, algorithms, etc. I would be interested to see how a developer approaches or go about solving a problem. I believe the best way to test the skills of a developer is to give home a practical problem to solve and review the solution that is created by the software developer.

Good programmers should write a lot of code, right?

Measuring the development skills of a programmer by the lines of code he can program is in my opinion a bad idea. Simply because “lines of code” is a poor metric, when a developer is coding new functionality (which is very different from fixing bugs, refactoring architecture, and solving tricky problems) the lines-of-code output depends on the natural “code bulk” of the language and the type of problem being worked on.

Simply said: lines of code are in general like working hours, you can work 100 hours and get nothing done and you can write 100 lines of code that do nothing.

In my professional career I’ve programmed a lot, worked with programmers and was leading developers as well and my message to myself and to other developers is always: write clean code, clean up code. The best line of code is the one that you don’t need to write. A software developer, working at Facebook for three years wrote: “I have added 391,973 lines to and removed 509,793 lines from the main repository. So if I coded 1000 hours a year, that’s about 39 net lines removed per hour!“

How to measure software development performance?

The main problem with performance metrics like this, is that humans are very good at gaming any system that measures their own performance to maximize that exact performance metric – usually at the expense of something else that is valuable.
Some examples how on how you could measure a developer and how a developer could game the system:

• Measure the number of lines a developer produces – (the developers just churn out mountains of boilerplate code, and other needless over engineering, or simply just inline every method)
• Measure the unit test failures – (the developer will not write any unit tests so no failed unit tests can be measured)
• Measure unit test coverage – (the developer will write weak tests that exercise the code, but don’t really test the produced code properly)
• Count the number of bugs found in the produced code – (the developer simply won’t do any coding, because of this his code does not generate any bugs)
• Count the number of bugs that were fixed by the developer – (the developer will choose the easy/trivial bugs to work on)
• Measure the actual time to that was used to finish a task based against the developers own estimate – (the developer will estimate more time to give more room)

And the list goes on.

I believe the best way to measure a developer performance is to have excellent managers in place that do their job well, have good specifications that accurately reflect the requirements and track everyone’s progress carefully against those specs.