http://www.programeter.com/ is some kind of joke, right?

Hi Mark, thanks for the come-back :-)

Go on then. The day before yesterday, my colleague solved an extremely dangerous bug that had been a risk to us for months. He did it by removing 1 character and 1 space from the source-code (which in the language we use, fixed an incorrect indentation and therefore pulled an existing statement back *outside* the scope of an if statement where it didn't belong.)

How would the software assess the "productivity" of this? (I mean long period of reading a file of about 2000 lines of horrible legacy spaghetti code, identifying the bug, and then making a two character change + adding a comment)

Obviously, I'd be interested to see the algorithms you use, but I can't imagine any algorithmic measurement which has much better than chance possibility of correlating with "productivity" or ROI. (Which is more productive? More code or less code? Higher comment / code ratio or lower? Higher complexity or lower?)

Ultimately the answers will depend on the essential complexity of the problem being solved. If the problem-space has essential complexity, then the program will need to be big and complex to capture that, and a shorter one would be incomplete. If the problem turns out to have low complexity, or is "compressible" through some clever higher-order or meta-programming tricks, then 30 short lines written in one afternoon can beat a framework of 10,000 that took 3 months.

Then there are the contextual and external values : timeliness of the solution, long-term maintainability of the code, the guy who looked over my shoulder and pointed out a crucial error before it went into production and lost us millions of dollars worth of credibility etc.

Unless your software has a way to capture information about the essential complexity of each specific real-world problem being addressed (note, it's not good enough to parameterize this on a "company" or "project" basis, it has to be done "per problem") then you'll never be able to answer the question as to whether the programmer who wrote more code was more productive than the programmer who wrote less.

What your tool does, if people take it seriously, is give non-technical managers (technical ones, I presume would just understand what their underlings are doing sufficiently well to assess it) a completely spurious set of measurements which will give them almost no insight into how well their programmers are performing but let them make everyone's lives miserable until the employees start learning to fake the metrics.

Programmers who wrote elegant short solutions and got reprimanded for idling, will simply learn to write less productive, more verbose boilerplate to fool the system. If you recallibrate the system to reward brevity, you'll kill the guy who realized the long-term necessity of adding a crucial abstraction layer between two modules and took a month to put it in place.

Software development is a horrendously delicate "satisficing" problem. It's hard enough for smart humans to do well. It's extremely unlikely your software will be able to improve on that.

phil jones

I too see this as a deeply broken model of programmer productivity - something I'd never use and strongly recommend against deploying. Off the top of my head:
* What about pair programming?
* What about time spent by lead developers mentoring?
* Code lasting does not necessarily mean good code. Good code gets refactored and updated continuously.
* All these metrics seem, as described, easy to game. And gaming metrics like this would _really_ hurt the application.
* The best developers often spend large amounts of their time removing code - by killing dead code or by refactoring for simplicity. By measuring the "amount of code produced" this looks unproductive!

I could go on. So far with every developer or manager I've show this to the reaction has been "a joke" (which was my first reaction - and if so - well played sir!) or sad dismay.

Mark,

good to see you coming back. I'd love to see your response to my fundamental point, though : that almost any metric you can think of for programmer behavior will be *problem specific*. Doesn't matter whether that's quantity of code, complexity, comment-to-code ratio, frequency of checkins etc. And it doesn't matter if you have one metric or a basket of them. It's the same problem.

Just as there's no way to measure how good poetry is by looking at the density or layout of words on the page, or how many poems got written per day. For the obvious reason that programming is a creative, "design" discipline, just like architecture, engineering etc.

Or even "management".

In fact, I wonder how many of the CxOs you're hoping to sell this software to would like their own performance to be evaluated based on how many emails they send, and how long the sentences are, as opposed to the quality of the *ideas* expressed in them?

:-)

Mark,

I'm going by what I've read on your web-site. If you want to tell me more about the details of the measurements you take and what you interpret from them, then I'll make a better assessment.

But, like I say, it certainly sounds from your site that four of the metrics are : Contribution Size ("amount of code"), Activity (I'm guessing, frequency of checkins), Complexity ("McCabe's Cyclomatic") and Density of Comments (ratio of comments to code). These are the are ones I've been discussing previously. And I still claim that there is no way on earth that, given these metrics for two programmers, independent of having some corresponding metrics of the *problem* that's being solved, you can assess which programmer was "better" or "more productive".

(And given that the site says : "Decide the value of programmer instantly, based on his contributions" ... it certainly sounds like you're claiming you can do that.)

The other metrics like "measurement of know-how" I haven't commented on because I honestly can't imagine what they really mean or how you'd measure what I think of as "know-how". Again, I'll be fascinated to know more.

The fact that you're gathering the measurements described above, and the fact you say they make program managers "smarter" suggests to me that you consider them to be genuinely informative of something. So you can't in the next breath turn around and say : but it's ok, "there is still human with brains evaluating the reports we provide".

Either those brains are ignoring the metrics (in which case the whole thing is pointless) or the brains are taking them as important evidence (which is what I'm disputing.)

cheers

phil

BTW : One thing I give you kudos for : having the courage to link this discussion from your site. Well done on that.

@Mark,

Thanks for opening up the discussion. I probably did not clarify my concerns adequately earlier. I have made up for this in an open letter to Programeter.

It is obvious to me that this software is aimed a catching 'Cheats' given the frequency of that words occurrence in these comments. Thus we get a good picture of where this coming from and why it is unlikely to find good usage in anything other than programmer sweatshops, probably best left ignored unless thats the kind of operation you run.

I think Mark has replied positively. Programeter is not for serious use

It is really an issue of hourly rate or per task rate. This needs to be estimated, bargained or extrapolated based on reputation and not on surveillance technology.

Risk is a permanent feature of doing business and trust is a finely balanced responsability. There is no place for Stanley Milgram machinery in building trust or reputation.

Vik

Evaluating the correctness of a program reduces to the halting problem - in general terms this is, of course, impossible to do - and provably impossible. See the Entscheidungsproblem on Wikipedia!

In specific domains, the problem might be solvable. But in that case, you can generate the correct solution by running your tests backwards. If you know what is 'right' - just have the tester write the code.

And we know from experience that code generation only works (but works well) in limited areas, like building GUIs, gluing layers together. 'Real' programming still needs people - for checking and for doing.

So based on this argument, I'd say this is either a joke (that hits a raw nerve, if you ask me) or a fundamentally flawed idea.

Having said that, from a business point of view, brilliant. With the right contacts and sales approach you could make a mint. I hope it doesn't though, as it is wasted money - and the only idiots really likely to buy this big time are government projects and big business. I don't want my taxes spent on this. :(