interstar's dashboard

Good support

Cool. Thanks. I shouldn't really have spaces in my paths, I know, but I foolishly uploaded these without checking – interstar, on July 24, 2008 23:35

Oops, there was an issue with spaces in URLs, like yours have. Thanks for reporting this!

I've fixed it in the code, it'll be online with the next update -- shouldn't be more than a few days away. I'll let you know.

my comment was related to your argument that programmers don't reject performance-measurement idea – Mark, on July 08, 2008 05:34

not agree with which of my claims? that "most programmers hate the idea"? Or that they hate it because they want to be assessed on an ability that the software doesn't measure? – interstar, on July 07, 2008 19:56

adrianb, Programeter is not about evaluating correctness of the programm. It is about collecting indicators about programmers themselves. – Mark, on July 07, 2008 14:07

"Programmers hate the idea ... not because they reject performance-measurement but, like anyone else, they want their performance to be assessed by what they actually work with. Programmers produce *ideas*, algorithms, abstractions, architectural patterns, mathematical entities. That's the criteria they want to be judged by. "

Considering the fact that most negative comments came from those who never tried our Programeter, I would not agree with you. It seems to me that just the idea of performance measurement is very provocative and generates lots of emotions.

We are helping lots of managers already, and they feel more informed with our indicators than without. So early enough warning, as you said, is something very valuable.

"Seriously Mark. Wouldn't it be much smarter just to have a good manager of the programmers who actually talks to them every day about what they're doing and has a pretty good understanding of what it is and how things are going? That's early enough warning if things are starting to go wrong. "

Yes, good manager is a good choice. But it is not always possible. Following could be the reasons for that:
* Additional managers are expensive. Especially if manager is "good"
* Not all managers have programmer background. So sometimes they need tools to help them out to understand what has really been done
* Managers and developers are often located in different offices, city or even countries. It makes it very difficult to meet developers on daily basis. Personally I have participated in project like this, and I have met my project managers once per few months.

So what you describe is an ideal case. But unfortunately, not all software is developed in ideal worlds.

Seriously Mark. Wouldn't it be much smarter just to have a good manager of the programmers who actually talks to them every day about what they're doing and has a pretty good understanding of what it is and how things are going? That's early enough warning if things are starting to go wrong.

Your software seems to be designed for a world where one person supervises dozens of programmers at once, without having any deep engagement with the work they're doing, and so needs some kind of simplistic statistical early warning system.

It assumes that running a software company is like running some stereotypical factory production line.

But how many successful software companies or projects are run like that? Not Google. Not Microsoft. None that I can think of.

Does the software development in your company work like this?

Programmers hate the idea ... not because they reject performance-measurement but, like anyone else, they want their performance to be assessed by what they actually work with. Programmers produce *ideas*, algorithms, abstractions, architectural patterns, mathematical entities. That's the criteria they want to be judged by.

They don't produce "lines of code" or "check-in events" any more than their managers produce "emails" and "spreadsheets" and would like to be judged on the amount of those they produce.

That's different from sales. Sales-people actually *do* produce "sales" and so that performance-measure is appropriate to them.

how does "frequency of check-ins" differ from "number of times programmer changes code components"? ... you mean one is "5 times a day" and the other is "2400 times over the last month"??? – interstar, on July 04, 2008 20:08

Evaluating the correctness of a program reduces to the halting problem - in general terms this is, of course, impossible to do - and provably impossible. See the Entscheidungsproblem on Wikipedia!

In specific domains, the problem might be solvable. But in that case, you can generate the correct solution by running your tests backwards. If you know what is 'right' - just have the tester write the code.

And we know from experience that code generation only works (but works well) in limited areas, like building GUIs, gluing layers together. 'Real' programming still needs people - for checking and for doing.

So based on this argument, I'd say this is either a joke (that hits a raw nerve, if you ask me) or a fundamentally flawed idea.

Having said that, from a business point of view, brilliant. With the right contacts and sales approach you could make a mint. I hope it doesn't though, as it is wasted money - and the only idiots really likely to buy this big time are government projects and big business. I don't want my taxes spent on this. :(

I think Mark has replied positively. Programeter is not for serious use

It is really an issue of hourly rate or per task rate. This needs to be estimated, bargained or extrapolated based on reputation and not on surveillance technology.

Risk is a permanent feature of doing business and trust is a finely balanced responsability. There is no place for Stanley Milgram machinery in building trust or reputation.

Vik

It is obvious to me that this software is aimed a catching 'Cheats' given the frequency of that words occurrence in these comments. Thus we get a good picture of where this coming from and why it is unlikely to find good usage in anything other than programmer sweatshops, probably best left ignored unless thats the kind of operation you run.

@Mark,

Thanks for opening up the discussion. I probably did not clarify my concerns adequately earlier. I have made up for this in an open letter to Programeter.

Mark,

I'm going by what I've read on your web-site. If you want to tell me more about the details of the measurements you take and what you interpret from them, then I'll make a better assessment.

But, like I say, it certainly sounds from your site that four of the metrics are : Contribution Size ("amount of code"), Activity (I'm guessing, frequency of checkins), Complexity ("McCabe's Cyclomatic") and Density of Comments (ratio of comments to code). These are the are ones I've been discussing previously. And I still claim that there is no way on earth that, given these metrics for two programmers, independent of having some corresponding metrics of the *problem* that's being solved, you can assess which programmer was "better" or "more productive".

(And given that the site says : "Decide the value of programmer instantly, based on his contributions" ... it certainly sounds like you're claiming you can do that.)

The other metrics like "measurement of know-how" I haven't commented on because I honestly can't imagine what they really mean or how you'd measure what I think of as "know-how". Again, I'll be fascinated to know more.

The fact that you're gathering the measurements described above, and the fact you say they make program managers "smarter" suggests to me that you consider them to be genuinely informative of something. So you can't in the next breath turn around and say : but it's ok, "there is still human with brains evaluating the reports we provide".

Either those brains are ignoring the metrics (in which case the whole thing is pointless) or the brains are taking them as important evidence (which is what I'm disputing.)

cheers

phil

BTW : One thing I give you kudos for : having the courage to link this discussion from your site. Well done on that.

Interstat,

In general, I agree that there is no way to give creativity one measure. But we don't try to do that. Instead we give different level of management additional information for their decisions.

What is important to understand is that manager with Programeter metrics is much smarter than manager without those. Don't forget that there is still human with brains evaluating the reports we provide. So if somebody cheets, it can bee understood easily.

Mark,

good to see you coming back. I'd love to see your response to my fundamental point, though : that almost any metric you can think of for programmer behavior will be *problem specific*. Doesn't matter whether that's quantity of code, complexity, comment-to-code ratio, frequency of checkins etc. And it doesn't matter if you have one metric or a basket of them. It's the same problem.

Just as there's no way to measure how good poetry is by looking at the density or layout of words on the page, or how many poems got written per day. For the obvious reason that programming is a creative, "design" discipline, just like architecture, engineering etc.

Or even "management".

In fact, I wonder how many of the CxOs you're hoping to sell this software to would like their own performance to be evaluated based on how many emails they send, and how long the sentences are, as opposed to the quality of the *ideas* expressed in them?

:-)

Hi Adrian,

I appreciate your comment. To make my comment useful for others too, I will try to skip your emotions and address important topics.
* "All these metrics seem, as described, easy to game" - You can't claim so untill you have done so. There were many programmers who tried to trick our algorithms. But that cheating was clear from the reports.
* "The best developers often spend large amounts of their time removing code" - that is very good comment. Which actually proves that our metrics are usefull. We have never claimed that lots of code is good ;)
* " By measuring the "amount of code produced" this looks unproductive!" - We don't measure just amount of code produced. We measure a set of metrics which all together bring a objective enough picture on what has been done by programmer.

If you would like to dispute on any of those topics, I would appreciate if you start a separate topic. I would be happy to answer all your questions.

Btw, if you would like to try out our service to make your claims more reputable - give me a note.

I too see this as a deeply broken model of programmer productivity - something I'd never use and strongly recommend against deploying. Off the top of my head:
* What about pair programming?
* What about time spent by lead developers mentoring?
* Code lasting does not necessarily mean good code. Good code gets refactored and updated continuously.
* All these metrics seem, as described, easy to game. And gaming metrics like this would _really_ hurt the application.
* The best developers often spend large amounts of their time removing code - by killing dead code or by refactoring for simplicity. By measuring the "amount of code produced" this looks unproductive!

I could go on. So far with every developer or manager I've show this to the reaction has been "a joke" (which was my first reaction - and if so - well played sir!) or sad dismay.