Turning "we want a good analyst" into scoring weights

"We want a good analyst" is not a specification. It is a feeling, and it is why two people on the same panel can interview the same candidate and disagree completely — they were quietly weighting different things. Here is how to turn it into criteria with numbers on them, using the weights our own product ships as a worked example.

The problem with "good"

Ask three people what a good analyst looks like and you will get three answers: one says SQL, one says "asks the right question", one says a degree in statistics. None of them is wrong. They are describing different dimensions of the same role and assuming everyone means the same thing by "good".

That disagreement does not disappear when you start interviewing. It just moves underground, and resurfaces as an argument about one candidate that is really an unresolved argument about the role.

A vocabulary of six

You cannot weight things you have not named. These are the six we use, and they are deliberately few — a rubric with twenty rows never gets filled in:

The same six, weighted three ways

These are the presets our product actually ships. They are not illustrations — this is the configuration:

Read the two extremes against each other, because the contrast is the whole lesson.

For an analyst, skills carry 30% and achievement 15%. The work is largely "can you actually do this with these tools", and a candidate who has done it competently without ever quantifying the impact is still a strong analyst.

For a sales hire, that reverses: achievement carries 35% and skills only 10%. Everyone claims the skills, the skills are teachable, and what you actually want to know is whether numbers moved when this person was responsible for them. A certificate is worth 5%, which is to say almost nothing.

Neither weighting is more rigorous than the other. They encode different beliefs about what predicts success in different work, and writing them down is what makes those beliefs arguable.

Building your own, in four steps

1. Write the one sentence. What will this person be doing in month three that nobody is doing today? If you cannot write it, the role is not defined enough to weight.
2. Pick your top two dimensions and say why. Only two. If everything is important, nothing is weighted, and you have re-created the argument you were trying to avoid.
3. Give those two half the total between them. Then distribute the rest. Round numbers are fine; precision here is false comfort.
4. Sanity-check with a real person. Take someone already doing the job well and score them against your weights. If your best current employee comes out mediocre, your weights describe a person you have not got and may not need.

That last step catches more bad rubrics than anything else. It is the rubric equivalent of checking whether your shortlist looks like your job description.

The trap: weighting achievement heavily

It is tempting, because impact is what you actually care about. But there is a measurement problem, and we can show it rather than assert it.

Across the 78 real CVs we have scored, Achievement & Impact is the flattest dimension we have — its average is roughly half of what Education averages, and nearly everyone clusters at the bottom. Not because the applicants achieved nothing, but because CVs almost never say what changed. "Handled the monthly reporting" and "cut the reporting cycle from five days to one" can describe the same person on different days of writing.

So if you weight achievement at 35% and your applicant pool writes duty-lists, you have built a rubric that mostly measures who has been taught to write a CV. That is a real signal for a sales role, where communicating your own value is part of the job. It is close to noise for a back-office role, where it is not.

Two ways to handle it: ask for impact explicitly in the posting, so candidates know to supply it; or keep the weight moderate and gather the evidence in the interview, where you can ask "what changed?" and hear the answer.

Checking your weights after one round

Weights are a hypothesis. After a round, test it — three questions, no software needed:

1. Did the people you shortlisted actually score well on your top dimensions? If your highest-weighted dimension does not predict who you wanted to talk to, it is the wrong dimension.
2. Which dimension did everyone score the same on? A dimension where everyone lands in the same place is not separating anyone, and it is spending weight it has not earned.
3. Did you overrule the ranking, and on what basis? If you pulled someone up from the middle, the reason you did it is a dimension you care about and did not write down. Add it.

That third question is the useful one. Overruling the ranking is not a failure of the ranking — it is information about what you actually value, arriving in the only form it ever arrives in.

What a weighting is not

It is not a decision procedure. A weighted score puts candidates in an order and shows its working; it does not tell you who to hire, and it should never be allowed to close anybody out by itself. What it buys you is that when your panel disagrees about a candidate, you are now having a specific argument — about whether experience should outweigh credentials for this role — instead of a vague one about whether someone feels right.