How to Score RFP Responses from IT suppliers

I’ve seen it several times now: a spreadsheet containing clauses from an RFP, evaluation criteria with importance weights from 1 to 5, and a scale from 1 to 10 to score the offerings from different suppliers.  Weight x Score = Weighted Score.  Sum over all criteria to create the Grand Total.

What’s not so grand is the usefulness of this method for scoring RFP responses from IT suppliers. For example four suppliers receive Grand Totals of 524, 507, 498 and 530. These total scores do not differentiate the suppliers convincingly, considering the margin for error, so the method is worthless. Why? With a large number of criteria and closely-spaced scores, the unders and overs cancel each other out, so you end up with very similar totals.

On the other hand, your experience and intuition tell you that supplier A’s solution is significantly better than B’s. 

One reason that your instinct disagrees is that a simple comparison of weighted scores is inappropriate for evaluating RFP responses from IT suppliers, because it allows good scores in several highly desirable areas to compensate for very poor scores in very important areas that should not be traded away.

If you really must use this method to quantify the merit of RFP responses, then this is how to do it:

1. Write a decent RFP with questions that really elicit the type of response that you can give a meaningful score to. Give each question an exponential weight like 5, 10, 25, 50, and give serious consideration to tossing out the questions that have a low weight.  If they are that unimportant, then why waste the time of X suppliers to respond, and waste your own organisation's time (multiplied by X) to evaluate responses to unimportant questions?

2. Use your team’s collective experience and knowledge of what is really important, to instinctively determine the merit of each response. Use an exponential scale to score the responses, e.g. 0, 10, 100, 1000.

3. Multiply and add, as usual.

4. If the grand totals don’t match your instinct, fiddle with the weights and calculate again. Repeat until you have designed a plausible evaluation system.

This way, the wretched spreadsheet quantifies and supports your instinct, but it doesn’t do the thinking for you.