In a December 21, 2009 editorial in the Suffolk News Herald, the city’s Director of Media and Communications gave the folks at the Suffolk News Herald a hand-slap. Ouch! I read the editorial and then decided to look at the Request for Proposals for the Obici House Renovation and Reuse. I wanted to answer the question: Was the actual RFP a fair tool?
To answer my question, I studied the scoring tool included in the RFP to determine its effectiveness as a fair tool for evaluating proposals. Why is this important? The scoring tool is what is used to award the points for each proposal – it is the place where all of the work done by firms is finally scored for merit. The scoring tool is the most important part of the RFP. So here goes.
The scoring “matrix” for the Obici House Renovation and Reuse proposal was included with the RFP handout. You see, the purpose of an evaluation tool is to ensure a valid, reliable, and bias-free evaluation of proposals while awarding points to projects. A valid scoring tool simply means that the tool evaluates what you wanted to evaluate. I would expect to see actual statements from the proposal included in the tool. A reliable scoring tool means that if you put the proposals in front of two or more people for scoring, the scores will be similar. A bias-free tool means that the scoring will be fair – no matter what persons, organization, or entity turned in a proposal. Each firm has an equal chance from the scoring side of the evaluation. The scoring tool for the Obici House Renovation and Reuse proposals follows:
The first thing I want to do is to address this part of the evaluation matrix – which is not really a matrix. This is simply a scoring guide. I want to commend the city for including a scoring guide in the proposal packet. That is evidence of working toward a fair process in terms of scoring the proposals. But, the scoring guide isn’t yet a valid, reliable, or bias-free scoring tool. Let me show you why.
In order to get an Outstanding, a firm must achieve 8-10 points. There is nothing in this scoring tool that tells firm members what has to be achieved in order to receive those points. What does a firm have to have in the proposal in order to get an 8? What makes it a 9? What makes it a 10? How does someone who is scoring the proposals know to give a firm an 8, 9, or 10? What makes an 8 different from a 9?
In the 4-7 Points category, a firm meets expectations and is fully qualified and has adequate experience. Why wouldn’t that be an outstanding in terms of points? What would make a firm lose points here? What is different in a proposal that receives a 4 from one that receives a 7?
In the 1-3 points category, a firm will have minimal experience. What designates minimal experience? Also what does “less than expectations” mean? What does”less than desired level/insufficient documentation” mean? What gives a firm a score point of 1? A score point of 2? A score point of 3?
In the 0 Points category, what does “not documented” mean? The whole proposal is not documented? A part of the proposal or section is not documented?
These areas do not correspond on a one-to-one basis from the proposal – as they should. Each entity (e.g., city, firms submitting a proposal) should know EXACTLY each score point looks like in each area being evaluated. The scoring tool is not yet valid because there is no connection between the tool and the requirements in the RFP.
For example, look at number 1: Obici House Renovation, Restoration and Re-Use Plan and Property Management Program to include Architectural and Historical Preservation. What does a firm have to specifically do from the RFP requirements to get a score of 10? A 9? An 8? Since the information about how a firm achieves each score point is not there, the scoring system is flawed. A firm trying to use this tool to guide its work cannot tell what it specifically needs to do to achieve a level 8, 9, or 10 – or any other level for that matter. There is nothing that aligns the overall score system to the RFP requirements.
Another thing I’d like to address with this score system is that the way it is designed; everything counts the same. A firm can earn just as many points for having a complete proposal as it does for the actual proposal ideas. A scoring tool is also a communication tool – communicating to firms what is most important to pay attention to because it’s most important to the city. The scoring tool does not do this at all. Again, the scoring tool is not aligned to the requirements of the RFP. Even if the information is included elsewhere in the RFP, if the scoring tool is not adequate and aligned, it is a flawed tool.
A high-quality and effective scoring guide will include an actual matrix, which clearly shows how each point can be attained. A high-quality scoring tool will also align to whatever it is the city is specifically looking for in its proposals. Without this, the scoring tool is arbitrary and open to the scoring whims of the scorers. Additionally, the scoring tool will reflect the weight the city places on the items it is scoring.
Now to answer my own question – was the actual RFP itself fair to those who used it to develop their proposals? I’d have to say no for the following reasons: (1.) the RFP includes an inadequate scoring tool making it an unreliable tool that is open to bias, (2.) The scoring tool is not aligned with the requirements in the RFP making it not valid, and (3.) The scoring tool does not reflect the weight of items in the RFP further impacting its validity.
I can offer a simple and easy solution. In the next version of the RFP (or any other RFP), simply create a valid and reliable scoring tool that can fairly be used by the city and firms responding to RFPs.
