Month: December 2009

Is the RFP Scoring Tool Fair?

In a December 21, 2009 editorial in the Suffolk News  Herald, the city’s Director of Media and Communications gave the folks at the Suffolk News Herald a hand-slap.  Ouch!  I read the editorial and then decided to look at the Request for Proposals for the Obici House Renovation and Reuse.  I wanted to answer the question: Was the actual RFP a fair tool?

To answer my question, I studied the scoring tool included in the RFP to determine its effectiveness as a fair tool for evaluating proposals.  Why is this important?  The scoring tool is what is used to award the points for each proposal – it is the place where all of the work done by firms is finally scored for merit.  The scoring tool is the most important part of the RFP.  So here goes.

The scoring “matrix” for the Obici House Renovation and Reuse proposal was included with the RFP handout.  You see, the purpose of an evaluation tool is to ensure a valid, reliable, and bias-free evaluation of proposals while awarding points to projects.  A valid scoring tool simply means that the tool evaluates what you wanted to evaluate.  I would expect to see actual statements from the proposal included in the tool.   A reliable scoring tool means that if you put the proposals in front of two or more people for scoring, the scores will be similar.  A bias-free tool means that the scoring will be fair – no matter what persons, organization, or entity turned in a proposal.  Each firm has an equal chance from the scoring side of the evaluation.   The scoring tool for the Obici House Renovation and Reuse proposals follows:

The first thing I want to do is to address this part of the evaluation matrix – which is not really a matrix.  This is simply a scoring guide.  I want to commend the city for including a scoring guide in the proposal packet.  That is evidence of working toward a fair process in terms of scoring the proposals.  But, the scoring guide isn’t yet a valid, reliable, or bias-free scoring tool.   Let me show you why.

In order to get an Outstanding, a firm must achieve 8-10 points.  There is nothing in this scoring tool that tells firm members what has to be achieved in order to receive those points.  What does a firm have to have in the proposal in order to get an 8?  What makes it a 9?  What makes it a 10?  How does someone who is scoring the proposals know to give a firm an 8, 9, or 10?  What makes an 8 different from a 9?

In the 4-7 Points category, a firm meets expectations and is fully qualified and has adequate experience.  Why wouldn’t that be an outstanding in terms of points?  What would make a firm lose points here?  What is different in a proposal that receives a 4 from one that receives a 7?

In the 1-3 points category, a firm will have minimal experience.  What designates minimal experience?  Also what does “less than expectations” mean?  What does”less than desired level/insufficient documentation” mean?  What gives a firm a score point of 1?  A score point of 2?  A score point of 3?

In the 0 Points category, what does “not documented” mean?  The whole proposal is not documented?  A part of the proposal or section is not documented?

These areas do not correspond on a one-to-one basis from the proposal – as they should.  Each entity (e.g., city, firms submitting a proposal) should know EXACTLY each score point looks like in each area being evaluated.   The scoring tool is not yet valid because there is no connection between the tool and the requirements in the RFP.

For example, look at number 1: Obici House Renovation, Restoration and Re-Use Plan and Property Management Program to include Architectural and Historical Preservation.  What does a firm have to specifically do from the RFP requirements to get a score of 10?  A 9?  An 8?  Since the information about how a firm achieves each score point is not there, the scoring system is flawed.  A firm trying to use this tool to guide its work cannot tell what it specifically needs to do to achieve a level 8, 9, or 10 – or any other level for that matter.  There is nothing that aligns the overall score system to the RFP requirements.

Another thing I’d like to address with this score system is that the way it is designed; everything counts the same.  A firm can earn just as many points for having a complete proposal as it does for the actual proposal ideas.  A scoring tool is also a communication tool – communicating to firms what is most important to pay attention to because it’s most important to the city.  The scoring tool does not do this at all.  Again, the scoring tool is not aligned to the requirements of the RFP.  Even if the information is included elsewhere in the RFP, if the scoring tool is not adequate and aligned, it is a flawed tool.

A high-quality and effective scoring guide will include an actual matrix, which clearly shows how each point can be attained.  A high-quality scoring tool will also align to whatever it is the city is specifically looking for in its proposals. Without this, the scoring tool is arbitrary and open to the scoring whims of the scorers.  Additionally, the scoring tool will reflect the weight the city places on the items it is scoring.

Now to answer my own question – was the actual RFP itself fair to those who used it to develop their proposals?  I’d have to say no for the following reasons:  (1.) the RFP includes an inadequate scoring tool making it an unreliable tool that is open to bias, (2.) The scoring tool is not aligned with the requirements in the RFP making it not valid, and (3.) The scoring tool does not reflect the weight of items in the RFP further impacting its validity.

I can offer a simple and easy solution.  In the next version of the RFP (or any other RFP), simply create a valid and reliable scoring tool that can fairly be used by the city and firms responding to RFPs.

Western Expansion

Here are a few materials for I.8.b – Distinguish between economic and geographic factors.  Enjoy – and let me know if you recommend any changes to these!

The first activity is what I call a Wall Activity – and when you look at it, you’ll see why.  This is a good introduction to geographic and economic factors and their role in the Western Movement.

Download the fifteen-page document that includes page-sized pictures to use in the activity.

The second activity is a t-chart in which students distinguish between geographic and economic factors that supported the Western Expansion.

Download the activity here.

Deb’s Stem Starters

If you write paper-and-pencil assessments, you’re going to love these sheets that I call Stem Starters.   Stem Starters are ideas of ways to write item stems for test items.  Think of these as handy reference sheets for those times when you are writing item stems – the question part of a test item.

I’ve designed Stem Starters for each of the four core content areas including Language Arts, Mathematics, Science, and Social Studies.  If you teach any other content area, just look at all of the sheets for ideas.

Stem Starters – LANGUAGE ARTS

Stem Starters – MATHEMATICS

Stem Starters – SCIENCE

Stem Starters – SOCIAL STUDIES

By the way, these are also good for asking oral questions in the classroom.  Download these NOW and use them often.  I’m pleased to share these with you.

Unpacking Standards – Measuring Length

I’ve been working with a few different ways to unpack standards so that they are easily accessbile to teachers.  This is a grade-four standard for measurement.  Actually, this is PART of the standard for measurement.  For this part of the “test”, I’ve focused on just the measurement of length in U.S. customary units.  (The part of the standard dealing with the measurement of length in the metric system will be a different post.)

Download a copy of the entire document, which includes links to other documents I’ve created or found for this part of the standard.

Here’s what you’ll find:  (1) The first “cut” of unpacking the measurement standard, (2) a link to the content card I designed for this part of the standard, (3) a link to analysis questions for the content card, (4) links to downloadable rulers, (5) links to sample released test items for parts of the standard where they exist, (6) a link to a foldable for that can be started with this part of the standard, and (7) a link to a record-keeping sheet entitled, Estimate Then Measure, that I created for one of the objectives in the unpacked standard.

You’ll find my unpacking results (which I’ll continue to tweak), a set of beginning ideas for instructional strategies and assessments, and a vocabulary list with definitions/descriptions.  When you look at the vocabulary list you’ll notice that I’ve included the core terms for the whole standard – and not just what is represented in this part of the unpacked standard.  So while this document deals with U.S. customary units, I’ve included vocabulary for the metric unit as well.  My reasoning for doing this is to provide the core vocabulary in one place.

So I’ve got the idea started.  What would you suggest to improve this?  Anything goes!

Content Card – Measuring Length

If you’ve ever worked with me, you know the importance I place on the content that students learn.  Of course we want students to think deeply, but we want them to think deeply about something – the content we want them to learn.  Here’s another example of a content card – this one for fourth-grade mathematics.  This one is for measuring length  in U.S. customary units.  A content card for measuring length in metric units will be next.  Don’t forget to subscribe to my blog and you’ll automatically be notified when I make any posts.

 Download a pdf copy of the content card.

I’ve also included a list of analysis questions for this content card.  Remember that the purpose of the analysis questions is to help students learn to work with the information on the card.  There are two pages to the following download.

Download a pdf of the Analysis Questions for the Content Card, Measuring Length.