HomeMy WebLinkAboutRESPONSE - RFP - 8433 COMMERCIAL & RESIDENTIAL ENERGY PROGRAMS THIRD PARTY CONSULTANTPrepared by
3934 NE Martin Luther King Jr. Blvd., Suite 300 | Portland, Oregon 97212
www.researchintoaction.com
PROPOSAL
City of Fort Collins
Commercial & Residential Energy Programs Third Party Consultant
RFP No. 8433
January 12, 2017, 3:00 pm Mountain Time
Proposal
Commercial & Residential Energy
Programs Third Party Consultant
RFP# 8433
January 12, 2017
Requested By:
Submitted By:
Research Into Action, Inc.
Mesa Point Energy
Apex Analytics, LLC
www.researchintoaction.com
PO Box 12312
Portland, OR 97212
Phone: 503.287.9136
Fax: 503.281.7375
Contact:
Jane S. Peters, President
Jane.Peters@researchintoaction.com
Research Into Action, Inc.
PO Box 12312
Portland, OR 97212
www.researchintoaction.com
503 287 9136 (main)
503 281 7375 (fax)
888 492 9100 (toll free)
Page i
January 12, 2017
Brian Tholl, Energy Services Supervisor
Fort Collings Utilities
222 Laporte Avenue
Fort Collins, CO 80521
btholl@fcgov.com
Dear Mr. Tholl:
Research Into Action, Inc., in association with our subcontractors, Mesa Point Energy, and Apex Analytics
LLC, is pleased to submit this response to City of Fort Collins Utilities (Utilities) request for proposal (RFP)
for Commercial and Residential Energy Programs Third Party Consultant, RFP #8433.
We are excited by the prospect of doing this interesting work for a client that share our passion for
energy efficiency and resource conservation, as well as best-in-class performance. The executives of our
three small teaming firms commit themselves and the resources of their companies to collaborating
with Utilities and its program implementation partners to further Utilities’ objective to lead the City and
its people in stewarding energy and water resources.
The Research Into Action team specializes in the proposed services and is prepared to provide
excellence in impact and process evaluation to meet Utilities’ needs. In addition to the thought
leadership in the arenas of impact and process evaluation methods and protocols for which our teaming
firms are known, we have the managerial skills to meet Utilities’ aggressive timeline and deliver on
budget.
We are pleased to offer Utilities a local presence, as our team members are located within commuting
distance of Fort Collins.
We acknowledge receipt of addendums 1, 2, and 3. We are prepared to sign prior to commencing
services the City’s standard Agreement without revision.
I am authorized to commit to the proposed work scope, budget and rates. Our proposed time-and-
materials not-to-exceed budget of $349,796 contained in this proposal is valid for 90 days. Please
contact me at the phone number listed above or at jane.peters@researchintoaction.com if you have any
questions.
We believe we offer an exceptional team with the skills, experience, commitment and passion to be the
collaborative evaluation partner Utilities seeks. We look forward to hearing from you.
Sincerely,
Jane S. Peters, Ph.D.
President and Owner of Research Into Action, Inc.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Table of Contents | Page I
TABLE OF CONTENTS
1. Introduction ................................................................................................................... 1
1.1. Team ............................................................................................................................................... 1
1.2. Proposed Approach ........................................................................................................................ 2
2. Tasks ............................................................................................................................... 4
2.1. Task 1: Project Initiation ................................................................................................................. 4
2.2. Task 2: Evaluation Plan ................................................................................................................... 4
2.3. Task 3: Impact Evaluation .............................................................................................................. 6
2.4. Task 4: Process Evaluation ........................................................................................................... 18
2.5. Task 5: Project Management and Reporting ................................................................................ 21
3. Company Description ................................................................................................... 24
3.1. Research Into Action .................................................................................................................... 24
3.2. Mesa Point Energy ....................................................................................................................... 24
3.3. Apex Analytics .............................................................................................................................. 24
4. Company Experience ................................................................................................... 25
4.1. Research Into Action .................................................................................................................... 25
4.2. Mesa Point Energy ....................................................................................................................... 28
4.3. Apex Analytics .............................................................................................................................. 30
5. Staff Qualifications and Project Management............................................................. 32
5.1. Staff Qualifications and Experience ............................................................................................. 32
5.2. Project Management .................................................................................................................... 36
6. Proposed Schedule of Deliverables ............................................................................. 37
7. Pricing ........................................................................................................................... 38
8. References .................................................................................................................... 40
9. Sustainability/TBL Methodology .................................................................................. 41
10. Proof of General Liability Insurance Coverage ............................................................ 43
Appendices ........................................................................................................................ 44
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Table of Contents | Page II
Appendix A. Background on Evaluation Terms............................................................... A-1
Appendix B. Methodological Detail for Free Ridership Estimation ................................ B-1
Appendix C. Resumes of Key Personnel ......................................................................... C-1
C.1. Research Into Action, Inc............................................................................................................. C-1
C.2. Mesa Point Energy ...................................................................................................................... C-1
C.3. Apex Analytics, Inc....................................................................................................................... C-1
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Introduction | Page 1
1. INTRODUCTION
Research Into Action, Inc. and its project partners Mesa Point Energy and Apex Analytics, LLC are please
to submit this proposal to the City of Fort Collins Utilities (Utilities) to serve as the third party consultant
tasked with conducting a comprehensive evaluation of Utilities’ energy programs. We are excited by the
prospect of doing this interesting work for a client that shares our passion for energy efficiency and
resource conservation, as well as best-in-class performance.
The executives of our three teaming firms commit themselves and the resources of their companies to
collaborating with Utilities and its program implementation partners and fully meeting and exceeding
their expectations. We understand that Utilities is under no regulatory requirement to conduct this
evaluation. Rather, Utilities seeks to demonstrate to the people of the City of Fort Collins its prudent use
of ratepayer money and suggested actions it might take to further improve program efficiency. Further,
Utilities seeks to continually improve its program offerings so that it might lead the City and the people
in stewarding energy and water resources.
With these objectives in mind, we present in this document our proposal to conduct a comprehensive
evaluation of the Utilities programs identified in RFP 8433.
1.1. Team
Research Into Action will lead the evaluation team, collaborating with Utilities and its program
implementation partners and assuring the project meets Utilities’ needs, highest evaluation standards,
reporting excellence, and strong project management, delivering on time and budget. We will conduct
all process evaluation activities, as well as generate estimates of free ridership, spillover, and leakage to
generate net savings estimates.
As a leading research firm in the clean energy and sustainable resource sector, Research Into Action
offers Utilities tailored information and insights in real time to advance its energy and water programs.
We help our clients:
Understand what motivates or hinders customer engagement and thus bring new customers to
the programs and services.
Identify opportunities to make programs more efficient in design and delivery.
Clarify what messages best align with customer needs and characteristics.
Identify how to integrate the programs within existing and evolving markets to facilitate more
efficiency investments.
We bring fresh, creative perspectives, grounded in a rigorous, scientific approach, as well as a deep
understanding of the energy users, hard-earned from decades of first-hand experience.
Research Into Action offers Utilities the following advantages:
Leader in process evaluation and market research, having contributed to the seminal research in
our field and conducted numerous trainings. Our founder, Jane Peters, received the Lifetime
Achievement Award from the International Energy Program Evaluation Conference (IEPEC) in
recognition of her process evaluation leadership.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Introduction | Page 2
Developer, with Energy Trust of Oregon, of a simplified yet rigorous self-report method to
estimate free ridership that we have successfully used in jurisdictions in Oregon, Washington,
Idaho, Montana, Missouri, Pennsylvania, and Ontario.
Led or played key role in energy program portfolio evaluations in Washington, Idaho, Montana,
Missouri, Iowa, Pennsylvania, and Ontario – most frequently through multiple contracts with
each of these clients.
Evaluation experience relevant to each of Utilities’ programs.
Proven working relationship with our team members Mesa Point Energy and Apex Analytics.
Mesa Point Energy will assess the impacts of the commercial programs, offering Utilities the following
advantages:
Strong technical understanding of energy baselining and savings calculations based on both
physical and empirical methods
Working knowledge of buildings energy using systems, including commissioning and conducting
field inspections
Local proximity and knowledge of local energy market providers, building practices, and overall
DSM climate
Apex Analytics will assess the impacts of the residential programs as well as conduct the cost-
effectiveness analyses with inputs from Research Into Action and Mesa Point Energy. Apex Analytics
offers Utilities the following advantages:
Leading role in developing residential evaluation protocols for lighting, appliance recycling, and
Home Energy Reports (behavioral programs), as published in the Department of Energy
Universal Methods Project (UMP) and various statewide Technical Reference Manuals (TRMs)
Extensive experience working with Colorado gas and electric utilities
Dual role as both advisor and evaluator has allowed Apex to remain at the forefront of impact
research
1.2. Proposed Approach
We propose a simple five-task structure for the project, where initiation and evaluation planning occur
in Tasks 1 and 2, the impact and process evaluations occur in Tasks 3 and 4, and management and
reporting constitute Task 5.
The Research Into Action team recognizes the balancing act necessary to deliver high-quality insightful
research within Utilities’ budgetary and time constraints. Our approach will mirror similar research that
our team members have performed and overseen in multiple jurisdictions, ensuring the evaluation
prioritizes research efforts that are most critical to Utilities.
Both our residential and commercial impact research will follow the same evaluation best-practices to
help identify key areas for program improvement and revisions to portfolio savings assumptions. Our
impact findings will be augmented with insights drawn from our careful process assessment to develop
useful and actionable recommendations.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Introduction | Page 3
Our research intends not simply to report ex post verified savings, but to provide the context behind any
adjustments, as well as an understanding of program processes and market response, to help support
Utilities’ efforts to provide their ratepayers with comprehensive and valuable energy savings programs
for the future.
For the residential impact research, the Apex team will perform a database verification, ex ante savings
and calibrated engineering review for ex post savings verification, coupled with participant surveys to
assess installation and persistence and improve the calibrated engineering models to reflect actual
demographic and household characteristics of program participants. We also propose optional billing
analyses for the HER and Peak Programs.
Mesa Point Energy will assess the business program impacts, specifically, those of Efficiency Works -
Business Audits, Business Tune Up, Business Rebates, and the Integrated Design Assistance Program.
Leveraging extensive experience in the evaluation, design and implementation of business programs,
the Mesa Point team is ideally suited to provide a comprehensive, yet cost effective evaluation.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 4
2. TASKS
2.1. Task 1: Project Initiation
Key members of our team will meet with the evaluation manager and program staff in Utilities’ offices
to initiate the project and gather the information we need to develop the evaluation plan that will
govern our activities.1 The kick-off meeting, which we envision will comprise an initial overview meeting
followed by interviews with program staff from Utilities, PRPA, and implementation contractor
managers, will last the better part of two days. The meeting with serve to:
Initiate and foster a strong collaboration between our team and Utilities’ project manager and
program staff (including those of Utilities’ program implementation partners), to ensure that our
consulting engagement and final reporting best meets Utilities’ needs,
Jump-start our evaluation plan development based on discussion of Utilities’ evaluation needs,
our proposed approach, and any modifications to our proposed approach or optional tasks that
Utilities would like us to pursue,
Enable us to develop a working understanding of Utilities’ energy programs, implementation
approaches, staff and implementation contractor roles, and so on to support our development
of the evaluation plan, data collection instruments, and sampling plan.
Provide us with program planning and procedural documents and participation databases, and
enable us to explore the methodological underpinnings of Utilities’ program offerings.
We will work with the evaluation manager prior to the meeting to identify a schedule to govern the kick-
off and to invite attendees. We will interview both teams and individuals, as appropriate to program and
topic. For example, we anticipate one or more interviews will address Utilities’ program planning and
cost-effectiveness activities, while other interviews will focus on the programs to be evaluated.
We will summarize in a memo implications for the evaluation plan,
2.2. Task 2: Evaluation Plan
Our team can begin work at contract award and will quickly mobilize to produce an evaluation plan by
the beginning of February, with the evaluation targeted for completion in June.
Our evaluation plan will describe our impact and process evaluation activities and present our final
sampling plan based on a prioritization of programs for evaluation. Tasks 3 and 4, below, describe our
proposed methods. Here, we discuss our proposed sampling plan.
We allocate the sample based on value-of-information principles, which state that more sample should
be allocated to those programs that have the greatest likelihood of affecting the portfolio realization
rate (gross savings expressed as a percentage of reported savings). Our sampling plan prioritizes
programs whose savings comprise a relatively larger proportion of portfolio savings and have relatively
higher degrees of uncertainty associated with their assumed savings (the ex ante estimates), as these
programs have the greatest likelihood of causing evaluated portfolio savings to differ from reported
1
The Mesa Point Energy Principal and Apex Analytics Lead Analyst both live within commuting distance of Utilities.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 5
portfolio savings. We will also take into the Utilities’ expectations for future program growth and
whether any special features of a program require additional evaluation effort.
Figure 2-1 shows the program level contribution to the 2015 Programs presented in the RFP. Note that
Home Energy Reports program comprises nearly half of portfolio first-year annual energy savings, while
the Business Rebate programs make up the next largest portion at over a quarter of the savings.
Although all programs are important, because of the large contribution of these programs to portfolio
achievements, high impact programs will potentially receive a higher level of attention during the
evaluation.
Figure 2-1: 2015 Programs as a Proportion of Portfolio Megawatt Hour Savings Per the RFP
A well-designed sampling approach is critical to ensuring surveyed participants are representative of the
larger participant pool across entire programs. We plan to use simple random sampling approaches for
much of our data collection activities, though discussion with Utilities may result in more advanced
stratified sampling for some of the programs. We will Identify and characterize the population of
interest for each program, and then determine the sample stratification approach (either simple random
sample or stratified). We will consider participation levels, measure types, energy savings, and
household types as it designs sample frames for the data collection task.
We agree with Utilities’ suggestion that 90/10 confidence/precision targets are appropriate for
validating impacts. The proposed sampling approach will provide robust measure verification while
ensuring site visits are used only for the highest impact projects. Installation verification and energy
savings validation will include a first wave of recruitment for web-based surveys, followed by telephone-
based surveys should the web surveys return lower-than-anticipated participation to achieve the
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 6
targeted sample sizes. Each program sample will also receive a more detailed engineering desk review,
with some of the larger commercial projects selected for site visits.
We will work with Utilities to refine our proposed sample sizes, which we show in Table 2-1.
Table 2-1: Program Sample Frame for All Programs to Achieve 90/10 Confidence and Precision
PROGRAM TOTAL 2014-2016
PROJECT COUNT
SURVEY
SAMPLE
SIZE
DETAILED FILE REVIEW
AND SITE VISIT SAMPLE
SIZE
Appliance Rebates 4,281 70
Appliance Recycling 1,607 70
Peak Partners 4,848 70
Home Audits 1,716
70*
Home Direct Install 1,304
Home Rebates 942
Help Loan Program 74 35
Home Energy Reports 154,211 70
Larimer County Conservation Corps 1,353 70
Midstream Retail Lighting 153,314 4-6**
Renewable Energy - Solar 251 53
Business Audits 654
70***
0
Business Rebates 1,137 64
Business Tune Up 12 10 5
Integrated Design Assistance 10 8 4
Total 1,813 670-672 73
* Combined sample for Audits, Direct Install, and Help.
** Note the 4-6 sample size for the lighting program reflects retailer interviews.
*** Most of the samples for the business audit program will be the same as those drawn from the Business rebate
program, thus the total sample size for audits and rebates, combined is approximately 70.
2.3. Task 3: Impact Evaluation
2.3.1. Savings Estimation
The impact evaluation will verify the savings resulting at the measure, program and portfolio level as
well as verifying savings by market sector. The work will be done in a structured manner as described
later in this section.
As programs proceed from planning phases, to implementation, to evaluation, the recorded savings may
be updated at different points in time, and may change for various reasons. In conducting the
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 7
evaluation, we will proceed by analyzing and reporting savings from four perspectives (or points in
time), as follows. (The reader for whom these concepts are unfamiliar will find them discussed in
Appendix A.)
Figure 2-2: Basic Components of an Impact Evaluation
Two primary and complementary goals of the impact are to (1) verify and adjust savings values, and (2)
to identify program improvements for improved performance and so that both realization rates and
NTG values are driven as close as possible to unity.
Impact evaluation results will be reported comparing reported and verified gross and net savings. Key
parameters that will be reported include, but are not limited to, NTG, realization rates, impact of
measures and programs, and discussion of results. (See Section 2.5 for an illustrative “Waterfall” graphic
that we intend to use to present impact findings.)
Suggestions for improvement of program tracking, calculation, verification and other aspects of program
design and operation will be presented and summarized. We firmly believe providing Utilities with key
insights as to the drivers behind savings adjustments is as critical a step as reporting the actual impacts.
2.3.2. Cost Effectiveness Assessment
An important element of the evaluation that is requested in the RFP is the calculation of program and
portfolio cost effectiveness. Cost effectiveness measures the value of the DSM programs as compared to
the alternative of doing nothing. A program is deemed Cost effective if benefits outweigh the costs, or,
put another way, when Benefit / Cost ratios exceed 1.
The RFP calls for analysis of three cost effectiveness tests, namely the Total Resource Cost (TRC) Test,
Utility Cost Test (UCT), and Participant Cost Test (PCT).2 We will calculate each of these cost tests using
the DSMore cost effectiveness analysis tool. We will complete these tests at the program and portfolio
level. Similar to the calculation of energy savings, our analysis takes care to use reliable estimates of
both costs (including incremental installation costs, audit costs, and other overhead and program
delivery cost) and benefits (avoided cost of energy and demand, and any other benefits).
When assessing program value, we recommend adopting the modified TRC, which has been used by the
Colorado gas utilities (Black Hills, Atmos Energy, Colorado Natural Gas) and Xcel Energy. The modified
2
In the TRC Test, all the actual costs and benefits are included, regardless of whether they are the participants or the
administrator’s (Utilities’) responsibility. The UCT measures program effectiveness accounting only for the administrator’s
costs and benefits; it excludes the cost to the participant. In the PCT, only the participant’s cost to implement DSM
measures is considered.
Savings
Targets
Gross
Savings,
Reported
Gross
Savings,
Evaluated
Net
Savings,
Evaluated
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 8
TRC includes a 5% benefits “adder” to account for non-energy benefits not directly quantified by the
energy savings alone.
2.3.3. Impact Evaluation Overview
The impact evaluation will follow industry best practices adapted to meet the evaluation needs of the
residential and commercial program portfolios. The primary purposes of an impact analysis is to assess
gross and net energy savings and the cost-effectiveness of the installed measures. Table 2-2 describes
the key research questions and principal evaluation activities associated with these purposes.
Table 2-2: Key Impact Research Issues
PROGRAM PLANNING AND
TRACKING
GROSS AND NET ENERGY SAVINGS COST EFFECTIVENESS
Key Research Issues
Are the programs collecting the
necessary information to support
impact estimation?
Are the savings assumptions
reasonable and accurate?
Are there more customized
algorithms that could be used to
determine savings?
Were gross program-reported
impacts achieved?
What factors resulted in any
discrepancies between reported
and evaluated savings?
What net impacts are
attributable to each program?
Is the portfolio cost-effective?
Are the programs cost-effective?
What are the utility and
incremental costs and expected
useful life of each measure and
program?
Principal Evaluation Activities
Program files and databases
reviewed for completeness and
accuracy
Measure savings assumptions
compared to relevant industry
sources
Savings algorithms compared to
engineering best practices
Opportunities identified for
enhanced planning and tracking
Measures verified to be installed
and working for Utilities
customers
Assumptions verified as
appropriate to customer
conditions (specific customers
for large projects; customer class
for smaller ones)
Savings calculations verified to
be accurate
Free ridership, spillover, and
leakage estimated from careful
analysis of customer self-reports
Measure costs verified
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 9
Table 2-3: Portfolio Impact Evaluation Research Tasks by Program
PROGRAM
DATABASE REVIEW
PROJECT FILE REVIEW
EX ANTE SAVINGS
REVIEW
ON-SITE SAVINGS
VERIFICATION
CALIBRATED
ENGINEERING
ESTIMATES
BILLING ANALYSIS
PARTICIPANT
SURVEYS*
EX POST SAVINGS
CALCULATION
NET SAVINGS
ANALYSIS
COST-EFFECTIVENESS
Appliance Rebates
Appliance Recycling
Peak Partners ●
Home Audits
Home Direct Install
Home Rebates
Help Loan Program
Home Energy Reports ●
Larimer County
Conservation Corps
Midstream Retail
Lighting
●
Renewable Energy –
Solar
Business Audits
Business Rebates
Business Tune Up
Integrated Design
Assistance
* Telephone surveys will be leveraged to collect installation verification, home/demographic characteristics, NTG, and
general process topics. signifies core research, while ● signifies optional research task.
2.3.4. Impact Research Tasks Common Across Programs
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 10
Figure 2-3: Impact Evaluation Workflow
The following sections describe the approach to each of the impact tasks.
Database Review
The impact evaluation of all programs begins with a review of all databases used to track measure
installation activities. The review will determine if the program databases are accurately capturing the
key metrics for each project necessary to document savings for each qualifying measure. It will also
verify measure installation rates and persistence and assumed (ex ante) savings. We will summarize the
key metrics regarding program operations, including number of participants by measure type, regional
average measure costs, and estimated savings impacts. Our review will include an assessment of current
tracking methods and suggest to Utilities’ opportunities for improvements.
Specific information that we will check includes customer and vendor information, measure type,
building/facility type, project forms, project timeline, and reported savings. We will review cost and
lifetime information and compare with reputable industry sources to help ensure that cost effectiveness
calculations are reliable.
Sample groups will be selected during the database review for each program as required with the goal
of achieving 90/10 confidence and precision in energy savings. The sample groups will be used for the
various file review, calibration and on-site verification activities.
File Review
The file review conducted is dependent upon the program type.
For residential programs, the file review will be tailored to those participants that were identified as part
of the statistical sampled with focus on the application documents and the details provided within them
as appropriate to each program. In some cases, the entire project level documentation may be included
in the database (copied over from the application), while in other programs additional application files
and documentation will be required for review. We will work with Utilities during project initiation to
understand document availability.
For the commercial programs, it is anticipated that project specific documentation such as application
forms, audit reports, project verification forms and so on will be available. We will select a sample of
projects for detailed desk review. We will check the files for consistency with the database entries, and
will review savings and database backup information for accuracy and appropriateness.
Database review
•Comprenhiveness
•Accuracy of input
•Draw samples
File review
•Program Level
Documents
•Project
documents in
Sample
Reported / Ex Ante
Savings Review
•Analysis of savings
algorithms and
assumptions
•Calibration and
comparison to
best practce
analysis
Progream Specific
Analysis
•Onsite verification
•Calibrated
engineering
estimates
•Billing analysis
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 11
Reported / Ex Ante Savings Review
We will review the sources, algorithms, assumptions and ex ante savings values for accuracy and
appropriateness. If we find opportunities to improve the assumptions or methods used by Utilities, we
will suggest alternate methods or assumptions and apply the alternate calculations to the savings to
calculate the gross savings estimate.
We will also compare the existing ex ante savings assumptions to those currently used by Xcel Energy if
sufficient data from Xcel are available. Our team has also developed for most of the programs in
Utilities’ portfolio algorithms in use in other jurisdictions, including Arkansas, Pennsylvania, and
Missouri. We will check all available algorithms/assumptions against these regions as well and make
recommendations for adjustments where necessary.
As relevant, we will assess the appropriateness and accuracy of inputs through research of the best
available information. We anticipate consulting these secondary sources:
Evaluations from neighboring utilities, such as Xcel Energy (Colorado),
Weather Station and Climate Zone databases,
Regional Technical Reference Manual (TRM) assumptions (including Xcel Energy, and possibly
the Illinois TRM as it is the most comprehensive and applicable to Colorado, and members of
our team serve on the IL TRM technical advisory committee), and
ENERGY STAR savings assumptions.
While the ex ante review will help to validate and potentially improve upon the savings assumptions
currently employed by Utilities, an additional step to enhancing the savings protocols will be to calibrate
the savings algorithms. This calibrated engineering approach is reviewed in greater detail below.
Onsite Verification
Because of the nature of many of the programs in the residential sector, onsite inspection of residential
programs will not be conducted.
For the commercial programs, we will conduct onsite inspections of a census or sample of the desk
reviewed commercial projects. The onsite inspections will verify the measure was installed and is being
operated as represented in the project database and files, as well as collecting and verifying facility
characteristics.
Calibrated Engineering Estimates
Residential and commercial measures will be calibrated to actual characteristics of participants, where
available and when applicable to the program. Where we collect information such as demographic or
household characteristics on actual participants in a program we will use that in the algorithms.
Otherwise, we rely on data that Ft. Collins provides, or use secondary data sources such as the American
Community Survey, RECS, CBECs, etc. As an example, we can use appliance capacity and configuration
data to update parameters used for existing unit energy consumption (UEC) regression models to derive
Utilities’ specific savings for the appliance recycling program.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 12
Billing Analysis
The Home Energy Report and Peak Partners program impact evaluations may include a billing analysis to
estimate energy consumption and savings values using actual customer billing data. This is the most
reliable and robust method to use if adequate billing data are available. Note, however, that we have
proposed these as optional tasks, as discussed in greater detail below.
Participant Surveys
Participant surveys are an ideal source for data that is unique to personal experience and preferences;
as we describe in Task 4, our process evaluation will conduct surveys for all programs except HER. We
are cautious in our use of surveys in support of impact analysis, as participants may not accurately
report the answers to highly detailed questions; Table 2-4 identifies the programs for which our
participant surveys will include impact-related questions. Unreliable data from participant surveys can
result in skewed inputs or inaccurate estimates. For example, participant responses can be reliable
regarding whether their home has any LEDs installed, however, responses are less reliable when asking
about the exact number and location of installed LEDs. We will include in the participant surveys (fielded
by the process team) questions to provide the following inputs to the impact assessment:
Installation verification and measure persistence
Housing type
Household characteristics (e.g., square footage, number of bathrooms, space and water heating
fuels, cooling equipment)
Demographic information (e.g., number of household members, age of household members)
Behavioral changes
Additional measure savings algorithm inputs not available in the tracking database
Free ridership, spillover, leakage
Combining the tasks outlined above, ex ante savings will be verified and the algorithms will be checked
for appropriateness and accuracy. Ex post savings will be calculated based calibrated engineering
estimates, using the results of the surveys (where available).
2.3.5. Residential Program Impact Evaluations
This section describes the impact evaluation approach on a program-by-program basis for the
residential programs.
Appliance Rebates
This is a simple rebate program and involves only two measures: high efficiency dishwashers and clothes
washers. We will limit our research to reviewing the engineering algorithm for the two measures, and
depending on the availability of measure characteristics, will ensure the correct and current baselines
are being applied to the ex ante savings estimates. We will calculate ex post electric, gas, and water
savings based on the most up-to-date engineering algorithms and input assumptions.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 13
Appliance Recycling
Refrigerator and freezer recycling programs have been offered now for many years, and since 2004, by
Utilities. The gross impact evaluation will focus exclusively on the ex ante savings assumptions and
calibrate the per-unit savings based on the Uniform Methods Project UMP) protocols.3 (It should be
noted that savings associated with refrigerators and freezers that are replaced will only be adjusted in
the net-savings estimates and not gross. Optional participant surveys will allow us to develop part-use
factors specific for Utilities (to account for those recycled units that were either only used seasonally or
not used at all). If surveys are not conducted, we will rely on default assumptions from the UMP, but still
calibrate savings to the actual characteristics of the program units (e.g., unit size, style).
Home Audits, Home Direct Install, and Home Rebates Programs
Audits are frequently the “gateway” program for participants to pursue more substantial home
improvements, including shell measures. We believe combining these three related programs together
for one comprehensive program evaluation. A phone survey has been designated as part of our core
research, and will investigate in-service rates, measure persistence, home characteristics, demographics
and attempt to collect any additional savings parameters to allow a more detailed – and accurate – ex
post calibrated engineering savings estimation.
Home Energy Reports
Utility behavior-based programs are one of most recent additional to energy efficiency programs, with
the most popular version including Home Energy Reports. The Utilities’ provider of Home Energy
Reports, Opower, has adopted the same savings validation as employed by evaluators. This has led to
many recent HER billing results to show ex post savings within one-percent of claimed savings.4
Therefore, we have instead focused the core evaluation task for the HER on examining the claimed
persistence and measure lifetimes. A number of recent studies are finding electric persistence (i.e.,
savings that would be continued to be claimed even if the reports are stopped) in the 80% range.5 This
means that there are multiple ways to optimize program design and claim savings (e.g., using a “crop
rotation” methodology where participants are cycled in and out of receiving reports in different years).
The evaluation, therefore, will review how Utilities is currently claiming savings from the reports and
offer alternative measure lifetimes and annual savings benefits.
If Utilities chooses to conduct a billing analysis for the Home Energy Report program, we will use the
randomized control trial (RCT) experimental design to conduct a “difference in differences” approach,
The HER impact analysis will include a fixed-effects regression analysis of the billing data to estimate the
energy used for each participant and to quantify the impacts of the HER’s on energy consumption, after
taking into the account the effects of weather and other factors. The HER model will explain the
observed energy usage in terms of variables representing recipient status (treatment vs control),
weather, and time-period. Additionally, the model controls for non-programmatic effects thorough use
3
Department of Energy, April, 2013. https://energy.gov/sites/prod/files/2013/11/f5/53827-7.pdf
4
Numerous recent EM&V HER reports demonstrating the high realization rate can be found here:
https://opower.com/resource_type/verification-reports/
5
Illinois TRM, version 5.0, February, 2016. Page 6 of 94.
http://ilsagfiles.org/SAG_files/Technical_Reference_Manual/Version_5/Final/IL-TRM_Effective_060116_v5.0_Vol_4_X-
Cutting_Measures_and_Attach._021116_Final.pdf
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 14
of customer-specific dummy variables. This is a Fixed Effects Model specification, in which the customer-
specific dummy variables account for non-programmatic impacts that cannot be explicitly controlled for
and is not relevant to the estimation of program savings. The specification of customer specific effects
allows the model to capture much of the baseline differences across customers while obtaining reliable
estimates of the impact.
We will use a post-only model with pre-usage controls. Other model specifications have not proven to
provide as high degree of precision. The model specification will use one year of pre-treatment data to
construct control variables which capture the primary drivers of a household’s energy use. The HER
billing analysis will follow standard evaluation protocols, including factoring in the potential for double
counting of other program savings, participant opt-outs, and equivalency testing between control and
treatment groups.
Peak Partners Program
This program allows Utilities to manage peak demand through Wi-Fi enabled thermostats and cellular
water heater controllers. The water saving devices will be assessed through a calibrated engineering
analysis, while the peak demand savings will be assessed through an optional billing analysis. We have
proposed optional billing analysis, since our understanding is that the focus of the evaluations is on
energy, rather than demand savings.
Should Utilities choose to assess the demand savings from the thermostat portion of the Peak Partners
Program, we would use the AMI data to examine energy use over 15 minute periods as a function of
weather and treatment (i.e., event) vs. control periods. In this way, the hotter days of the summer
where there was no event called serve as the counterfactual to the event periods. We would also use
the run time data from the thermostats, examining the decrease in run-time during the event periods
vs. the control periods. Again, this would be regressed with weather. Combining the regressed energy
use and run-time then allows us to back out the demand savings, since we can divide the energy use by
the run time to get the demand of the units (i.e., dividing kWh by hours provides kW as the quotient).
Home Efficiency Loan Program
This is a residential efficiency financing program, supporting customer installation of larger retrofit
projects. Because this program is not claiming any savings, the evaluation will be strictly process
oriented – and therefore is not covered under this impact section.
Larimer County Conservation Corps
This is an additional residential direct install program, targeting lower-income households (though with
no income requirement). The participant home characteristics may differ for the Larimer County
Conservation Corps (because this program targets lower income population), so we will integrate low
income household and demographics characteristics where available to help calibrate energy and water
savings algorithms. For example, lower-income homes are typically smaller (leading to lower savings)
and more energy intensive (leading to higher potential savings) than non-low income homes.
Midstream Lighting
Up-or midstream lighting programs have been a very effective means of helping increase both the
penetration (presence of at least one bulb) and saturation (the percentage of lighting sockets) of
efficient lighting in households (and to a lesser extent commercial buildings). We will interview up to 4-6
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 15
key representatives from participating retailers. The Apex team is well versed in lighting evaluations,
having authored the Uniform Methods Project lighting chapter and is currently leading lighting
evaluations in five states throughout the U.S.
Solar
The Solar Program offers Utilities ratepayers several opportunities for purchasing solar power: either
through offsetting the cost of installation of their own panels, buying into community solar system, or
supporting larger commercial solar installation through the purchasing of long-term power purchase
agreements. We will review and perform a high-level engineering analysis of the anticipated versus
realized generation from the installed capacity of the Solar Program (since all Solar Program panels are
effectively metered, we assume the MWh generated is actual relative to estimated savings from
deemed measure assumptions for efficiency programs).Otherwise, the primary focus for the Solar
Program will be process oriented (gauging satisfaction, realized savings relative to anticipated, areas for
improvement for the program delivery).
2.3.6. Commercial Program Impact Evaluations
This section describes the impact evaluation approach on a program-by-program basis for the
commercial programs.
Efficiency Works – Business Audits
Per the RFP, the business audits program focuses on providing free ASHRAE Level 1 audits, and provides
a gateway to other programs.
We will review a sample of audits to ensure quality and applicability of savings. In addition to reviewing
for general quality, the following attributes will be explicitly reviewed and reported:
Savings uniformity, approach and rigor between providers
Checking savings calculation results to ensure reliability
Methods, algorithms and assumptions as compare to "best practice" or industry standard
methods and assumptions
To the extent the audit savings directly feeds into the other Efficiency Works program, will also compare
the savings values to those recorded in the Rebates databases and program materials.
It is anticipated that a significant number of participants in the Audit program also participated in the
Business Programs. As such, the audits reviewed will be taken from the sample drawn for the Business
Rebates program.
The RFP shows that that there is savings associated with Efficiency Works – Business Audits program.
We will investigate the attribution of savings to the audit program and other programs to ensure that
the savings are correct and understood, and not duplication in other programs. The review will focus on
audits of facilities that went on to participate in other part of Efficiency Works, but some audits that did
not result in project savings will also be reviewed.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 16
Efficiency Works – Business Rebates
In terms of energy savings, the Business Rebases program, making up 28% of the portfolio savings, is by
far the largest contributor to, and conduit for, incentive dollars. As such, this program will be the largest
focus of the commercial impact evaluation.
The RFP (Table 2, pg. 9) indicates that there were 378 rebates completed in 2015, assuming that all
three years, 2014, 2015 and 2016 had similar participation levels, we assume that there were some
1134 total projects.
To manage costs, a sample of projects will be reviewed in detail. The sample will be drawn to achieve a
90/10 level of reliability based on energy savings (such as kWh, therms, or gallons of water as
appropriate). Assuming a coefficient of variation of 0.5 and that savings is normally distributed the initial
sample size is estimated to be 64 projects. As we learn more about the population characteristics, such
as size and stratification of savings, the sample group will be adjusted to ensure statistical criteria are
met in an efficient and effective way.
Detailed desk reviews and site visits will be conducted at the sampled participant facilities that
participated in the prescriptive program and in the custom program. Specific elements of the review will
include the following:
Reporting – are project savings and other information properly tracked?
Ex Ante savings – Do ex ante savings methods comport with industry practice and are savings
properly calculated?
Program Administrator Verification – What methods are used to provide program quality
assurance during implementation?
Installation verification – Were projects installed, and to what extent are the installed measures
in keeping with installation tracking?
Measure impact analysis – which measures are most common, and what is the relative
contribution to savings of implemented measures by type?
Efficiency Works – Business Tune Up
Th Tune Up program is relatively small, contributing an estimated 2% of program savings. Assuming that
savings is from low-cost measures such as weatherization, equipment repair and refurbishment and
controls, verification will focus on field verification of low-cost measure implementation and operation.
A sample, which may include up to 14 sites, will be drawn and each of the project elements described
above in the Rebates evaluation section will be checked.
The measures in the Tune Up program will be evaluated to be sure that they are appropriate for the
application, persistent, and result in real savings. Like the Audit program, savings, calculations and
reporting will be checked for persistence and reliability.
Integrated Design Assistance Program
The integrated design assistance program provides expert energy design assistance for new commercial
construction. The program, which contributes and estimated 7% of portfolio savings (making it the third-
largest program in terms of savings) will be evaluated covering elements similar to those in the Rebates
program. After a review of the tracking systems and total savings, a sample, which may include up to 12
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 17
of 15 projects will be selected and reviewed. The reviews will include a desk audit, and site inspections
as needed to achieve statistical requirements.
Because the program focuses on new construction, certain special evaluation parameters will be
considered. Specifically, the evaluation will consider:
Review of early, 100% construction document, and as-built phases of the project to ensure
energy savings features were included and to understand how the design assistance fed the
construction project.
Site visits of a sample of projects will be conducted to verify that measures were implemented
as envisioned by the program.
Calculated savings will be reviewed, including the review of computer energy models as needed.
Reporting and analysis will be reviewed for uniformity, approach, rigor and reliability.
Comparisons against prevailing energy code and local standard construction practice will be
made, and such comparisons will include a quantitative aspect focusing on savings realization
rates.
2.3.7. Net Savings Estimation
The issues of free riders and spillover, used to report net impacts, are thorny. Dr. Jane Peters, President
of Research Into Action, was one of the first evaluators to specify a decision-making framework for
estimating free riders.6 Decision-making frameworks since have become the “best practices” approach
to estimating free riders and spillover and are the required approach in some states.
Collaborating with Energy Trust over several years, Research Into Action developed a streamlined
approach to estimating the proportion of free riders while reflecting the complexity of customer
decision making and the multi-faceted influence of efficiency programs. Energy Trust has gone on to use
this method for all of its programs, and Research Into Action has successfully used the approach in
multiple evaluations undertaken for each of the following: Avista Utilities, Ameren Missouri,
NorthWestern Energy, Independent Energy System Operator (IESO, in Ontario), and Consumers Energy.
Our process team (Research Into Action) will develop per-participant estimates of free ridership,
spillover, and leakage for the impact teams to use in estimating program-specific net impacts, weighting
the per-participant estimates with the participant’s share of the sample’s total project savings. For each
program, the program-specific net-to-gross ratio is calculated from the program-specific weighted
average inverse free ridership (FR) score (that is, 1 – FR, weighted by project savings), plus additional
spillover savings, minus savings lost to leakage, where the spillover and leakage terms are expressed as
proportions of program savings.
We develop these estimates using a self-report method through surveys with a statistically valid sample
of participants. The self-report free ridership method asks participants a series of carefully constructed
survey questions to learn what the participants would have done in the absence of the program.
6 Peters, J. S. & Windell, P. “Innovativeness, Predisposition and Freeridership.” Proceedings of the ACEEE Summer Study on
Energy Efficiency in Buildings. Pacific Grove, CA, August 1994. pp. 1:201-1:208.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 18
We assess attribution (the extent to which the program can be attributed with inducing the efficiency
action, the converse of which is free ridership) using a brief instrument that assesses two components of
free ridership: 1) intention to carry out the energy-efficient project without program funds (project
change in the absence of the program); and 2) influence of the program in the decision to carry out the
energy-efficient project (program influence). The project change and program influence scores are
summed for each respondent, resulting in a total free ridership score ranging from 0 to 100. The number
is interpreted as the percentage likelihood that a given respondent is a free rider. Appendix B further
describes our free ridership methodology.
For spillover, we ask participants and nonparticipants about efficiency measures installed and actions
taken outside of Utilities’ programs. For all reported spillover actions, we ask the extent to which
Utilities energy program activities – its rebate programs, advertising, website, call center, audits, et
cetera – have influenced them to undertake the efficiency action outside of the program.
For leakage, we ask participants where the equipment was initially installed and whether it has been
subsequently moved to a new location.
We will pursue free ridership estimation for the midstream retail program through retailer interviews.
Retailers offer important insights into what energy efficient lighting sales would have been without the
instant incentives, marketing, in-store signage, and sales associate training. These retailer contacts have
the unique knowledge of both program and non-program lighting sales at their company, and therefore
can estimate the impact of the program on all the entirety of their company LED sales.
We use responses from the following questions to estimate NTG from the retailer interviews:
Total number of energy efficient bulbs sold: Respondents estimate the percentage of total
store-level lighting sales the Midstream Retail Lighting Program sales represented. We combine
that estimate with the program implementations database to extrapolate store-level lighting
sales.
Likelihood to sell efficient bulbs in absence of the discount: We ask respondents about their
likelihood of selling the same number of LEDs if they had not been discounted: “Do you think
your sales of these LEDs would have been the same, lower, or higher without the Utilities Utility
Program?”
Magnitude of impact: For those that indicated sales would have been higher or lower in
absence of the program, we ask “By what percentage would store sales of LED products have
been higher/lower without the program?”
2.3.8. Impact Evaluation Reporting
Impact estimates will be rolled up from the measure, to the program and then to the portfolio levels.
Our reporting will ensure that the portfolio and program impacts are clearly documented and explained.
In additional, the report will delineate suggested program refinements to help with program continuous
improvement. More details on the reporting is presented in the Task 5 Description.
2.4. Task 4: Process Evaluation
The process evaluation, like the impact evaluation, will follow industry best practices. The primary
purposes of the process investigation are to assess whether the programs are meeting their objectives
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 19
and market needs and to identify opportunities to enhance program design or operations to increase
effectiveness and efficiency (cost-effectiveness). Table 2-4 describes the key research questions and
principal evaluation activities associated with these purposes.
Table 2-4: Key Process Research Issues
PROGRAM OBJECTIVES AND
DESIGN
PROGRAM OPERATIONS MARKETPLACE INTERACTIONS
Key Research Issues
What are the program
objectives? What are key market
barriers (demand and supply) the
program seeks to address?
How effective is the program in
addressing market barriers?
What barriers are limiting
program participation?
How well are Utilities’ programs
achieving their goals and
objectives?
What are opportunities to better
align program activities and
objectives?
What evidence suggests specific
programs should be continued
or discontinued?
What program implementation
activities does Utilities engage
in? What are their procedures
and operations?
What activities, procedures, and
operations do the
implementation contractors
engage in?
What are opportunities to
improve operational efficiency
and effectiveness?
How do customers and trade
allies learn about the programs?
What motivates them to
participate? Why do they
hesitate?
What steps and activities do
participants and trade allies
engage in? Are these interactions
smooth?
Are the programs meeting their
expectations and needs?
What are opportunities to better
serve customers and to advance
sustainable energy?
Principal Evaluation Activities
Interviews with Utilities and PRPA
program staff
Review of program plans, logic
models
Review of any planning studies,
market feedback, etc.
Review of innovative program
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 20
2.4.1. Interviews and Surveys
The interviews we conduct during Task 1 will contribute to our process evaluation investigation. As
necessary to develop a full understanding of Utilities’ energy programs and operations, we will conduct
up to six additional hour-long interviews with program staff – including those at Utilities, PRPA, and the
implementation contractors. We anticipate reaching out to staff over the course of the evaluation as
needed to ensure an accurate and full understanding of the programs.
Research Into Action staff will conduct the residential and commercial participant surveys following the
sampling plan presented in Table 2-1. The surveys will investigate the process issues of importance to
Utilities, as summarized in Table 2-4 above. The surveys will collect information requested by the impact
teams to support the estimation of gross savings. Finally, the surveys will include the self-report battery
of questions that Research Into Action has developed and honed in numerous projects to collect the
information necessary to estimate net impacts (free ridership, spillover, and leakage components).
Our residential and commercial nonparticipant surveys will obtain information on their awareness,
knowledge, motivations, barriers, and experiences relevant to Utilities’ research objectives. The surveys
will also explore program spillover and purchase of instant-incentive lighting products offered through
the Midstream Retail Lighting Program.
We anticipate that Utilities has email addresses for most of its program participants. In our experience,
many contacts appreciate the ease of replying to surveys via the web. We also experience a lower web
response rate than phone response rate. After issuing the survey request via email and nudging non-
respondents four times, we will then call non-respondents with a request to complete the survey by
phone as necessary to obtain our sample quotas.
2.4.2. Quality Assurance for Data Collection and Analysis
Research Into Action has developed and employ strong quality assurance procedures for our process
investigations. We use the research questions to identify the target research issues that drive the specific
questions used in interviews and surveys. We formulate questions capable of yielding reliable
information and insights.
All our consulting staff have masters or Ph.D. degrees in the social sciences, and we use only
experienced interviewers to conduct in-depth interviews. Our interviewers are project staff that are
involved from the beginning to the end of the project. They are familiar with the work plan, research
objectives, and all project activities. They conduct interviews, analyze, and write report sections. With
permission, we audio record our interviews and use the recording to ensure the accuracy and
completeness of our notes. With permission, we re-contact interviewees when we need to clarify or
amplify a point discussed.
We have an internal instrument review process. We have designated group of staff on a rotating basis
review each data collection instrument relative to a checklist of criteria regarding such things as
question wording, skip patterns, instructions, flow, and clarity. We will conduct this internal review of
instruments prior to sending them for Utilities’ review. We pretest the CATI-programmed surveys and
initiate web surveys with a soft launch, sending invitations to and following up with a small number of
contacts to ensure the web link and programmed survey are working as expected.
We will use the research questions again to guide the analysis and reporting to clearly address Utilities’
objectives. Research Into Action employs best practices in our research and analysis. We analyze in-
depth interviews, as well as program documentation, using NVivo qualitative analysis software. The
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 21
project manager will begin by developing codes based on our analytical framework and research issues.
When all sources have been initially coded, we begin detailed analyses by topic in preparation of report
writing. Prior to analyzing survey data, we follow protocols we have established to appropriately clean
the data. Our data cleaning steps include, assessing data completeness, checking variable definitions,
and identifying and handling outliers.
Prior to initiating report writing, we discuss findings as a team; the project manager will determine the
best organization for the material. Our initial drafts include marginal comments identifying sources, and
we check the raw data when we need to better understand or corroborate the finding as initially
drafted. After writing our findings, we meet as a team to generate conclusions and recommendations.
The project manager and principal-in-charge, with input from the team, then draft the conclusions and
recommendations, which are reviewed by and discussed with the president.
2.4.3. Best Practices
The National Energy Efficiency Best Practices Study (www.eebestpractices.com) is an excellent
compendium of best practices for energy programs. We used this resource to identify program strengths
and opportunities for improvement in the portfolio evaluation we conducted for NorthWestern Energy.
We will consult this study and others in our assessment of Utilities’ program designs and
implementation approaches.
We anticipate the Utilities is also interested in identifying innovations in program design and delivery.
Research Into Action and others have contributed to this field by advising our clients in ways to apply
lessons from behavioral economics and neuropsychology to their efficiency programs, such as including
prompts and triggers, opt-out defaults, anchor points, carefully constructed choice architecture and
other techniques.
We will consult a variety of sources to investigate program innovations, including ACEEE, IEPEC, US EPA,
and US DOE (including the betterbuildingssolutioncenter.energy.gov).
We also propose identify innovative approaches through an investigation of the efficiency portfolios of
three program administrators of Utilities choosing. For example, SMUD and Austin Energy are
recognized leaders in efficiency and, as municipal utilities, might constitute point of comparison for
Utilities.
Our best practices and program innovation research will aim to identify Utilities’ program strengths,
opportunities for improvements in effectiveness and efficiency, and innovative programs or program
elements that Utilities may want to consider adopting.
2.5. Task 5: Project Management and Reporting
Two words describe our approach to project management and reporting: No surprises!
Bi-Weekly Memo Status Reporting: Our Evaluation Plan will include a schedule of milestones and
deliverables. At the middle and end of each month, we will provide Utilities with a memo that describes
activities undertaken in the period and planned for the next period, as well as any support needed from
Utilities. The memo will include an updated milestones schedule showing dates of completed
accomplishments and any needed revisions to planning dates. The memo will provide a summary of key
findings obtained during the period.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 22
Monthly Project Management Meetings: Monthly throughout the project duration, we will lead a
project management meeting (by phone or webinar) attended by the key staff from our team. We will
discuss our progress and findings to date, answer Utilities’ questions, and seek Utilities’ feedback and
direction. In a memo that accompanies our invoice, we will detail our accomplishments, plans, and
budget expended and remaining.
Preview of Findings and Conclusions: As our analysis is nearing completion and report writing is
initiated, we will meet (by phone or webinar) with Utilities to discuss our key findings and our
interpretations (that is, the conclusions we are drawing). We will seek Utilities’ insights into and
interpretation of the findings to ensure that the recommendations we develop are grounded in a solid
understanding of program operations and primary data.
Evaluation Reporting: We will report on the full evaluation (all programs investigated) in three formats:
Written report – A concise report presenting in graphic, tabular, and narrative form the essence
of the evaluation research, findings, conclusions and recommendations, with details relegated
to appendices (MS Word report; body of report limited to 40 pages)
PowerPoint – A PowerPoint slide deck summarizing the written report
Presentation – A presentation of the PowerPoint at Utilities’ offices, attended by key evaluation
staff
We appreciate Utilities’ desire to present evaluation findings graphically as much as possible. Other
utilities are similarly seeking to portray evaluation findings in a manner most easily understood by a
diverse audience. In a recently released white paper, Pacific Gas & Electric (PG&E) efficiency evaluation
staff suggest evaluators use a “waterfall” graphic to convey impact evaluation findings, as shown in
Figure 2-4. We plan to use informative graphics such as this in our reporting.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Tasks | Page 23
Figure 2-4: Illustrative Waterfall Graphic of Efficiency Savings as Designed by PG&E
Source: Whitepaper: Development of Order-Independent Waterfall Graphics to Enable Comprehensive Understanding
of Impact Evaluation Results, by R. Kasman, A. Scheer, R. Allen, R. Friedmann, and J. Berman, Pacific Gas and Electric
Company, San Francisco, CA, September 12, 2016. Available at http://www.energydataweb.com/cpuc/search.aspx,
search “waterfall”.
Our firms frequently work with confidential data and our staff are trained in and follow procedures for
data security. We will exchange files with personally identifying information through a password
protected SharePoint site. We notify all interview and survey respondents that we will maintain the
confidentiality of their responses to the limit of the law. If requested by Utilities, we will use encryption
for customer data and ask employees to sign nondisclosure agreements or clients. We follow the ethics
and standards of Council of American Survey Research Organizations (CASRO) for survey
implementation and the American Evaluation Association (AEA) for evaluations.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Description | Page 24
3. COMPANY DESCRIPTION
3.1. Research Into Action
Research Into Action, Inc. is a leading provider of program evaluation and market research services for
energy, demand response, and renewable energy programs. Since its founding in 1996, Research Into
Action’s clients have included public and investor-owned utilities, as well as national and international
energy, government, and nongovernmental agencies. We conduct all phases of qualitative and
quantitative research in the design, delivery, analysis, and interpretation of program, product, and
service development. Research Into Action specializes in conducting field research to assist
organizations in three areas:
1. Understanding people's behaviors and motives through reliable research;
2. Developing research findings to support decision-making and implementation; and
3. Working with clients to understand and remedy organizational barriers to action.
Our research spans all customer sectors (residential, low-income, multifamily, commercial, industrial,
agricultural), both new and existing facilities, and all program types – rebate and custom incentives,
financing options, information and behavior change (pricing, assessments, home energy reports,
education, training), demand response and load management, direct install, upstream, community-
based, market transformation, and marketing.
3.2. Mesa Point Energy
Mesa Point Energy’s business approach is to provide tailored, client-specific strategic and tactical
support to best meet client needs. Mesa Point excels in the technical aspects of energy efficiency
programs and prides itself on providing advanced engineering and analysis. Our clients include
regulators, utilities, building owners, government, and energy service companies.
Mesa Point Energy a comprehensive array of services in the area energy efficiency program design,
implementation, and evaluation, litigation support, portfolio energy efficiency performance, potential
studies, technical resource manual review, M&V, training, energy auditing, building modelling,
commissioning and retro commissioning, project management, and program oversight.
3.3. Apex Analytics
Apex Analytics was formed in 2011 with the goal of providing the highest quality planning, potential, and
evaluation studies for energy efficiency, load management, and market transformation programs. Apex
has grown to a staff of five consultants with backgrounds in economics and market research. Our
projects range from leading impact and process evaluations for some of the largest program
administrators in the North America (PG&E, SCE, NYSERDA, Energy Trust of Oregon) to developing
program plans and filings (SourceGas, Black Hills Energy, Atmos Energy). Apex staff have extensive
experience in Colorado going back to 2008, having designed and evaluated program for Xcel Energy,
Atmos Energy, Black Hills Energy, and Colorado Natural Gas. Apex staff also provide technical and
analytical oversight for portfolio evaluations and potential study efforts in Arkansas, Connecticut,
Illinois, Massachusetts, Missouri, and Pennsylvania. In addition, Apex has spearheaded the development
of the Consortium for Retail Energy Efficiency Data (CREED), a consortium of program administrators,
regulators, and government that is working to make retail point-of-sale data available for improved
program planning and evaluation efforts.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 25
4. COMPANY EXPERIENCE
We illustrate the relevance of our prior experience using the following key:
Appliance Rebate
Appliance Recycling
Demand Response –
Peak Partners
Efficiency Works –
Home Audits
Efficiency Works –
Home Direct Install
Efficiency Works –
Home Rebates
Efficiency Works –
Business Audits
Efficiency Works –
Business Rebates
Efficiency Works –
Business Tune Up
Help Loan Program
Home Energy Report
Integrated Design
Assistance Program
Larimer County
Conservation Corps
Midstream Retail
Lighting
Renewable Energy –
Solar
4.1. Research Into Action
State of Colorado, Colorado Governor's Energy Office, American Recover and Reinvestment Act
Evaluation (2011). As a subcontractor to Nexant, we conducted the free ridership analysis for a
comprehensive impact evaluation of the State of Colorado's Governor's Energy Office's (GEO)
residential, commercial, and industrial energy efficiency programs. We developed and fielded the
attribution surveys for all programs necessary to conduct the program-specific analyses.
NorthWestern Energy, Commercial and Residential Portfolio Evaluations (two contracts). As a
subcontractor to Nexant in 2007 and another firm in 2013, we conducted the process evaluation portion
of a comprehensive analysis of the residential, commercial, industrial, and agricultural electric and
natural gas demand side management (DSM) and renewable energy (photovoltaic and wind) programs
of NorthWestern Energy (NWE). In 2013, we evaluated each of NWE’s more than 30 programs, including
audits, incentives, and training. We also developed estimates of free ridership, spillover, and leakage.
Avista Utilities, Commercial and Residential Portfolio Evaluations (two contracts). As a subcontractor
to Nexant in 2014-2015 and again in 2016-2017, we conducted the process evaluation of Avista Utilities’
17 commercial and residential energy programs. As part of the portfolio evaluation, we conducted a
special impact study of Avista’s Home Energy Report (HER) Program. Using a quasi-experimental design,
we analyzed average daily household energy usage for 90,000+ Avista households over a 20-month
intervention period and a 12-month pre-intervention period. We found that households with both HERs
and rebates had significantly higher savings than the sum of the average savings attributed to the rebate
programs alone and HER alone.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 26
National Grid, Single-Family EnergyWise (2015). We conducted a process evaluation of National Grid’s
2015 Single-Family EnergyWise program, and the HEAT Loan offering. Our process evaluation and best
practices study examined program processes and the customer experience, the effectiveness and equity
of the loan offering, and opportunities to learn from other programs across the country.
Snohomish County PUD, Residential Portfolio Review (2010, 2011 [two contracts]). We conducted
process evaluations of Snohomish County PUD’s (Everett, WA) residential program portfolio, including
heat pump loans and incentives, lighting (three programs with dealer and consumer incentives),
weatherization (single- and multi-family programs), and appliances (rebates and appliance recycling).
The studies included a “best practices” analysis, investigating comparable program categories of other
utilities to identify and understand other tested opportunities for program expansion.
Bonneville Power Administration, Simple Steps Smart Savings Process Evaluation (2015-2016). As a
subcontractor, we contributed to the process evaluation of BPA's Simple Steps Smart Savings program.
The program primarily works upstream with manufactures and retailers to reduce the store price of
CFLs, LEDs, lighting fixtures, showerheads, and selected appliances. The program also conducts direct
installation of measures and bulk purchases. We identified the value of the program to the region,
program facets most appealing to participating utilities, and opportunities for program improvement.
Northwest Energy Efficiency Alliance, Retail Product Portfolio Evaluation (2015-present). We
evaluated NEEA’s residential products portfolio (RPP) pilot to assess pilot influence on retailer
merchandizing behavior, and whether a larger Initiative had potential to influence retailers further. We
learned that a midstream program like RPP has the potential to influence retailer behavior, but it
requires national-level scale to do so. To assess whether this level of scale is possible, we interviewed
participating program administrators in California and at the EPA, as well as potential program
administrators who recently decided not to participate in a national RPP program.
New York State Energy Research and Development Authority Workforce Training Studies (multiple
contracts). In support of the PV Program, we conducted research and analysis on the effect of workforce
training on NYSERDA’s PV projects as well as on the development of the photovoltaic (PV) market in the
U.S. In separate research, we conducted a process evaluation of NYSERDA’s Workforce Development
Training Partnerships for Energy Efficiency (WFD) program. WFD provides funds for career pathways
training for underserved and underemployed populations, technical training to improve the skills of
those already in the building industry, and internship, apprenticeship, and certification programs.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 27
Energy Trust of Oregon, Commercial and Industrial Solar Marketing Strategy Research Support (2013-
2014). To guide Energy Trust’s commercial and industrial solar marketing strategy, we analyzed the
“most likely to consider solar” market segments. We characterized participants and partial participants
in the program databases, including tabulating business characteristics, geographic distribution, and
other data. We identified issues on commercial and industrial solar adoption from a literature review
identify issues. In in-depth interviews with current and potential solar adopters in the commercial and
industrial sectors, we explored issues to solar adoption that we had identified from a literature review.
Independent Electricity System Operator (IESO), Commercial and Residential Program Portfolio
Evaluations (2011-2016 [four contracts]). Twice we led a team conducting process and impact
evaluations of IESO Consumer Program portfolio and twice served as the process subcontractor on a
team lead by Nexant to evaluate IESO’s Business Program portfolio. The portfolios included home and
business audits and rebates, direct install, appliance recycling, midstream incentives, and business
retrocommissioning. We addressed program-specific and cross-program and marketing effects,
including motivations for behavior and decision-making processes, the conversion of audits to retrofit
projects, the appropriateness of incentive levels, non-energy benefits, net-to-gross savings, greenhouse
gas effects, and direct employment effects.
Portland General Electric, Solar Payment Option Program Evaluation/Feed-in-Tariff (FIT) Evaluation
(2010-2015). We conducted a five-year combined market assessment, process evaluation, and impact
evaluation for Portland General Electric of its pilot Solar Payment Option program for residential
customers. The program was the first in Oregon and one of the first in the U.S. to test a FIT policy
mechanism that requires utilities to buy solar electricity at a premium from customers. We surveyed
customers within three months following successful interconnection and one year later.
Energy Trust, Production Efficiency Program Process Evaluation (2012-2013). We led a process
evaluation of Energy Trust's 2011-2012 Production Efficiency Program for the industrial sector. Our
process evaluation studied the program as well as several of its individual tracks: Custom,
Retrocommissioning, Boiler Tune Ups, Corporate Strategic Energy Management, Refrigeration Operator
Certification, Small Industrial and Prescriptive, Green Rewind, and Lighting. The Corporate Strategic
Energy Management portion of the program was a behavior change effort.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 28
4.2. Mesa Point Energy
Custom Energy Efficiency Program Design and implementation for Black Hills Energy, Atmos Energy,
and Colorado Natural Gas (2014-present). Mesa Point designed and is implementing the Custom Energy
Efficiency Program (CEEP) for Black Hills Energy (formerly SourceGas), Atmos Energy, and Colorado
Natural Gas. The CEEP provide technical and financial incentives for the identification and installation of
gas savings project in their Colorado service territory. We provide program design, administration,
tracking, marketing and outreach, onsite inspections, and technical analysis services for the program,
including: Calculating deemed energy savings for various energy efficiency measures, calculating
incentive levels based on incremental cost and/or energy savings. Evaluating net present value (NPV)
based on measure life, determining project and program level cost effectiveness using TRC and other
cost tests
Commercial Building Energy Efficiency Market Study for the Colorado Governor’s Energy Office (2012).
We performed a commercial buildings market assessment and potential study for the Governor’s Office.
The project characterized and quantified commercial market energy efficiency potential based on
available secondary data and surveying. The report included information about commercial buildings in
Colorado (by building type, customer segment and energy end-use) and provided quantified analysis and
projections for technical, economic and achievable potential. The information helped formulate policy
and define cost-effective measures to best direct the State towards savings in commercial buildings by
identifying gaps between efficiency potential and the existing level of activity.
Philips Lighting M&V Support (2013-present). We provide expert energy analysis and M&V support to
Philips Lighting for performance contracting projects. Mesa Point Energy is currently helping to develop
and implement a detailed and comprehensive parking garage M&V plan for the installation of advanced
LED lighting systems and controls for all the Washington Metropolitan Transit Authority, Washington DC
parking facilities.
Xcel Energy, Impact Evaluation of Whole House Energy Efficiency and Comfort Pilot Program (2013).
Mesa Point performed a detailed impact evaluation of the Whole Home Energy Conservation pilot
program which was implemented in Xcel Energy’s Colorado service Territory in 2011/2012. Work
involved project verification, detailed billing analysis, TRM calculation analysis and critique, report
development and presentation of findings at Xcel Energy’s quarterly Colorado Roundtable.
Arkansas Public Service Commission, Independent Evaluation Monitor Team (2011-present). Mesa
Point serves on a team in the role of Independent Evaluation Monitor (IEM) for the Arkansas Public
Service Commission (AR PSC). The IEM team provides oversight and quality insurance across three
separate evaluation contractors working on behalf of six IOU’s. Work on this project is very similar to the
work requested in the RFP. Specific tasks completed by Mesa Point include:
Review and analysis of utility DSM plans, goals and programs.
Review and input on development of evaluator EM&V plans prior to them being implemented.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 29
Detailed review of utility annual reports checking for consistency and accuracy of filings,
comparison to evaluation reports and plans.
Detailed review, input and critique of evaluation reports. Tracking of evaluator findings and
recommendations and detailed analysis of statewide program overall performance over all
utilities and administrators.
Management and development of the Arkansas TRM versions 2 through 5.
Support on special projects and requests, assistance in interpretation of TRMs, program data
and other information.
Development if Independent Evaluation Monitor (IEM) reports of findings and
recommendations.
Development and delivery of testimony for the Arkansas Public Service Commission.
Regular participation, including extensive presentation roles, in both in person and
teleconference meeting with commission staff, utilities, evaluators, interveners and other
interested parties.
The work in Arkansas, along with work on other projects has included in depth review and analysis of
several TRMs around the country. Some examples of TRMs we have worked on or with include Xcel
Energy’s TRM for PSCO and Northern States Power, the Illinois, Ohio, Pennsylvania, Texas and other
states TRMs, the DEER data bases, the Northwest’s Reginal Technical Forum TRM documents, and
others.
ECube, Building Energy Modelling (2012-present). Mesa Point provides energy modelling services to E-
Cube clients throughout the Country in support of their commissioning and project management
engineering practice.
Most models are completed in eQuest 3.65, and facilities have included hospitals, outpatient health
care facilities, office buildings, newspaper printing facilities, pharmaceutical research labs, and
manufacturing facilities.
A significant portion of the work focused on energy modelling for LEED certification. As a result, Mesa
Point has extensive experience working with LEED buildings and has developed deep knowledge and
understanding of recent energy codes such as ASHRAE 90.1 from 2007 to present and the International
Energy Conservation Code (IECC) from 2009 – 2015.
Federal Energy Management Program (FEMP) Measurement and Verification Guidelines, Version 3.0
(2008). Dr. Bradford is a co-author of the FEMP Measurement and Verification Guidelines, Version 3.0.
In addition to providing expert EM&V support to FEMP for specific projects, Jim provided authorship and
input on a wide range of EM&V approaches for various efficiency measures and provided authorship
and input on sampling, uncertainty and confidence and precision analysis.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 30
4.3. Apex Analytics
Colorado Gas Utility Consortium EM&V and DSM Plans (2008-present). Apex Analytics led an initial
potential assessment (in 2008) for a consortium of three Colorado gas utilities: Atmos Energy, Colorado
Natural Gas, and Black Hills Energy (formerly SourceGas). Since then, Apex staff have developed four
consecutive tri-annual DSM plans for members of the consortium. Each plan included developing a
measure database for all potential DSM measures, selecting which measures to include in programs,
designing the incentive structures and program strategies, running program cost-effectiveness models,
and preparing regulatory filings. Apex staff have also led the most recent impact evaluation for the
consortium, conducting billing analysis, surveys, and engineering desk reviews.
Duke Energy Progress Energy Efficient Lighting Program Evaluations (2010-2015). Apex Analytics
managed the evaluation of the Duke Energy Progress (formerly Progress Energy Corporation) Energy
Efficient Lighting Program. The evaluation efforts included comprehensive impact and process
evaluations, utilizing customer telephone surveys, intercept surveys, a shelf stocking study, in-home
lighting audit and metering study, and retailer and manufacturer interviews. The research also included
the use of a longitudinal sampling approach to estimate CFL installation rates over multiple years. The
results from these efforts have been used to develop both gross and net impact parameter inputs,
including an estimate for cross-sector sales that led to a large increase in program savings.
U.S Department of Energy – Uniform Methods Protocols (2013-present). Though not directly
evaluation of program activity, since 2013, Apex has authored the comprehensive lighting program
evaluation protocols established by the DOE in the Uniform Methods Project (UMP). The lighting
chapter details the recommended approaches that should be followed to evaluate lighting programs,
and has been referenced in evaluations in jurisdictions across the country. Apex is slated to manage the
2017 updates to the UMP lighting protocols.
Report: U.S. DOE Uniform Methods Project: Chapter 21: Residential Lighting Evaluation Protocol
http://energy.gov/sites/prod/files/2015/02/f19/UMPChapter21-residential-lighting-evaluation-
protocol.pdf
Smart Thermostat Pilot Evaluations, Energy Trust of Oregon (2014-current). Apex Analytics was
engaged by Energy Trust of Oregon to help evaluate three smart thermostat pilot programs. The first
pilot involved testing a Nest thermostat in heat pump heated homes, the second pilot involved testing
both a Nest and Honeywell Lyric smart thermostat in gas furnace heated homes, and the third and
current evaluation seeks to understand satisfaction with and savings generated from the Nest Seasonal
Savings pilot. The evaluation objectives were to determine if installing the Nest thermostat is a viable
strategy for properly controlling central electric heat pump operation in residential settings; the
electricity and gas savings associated with smart thermostats; how customers interact with the device;
and customer satisfaction with the device and its control of the comfort of their homes. Each evaluation
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Company Experience | Page 31
included two web-based participant surveys – conducted shortly after installation and then
approximately six months after installation – coupled with a billing analysis to determine energy savings.
Report: Nest Thermostat Heat Pump Control Pilot Evaluation
http://energytrust.org/library/reports/Nest_Pilot_Study_Evaluation_wSR.pdf
Report: Smart Thermostat Pilot Evaluation
http://assets.energytrust.org/api/assets/reports/Smart_Thermostat_Pilot_Evaluation-Final_wSR.pdf
Arkansas Program Evaluation Oversight and Protocol Development (2012-present). Apex has led the
development of the evaluation protocols for behavior programs for the Arkansas Technical Reference
Manuals (TRM) and played an active role in the oversight of all evaluation methods and reported
findings. The protocols present the recommended approaches for evaluating savings for all residential
and behavioral programs run by both the gas and electric utilities in the state. Working on behalf of the
Arkansas Public Service Commission, Apex also reviews and manages behavior and residential program
M&V efforts, ensuring that they meet the requirements of the protocols.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Staff Qualifications and Project Management | Page 32
5. STAFF QUALIFICATIONS AND PROJECT MANAGEMENT
5.1. Staff Qualifications and Experience
5.1.1. Research Into Action Key Staff Qualifications
Marjorie McRae, Ph.D. Vice President. Dr. McRae has been active in the assessment of energy
efficiency and demand response programs since 1980. She holds a Ph.D. in psychology from The Wright
Institute, an M.A. in economics from University of California, Berkeley, and a B.A. in economics from
Goucher College. Dr. McRae focuses on the design and implementation of process and market research
studies for energy efficiency, demand response, renewable energy, market transformation, information,
and new technologies programs in all sectors. She is the author of DSM Evaluation: Six Steps for
Assessing Program Effects, published by the Electric Power Research Institute (EPRI), and has provided
expert testimony on process evaluation findings in regulatory proceedings. Dr. McRae led the following
projects described in Section 4.1: Northwestern Energy, Avista, National Grid, Snohomish PUD,
Bonneville Power Administration, NYSERDA, and Energy Trust. She also led the evaluation of DOE’s
Better Buildings Neighborhood Program, which gave grants to local and state governments to design
and implement whole-building upgrades in their communities.
Role: Dr. McRae serve as the Principal-in-Charge for the project. She will oversee all aspects of
Research Into Action’s activities in this project and ensure we remain on time and on budget.
Joe Van Clock, M.A., M.S., Senior Consultant 1. Mr. Van Clock has ten years of research experience,
seven of which are in energy efficiency and renewable energy. He has an M.A. in global communications
from the University of Southern California, an M.S. in global media from the London School of
Economics, and a B.A. in international affairs and Hispanic studies from Lewis & Clark College. Mr. Van
Clock is an experienced researcher who excels at all aspects of program evaluation and market research
projects, including conducting in-depth interviews, analyzing qualitative data, and reporting and
presenting research findings. He has lent his expertise to a wide range of projects, including market
characterizations of business and consumer electronics products, comparison research into the role of
evaluation in efficiency organizations throughout North America, and evaluations of efforts to promote
whole-house retrofits in California and across the country. Among his recent work, he served as the
project manager for Research Into Action’s work on for National Grid, Northwest Energy Efficiency
Alliance, and Bonneville Power Administration.
Role: Mr. Van Clock serve as the overall project manager. He will coordinate day-to-day project
activities for the Research Into Action team and be Utilities’ point-of-contact for all aspects of
the project. He will work with the team to develop survey and interview guides, oversee fielding
of interviews and surveys, direct the analysis and reporting, and develop conclusions and
recommendations.
Doré Mangan, M.P.A., Consultant 3. Ms. Mangan specializes in qualitative and quantitative market
research and analysis to evaluate energy and behavioral programs. She has an M.P.A. from the
University of Washington, Evans School of Public Affairs, with a focus on environmental and natural
resource management, as well as a B.A. in psychology with an emphasis on animal behavior, and a B.A.
in business, both from the University of Puget Sound. Her experience includes cost-benefit analysis,
policy analysis, program evaluation, economic analysis, and advanced statistical analysis. She leverages
her diverse background in psychology, public administration, behavior, market research, program
evaluation, data collection, and data analysis to evaluate commercial and residential programs and help
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Staff Qualifications and Project Management | Page 33
solve complex behavior-based problems. Ms. Mangan has served as lead analyst and support to the
project manager on projects for Energy Trust, National Grid, Northwest Energy Efficiency Alliance,
NYSERDA, PG&E, and U.S. DOE.
Role: Lead Analyst. Ms. Mangan will lead the analysis, including contributing to the data
collection instruments, leading associated data collection and analysis, and contributing to
report writing and progress tracking/reporting.
Jennifer Loomis, Ph.D., Consultant 2. Dr. Loomis is a social scientist with a background in
environmental sociology, qualitative research, and social inequality. She received a Ph.D. in sociology
from Portland State University, and her M.A. and B.A. in sociology from Colorado State University, Fort
Collins. Prior to joining Research Into Action, she studied the nexus of human-environment interactions,
primarily focusing on how organizations effectively manage natural resources. She has engaged in
qualitative and mixed-method research in the United States, Peru, and El Salvador. She has published
articles on issues relating to disaster recovery and social sustainability and presented her work at
regional and international conferences. She managed our evaluations for Southern California Edison of
the Local Government Partnership program. Her expertise in qualitative research enables her to develop
rapport with respondents and capture rich, candid data. She delivers results in a way that accurately
describes participant experiences and provides relevant information to guide informed decision-making.
Role: Lead, Interviews and Support, Analysis. Dr. Loomis will conduct in-depth and telephone
interviews, and work closely with Ms. Mangan on analysis and reporting.
Jun Suzuki, M.P.A., Senior Consultant 2. Mr. Suzuki has more than 12 years of experience in market
research for and evaluation of clean energy technologies and sustainability programs. Mr. Suzuki earned
his M.P.A., Public Administration from Monterey Institute of International Studies, Monterey, California
and his B.S., Business Administration from Aichi University, Japan. He studied advanced statistics and
survey methodologies from Portland State University’s Urban Studies doctoral program. He manages
our Qualtrics and SPSS software services. Mr. Suzuki’s work has emphasized survey research projects
that require large sample sizes, complex methodologies, and advanced statistical analysis. He ensures
sound survey methodology accompanies all data collection and that question design maximizes the
reliability and validity of survey and interview data. He also employs modern and emerging data
collection techniques, including multi-mode and mobile approaches, to optimize results. He has
managed surveys for process evaluations and market research for Smart Grid Consumer Collaborative’s,
The Empowered Consumer, Portland General Electric’s solar payment pilot, NYSERDA, SCE’s Flex Alert
and Save Power Day programs, PG&E’s Low-Income EE program, and BPA Residential Building Stock
Assessment Database analysis. He also led five rounds of implementation of the Residential Customer
Awareness survey for Energy Trust of Oregon.
Role: Mr. Suzuki will serve as the survey manager overseeing the operational aspects all
quantitative data collection efforts. This includes leading and coordinating online survey data
collection activities, including data management, oversight, and quality control, to ensure that
all data are managed for security and data accuracy.
5.1.2. Mesa Point Energy Key Staff Qualifications
James D. Bradford, Ph.D., P.E., President. Dr. Bradford is the president and founder of Mesa Point
Energy. Dr. Bradford is a licensed engineer in the state of Colorado and holds a Ph.D. and M.S. in Civil,
Environmental and Architectural Engineering from the University of Colorado and a B.S. in Mechanical
Engineering from Montana State University. Dr. Bradford has over 27 years of experience in commercial
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Staff Qualifications and Project Management | Page 34
building systems, energy performance metrics, energy systems design and analysis, performance
contracting, DSM utility programs, energy project measurement and verification (M&V), energy policy
development, and expert witness support. Dr. Bradford currently serves on the Louisville Colorado
Sustainability Advisory Board and the International Performance Measurement and Verification Protocol
(IPMVP®) committee tasked with developing protocol and setting certification standards.
Donna Scott, M.S., Project Manager. Ms. Scott has over 20 years of experience in the environmental
field working with municipalities and consulting services. Ms. Scott holds a Master of Environmental
Science from the University of Colorado, Denver. She has expertise in project management, public
outreach and education, marketing, website development, grant writing, as well as program design,
implementation and evaluation. While working at the City of Boulder, Ms. Scott conceived of and
launched a number of nationally recognized, regional, cooperative programs including the Boulder
County Household Hazardous Waste Program, Partners for a Clean Environment (PACE), and Keep it
Clean Program (KICP) and successfully navigated council approval of several ordinances and design and
construction standard requirements.
Gang Tan, Ph.D., P.E., Senior Engineer. Dr. Tan is an Assistant Professor, Department of Civil and
Architectural Engineering, University of Wyoming, Laramie, WY. Dr. Tan provides engineering modeling
support. He has a Ph.D. in Building Technology, Massachusetts Institute of Technology (MIT), Cambridge,
MA. Dr. Tan has worked on many DSM related projects such as potential studies, evaluations and tool
design for organizations such as the Energy Information Administration/U.S. Department of Energy
(DOE) New Jersey Public Service Enterprise Group’s (PSEG), New York City Green House Gas Reduction
Long-term Plan project, State of Florida, National Grid, ComEd Florida Power & Light’s (FP&L), Pacific
Gas and Electric Company’s (PG&E), USGBC, New York Power Authority (NYPA) The Georgia Power
Company The American Society of Heating, Refrigerating, and Air-conditioning Engineers (ASHRAE), New
York Energy $martSM Program, New York State Energy Research and Development Authority
(NYSERDA), and MidAmerican Energy.
5.1.3. Apex Analytics Key Staff Qualifications
Scott Dimetrosky, M.B.A. Mr. Dimetrosky has over 25 years of experience in leading evaluations of
energy efficiency programs, including extensive experience in Colorado. He has an M.B.A. in Marketing
Research & Quantitative Methods from Cornell University and a B.A. (Magna Cum Laude) in Sociology
from the University of Michigan. He has designed and evaluated program portfolios for a consortium of
Colorado gas utilities (Atmos Energy, Black Hills Energy – formerly Source Gas, and Colorado Natural
Gas) since 2009. He also led the 2009-2011 residential evaluations for Xcel Energy. He has held
numerous meetings, as well as testified, with the Colorado Public Utilities Commission (CO PUC)
Mr. Dimetrosky is also a nationally recognized expert in residential lighting programs, serving as the lead
author for the Department of Energy Uniform Methods Project (UMP) Residential Lighting Evaluation
Protocols. He has led evaluations of lighting programs throughout the U.S., including California, North
Carolina, and Wisconsin, and is currently managing lighting evaluations in four states, including
Massachusetts, Connecticut, Arkansas, and Missouri.
Prior to forming Apex Analytics, Mr. Dimetrosky was a founding member and principal at Quantec, LLC,
which merged with the Cadmus Group in 2008. During his 13 years and Quantec and Cadmus, Mr.
Dimetrosky led some of the largest evaluations in the United States. Mr. Dimetrosky has delivered
papers at over 25 energy efficiency conferences, and taught principles of Demand-Side Management
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Staff Qualifications and Project Management | Page 35
(DSM) and DSM evaluation courses. He is on the Planning Committee for the IEPEC and a former board
member of the Rocky Mountain Chapter of the Association of Energy Service Professionals.
Role: Mr. Dimetrosky will provide overall project oversight, quality assurance, lead technical
impact advisor.
Noah Lieb, B.A., Associate. Mr. Lieb brings more than 13 years of experience in the energy industry,
offering strong quantitative and analytical skills, particularly with large, complex datasets and market
characterization studies. Mr. Lieb holds a bachelor’s degree in honors international economics and
business, graduating summa cum laude from the University of Colorado. He specializes in residential
impact evaluations, having led the recent evaluation of the Wisconsin Focus on Energy Lighting Program,
the NYSERDA ENERGY STAR Program, and the impact evaluation of the California Multifamily Program,
which includes both prescriptive and “whole building” approaches and is one of the largest multifamily
efforts in the United States. He is proficient in SAS, Stata, and web-based programming, and has
managed the technical development and analysis for multiple conjoint survey studies. Before joining
Apex, Mr. Lieb was a Senior Associate at the Cadmus Group (formerly Quantec LLC), where he managed
large portfolio evaluations, specifically focused on database integration and management. He also led
several residential lighting and appliance conjoint studies, cost effectiveness analysis, and residential
lighting hours of use studies.
Role: Mr. Lieb will provide project management, program evaluation lead, will oversee all
residential ex post impact savings analysis
Maegan McKee, B.A, Analyst. As an analyst at Apex Analytics, Maegan McKee contributes to practical
applications of quantitative and qualitative research. Ms. McKee holds a bachelor’s degree in
Environmental Studies with a concentration in Mathematical Sciences from Lewis & Clark College. She
has contributed to a broad range of projects including program evaluation and Net-to-Gross analysis.
Current projects include the DSM 2017-2019 Plan, 2015 Upstream Lighting Pilot Program Evaluation for
the Independent Electricity System Operator in Ontario and the Impact Evaluation for Colorado Partners
in Energy Savings. Prior to joining Apex, Ms. McKee was an analyst at RRC Associates. She joins Apex
with a diverse background in market research, specializing in data collection, analysis and reporting. Her
experience has provided a comprehensive perspective on planning and research in the outdoor
recreation, tourism and ski industries.
Role: Ms. McKee will be supporting the residential impact evaluation, from application review to
calibrate energy analysis.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Staff Qualifications and Project Management | Page 36
5.2. Project Management
The prior section describes the roles of our team members. The following figure illustrates our project
management structure.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Proposed Schedule of Deliverables | Page 37
6. PROPOSED SCHEDULE OF DELIVERABLES
Table 6-1 provides our proposed project schedule.
Table 6-1: Schedule of Milestones and Deliverables
Jan.
30-Jan
6-Feb
13-Feb
20-Feb
27-Feb
6-Mar
13-Mar
20-Mar
27-Mar
3-Apr
10-Apr
17-Apr
24-Apr
1-May
8-May
15-May
22-May
29-May
5-Jun
12-Jun
19-Jun
26-Feb
Task Start Date End Date
Kick-off Meeting & Interviews February 1, 2017 February 2, 2017
Draft Evaluation Plan February 1, 2017 February 10, 2017
Final Evaluation Plan February 22, 2017 February 22, 2017 p
Project Files Received February 10, 2017 February 10, 2017 p
Draft Data Collection Instruments February 13, 2017 February 24, 2017
Final Data Collection Instruments March 3, 2017 March 3, 2017 p
Monthly Phone Meeting March 3, 2017 First Friday of the Month p p p p
File Review February 10, 2017 May 12, 2017
Reported/ Ex Ante Savings Review February 10, 2017 May 12, 2017
Bi-weekly Status Update February 17, 2017 Every Other Friday p p p p p p p p p p
Surveys and Interviews March 6, 2017 April 14, 2017
Onsite Verification March 13, 2017 May 12, 2017
Calibrated Analysis April 3, 2017 May 12, 2017
Billing Analysis (optional) April 10, 2017 May 12, 2017
Net Savings Analysis April 17, 2017 May 12, 2017
Cost-effectiveness Analysis May 8, 2017 May 26, 2017
Draft Report (internal) May 8, 2017 June 9, 2017
Discussion of Findings with Utilities May 26, 2017 May 26, 2017
Delivery of Draft Report June 7, 2017 June 7, 2017 p
Draft PowerPoint June 14, 2017 June 14, 2017 p
Final Report and PowerPoint June 23, 2017 June 23, 2017 p
Presentation of Research June 28, 2017 June 28, 2017 p
June
Week of:
February March April May
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Pricing | Page 38
7. PRICING
We will conduct the comprehensive evaluation for Utilities we describe in Section to for a not-to-exceed
price of $349,796. Table 7-1 provides our proposed budgets per program for Utilities, with options for
PRPA programs and two optional tasks described in Table 2-3. We wish to clarify that our per-program
cost estimates reflect economies of scale from conducting the portfolio evaluation. Were we to evaluate
fewer programs, our costs per program would increase. Further, we do not intend for Utilities to
interpret our per-program costs as individual not-to-exceed estimates. Our actual costs for any specific
program may vary somewhat from our estimated program budget.
Table 7-1: Price Proposal by Program, by Utilities and Options
PROGRAM UTILITIES PRPA OPTIONAL RESEARCH
TOTAL $349,796 $88,611 $50,000
Appliance Rebates $13,653
Appliance Recycling $16,979
Peak Partners $28,571 $25,000
Home Audits $14,659 $7,330
Home Direct Install $12,178 $6,089
Home Rebates $24,794 $12,397
Business Audits $14,084 $7,042
Business Rebates $68,777 $34,389
Business Tune Up $22,761 $11,380
Help Loan Program $11,868
Home Energy Reports $19,769 $25,000
Integrated Design
Assistance $40,769
Larimer County
Conservation Corps $15,354
Midstream Retail Lighting $19,969 $9,984
Renewable Energy - Solar $25,611
We also provide Utilities with our price proposal by task and team member in Table 7-2.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Pricing | Page 39
Table 7-2: Price Proposal by Task and Team Member
Task 1 Task 2 Task 3 Task 4 Task 5
Rate/
Hour
Kick-off &
Interviews
Evaluation
Plan
Impact
Evaluation
Process
Evaluation
Project Mgt,
Reporting
Total
Hours
Total
Amount
Opt. Billing
Analyses
Research Into Action Hours
Executive Consultant $ 225 20 4 20 20 64 $ 14,400
Sr. Consultant 2 $ 153 54 12 66 $ 10,098
Sr. Consultant 1 $ 144 20 12 84 40 156 $ 22,464
Consultant 3 $ 133 20 104 40 164 $ 21,812
Consultant 2 $ 118 168 40 208 $ 24,544
Data Collection Assoc. $ 107 80 80 $ 8,560
Research Associate $ 95 106 40 146 $ 13,870
Hours per Task 60 20 0 621 208 909
Labor cost per Task $ 10,040 $ 2,928 $ - $ 77,519 $ 27,136 $ 117,623
Sub-contractor Hours
Mesa Point Energy Principal $ 190 16 4 12 24 56 $ 10,640
Mesa Point Energy Project Manager $ 150 16 20 310 30 376 $ 56,400
Mesa Point Energy Sr. Engineer $ 150 8 350 24 382 $ 57,300
Apex Analytics Principal $ 225 4 2 18 4 28 $ 6,300 $2,600
Apex Analytics Project Manager $ 160 16 20 182 40 258 $ 41,280 $22,400
Apex Analytics Lead Analyst $ 120 8 350 40 398 $ 47,760 $25,000
Sub-contracting Hours per Task 68 46 1222 0 162 1498
Total Sub-contracting Cost/Task $ 11,060 $ 7,410 $ 176,450 $ 24,760 $ 219,680
Other Direct Costs (ODC)
Travel: Research Into Action $ 715 $ 715 $ 1,430
Travel: Mesa Point Energy $ 7,317 $ 7,317
Travel: Apex Analytics $ -
Other $ 1,000 $ 1,000
ODC by Task $ 715 $ 7,317 $ 1,000 $ 715 $ 9,747
G&A on Subs @ 1.25% $ 138 $ 93 $ 2,206 $ - $ 310 $ 2,746
Total Cost by Task $ 21,953 $ 185,973 $ 78,519 $ 52,921
Project Total $ 349,796 $50,000
Hours Per Task
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
References | Page 40
8. REFERENCES
Please feel free to contact any of our references.
Dan Johnson, P.E., Director, Energy Efficiency, Avista Utilities, (509) 495-2807,
dan.johnson@avistacorp.com
Deb Young, Director of Programs Energy Services, NorthWestern Energy, (406) 497-2339,
deb.young@northwestern.com
Phil Bosco, IESO Independent Electricity System Operator, Phil Bosco, Evaluation Manager (416)
969-6095, phil.bosco@powerauthority.on.ca
Phil Degens, Evaluation Manager, Energy Trust of Oregon, (503) 493-8888, ext. 220,
phil.degens@energytrust.org
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Sustainability/TBL Methodology | Page 41
9. SUSTAINABILITY/TBL METHODOLOGY
Research Into Action is extremely aware of how life-styles of businesses and individuals impact the
environment. We have extensive sustainability practices in place. We truly walk the talk and enjoy
working for and with companies that share our sustainable culture and mission.
At Research Into Action, workplace sustainability is foundational to our company philosophy. We
research sustainable solutions for governments and utilities for a living, and we make sure we follow our
own recommendations! We have an empowered volunteer green team that actively works on making
our company as sustainable as possible, even when sustainable choices/actions are more costly. Our
president and management team supports the green team 100% and consistently approves the green
team’s plans. About one-third of our Portland staff is on the green team, so we have lots of people to
help generate, plan, and implement our ideas. Thanks to the hard work of the green team, our staff, and
the support of our president and management team, we have implemented many sustainability
practices, discussed below.
Certifications and Awards: We are Gold Certified by the City of Portland’s Sustainability at Work
certification program, we have consistently been in the top eight of the 100 Best Green
Companies list, and we are working on finalizing our B-Corp certification.
Our offices are in a LEED® Gold Building with a geothermal heating system
Waste Reduction
We recycle all materials the city collects curb-side, including food waste (compost)
We encourage employees to bring in non-curb-side recyclable materials and take them,
along with our office-generated waste, to specialized recycling facilities (odd-sized plastics,
corks, etc.)
Our lunchroom, kitchen, and bathrooms are stocked with cloth napkins and towels to
minimize the use of paper ones
We take electronic-related items are no longer of use to us to the nonprofit Free Geek,
which repairs and repurposes them
We encourage employees to bring in hazardous waste (CFLs, batteries, etc.), which we take
to the appropriate Metro disposal facility
We provide lunch fixings for staff meetings three days a week, minimizing “to go” food
containers
We worked with the neighboring coffee shop to develop a punch card for staff when they
bring their own or a company-supplied mug; when the punch card is full, we provide them
with a gift card to that shop
Reducing Chemicals and Toxins
We use environmentally safe products in our office and engage a cleaning crew that
similarly uses safe products
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Sustainability/TBL Methodology | Page 42
Clean Transportation
We offer a $50/month bonus for employees that do not arrive to work as the single
occupant of a vehicle (type of alternative transit mode is their discretion)
We offer a $50/month bonus for the purchase of a highly efficient automobile
Energy Use and Conservation Efforts
We frequently have the lights off or on the lowest setting
We continually strive to “reduce, reuse, and recycle” in all things, efforts that have been
recognized by the City of Portland
Office Supplies and Equipment
We by Energy Star equipment, recycled paper, and metal desk and office supplies instead of
plastic (for recyclability)
Green Events
We host two to three green events a year, including competitions and neighborhood clean-
ups
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Proof of General Liability Insurance Coverage | Page 43
10. PROOF OF GENERAL LIABILITY INSURANCE COVERAGE
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Appendices | Page 44
APPENDICES
Appendix A. Background on Evaluation Terms ...................................................... A-1
Appendix B. Methodological Detail for Free Ridership Estimation ....................... B-1
Appendix C. Resumes of Key Personnel ................................................................. C-1
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Background on Evaluation Terms | Page A-1
APPENDIX A. BACKGROUND ON EVALUATION TERMS
As programs proceed from planning phases, to implementation, to evaluation, the calculated savings
may be updated at different points in time, and the recorded savings may change for various reasons. In
conducting the evaluation, we will proceed by analyzing and reporting savings from four perspectives (or
points in time), as follows. (The reader for whom these concepts are unfamiliar will find them discussed
in Appendix A.)
Each of these four savings perspectives are as follows.
Savings Targets – Target values are annualized savings goals for the particular measure,
program, sector or portfolio. Savings targets are usually set outside of the evaluation process,
but can be informed by evaluation results. Often potential studies and/or regulations set
program targets and budgets. Targets can be specific to measures and programs, or may provide
a more overarching goal for a portfolio of programs.
The RFP presents budgets and savings. However, the full picture of the budgeting and target
setting is not known by us. An exploration of budgets and energy savings targets will be
completed at the beginning of the impact evaluation.
Reported Gross Savings – Reported Gross savings, often known as ex ante gross savings, are
annualized savings either reported by the program implementer, or as calculated by applying
tracked program activity to deemed savings values.
Gross savings represents the savings calculated without consideration of free ridership, leakage,
spill over or market effects. It is assumed that the savings values presented in the RFP are Ex
Ante Gross savings.
Verified Gross Savings – Verified Gross Savings, often known as ex post gross savings, is
calculated during the evaluation phase. We will calculate the Verified Gross savings at the
measure, program and portfolio levels.
Verified Gross Savings is related to Reported Gross Savings by the “realization rate,” which is
simply the ratio of ex post savings to ex ante Savings. Realization rates of 1 indicate that the
savings reported by the program is the same as the savings found during the evaluation.
Realization rates greater than one indicate that the reported savings underestimated the actual
savings, while realization rates less than one indicate savings are being overestimated by the
program implementer.
Evaluated Net Savings – Evaluated Net savings, often known as Ex post net savings, is the
Verified Gross Savings multiplied by the net-to-gross (NTG) ratio, which accounts for free
ridership, spillover effect, leakage and market effects. The NTG will primarily be estimated
through a series of participant survey questions for most programs.
A comparison of these various savings values can be represented graphically as shown in Figure A-1.
Savings
Targets
Gross
Savings,
Reported
Gross
Savings,
Evaluated
Net
Savings,
Evaluated
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Background on Evaluation Terms | Page A-2
Figure A-1: Savings Comparison Illustration for a Hypothetical Program
In the example shown in Figure A-1, the implementer reported achieving 110% of goal, and the
evaluation showed that actual savings is 15% greater than reported. Finally, the net to gross ratio (NTG),
was found to be 90%. Therefore, the program, overall, was evaluated to achieve 114% of its target.
Some typical reasons for realization rate and NTG variations from unity include, but are by no means
limited to; database entry or calculation inaccuracies, improper or improperly used deemed savings
values, and field conditions diverging from reported field conditions
In equation form, these key program indicators are calculated as follows:
𝑅𝑒𝑝𝑜𝑟𝑡𝑒𝑑 𝑁𝑒𝑡 𝑆𝑎𝑣𝑖𝑛𝑔𝑠 = 𝑅𝑒𝑝𝑜𝑟𝑡𝑒𝑑 𝐺𝑟𝑜𝑠𝑠 𝑆𝑎𝑣𝑖𝑛𝑔𝑠 ∗ 𝑅𝑅 ∗ 𝑁𝑇𝐺
The calculation of these four primary perspectives is, quantitatively, at the core of the impact
evaluation.7 Comparisons of these values is important to understanding and reporting program
performance to oversite bodies such as the City of Fort Collins.
Two primary and complementary goals of the impact are to (1) verify and adjust savings values, and (2)
to identify program improvements to help modify programs for improved performance and so that both
realization rates and NTG values are driven as close as possible to unity.
Impact evaluation results will be reported comparing reported and verified gross and net savings. Key
parameters that will be reported include, but are not limited to, NTG, realization rates, impact of
measures and programs, and discussion of results. Suggestions for improvement of program tracking,
calculation, verification and other aspects of program design and operation will be presented and
summarized. We firmly believe providing Utilities with key insights as to the drivers behind savings
adjustments is as critical a step as reporting the actual impacts.
7
Often an additional value, the Reported (ex ante) net savings is often also presented, when the program implementer is
using ex ante NTG ratio values to calculate reported net savings.
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
MWH
Savings Comparison Illustration for a
Hypothetical Program
Target Reported Gross Evaluted Gross Evaluated Net
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Methodological Detail for Free Ridership Estimation | Page B-1
APPENDIX B. METHODOLOGICAL DETAIL FOR FREE RIDERSHIP ESTIMATION
We assess attribution (the extent to which the program can be attributed with inducing the efficiency
action, the converse of which is free ridership) using a brief instrument that assesses two components of
free ridership: 1) intention to carry out the energy-efficient project without program funds; and 2)
influence of the program in the decision to carry out the energy-efficient project.
We assess intention through three brief questions:
Had the respondent ever considered replacing the measure in question before being contacted
by the program representative?
Had the respondent planned to replace the measure in question before being contacted by the
program representative?
How the project likely would have differed if the respondent had not received the program
incentive, from no change (would have done the project exactly as it was done), to reduced
project scope or size, or used less expensive or efficient equipment, to cancelled the project
altogether.
We assess program influence by asking the respondent how much influence – from “1” (no influence) to
“5” (great influence) – the program incentive, the assessment, and the respondent’s interaction with the
contractor had on the decision to do the project the way it was done.
We apply the following algorithms to the responses to the two sets of questions to generate a project
change score and a program influence score.
Project Change: Respondents receive a project change free ridership score ranging from 0 to 50. They
received a maximum project change free ridership score of 50 if:
They had already installed the equipment before contact with the program.
They had planned to install the equipment before contact with the program and did not
describe how the project would have been different without program contact.
They would not have done the project any differently without program contact and said their
firm would have paid the entire cost of the project or did not answer whether the firm would
have paid the entire cost.
Respondents receive a project change free ridership score of 37.5 if:
They would not have done the project any differently without program contact and said their
firm probably would not have paid the entire cost of the project or said they lacked sufficient
funds without the incentive.
Respondents receive a project change free ridership score of 25 if:
They would not have done the project any differently without program contact and said their
firm definitely would not have paid the entire cost of the project.
They would have done a smaller project or one with less costly or less efficient equipment.
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Methodological Detail for Free Ridership Estimation | Page B-2
They responded to other survey questions but not project change questions.
Respondents receive a minimum “project change” free ridership score of 0 if:
They would not have done a project, kept using existing equipment, and cancelled or postponed
at least one year any planned projects.
Program Influence: Program influence free ridership score is calculated as the maximum influence score
given for any of the three influence factors. There are three possible sources of program influence for
the program: the incentive, support from program representative(s), and technical information, study or
audit. Each is scored on a 0 ("no influence") to 5 ("critical influence") scale. Respondents receive a
program influence free ridership score ranging from 0 to 50, based on their maximum influence score, as
follows:
Maximum influence = 1, program influence free ridership = 50
Maximum influence = 2, program influence free ridership = 37.5
Maximum influence = 3, program influence free ridership = 25
Maximum influence = 4, program influence free ridership = 12.5
Maximum influence = 5, program influence free ridership = 0
Commercial & Residential Energy Programs Third Party Consultant – RFP# 8433
Resumes of Key Personnel | Page C-1
APPENDIX C. RESUMES OF KEY PERSONNEL
C.1. Research Into Action, Inc.
Marjorie McRae, Ph.D.
Jun Suzuki, M.P.A.
Joe Van Clock, M.Sc., M.A.
Doré Mangan, M.P.A.
Jen Loomis, Ph.D.
C.2. Mesa Point Energy
Jim Bradford, Ph.D., P.E.
Gang Tan, Ph.D., P.E.
Donna Scott, M.S.
C.3. Apex Analytics, Inc.
Scott Dimetrosky, M.B.A.
Noah Lieb, B.A.
Maegan McKee, B.A.
Marjorie McRae Resume | Page 1
Dr. McRae has provided impact and process assessment and
regulatory support for energy efficiency and demand
response programs since 1980. Since 2000, she has focused
on the design and implementation of process and market
research studies for energy efficiency, demand response,
renewable energy, market transformation, information, and
emerging technology programs in all sectors. Her experience
includes program planning and the estimation of free-
ridership and program impacts. Her work has been featured
in many energy publications, and she is the author of DSM
Evaluation: Six Steps for Assessing Program Effects, a guide
to process and impact evaluation and market research
published by the Electric Power Research Institute (EPRI). Dr.
McRae has provided expert testimony in regulatory
proceedings.
She is a member of the Evaluation, Measurement, and
Verification (EM&V) training team for the Association of
Energy Services Professionals, and has conducted training as part of the International Energy Program
Evaluation Conference.
Representative Experience
Dr. McRae’s has led numerous assessments of clean energy technologies programs, including:
Led the process evaluation of Avista’s energy efficiency program portfolio.
Led numerous process evaluations of NYSERDA programs starting in 2003, including zero net energy
market assessment and baseline, workforce development, the multi-million-dollar evaluation of
Multifamily Performance Program, PV process and workforce certification, Cleaner, Greener
Communities, and Green Jobs Green New York.
Led the process evaluation for NorthWestern Energy’s commercial and residential demand side
management program evaluations (for two seven-year planning cycles).
Led a four-firm team in a four-million-dollar, five-year comprehensive evaluation of U.S. DOE’s $500
million stimulus-funded Better Buildings Neighborhood Program.
Leading the California Energy Commission’s EnSEED: Accelerating the Deployment of Advanced
Energy Communities (an EPIC Grant) program.
Led combined impact and process evaluations of two innovative U.S. DOE Lab-Corps and Small
Business Voucher pilots to increase National Laboratory involvement in the commercialization of
new technologies.
Marjorie McRae, Ph.D.
Vice President
Education
Ph.D., Psychology: The Wright
Institute, Berkeley, CA
M.A., Economics: University of
California, Berkeley, CA
B.A., Economics: Goucher College,
Towson, MD
Joe Van Clock Resume | Page 1
Mr. Van Clock has ten years of research experience, including
seven in the clean technology sector. He excels at all aspects of
program evaluation and market research, including qualitative
analysis, in-depth interviews, and the analysis and
interpretation of results. Through his work at Research Into
Action, Mr. Van Clock has developed extensive experience in
comparison research into the role of evaluation in efficiency
organizations throughout North America, market
characterizations of business and consumer electronics
products, and evaluations of whole-house retrofit programs.
Representative Experience
Mr. Van Clock manages and coordinates day-to-day process
evaluation and market research activities, develops survey and
interview guides, oversees fielding of interviews and surveys,
analyzes interviews and surveys, and develops conclusions and
recommendations. His representative projects include
Managed the evaluation of a home energy audit and weatherization program for National Grid’s
EnergyWise Rhode Island Single Family Process Evaluation.
Conducted and analyzed in-depth interviews with utilities participating in a regional midstream buy-
down program promoting efficient products for Bonneville Power Administration’s Simple Steps,
Smart Savings Program.
Conducted and analyzed in-depth interviews with retail merchants and sustainability specialists and
integrated findings with data from interviews with participating and non-participating utilities,
program staff, and data management providers for the retail portfolio evaluation for the Northwest
Energy Efficiency Alliance.
Conducted research, analyzed, and contributed to the final report for NorthWestern Energy’s
commercial and residential portfolio evaluations.
Developed the framework for and led the creation of a catalog of programs serving the multifamily
market across the U.S., with a focus on programs targeting low-income multifamily properties for
California Energy Savings Assistance Program Multifamily Segment Study.
Conducted qualitative interviews and data analysis in support of a process evaluation of California’s
statewide whole-house retrofit program, for a process evaluation of SDG&E and SCG’s Residential
Energy Efficiency programs.
Joe Van Clock, M.A., M.S.
Senior Consultant 1
Education
M.A., Global Communications:
University of Southern California
M.S., Global Media: London School
of Economics, U.K.
B.A., International Affairs and
Hispanic Studies: Lewis & Clark
College, Portland, OR
Doré Mangan Resume | Page 1
Ms. Mangan applies her analytical skills to support market
research for and evaluation of energy programs. With a diverse
background in psychology, behavioral and market research,
program evaluation, data collection, and data analysis, she is
well-versed in all aspects of the research process. She is skilled
in a variety of analytical methods, including cost-benefit
analysis, policy analysis, economic analysis, and advanced
statistical analysis. Prior to obtaining her master’s degree in
public administration, she worked as a data analyst for a
business consulting firm.
Representative Experience
Ms. Mangan leads and contributes to the development of
survey instruments, data collection efforts, and analysis for
residential and commercial program evaluations and market
research studies. Representative projects include:
Conducted in-depth interviews and qualitative analysis and
contributed to reporting for National Grid’s EnergyWise
process evaluation.
Managed the online market intelligence survey through
Qualtrics, and analyzed and reported on in-depth interviews
with key foundational stakeholders to document pilot design and evolution for PG&E’s Step Up and
Power Down (SUPD) Residential and Commercial campaigns.
Conducted and analyzed in-depth interviews of retail merchants and sustainability specialists for
NEEA’s Retail Product Portfolio Evaluation.
Conducted in-depth interviews of program participants, assisted with preliminary data analysis,
conducted other data analysis tasks to support process evaluation, and contributed to report writing
for NYSERDA’S Multifamily Performance Program.
Developed guide, managed data collection, analyzed data, and contributed to reporting for Energy
Trust’s Annual Residential Awareness Survey.
Conducted in-depth interviews and surveys of trade allies, participants, and nonparticipants for
Energy Trust’s Savings Within Reach on-bill repayment pilot.
Developed survey instruments, conducted primary and secondary data analysis, contributed to
report writing, and assisted with project management to support a process evaluation of 41
federally funded, primarily residential, energy efficiency programs for U.S. DOE’s Better Buildings
Neighborhood Program.
Doré Mangan, M.P.A.
Consultant 3
Education
M.P.A., Environmental/Natural
Resource Management: University
of Washington, Evans School of
Public Affairs, Seattle
B.A., Psychology (Animal Behavior):
University of Puget Sound, Tacoma,
WA
B.A., Business: University of Puget
Sound, Tacoma, WA
Jennifer Loomis Resume | Page 1
Dr. Loomis is a social scientist with a background in
environmental sociology, qualitative observational research,
and social inequality. Prior to joining Research Into Action, she
studied the nexus of human-environment interactions, primarily
focusing on the management of fresh water resources and food
systems. She has engaged in qualitative and mixed-method
research in the United States, Peru, and El Salvador. In these
locations, she collaborated with community-based
organizations to ensure solutions were compatible with the
local culture and constraints. She has published articles on
issues relating to disaster recovery and social sustainability and
presented her work at regional and international conferences.
At Research Into Action, she uses her analytical and
methodological expertise to contribute to market research for
and evaluation of clean energy technologies programs and
products.
Representative Experience
Dr. Loomis is involved in the development of interview
instruments, data collection, qualitative and quantitative
analysis, and report writing. Representative projects include:
Conducted analysis of survey and group interview data from manufacturers and distributors of
commercial and residential HVAC equipment for the evaluations of Independent Electricity System
Operator’s business and consumer’s programs.
Interviewed project managers for an in-depth case study of key program strategies, conducted data
analysis in Dedoose, assisted in final process evaluation documentation, and managed database for
the comprehensive evaluation of the U.S. DOE’s Better Buildings Neighborhood Program.
Contributed to literature review on topics relating to effective messaging strategies for African
Americans, Hispanic Americans, and Chinese-Americans to encourage uptake in the HERO (Home
Energy Renovation Opportunity) program for California Center for Sustainable Energy.
Contributed to logic model development, collected baseline market data, provided note-taking
during interviews with multifamily affordable housing governmental actors, contributed to
development of interview guides, and conducted interviews with market actors and market
observers for Los Angeles Department of Water and Power’s market transformation study.
Jennifer Loomis, Ph.D.
Consultant 2
Education
Ph.D., Sociology: Portland State
University, Portland, OR
Dissertation: Activist Doctors:
Explaining Physician Activism in the
Oregon Movement for Single-payer
Healthcare
M.A., Sociology: Colorado State
University, Fort Collins
B.A., Sociology: Colorado State
University, Fort Collins
Jun Suzuki Resume | Page 1
Mr. Suzuki has more than 12 years of experience in evaluation
and market research, including nine years focused on clean
energy technologies. He specializes in market research projects
requiring large sample sizes, complex survey methodologies,
and advanced statistical analysis. He plays an instrumental role
on Research Into Action’s quantitative analysis team by
ensuring that sound survey methodology accompanies all data
collection and storage, and by developing and documenting
syntax for SPSS and MS Excel files. Mr. Suzuki also monitors all
of our CATI, web-based, and email survey procedures using
Qualtrics to maintain our rigorous data collection standards,
and manages the company’s data security policy to protect
sensitive information. He is fluent in Japanese.
Representative Experience
Mr. Suzuki manages and leads the firm’s survey data collection
activities, including programming and fielding online surveys
and coordinating with phone survey fielding houses. These
activities include data management, oversight, and quality
control to ensure that all data, from contact lists to databases,
are secure and maintain data accuracy. Representative projects include:
Managed the process evaluation of Energy Trust’s consumer products program and led the primary
data collection activities that focused on efficient appliance rebates, refrigerator/freezer recycling,
and lighting/showerhead buy-downs.
Conducted online surveys of program participants, program dropouts, and installer/contractors for
fast feedback to the client, analyzed data and contributed to reports for an evaluation of Portland
General Electric’s Solar Payment pilot residential and commercial feed-in-tariff photovoltaic
installation programs.
Estimated the penetration of targeted measures among large industrial customers for NYSERDA’s
New York Energy $mart production efficiency program that targeted large energy savings from
improvements to industrial processes.
Lead the survey design development, data collection, and data security for National Grid’s
EnergyWise Rhode Island Single-Family process evaluation.
Managed the project, which used a conjoint analysis to understand benefits of and barriers to smart
grid enabled technologies pertinent to different residential segments, including low- to moderate-
income for the Smart Grid Consumer Collaborative Empowered Consumer Study.
Jun Suzuki, M.P.A.
Senior Consultant 2
Education
M.P.A., Public Administration:
Monterey Institute of
International Studies, Monterey,
CA
B.S., Business Administration:
Aichi University, Japan
Studied advanced statistics and
survey methodologies from
Portland State University’s Urban
Studies doctoral program
MPE Resumes
James D. Bradford, PhD, PE
Mesa Point Energy, Inc.
jbradford@mesapointenergy.com
303.661.0159 (Office), 720.232.9634 (Mobile)
Dr. Jim Bradford has over 27 years
of experience in the areas of
building energy analysis,
monitoring and energy efficiency.
Jim has worked in many
capacities in the field of building
systems and has extensive
experience in; DSM program
design, implementation and evaluation, Technical
Reference Manual (TRM) development, regulatory
support, HVAC system design, scholarly research in
the building sciences, energy auditing, and
measurement and verification (M&V). Dr. Bradford
regularly authors papers and delivers presentations in
the area of building energy efficiency.
Dr. Bradford’s company, Mesa Point, Inc. provides a
comprensive array of services in the area of building
and industrial systems energy conservation. such as
program design and implemention, detailed
technical analysis, enregy efficiency program design,
implementation and evaluation, litigation support,
M&V, training, project management and program
oversight.
Areas of Expertise
Measurement and Verification: Jim is an industry-
recognized leader in the development of
methods and techniques for the measurement
and verification (M&V) of energy use and savings
for performance contracts and other energy-
related projects. Jim is a co-author of the Federal
Energy Management Program (FEMP)
Measurement and Verification Guidelines, Version
3.0, 2008. Jim is a member of the International
Performance Measurement and Verification
Protocol (IPMVP) technical committee. Jim has
extensive experience in the design and
implementation of M&V plans for commercial
and industrial facilities of all types and sizes.
Recent clients in the area of M&V include Philips
Lighting, Honeywell, Xcel Energy and the Energy
Efficiency Business Coalition.
Education and Licensing
PhD, Civil, Environmental and
Architectural Engineering,
University of Colorado, 1998.
Dissertation: Optimal Supervisory
Control of Cooling Plants Without
Storage.
MS, Civil, Environmental and
Architectural Engineering,
University of Colorado, 1995.
Thesis: Optimal Design of Air
Cooling and Dehumidifying
Systems.
BS, Mechanical Engineering,
Commissioning and Retro-commissioning: Jim has been a technical and
administrative leader in the design and implementation of many commissioning
and RCx projects for new, retrofit and existing building systems. Two example
projects include the commissioning of a 2,000-ton expansion to the central chiller
plant at the University of Colorado Health Sciences Center, and the
commissioning of a new 800-ton chiller and cooling tower retrofit for the Qwest
Advanced Technologies building in Boulder, Colorado.
Energy Program Design and Evaluation: Jim has extensive experience in the
design, implementation and evaluation of energy efficiency, demand side
management (DSM) and energy cost reduction programs. This work is done
utilities, units of government, program administrators, and end users. This
experience also includes regulatory and code compliance support. Some
example clients include Xcel Energy, and Colorado Natural Gas. Additionally, Jim
is on the Louisville, Colorado Sustainability Advisory Board where he helps direct
city sustainability initiatives.
Energy Analysis: Jim has performed advanced research, simulation, monitoring
and evaluation of energy using systems and programs in both acedemic and
professional settings. Jim also has extensive experience in the commerical
energy analysis of buildings. Some examples of his experience include:
developing computer simulation models of building energy consumption;
metering, measurement, and verification of system and measure-specific
energy/demands and savings; analysis of building and process loads;
conducting cost-benefit analyses of energy conservation measures. Example
clients include ESource, HDR, and the University of Colorado.
Engineering: Jim’s engineering experience includes: designing, implementing,
and assessing industrial mechanical systems and HVAC technologies; evaluation
of generation and cogeneration systems; commissioning buildings and controls
systems; monitoring power systems and devices; performing technical feasibility
studies and assessing new technologies. Sample clients include the Town of
Mountain Village (Telluride) Colorado, Eagle County Housing Authority and
Multiprop, Inc.
Gang Tan, Ph.D., P.E., ERP®, LEED® AP
Mesa Point Energy, Inc
tangang@gmail.com
Gang Tan provides advanced
technical support and engineering
analysis for building energy and
EM&V projects. Dr. Tan has over 9
years of experience working as a
program designer, evaluator and
researcher in the United States. Gang
leverages his considerable technical
abilities to provide high quality
engineering analysis and reporting. Prior to joining
Mesa Point Energy, Gang worked for both KEMA and
Nexant as a DSM engineer. Gang is also an Assistant
Professor at the University of Wyoming where the
conducts building energy-related research and
teaches university level engineering courses. Gang
has published over 30 scholarly research paper
Experience and Qualifications
Dr. Tan is an Assistant Professor, Department of Civil
and Architectural Engineering, University of
Wyoming, Laramie, WY. He has a Ph.D. in Building
Technology, Massachusetts Institute of Technology
(MIT), Cambridge, MA, 2005. Mr. Tan provides
engineering modeling support.
Gang has worked on many DSM related projects
such as potential studies, evaluations and tool
design for organizations such as the Energy
Information Administration / U.S. Department of
Energy (DOE) New Jersey Public Service Enterprise
Group’s (PSEG), New York City Green House Gas
Reduction Long-term Plan project, The State of
Florida, National Grid, ComEd Florida Power &
Light’s (FP&L), Pacific Gas and Electric Company’s
(PG&E), USGBC, New York Power Authority (NYPA)
The Georgia Power Company The American
Society of Heating, Refrigerating, and Air-
conditioning Engineers (ASHRAE), New York Energy
$martSM Program, The New York State Energy
Research and Development Authority (NYSERDA),
and MidAmerican Energy.
Education and Licensing
Ph.D. in Building Technology,
Massachusetts Institute of Technology
(MIT), Cambridge, MA, 2005
Certificate in Financial Technology
Option, a certificate program offered
by MIT Sloan School of Management
and EECS Department, Cambridge,
MA, 2004
.
M.S. Department of Thermal
Engineering (HVAC major), Tsinghua
University, Beijing, China
B.S., Department of Thermal
Engineering, Tsinghua University,
Beijing, China
Work History
Mesa Point Energy, Principal
Representative Clients and Projects
An innovative compact cooling system by integrating high efficiency thermoelectric
devices with phase change thermal storage materials, 2011 - present
Research and development of a combined PV/T/TE device for net-zero energy buildings,
2012 - present
Monitoring detailed indoor environment using Proper Otholgonal Decomposition and
Linear Stochastic Estimate (LSE-POD) technique with limited measurements, 2010 -
2013
Project of “Energy End-Use Consumption Estimates for Commercial Buildings” for the
Energy Information Administration (EIA) of U.S. Department of Energy (DOE) based on
2003 Commercial Buildings Energy Consumption Survey (CBECS). Developed
engineering models to estimate energy end-use consumption for creating conditional
demand analysis econometric models, 2007 - 2008
New Jersey Public Service Enterprise Group’s (PSEG) Advanced Metering Infrastructure
(AMI) business and technical consulting project. Conducted audits to the business model
and contributed to the implementation plan development, 2007 - 2008
New York City Green House Gas Reduction Long-term Plan project (30% CO2 reduction in
2017) – Assessment of City Operations and Maintenance Practices, 2008 - 2009
The commercial end-use survey project for Florida state. Surveyed 600 sampled
commercial buildings in Florida, analyzed equipment saturation and energy efficiency
potentials, 2008 - 2009
Energy modeling and technical support to National Grid’s energy efficiency programs in
Massachusetts and Rhode Island, 2007 - 2008
ComEd Company’s Energy Efficiency Program in the great Chicago area technical support
projects, 2008 - 2009
Florida Power & Light’s (FP&L) CO2 calculator development project, to help FP&L promote
renewable energy credit offsets, 2008
California Pacific Gas and Electric Company’s (PG&E) Enhanced Automation program
technical support projects, 2008
USGBC, LEED project review, especially for building energy credits, 2008
LEED related retro-commissioning project for a manufacturing and test center of Carrier
– A United Technologies Company, Charlotte, NC, 2008 - 2009
Two energy conservation market (ECM) assessment projects for the New York Power
Authority (NYPA) and the Georgia Power Company. Analyzed the load profiles, equipment
saturations, and potential opportunities of approximately 60 municipal customers for
NYPA. Developed an ECM model for the Georgia Power Company to quantitatively study
the technical, economic, and achievable energy savings potentials through the proposed
energy efficiency programs under four financial incentive scenarios, 2006 - 2007
Project “Protocol Study for Measuring and Reporting the On-site Performance of
Buildings except Low-rise Residential Buildings” for the American Society of Heating,
Refrigerating, and Air-conditioning Engineers (ASHRAE), by collaborating with University of
Colorado at Boulder, 2006 – 2007
Donna M. Scott, M.S.
Mesa Point, Inc.
dscott@mesapointenergy.com
303.661.0159
Donna Scott has over 20 years of
experience working in the
environmental field. Donna has
expertise in project management,
modeling, policy development,
public education and marketing,
grant writing, as well as program
design, program implementation,
and evaluation for local governments and utility
companies.
Areas of Expertise
Program Design and Implementation: Donna has been
successful launching, directing and managing large
scale environmental programs. She has lead high
performing teams within difficult program deadlines
and multiple objectives.
DSM Program Administration: Donna has experience
administering DSM custom energy efficiency incentive
programs for a number of utilities in Colorado. Activities
include designing marketing materials and outreach
strategies, evaluating program effectiveness, and
technical analysis (i.e. calculating savings and
incentives based on industry standards). Some example
clients include: Colorado Natural Gas, Black Hills Energy
(formerly SourceGas), and Atmos Energy.
Facilitation, Public Process and Marketing: Donna has
substantial experience successfully facilitated a number
of complex stakeholder process for projects with
conflicting objectives, controversial, and/ or technical
issues. She has experience using a number of
stakeholder involvement approaches including:
workshops, third-party facilitation, and technical
advisory round-table discussions. These approaches
were used to foster good communication and gather
input from participants with divergent objectives and to
resolve conflict and identify core values and goals.
Education
M.S. Environemental Science/
Engineering
University of Colorado at Denver,
1989. Thesis: Amonia Modeling of
Wastewater Discharges to Big Dry
Creek, Colorado.
BS, Chemistry and Biology,
Metropolitan State State University,
1984.
Work History
Mesa Point Energy
Louisville, CO
Program Manager, (2015 to Present)
City of Boulder
Boulder, CO
Enviroronmental Program
Supervisor (2006 -2013)
Environmental Program Specialist
(2000-2006)
Scott Dimetrosky, President
Experience
Apex Analytics
President (2011-present)
Design and evaluate energy efficiency and demand response programs for utilities, utility consortiums,
and state agencies. Design services include creating new programs, refining existing programs, and
preparing regulatory filings and testimony. Evaluation services include process, impact, and market
characterization/assessment evaluations of resource acquisition and market transformation programs.
Opinion Dynamics Corporation
Vice-President (2010-2011)
Directed market research and program evaluation of energy efficiency and demand response projects,
including all aspects of project management, including staff/budget management, client relations, and
overall quality control. Responsibilities included developing project methodologies and approaches,
devising questionnaires and discussion guides, selecting and overseeing all data analysis, writing reports,
maintaining client contact, managing project budgets, interpreting results, and presenting results and
recommendations to clients and stakeholder groups. Core projects focused on both process and impact
evaluation of energy efficiency and demand response programs, marketing strategies, communications
efforts, market structure and function, program theories and logic models, and program design and
planning.
The Cadmus Group, Inc. (formerly Quantec, LLC)
Principal (1998-2010)
Founding member and Principal of Quantec, LLC, a demand side management (DSM ) consulting firm,
specializing in the planning and evaluation of energy efficiency and demand response programs. Grew
firm from three employees in 1998 to over 50 employees (and $10 million in revenue) in 2008, at which
time firm was sold to Cadmus. Managed all aspects of DSM projects, including assessing potentials for
energy efficiency and demand response in Arizona, Illinois, and Iowa; designing, preparing regulatory
filing, and testifying for DSM plans for utilities in Colorado and Iowa; leading impact and market
characterization and assessment evaluations for DSM portfolios in California, Illinois Missouri,
Maryland, and Oregon.
Barakat & Chamberlain, Inc.
Associate (1992 to 1997)
Conducted impact and process evaluations for utility- and consortium-sponsored energy efficiency,
market transformation, and demand response programs. Developed and implemented all aspects of
research methodology: designed survey instruments; created sampling plans; and analyzed billing,
metering, and survey data. Evaluated efficacy of regional market transformation programs by assessing
changes in market indicators.
Response Analysis Corporation (Roper Corporation)
Analyst/Programmer (1991 to 1992)
Evaluated the impacts of energy-efficiency programs for low-income households through survey and
billing data analysis. Prepared survey of utility arrearages, bad debt, and service terminations for 1991
LIHEAP Report to Congress. Identified the eligible population for the Weatherization Assistance
Program (WAP) using data from the Residential Energy Consumption Study (RECS).
ICR Survey Research, Inc.
Project Director/Programmer (1987 to 1989)
Supervised all aspects of market research studies, including questionnaire design, sampling, interviewing,
and analysis. Developed tracking database for national demographic census. Programmed software
specifications for EXCEL, a national marketing and opinion poll interviewing 2000 people weekly, using
CFMC CATI system.
Education
M.B.A. Marketing Research & Quantitative Methods, Cornell University. Developed and programmed
logistic regression model for National Institute of Health (NIH) micro simulation model.
B.A. Sociology, May 1987, University of Michigan, Magna Cum Laude. Honors thesis, “Parental Coping
with Childhood Cancer: The Role of Self- Help Groups,” with the Institute for Social Research (ISR).
Selected Conference Papers and Presentations
“Uniform Methods for Upstream Lighting Program Evaluation”, International Energy Program
Evaluation Conference, Chicago, IL August 2013.
“Study It ‘til You’re Sick of It: CFL Research as an Example for Other Efficiency Markets”
International Energy Program Evaluation Conference, Chicago, IL August 2013.
“Technical Reference Manuals: Total Research Madness.” Panel moderator at the AESP National
Conference, Orlando, FL, January 2013.
“Are Savings from Behavioral Programs Ready for TRM Prime Time?” Behavior, Energy, and Climate
Change Conference (BECC), Sacramento, CA, November 2012.
“The Lights They are a Changing: Early Results from EISA 2007” ACEEE Summer Study, Pacific
Grove, CA August 2012
“Can Market Effects from CFL Programs be Measured? Let Us Count the Ways…” International Energy
Program Evaluation Conference, Boston, MA, August 2011.
“Finding the Low Hanging Fruit: Efficiency Supply Curves as a Means to Understanding and Targeting
Efficiency Savings.” AESP National Energy Services Conference, San Diego, CA, January 2009.
“Evaluating the Availability of Market Share Tracking Data for the Residential Sector.” ACEEE Market
Transformation Conference, Washington, D.C., March 2007.
“A Comparison of the Practices Used to Track ENERGY STAR Market Share.” AESP National Energy
Services Conference, Las Vegas, NV, January 2007.
“Assessing Impacts from Energy Efficiency Investments.” United States Association for Energy
Economics/International Association for Energy Economics North American Conference, Denver,
Colorado, September, 2005.
“Leveraging National ENERGY STAR® Survey Data for State-Level Evaluation.” The International
Energy Program Evaluation Conference, New York, NY, August 2005.
Noah Lieb, Associate
Experience
Apex Analytics
Associate (2013-present)
Manage energy efficiency program evaluations and assist in advisory services for utilities, utility
consortiums, and state agencies. Primary tasks include workplan and survey development, survey and
interview administration and analytics, report writing, and client relationship management. Advisory
services includes drafting NTG protocols, measure review and revisions for state TRMs, conducting
primary and secondary market research in support of client’s needs, all with a focus on quantitative
methods and analytics. Noah also directs most of the IT-related tasks for Apex.
The Cadmus Group, Inc. (formerly Quantec, LLC)
Senior Associate (2008-2013)
Directed all database-related aspects of large multi-state and multi-year evaluation efforts, including
California (2008-2009), Wisconsin (2011-2013), EmPower Maryland (2010-2013), Texas (2012-2013)
among others. Also directed multiple online conjoint studies, including drafting survey, programming
online survey tool and developing all findings. Position included direct client outreach with program
managers and IT/database personnel.
Ventyx (formerly Global Energy Decisions)
Associate (2003-2008)
Early staff member of start-up energy database software and consulting group. Began career in database
software group, with exposure to all aspects of power generation industry (down, mid, and upstream).
Managed power plant group, involved overseeing small staff and ensuring data integrity for both database
and geospatial software. Transitioned into consulting group, where focus was on upstream fossil fuels
forecasting and assisting utilities with integrated resource planning. Managed weekly and monthly online
subscription service for coal, oil, and gas market reports.
Education
B.A. International Economics, minor Business, May 2003, University of Colorado, Summa Cum Laude.
Honors thesis, “Migration and Employment in Colorado: An Empirical, Econometric and Geo-Spatial
Approach to Understanding the Determinants and Consequences of Migration and Employment Growth
in Colorado”.
Technical Qualifications
Twelve years of database experience (Oracle, SQL Server, Access) and has managed Web-based
client applications. Software experience includes SQL and PL/SQL, HTML, PHP, ARC-GIS,
Velocity Suite, Sawtooth CBC conjoint software, SAS, and Stata programming.
Publications
Co-Author: NREL. Uniform Methods Project: Residential Lighting Protocol. 2015
Conference Papers and Presentations
“The Oracle Peers through a Window: Using a Delphi Approach to Estimate Impacts of Windows
Programs”, International Energy Program Evaluation Conference, Long Beach, CA August 2015.
“Just How Smart: A look at Energy Trust of Oregon Nest Thermostat Pilot Program” Association Energy
Service Professionals Conference, Orlando, FL February 2015.
Co-Author: Over the Hill? A Look at a Mature Mid-Stream Residential Appliance Program. International
Energy Program Evaluation Conference, Long Beach, CA August 2015
Co-Author: What’s the Point (of Sale)? Program Activity Impacts Efficient Bulb Sales—Proof Across 44
States and Five Years. International Energy Program Evaluation Conference, Long Beach, CA August
2015
Maegan McKee, Analyst
Experience
Apex Analytics
Analyst (2016-present)
As an analyst at Apex Analytics, Maegan has been assisting with the planning of utility demand side
management plans, for Colorado gas utilities specifically, and she supports colleagues in evaluating a
variety of energy efficiency programs for utilities. Other projects have included reviewing submitted
evaluations and TRM reviews. Primary tasks include interview administration to program participants and
corporate entities, quantitative and qualitative data analysis, report writing and presenting key findings to
internal teams and clients. She also provides administrative support where needed.
RRC Associates
Analyst (2013-2015)
As an analyst at RRC Associates, Maegan assisted with market research projects through survey
development and administration, data cleaning and analysis, visualizing data results in innovative
approaches and reporting findings to senior staff. Her projects ranged from evaluating event research and
other customer experience research, specifically for local and national ski resort areas, to developing
plans for small towns and city planning organizations, such as the City of Boulder. She primarily worked
in SPSS, Excel and Tableau in her time at RRC.
Survey Research Management
Project Coordinator (2011-2012)
As a project coordinator at Survey Research Management, Maegan worked on one large-scale data
collection effort for a Department of Education research initiative. Primary tasks included database
development and maintenance, creating protocols and final deliverable templates for the client. She
primarily worked in in Excel and Access.
Education
B.A. Environmental Sciences, minor Mathematical Sciences, May 2007, Lewis & Clark College,
Stormwater Quality Specialist
(1989 - 2000)
Boulder County
Boulder, Colorado
Alternative Transportation
Coordinator (1986 - 1989)
Boards and Commisions
Colorado Water Quality Control
Commission – E.Coli Workgroup,
Permitting Workgroup, Stream
Standards Workgroup (2000-2013)
Broomfield Open Space Board
(2000-2004)
Colorado Bicycling Advisory
Board (1987-1989)
Engineer, 2012 – present
University of Wyoming -
Department of Civil and
Architectural Engineering, Assistant
Professor,, Laramie, WY; 2009 -
present
KEMA Services Inc., Senior
Engineer, Burlington, MA 2007 -
2009
Nexant Inc., Project Engineer, White
Plains, NY, 2005 – 2007
MIT, Postdoctoral Research
Associate, Cambridge, MA
Montana State University, 1986.
Registered professional
engineer since 1992. Current
State licenses:
CA - M34006
CO - 30138
TX - 86573
Work History
Mesa Point Energy
Louisville, CO
President, (2011 to Present)
Nexant, Inc.
Boulder, CO
Senior Vice President (2001 -2011)
Project Manager / Principal (1997 -
2001)
EG&G Idaho, Inc.
Idaho Falls, ID
Senior Engineer (1990 - 1993)
Garvin Engineering
Helena, MT
Staff Engineer (1986 -1989)
designs/approaches (best
practices from elsewhere)
Interviews with Utilities and
PRPA program staff
Interviews with program
implementation contractors
Review of program operations
manuals, flow diagrams,
assignment of responsibilities
Review of data tracking systems
and procedures
Surveys with participants,
nonparticipants, and trade allies
Interviews with Utilities and
PRPA program staff
Review of program marketing
materials and methods
Review of program application
requirements and forms
Research Into Action has conducted multiple evaluations of each of the program types Utilities offers,
with the exception of the Larimer Conservation Corps, for which our relevant experience is not this
specific program type but rather with adult training programs, as well as direct install. Our process
surveys will explore topics both similar across programs – such as awareness, motivations, and barriers,
and topics unique to each program. For example, our evaluations of the Integrated Design program of
the Northwest Energy Efficiency Alliance found it challenging for the program to involve the relevant
decision makers both at the outset – when teams are in formational stages and decision processes may
be unclear, as well as throughout the process, when the relevant people may be present but differ in
their commitment to the planning methods and desired outcome.
The impact evaluation work flow will follow a structured approach as shown in Figure 2-3.
Measure useful life compared to
relevant industry sources
Program cost-effectiveness
assessed from multiple
perspectives
Portfolio cost-effectiveness
assessed from multiple
perspectives
Table 2-3 describes the basic activities we will conduct to estimate gross savings for each program.
Utilities and PRPA implement a wide range of diverse programs and our evaluation will be tailored to
each program. Section 2.3.7 describes our net savings estimation methods.