Loading...
HomeMy WebLinkAboutWORK ORDER - RFP - 7226 SURVEY SERVICES - MISC PUBLIC OPINION SURVEYS (2)EXHIBIT "A" WORK ORDER FORM PURSUANT TO AN AGREEMENT BETWEEN THE CITY OF FORT COLLINS Nv+tunul PSND tau7�n Lein �Pr DATED: ! I Work Order Number. i Purchase Omer Number: Q% Project Title: Ar-Qyrll Ssl;d War SuryPZFP/ 7.2z9) Commencement Date: S-434.I Completion Dam: q/Z 3/l! Maximum Fee:(time and relmbursabie direct coats): �7f9�9lee- Projectoescripnon: G74uct ci;r jual4ulsmlrri WD-0C— S--un2fA � Q f'PS u i:s t papt'a Ce Dom} Professional agrees to perform the services identified above and on the attached forms In accordance with the terms and conditions contained herein and In the Professional Semces Agreement between the parties. In the event of a conflict between or ambiguity in the terms of the Professional Services Agreement and this work order (including the attached fors) the Professional Services Agreement shall control. The attached forms consisting of ExNbits consisting of_ U pages are hereby accepted and Incorporated herein, by this remrence, and Notice to Proceed Is hereby given, Professional i By: i SONy/Y (yyT/NLfL/ Se Nic rt p5S4!/NlF Dam: 5/ag/zou City of Fort Collins Submitted Byz Pm act Manager Date: S.; Cx 1 I Reviewed by: Approved by: Approved by: Director of Purchasing & Risk Management (if over seum.) Data: P T P J U O V N O O O O O O O O O O O o O O O O O O O b N O � ^ P N O Sl m o � o � o ,on a^ o 0 0 0 0 0 0 0 0 0 0 0 o n o a W '^ � _ 6 = b � V `a O o d d o a m a o 0 6 d e c p O b a m � n o x s a a 9 `o_ o>> n` S>> v o v jW V1 a v1 a O O O N g 0 a g a s E fL d w m v a= ni u .c Y j✓ N N N m b P b Y b O M N b N P M b d a\ b J V W O c o � c o 0 0 M w o 0 0 0 0 0 0 0 0 0 0 0 o N o Ci O p O O O O O O O O O m � Q 2 R O O O O O O b O O O O O O O O l0 N N I� O O O O b u a 'o c ° ¢ o n b9 � R P x a v m d d a v c o 0 i o o n° i o v ao s c c 2 c z « c d d ¢ L r v z o a t N T q d O d O ° O B r _ �.' �'n jN 9v G gyp[ u d y 1/dl J W O d d O O O d d 1d' N E At .� � �o a 00 � | $ ° ,;!!!!!!2!`:! ! y -- � �!!! ' I `/ ��!!!!!!!!!«! !!|�!!! f)|. .. !, !_I!!l�..:; ±|!�! _ { !f|.°, ���!\\!!!!!!�� �.. § , ' � / !:_ !�! ,. \ ., :!!!!#l=;;,�! !!! �-� ) } ) )))\\\)|)� � !!� !!!|; i!!| if ! (ji�)||§|�)) � � _- /\ (l,,l•!!!!!!}) | \ ) \ r r City of Fort Collins Air Quality and Solid Waste Survey Survey 1: 2011 Air Qualify/Solid Waste Survey Task A: Project Initiation It will be important to start this project with clear expectations, explicated barriers and incentives that could motivate potential respondents and understood resources, including time and personnel. We recommend, and note that the RFP requests, that we have a project initiation meeting to discuss the overall scope of the project, and to begin refining the questions for the survey This meeting also will serve to spell out the desired communication processes between NBC and the City of Fort Collins for the duration of this project (e.g., points of contact, types and frequency of updates to be provided throughout the project, intermediate timelines, etc.). Shortly after this meeting, we will produce a timeline for the City that indicates project milestones and tasks, with dates. We have provided a preliminary timeline in this proposal, but would work with the City to determine what the important deadlines are, and craft a project schedule to meet the City's goals. Task B: Choosing a Survey Administration Method Based on NRC's experience and that of other leading researchers, we have created a table that compares the strengths of various survey administration modes. A previous version of this table was included in the book Citizen Surveys; How to do them, how to use them, what they mean (lCMA, 2000). Comparison of administration methods for general population surveys" have Expense per completed survey Moderately expensive Mail Moderately inexpensive Web Moderately inexpensive* Speed ofadministration Moderately fast Moderately slow Moderately fast Typical response rate Fair Excellent Poor Ability to obtaining candid responses Fair Excellent Excellent Elimination of interviewer bias Fair Excellent Excellent Ability to get at in-depth topics I Good Good Fair Use of visual aids Poor Good Good Enforcement of question order Excellent Poor Good Inclusion of illiterate respondents Good Poor Poor Inclusion of young adults Poor Poor Fair Inclusion of respondents of lower socioeconomic status Fair Fair Poor Specifying sub -geographic locations (districts etc) Poor Excellent Poor ® Authors/ Research center, Inc., 2011 Asa stand-alone data collection method with recruitment We mailed postcard, roe response rote for Web is surprisingly low The large mailing size required to achieve the desired number of completes significantly increases the cost ofdals collection We Web. Web is more economical wham paired with a mailed survey 1permiding responses via either mehod) or when a complete list of email addresses is available for the population ofinterest as often is Me case for members of organizations. Some of the advantages of a mailed survey, especially for sampling within geographic areas, include: • Mailed surveys garner higher response rates than phone surveys. • Mailed surveys are more efficient for sampling within geographic boundaries. (Precise household locations can be identified in advance.) National Research Center, Inc. r City of Fort Collins Air Quality and Solid Waste Survey • Mailed surveys have less "coverage" error than phone surveys. (No loss of cell -phone only households.) • Mailed surveys are less expensive than phone surveys per completed interview. • Mailed surveys elicit more candid responses than do phone surveys. In particular this is true for questions that may be considered "sensitive" like crime victimization or participation in socially un/desirable behaviors (such as making trips by driving alone), but even for service evaluation questions. • Mailed surveys permit more thoughtful responses than phone surveys. • Mailed surveys will not disturb residents during their dinner hour (or any other time). Advantages of phone surveys include: • Faster completion of the study. (Time is not needed to print and mail the surveys, or wait for surveys to be returned by mail.) • Inclusion of residents who do not have facility with a written language. • Greater credibility from stakeholders used to receiving poll data from telephone data collection. For this project, we would recommend mailed survey data collection because of it's advantages and because it was the method used in the previous implementation in 2007. This will allow comparisons over time without worries about differences due to mode effect Mailed surveys have a lower cost while allowing for both more respondents and a longer survey; they have higher response rates which reduces concerns about bias; they facilitate geographic comparisons (if desired), they permit more efficient oversampling of populations who tend to respond at lower rates (younger and lower income residents who are more likely to be in multi -unit housing) and they engender more candid responses. The tasks outlined below and our cost estimate assumes a mailed survey methodology; we are happy to provide information on conducting a telephone survey, if desired. Task 1: Development of Survey Questions It is our experience that, to achieve the best results, time must be invested up -front to craft the right survey instrument and to tie survey design to the overall goals and objectives of the project. We regard the questionnaire development process as an iterative one, with drafts being reviewed and revised by a project oversight committee or point person and our team. NRC will work with the City to develop a comprehensive questionnaire that covers the objectives of the study. The obvious starting point will be the 2007 survey instrument, but we have conducted a number of air quality and solid waste surveys and can draw on our own library for example surveys. If desired, we can also look for other similar questionnaires used for other jurisdictions. We will work with the City so that we fully understand the use to which the data collected from this effort will be put. NRC will ensure that each question is easily comprehended by respondents and contributes to the overall analysis goals. For a mailed survey, we would recommend that the survey be no longer than 5 pages in length, and this is the assumption we have used in crafting our cost estimate. Task 2: Sampling Selecting survey recipients In survey research, it is desirable for every household (or every adult person living in a household) to have a known and equal chance of participating in the survey. A sample of all households is selected to be given an opportunity to participate. Sampling is based on probability theory; the idea National Research Center, Inc. City of Fort Collins Air Quality and Solid Waste Survey that if respondents are chosen randomly from the larger population, the results from that random sample will be very close to what we would get by interviewing every member of the population. The challenge is to find a list of all households from which survey recipients can be randomly chosen. For a mailed survey, mailing lists exist from the United States Postal Office which include all households with a mailing address. All households located within the boundaries that define the City of Fort Collins, or whatever boundaries the City determines are appropriate for this study, would be eligible for the survey. We will geocode the location of each address to assure it is within the study boundaries (as well as any desired subareas within the overall boundaries, if desired). We will use systematic sampling from the list of eligible households to select the sample of households, Systematic sampling is a procedure whereby a complete list of all possible addresses is culled, selecting every Nth one until the appropriate number of households is sampled. Not only will NBC scientifically and randomly sample households, but we also will select without bias the household member to participate. In cases where more than one household member is 18 or older, a procedure called "the birthday method" will be used to select the survey respondent without bias. The birthday method requests that the respondent be the eligible adult in the household who most recently had a birthday (irrespective of the year of birth). This methodology helps ensure the attitudes expressed by our respondent "sample" closely approximate the attitudes of all adult residents living in Fort Collins. Without this step in survey research, it is likely that results will be biased towards those who are more sedentary and those without jobs (who may have different opinions about some services). Choosing the number of completed surveys ("sample size") The relationship between sample size and precision of estimates (the 95% confidence interval or margin of error) is shown in the adjacent table. The margin of error or 95% confidence interval is determined solely by the number of completed surveys, and not the size of the population from which the sample is drawn, as long as the "population' size is greater than about 4,000. The table can be interpreted as follows: If 61% of 400 respondents said they would support a mandatory vehicle emissions program in Fort Collins even though it is not Federally required, we can be confident that, had we gotten ratings from all residents, somewhere between 56%and 66%would haves rtd dt hi Sample SW Margin of Error (95% Confidence Interval) 100 ±9,8% zoo ±6.9% 400 ±4.9% Goo ±4.0% Boo ±3.5% a,zoo ±a.8% z,000 ia.a% uppo a man a ory ve c e emissions. In the 2007 survey, surveys were sent to 1,500 households, from which 577 completed surveys were returned. As response rates have been declining over time, we would recommend that surveys be sent to 1,600 households, for a likely final completed sample size between 550 and 600, which would provide a 95% confidence interval of plus or minus 4 percent. Task 3: Survey Administration Below is outlined our survey administration procedures. We encourage our clients to inform the public about the survey through local newspapers, newsletters, community meetings and any other appropriate media. The publicity provides an opportunity to inform them about the purposes of the survey and the uses - for programming and policy- to which it will be put. The publicity, furthermore, will assist the survey process by increasing willingness of potential respondents to participate. National Research Center, Inc. City of Fort Collins Air Quality and Solid Waste Survey Mailed Survey Data Collection As described above, a sample of addresses from within Fort Collins will be selected to be part of the survey. This database of addresses will be processed for certification and verification. NRC vendors use CASS' /NCOA software that relies on the USPS National Directory information to verify and standardize the address elements and assign each a complete, nine -digit zip code where possible. In addition, the software will sort and barcode the addresses, allowing significant postage discounts. We propose mailing all materials pre-sorted first class. This ensures a fast delivery time of the materials, but provides an opportunity for a discount on the full first-class postage rates. Our mailhouse will oversee the printing and preparation of the prenotification postcards, survey packets and reminder postcards. As a part of the quality control process, proofs of the survey materials are reviewed by NRC staff before the final job is printed. We will include a point person at Fort Collins as a "seed" in the mailing list, so they will receive the mailing materials at the same time as the sampled recipients. We propose three contacts with each sampled household: 1) A prenotification announcement, informing the household members that they have been selected to participate in the community survey will be sent to each household. This announcement should be scheduled to arrive about a week before the survey packet. 2) Approximately one week after mailing the prenotification, each household would be mailed a survey containing a cover letter (ideally signed by the mayor) enlisting participation. The packet would also contain a postage paid return envelope that would send all completed surveys directly for data entry. 3) A reminder letter and survey, will be scheduled to arrive 1 week after the first survey. The second cover letter asks those who have not completed the survey to do so and those who have already done so to refrain from turning in another survey. We mail the survey twice because anonymity is promised in the cover letter to enhance the likelihood of honest responses. We take this implicit contract with respondents as a serious principle of the survey trade, which, if violated, harms the survey research industry no less than the client or respondent. We can also check for duplicates if we include a survey code for later mapping of responses. Generally we find only one or two duplicates in our responses. When the survey administration mode has been chosen, a sample size selected and the survey materials finalized, NRC will produce a memo for the City outlining the survey protocols and containing the final survey materials. Task 4: Data Tabulation Survey processing Mailed surveys will be returned to an NRC affiliate directly via postage -paid business reply envelopes where staff will assign a unique identification number to each survey. Additionally, each survey will be reviewed and "cleaned' as necessary. For example, a question may have asked a respondent to pick two items out of a list of five, but the respondent checked three; NRC staff would choose randomly two of the three selected items to be coded in the dataset. We have found that very little cleaning is needed on most surveys due to our expertise in question construction and survey formatting. However, extensive cleaning tends to be needed in surveys containing complicated question structure or skip patterns. Once all surveys have been assigned a unique identification number, they will be entered into an electronic dataset This dataset will be subject to a data entry protocol of "key and verify," in which survey data are entered twice into an electronic dataset and then compared. Discrepancies are National Research Center, Inc. City of Fort Collins Air Quality and Solid Waste Survey evaluated against the original survey form and corrected. Range checks as well as other forms of quality control will also be performed. Weighting the Data The last step in preparing the data for analysis is to weight the data to reflect the demographic profile of the Fort Collins population. Weighting is an important method to adjust for potential non - response bias. In general, residents with certain characteristics (for example: those who are younger or rent their homes) are less likely to participate in surveying, whatever the data collection mode. Weighting allows us to look at the demographic profile of residents who returned the survey compared with the US Census profile. We consider these disparities along with others and accordingly increase or decrease the weight of each respondent to mimic as closely as possible the population demographics. The weighting variables to be considered will be all those demographics included on the survey, as well as any geographic variable used for sampling. NRC has extensive experience with complex weighting schemes required with stratification. if the 2007 survey data are available in a useable format, we will weight those data as well to enable meaningful comparison to the most recent past survey. Basics of Data Analysis and Reporting It is paramount that the result of this study be more than a nice looking report that sits on a shelf, and as such we will work with the City of Fort Collins to ensure that we ask the right questions to feed into an actionable analysis plan. For quantitative analysis, we rely on the Statistical Package for the Social Sciences (SPSS). We believe that analysis must be replicable and leave a clear path. To this end, we keep every label and command run in SPSS in a syntax file available for audit and re -running as necessary. We also have trained clients on SPSS analysis and, for small recurring analyses, how to use Microsofts Excel. We will code open-ended responses using both an emergent approach, where themes are revealed through the analysis, combined with a deductive approach, where a scheme or codes are predetermined and applied to the data. Techniques include word counts, content analysis, comparative analysis, componential analysis and taxonomic analysis, among others and codes are applied based on the unit of analysis. Our qualitative analysis of verbatim responses will be done using Microsoft® Access, Microsoft® Excel or QSR qualitative software, depending on the types and complexity of the open-ended questions. The data and report will undergo a thorough quality assurance review. We will audit the original data files, our syntax/analysis files, compare automatically generated output to the formatted output in the report and data check all numbers and text prior to submitting the report We will provide basic descriptive statistics such as frequencies and means summarizing the responses to the survey questions. If it makes sense to create scales or indices that summarize information from a number of survey items, we can use factor analysis and other psychometric testing to create valid factors. Additionally, NRC staff often have employed other advanced data analysis techniques to transportation survey findings to determine what the key drivers of respondent satisfactions are (using linear or logistic regression to demonstrate the factors that are most important in respondent satisfaction), cluster analysis (market segmentation) to identify groupings of respondents whose members share characteristics but whose clusters differ in important way. For example, in a study we did for the Regional Air Quality Council on ground level ozone pollution, we found there were four clusters or segments of the population in regards to attitudes towards and behaviors to ameliorate ground level ozone, ranging from the "Negative National Research Center, Inc. r r City of Fort Collins Air Quality and Solid Waste Survey Normans"(comprising 12%of the population) who did not feel ground level ozone was a problem and were unlikely to make any behavior changes to decrease ground level ozone to'"rhe Eager' (comprising 22% of the population) who reported already making lifestyle changes to reduce ground -level ozone pollution and who were ready to be influenced to change behavior. Upon completion of the data analysis, NRC can provide an electronic copy of the dataset to the City of Fort Collins in a variety of formats. Typically, clients prefer to receive the data as an Excel worksheet, but we can also provide an ASCII dataset, an Access dataset, or an SPSS dataset. We will provide information about the layout and value labels of the data so that you will have no trouble accessing the raw data. As a note of caution: while we can provide the dataset in Excel, including the appropriate weight for each case, applying the weights in Excel is quite difficult. SPSS or another statistical application may be required to apply the statistical weights and derive the identical results to what would be displayed in the report of results. However, we keep all the data files on our own server, and will be happy to run additional crosstabulations for you (at no additional cost) as questions come up as you use and disseminate the report. Task 5: Reporting the Results Report of Results NRC understands that most readers want to understand the broader patterns and interpretations of data, so our professionally designed graphics are engaging and pop with clear information. Our reports are comprehensive and include technical and detailed numbers and information, but not at the expense of the day-to-day usefulness of the documents. We are careful about our communication. You will not need a degree in statistics to understand the survey results. All the technical details are in appendices for those who wish to read them (and so that the survey can be repeated by any vendor you choose), but we make sure the report body tells the story of the survey results in a stylish, colorful, informative way. For further ease, we also include an executive summary that gives a quick overview of results while highlighting key findings. We will prepare the report in Microsoft® Word (as well as convert documents into a PDF format). Drafts of the report will be provided to the City and the final report will incorporate any comments we receive from you. We will meet with you to review the draft report, and answer any questions or comments you have. We will provide an electronic copy of the report in Microsoft Office Word® format along with a version in PDF format (readable by Adobe Acrobat). We will also provide a hard copy of the report. Recommendations In order to best use the resources available for this project, we are proposing not to provide recommendations, but to devote the resources to conducting the research and analyzing the results. Our expertise is in research methods, analysis and facilitation. The best recommendations are likely to come from the City's own review and discussion of the results. For an extra cost, we could provide a half day "next steps" workshop, facilitated by Dr. Miller. In this workshop, he will work with up to 20 key staff, elected officials and/or resident decision -leaders to describe survey results and to plan for next steps that the City of Fort Collins should take. The workshop will result in identification of key findings and an action plan for moving forward with each. The cost of such a workshop is $1,800. National Research Center, Inc. 4 City of Fort Collins Air Quality and Solid Waste Survey Consulting and Continued Support We know that an effective report will be reviewed and discussed by many people in Fort Collins after we have completed the survey projects, and that new questions may arise. We commit to supporting these discussions by offering crosstabulations and other additional analyses of results when needed. And, though we will not abandon you with your data, we always will provide you with the electronic dataset of your responses for your future reference and use. National Research Center, Inc. r r City of Fort Collins Air Quality and Solid Waste Survey Survey 1: 2011 Air Quality/Solid Waste Survey Task A: Project Initiation [twill be important to start this project with clear expectations, explicated harriers and incentives that could motivate potential respondents and understood resources, including time and personnel. We recommend, and note that the RFP requests, that we have a project initiation meeting to discuss the overall scope of the project, and to begin refining the questions for the survey This meeting also will serve to spell out the desired communication processes between NRC and the City of Fort Collins for the duration of this project (e.g., points of contact, types and frequency of updates to be provided throughout the project, intermediate timelines, etc.). Shortly after this meeting we will produce a timeline for the City that indicates project milestones and tasks, with dates. We have provided a preliminary timeline in this proposal, but would work with the City to determine what the important deadlines are, and craft a project schedule to meet the Cityrs goals. Task B: Choosing a Survey Administration Method Based on NRC's experience and that of other leading researchers, we have created a table that compares the strengths of various survey administration modes. A previous version of this table was included in the book Citizen Surveys; How to do them, how to use them, what they mean (fCMA, 2000). Comparison of administration methods for general population surveys" Expense per completed survey Moderately expensive Moderately Inexpensive Moderately •Inexpensive* Speed of administration Moderately fast Moderately slow Moderately fast Typical response rate Fair Excellent Poor Ability to obtaining candid responses Fair Excellent Excellent Elimination of Interviewer bias Fair Excellent Excellent Ability to get at in-depth topics Good Good Fair Use of visual aids Poor I Good Good Enforcement of question order Excellent I Poor Good Inclusion of illiterate respondents Good Poor Poor Inclusion of young adults Poor Poor Fair Inclusion of respondents of lower socioeconomic status I Fair Fair Poor Specifying sub -geographic locations (districts etc) Poor Excellent Poor ®Nalsmal ReseamN center, Inc, 2011 As a Stan -alone data collecdan method will mcrultment vie mailed postcard, the response late M web is surprlsirply b .. The Mtge mailing slm requlmdfo achieve the deslmd number arcompletes significantly imaeases the cost or date crol iftin We Web. Web u more economical when pelretl with a maim survey (parmittbp resporwes Wa eifhermeMod) or when a complete fist otemall addresses is available torthe population of mmmst as 'tan is the case farmeessim aorgansusions. Some of the advantages of a mailed survey, especially for sampling within geographic areas, include: • Mailed surveys garner higher response rates than phone surveys. • Mailed surveys are more efficient for sampling within geographic boundaries. (Precise household locations can be identified in advance.) National Research Center, Inc. City of Fort Collins Air Quality and Solid Waste Survey Proposed Time line This timeline has a 4 month project length; this can be compressed or expanded, if needed. Task Contract awarded Finalize prenotification Date .................. I...... May 23, 2011 1 week of contract signing ...................................... week 3 Other survey materials and questionnaire finalized.....................................................................................week 4 Mail prenotification postcard.................................................................................................................................. week4 Mail first wave survey................................................................................................................................................week 5 Mail second wave of surveys................................................................................................................................... week 6 Allow surveys to be returned.............................................................................................................................. week 10 Electronicdata entry ..............................................................................................................................through week 11 Data analysis and report writing.......................................................................................................through week 14 Draft report for review by City............................................................................................................................ week 14 Review by City, meeting with NRC...................................................................................................through week 16 Final report and letter delivered........................................................................................................................ week 17 National Research Center, Inc. t City of Fort Collins Air Quality and Solid Waste Survey • Mailed surveys have less "coverage" error than phone surveys. (No loss of cell -phone only households.) • Mailed surveys are less expensive than phone surveys per completed interview. • Mailed surveys elicit more candid responses than do phone surveys. In particular this is true for questions that may be considered "sensitive" like crime victimization or participation in socially un/desirable behaviors (such as making trips by driving alone), but even for service evaluation questions. • Mailed surveys permit more thoughtful responses than phone surveys. • Mailed surveys will not disturb residents during their dinner hour (or any other time). Advantages of phone surveys include: • Faster completion of the study. (Time is not needed to print and mail the surveys, or wait for surveys to be returned by mail.) • Inclusion of residents who do not have facility with a written language. • Greater credibility from stakeholders used to receiving poll data from telephone data collection. For this project, we would recommend mailed survey data collection because of it's advantages and because it was the method used in the previous implementation in 2007. This will allow comparisons over time without worries about differences due to mode effect. Mailed surveys have a lower cost while allowing for both more respondents and a longer survey; they have higher response rates which reduces concerns about bias; they facilitate geographic comparisons (if desired), they permit more efficient oversampling of populations who tend to respond at lower rates (younger and lower income residents who are more likely to be in multi -unit housing) and they engender more candid responses. The tasks outlined below and our cost estimate assumes a mailed survey methodology; we are happy to provide information on conducting a telephone survey, if desired. Task 1: Development of Survey Questions It is our experience that, to achieve the best results, time must be invested up -front to craft the right survey instrument and to tie survey design to the overall goals and objectives of the project We regard the questionnaire development process as an iterative one, with drafts being reviewed and revised by a project oversight committee or point person and our team. NRC will work with the City to develop a comprehensive questionnaire that covers the objectives of the study. The obvious starting point will be the 2007 survey instrument, but we have conducted a number of air quality and solid waste surveys and can draw on our own library for example surveys. If desired, we can also look for other similar questionnaires used for other jurisdictions. We will work with the City so that we fully understand the use to which the data collected from this effort will be put. NRC will ensure that each question is easily comprehended by respondents and contributes to the overall analysis goals. For a mailed survey, we would recommend that the survey be no longer than 5 pages in length, and this is the assumption we have used in crafting our cost estimate. Task 2: Sampling Selecting survey recipients In survey research, it is desirable for every household (or every adult person living in a household) to have a known and equal chance of participating in the survey. A sample of all households is selected to be given an opportunity to participate. Sampling is based on probability theory; the idea National Research Center, Inc. s City of Fort Collins Air Quality and Solid Waste Survey that if respondents are chosen randomly from the larger population, the results from that random sample will be very close to what we would get by interviewing every member of the population. The challenge is to find a list of all households from which survey recipients can be randomly chosen. For a mailed survey, mailing lists exist from the United States Postal Office which include all households with a mailing address. All households located within the boundaries that define the City of Fort Collins, or whatever boundaries the City determines are appropriate for this study, would be eligible for the survey. We Will geocode the location of each address to assure it is within the study boundaries (as well as any desired subareas within the overall boundaries, if desired). We will use systematic sampling from the list of eligible households to select the sample of households. Systematic sampling is a procedure whereby a complete list of all possible addresses is culled, selecting every Nth one until the appropriate number of households is sampled. Not only will NRC scientifically and randomly sample households, but we also will select without bias the household member to participate. In cases where more than one household member is 18 or older, a procedure called "the birthday method" will be used to select the survey respondent without bias. The birthday method requests that the respondent be the eligible adult in the household who most recently had a birthday (irrespective of the year of birth). This methodology helps ensure the attitudes expressed by our respondent "sample" closely approximate the attitudes of all adult residents living in Fort Collins. Without this step in survey research, it is likely that results will be biased towards those who are more sedentary and those without jobs (who may have different opinions about some services). Choosing the number of completed surveys ("sample size") The relationship between sample size and precision of estimates (the 95%confidence interval or margin of error) is shown in the adjacent table. The margin of error or 95%confidence interval is determined solely by the number of completed surveys, and not the size of the population from which the sample is drawn, as long as the "population" size is greater than about 4,000. The table can be interpreted as follows: If 61%of 400 respondents said they would support a mandatory vehicle emissions program in Fort Collins even though it is not Federally required, we can be confident that, had we gotten ratings from all residents, somewhere between 56%and 66%would have supported mandatory vehicle emissions. In the 2007 survey, surveys were sent to 1,500 households, from which 577 completed surveys were returned. As response rates have been declining over time, we would recommend that surveys be sent to 1,600 households, for a likely final completed sample size between 550 and 600, which would provide a 95%conndence interval of plus or minus 4 percent. Task 3: Survey Administration Below is outlined our survey administration procedures. We encourage our clients to inform the public about the survey through local newspapers, newsletters, community meetings and any other appropriate media. The publicity provides an opportunity to inform them about the purposes of the survey and the uses - for programming and policy- to which it will be put The publicity, furthermore, will assist the survey process by increasing willingness of potential respondents to participate. National Research Center. Inc. t City of Fort Collins Air Quality and Solid Waste Survey Mailed Survey Do to Collection As described above, a sample of addresses from within Fort Collins will be selected to be part of the survey. This database of addresses will be processed for certification and verification. NRC vendors use CASS-/NCOA software that relies on the USPS National Directory information to verify and standardize the address elements and assign each a complete, nine -digit zip code where possible. In addition, the software will sort and barcode the addresses, allowing significant postage discounts. We propose mailing all materials pre-sorted first class. This ensures a fast delivery time ofthe materials, but provides an opportunity for a discount on the full first-class postage rates. Our mailhouse will oversee the printing and preparation of the prenotification postcards, survey packets and reminder postcards. As a part of the quality control process, proofs of the survey materials are reviewed by NRC staff before the final job is printed We will include a point person at Fort Collins as a "seed" in the mailing list, so they will receive the mailing materials at the same time as the sampled recipients. We propose three contacts with each sampled household: 1) A prenotification announcement, informing the household members that they have been selected to participate in the community survey will be sent to each household. This announcement should be scheduled to arrive about a week before the survey packet 2) Approximately one week after mailing the prenotification, each household would be mailed a survey containing a cover letter (ideally signed by the mayor) enlisting participation. The packet would also contain a postage paid return envelope that would send all completed surveys directly for data entry. 3) A reminder letter and survey, will be scheduled to arrive 1 week after the first survey. The second cover letter asks those who have not completed the survey to do so and those who have already done so to refrain from turning in another survey. We mail the survey twice because anonymity is promised in the cover letter to enhance the likelihood of honest responses. We take this implicit contract with respondents as a serious principle of the survey trade, which, if violated, harms the survey research industry no less than the client or respondent We can also check for duplicates if we include a survey code for later mapping of responses. Generally we find only one or two duplicates in our responses. When the survey administration mode has been chosen, a sample size selected, and the survey materials finalized, NRC will produce a memo for the City outlining the survey protocols and containing the final survey materials. Task 4: Data Tabulation Survey processing Mailed surveys will be returned to an NRC affiliate directly via postage -paid business reply envelopes where staff will assign a unique identification number to each survey. Additionally;Each survey will be reviewed and "cleaned" as necessary. For example, a question may have asked a respondent to pick two items out of a list of five, but the respondent checked three; NRC staff would choose randomly two of the three selected items to be coded in the dataset. We have found that very little cleaning is needed on most surveys due to our expertise in question construction and survey formatting. However, extensive cleaning tends to be needed in surveys containing complicated question structure or skip patterns. Once all surveys have been assigned a unique identification number, they will be entered into an electronic dataset This dataset will be subject to a data entry protocol of"key and verify," in which survey data are entered twice into an electronic dataset and then compared. Discrepancies are Notlorwl Research Center, Inc. t t City of Fort Collins Air Quality and Solid Waste Survey evaluated against the original survey form and corrected. Range checks as well as other forms of quality control will also be performed. Weighting the Data The last step in preparing the data for analysis is to weight the data to reflect the demographic profile of the Fort Collins population. Weighting is an important method to adjust for potential non - response bias. In general, residents with certain characteristics (for example: those who are younger or rent their homes) are less likelyto participate in surveying, whatever the data collection mode. Weighting allows us to look at the demographic profile of residents who returned the survey compared with the US Census profile. We consider these disparities along with others and accordingly increase or decrease the weight of each respondent to mimic as closely as possible the population demographics. The weighting variables to be considered will be all those demographics included on the survey, as well as any geographic variable used for sampling. NBC has extensive experience with complex weighting schemes required with stratification. If the 2007 survey data are available in a useable format, we will weight those data as well to enable meaningful comparison to the most recent past survey. Basics of Data Analysis and Reporting It is paramount that the result of this study be more than a nice looking report that sits on a shelf, and as such we will work with the City of Fort Collins to ensure that we ask the right questions to feed into an actionable analysis plan. For quantitative analysis, we rely on the Statistical Package for the Social Sciences (SPSS). We believe that analysis must be replicable and leave a clear path. To this end, we keep every label and command run in SPSS In a syntax file available for audit and re -running, as necessary. We also have trained clients on SPSS analysis and, for small recurring analyses, how to use Microsoft® Excel. We will code open-ended responses using both an emergent approach, where themes are revealed through the analysis, combined with a deductive approach, where a scheme or codes are predetermined and applied to the data. Techniques include word counts, content analysis, comparative analysis, componential analysis and taxonomic analysis, among others and codes are applied based on the unit of analysis. Our qualitative analysis of verbatim responses will be done using Microsoft® Access, Microsoft® Excel or QSR qualitative software, depending on the types and complexity of the open-ended questions. The data and report will undergo a thorough quality assurance review. We will audit the original data files, our syntax/analysis files, compare automatically generated output to the formatted output in the report and data check all numbers and text prior to submitting the report. We will provide basic descriptive statistics such as frequencies and means summarizing the responses to the survey questions. If it makes sense to create scales or indices that summarize information from a number of survey items, we can use factor analysis and other psychometric testing to create valid factors. Additionally, NBC staff often have employed other advanced data analysis techniques to transportation survey findings to determine what the key drivers of respondent satisfactions are (using linear or logistic regression to demonstrate the factors that are most important in respondent satisfaction), cluster analysis (market segmentation) to identify groupings of respondents whose members share characteristics but whose clusters differ in important way. For example, in a study we did for the Regional Air Quality Council on ground level ozone pollution, we found there were four clusters or segments of the population in regards to attitudes towards and behaviors to ameliorate ground level ozone, ranging from the "Negative National Research Center, Inc. r r City of Fort Collins Air Quality and Solid Waste Survey Normans"(comprising 12% of the population) who did not feel ground level ozone was a problem and were unlikely to make any behavior changes to decrease ground level ozone to "The Eager" (comprising 22%of the population) who reported already making lifestyle changes to reduce ground -level ozone pollution and who were ready to be influenced to change behavior, Upon completion of the data analysis, NRC can provide an electronic copy of the dataset to the City of Fort Collins in a variety of formats. Typically, clients prefer to receive the data as an Excel worksheet, but we can also provide an ASCII dataset, an Access dataset, or an SPSS dataset. We will provide information about the layout and value labels of the data so that you will have no trouble accessing the raw data. As a note of caution: while we can provide the dataset in Excel, including the appropriate weight for each case, applying the weights in Excel is quite difficult SPSS or another statistical application may be required to apply the statistical weights and derive the identical results to what would be displayed in the report of results. However, we keep all the data files on our own server, and will be happy to run additional crosstabulations for you (at no additional cost) as questions come up as you use and disseminate the report. Task 5: Reporting the Results Report of Results NRC understands that most readers want to understand the broader patterns and interpretations of data, so our professionally designed graphics are engaging and pop with clear information, Our reports are comprehensive and include technical and detailed numbers and information, but not at the expense of the day-to-day usefulness of the documents. We are careful about our communication. You will not need a degree in statistics to understand the survey results. All the technical details are in appendices for those who wish to read them (and so that the survey can be repeated by any vendor you choose), but we make sure the report body tells the story of the survey results in a stylish, colorful, informative way. For further ease, we also include an executive summary that gives a quick overview of results while highlighting key findings. We will prepare the report in Microsoft® Word (as well as convert documents into a PDF format). Drafts of the report will be provided to the City and the final report will incorporate any comments we receive from you. We will meet with you to review the draft report, and answer any questions or comments you have. We will provide an electronic copy of the report in Microsoft Office Word® format along with a version in PDF format (readable by Adobe Acrobat). We will also provide a hard copy of the report, Recommendations In order to best use the resources available for this project, we are proposing not to provide recommendations, but to devote the resources to conducting the research and analyzing the results. Our expertise is in research methods, analysis and facilitation. The best recommendations are likely to come from the City's own review and discussion of the results. For an extra cost, we could provide a half day "next steps" workshop, facilitated by Dr. Miller. In this workshop, he will work with up to 20 key staff, elected officials and/or resident decision -leaders to describe survey results and to plan for next steps that the City of Fort Collins should take. The workshop will result in identification of key findings and an action plan for moving forward with each. The cost of such a workshop is $1,800. National Research Center, Inc. MMMMMr r �Clty.f Fort Collins Air Quality and Solid Waste Survey Consulting and Continued Support We know that an effective report will be reviewed and discussed by many people in Fort Collins after we have completed the survey projects, and that new questions may arise. We commit to supporting these discussions by offering crosstabulations and other additional analyses of results when needed. And, though we will not abandon you with your data, we always will provide you with the electronic dataset of your responses for your future reference and use. Notional Reseorch Center, Inc. r Clty of Fort Collins Air Quality and Solid Waste Survey Proposed Time line This timeline has a 4 month project length; this can be compressed or expanded, if needed. Task Date Contractawarded .............................................. _.................. _................... _.................... .............. ................ May 23, 2011 Kickoff meeting...................................................................................................... within 1 week of contract signing Finalize prenotification postcard ...................................... _.................... ..... __............. _....... .......... ..... ........ week 3 Other survey materials and questionnaire finalized ........ _.................... _...................... ._....................... week4 Mail prenotification postcard ....................................... _......... _.............................. _........... .................................. week4 Mailfirst wave survey............................... _.................................................................... ...... _............ ........ ......... week 5 Mail second wave of surveys................................................................................_.._......_................................... week 6 Allow surveys to be returned....................................................................._............._.._................................... week 10 Electronic data entry ..........................................................................___..._.._......_..._........_.............through week 11 Data analysis and report writing.............................................................. _... _.... _............................ through week 14 Draft report for review by City.................................... _..................... _.............................................................. week 14 Review by City, meeting with NRC................................................. _ .......................... _.................. _through week 16 Final report and letter delivered ................................ _............................ __........... .......................................... week 17 Notlonal Research Center, Inc.