B.4 COLLECTION METHODS AND FOLLOW-UP B.4.a COLLECTION METHOD The primary collection method proposed is a telephone survey, although the ENERGY STAR Household Survey will be conducted as a mail survey. The telephone surveys will be administered by persons that have conducted numerous surveys and/or have supported various ENERGY STAR programs. The telephone survey method was chosen because it helps ensure that appropriate follow-up questions are asked of respondents, allows 'realtime' recording of responses, and is considered to be an appropriate collection method given the length of the survey. EPA chose to administer the ENERGY STAR Household Survey by mail because it permits the respondent to view the ENERGY STAR label, is cost effective given the size of the sample (approximately 1,000), and it allows the respondents to choose a time that is convenient for them to reply. The associated survey instrument is designed with clear instructions and graphics to minimize ambiguity in the responses and promote overall ease in responding. For example, respondents are asked to mark check boxes with an “X”, ample lines and spacing are provided for the open ended questions, and skip patterns are explained clearly and referenced by question numbers to provide adequate instructions to the respondent. B.4.b SURVEY RESPONSE AND FOLLOW-UP The target response rate for these surveys varies by survey type and is listed in Exhibit B-2-1. Response rate has been defined as the ratio of responses to eligible or valid respondents. The difference in the number of eligible or valid respondents and the total number of potential respondents included in the respective sample frame can be explained by unretrievable or unuseable responses. Such responses result from data quality discrepancies (e.g., inaccurate or out of date addresses), intentional ‘sanitizing' by the respondent of stratum codes, late responses received after data analysis is complete, or other unforseen circumstances such as tearing or smearing of responses received by mail. To achieve the highest response rate for the surveys, EPA has or will take the following steps. For the ENERGY STAR Household Survey, EPA will send reminder post cards to respondents that do not reply within approximately 2 weeks of the initial mailing. In addition, a second mailing will be conducted 4 weeks after the initial mailing. For the ENERGY STAR Household Survey, EPA has designed a survey that is a pre-addressed and postage-paid self-mailer to minimize the burden on respondents and ensure that the reply is directed to the appropriate address. EPA will provide a toll-free telephone number for respondents to call in case they have any questions about the survey. For all surveys and information collections, EPA will make the results available to the respondents and other interested parties such as utility companies and trade associations. The selection of an option for a particular variable will depend on how critical it is to the key objectives of the analysis and the likelihood of its success. For example, the percentage of efficient equipment sold by participating and nonparticipating HVAC contractors is a key component of the market transformation analysis for the Residential HVAC program. If this item is missing or has an improbably high or low value, it is likely that a call will be placed to the respondent to obtain or verify the information. If respondents to the consumer survey choose not to include certain kinds of demographic information, such as income, on their questionnaires, these values can be imputed using census information specific to customer's ZIP Code cluster. This approach may prove useful in analyses of the association of demographic characteristics with recognition and use of the ENERGY STAR label. Response Rate Tracking EPA will establish a receipt tracking system for all data collection instruments. The primary objective for tracking mail survey receipts is to ensure that completed forms submitted by respondents are logged promptly and given proper chain-of-custody. A secondary objective is to monitor cumulative receipts by date to identify potential issues with the response rate. Additionally, mail survey receipts will be sorted by stratum based on a coding system on the return address that will increase the ability to monitor response rates at the stratum level. B.5 ANALYZING AND REPORTING SURVEY RESULTS B.5.a DATA PREPARATION To promote the quality of data that is collected and to preserve this quality throughout the analysis process, EPA will record responses in an electronic data base. Use of CATI telephone software will minimize the need for any follow-up calls to respondents to verify (or collect additional) information. CATI software also features real-time data archiving—that is, it logs and records responses as they are received. CATI systems have built-in data control mechanisms that obviate the need for postsurvey data entry; however, CATI interviewers are monitored by experienced professionals to ensure productivity and accurate data entry. For the household mail survey, prior to entry into the data base, the EPA contractor will screen the completed mail survey instruments for legibility, completeness, and internal consistency. Any instrument that does not pass this initial screening will be set aside. Reviewers will also code the openended questions in preparation for entry into the data base. Data from the survey instruments will be keyed into the data base only after the returned surveys have passed the initial screening. After data are keyed, sorting will be done to ensure that only valid responses have been recorded. After data are electronically recorded, data quality will be verified by a random selection of surveys from all telephone survey types for independent verification. The verification will be conducted by an analyst other than the one that conducted the initial survey interview. If any discrepancies are discovered, the original analyst responsible for conducting the interview will be consulted. Since the household mail survey is completed by the respondent in written form, data entry accuracy is paramount to ensuring data quality. "Spot checks" will be conducted for the mail survey. If discrepancies are discovered in this survey, they will be examined by two analysts before any data processing decision is made. EPA also acknowledges the possibility of conducting its analysis with incomplete data. The basic options available for treating incomplete data in completed interviews are as follows: Re-contact the respondent to obtain missing data items, provided the data were not Impute the value of the missing data point based on analysis of the variable among respondents for which a valid value is available. This can be accomplished through a number of analytical methods which include using the simple average or modal value of the variable; using the average or modal value of the sample stratum from which the respondent was drawn; more sophisticated approaches to estimate the variable's value using regression or other statistical procedures; or the use of population descriptors from outside sources. Simply report that the information is missing. In many cases involving descriptive statistics, this will be an adequate approach, if the number of cases missing is relatively small (less than 10 to 15 percent of the sample). B.5.b ANALYSIS EPA will prepare a report that tabulates the results of the surveys and explains the precision of the summary statistics. Examples of general statistics that will be produced include: Percentage of persons in each surveyed market segment able to positively identify the Changes in the percentage of ENERGY STAR compliant equipment sold over the last two years. “Pie” charts illustrating the key decision factors in product selection. Mean and median statistics on percentage of ENERGY STAR of ENERGY STAR compliant equipment shipped with power management features enabled over the last two years. EPA anticipates using the following statistical procedures in the analysis of data collected through the efforts described above: Estimation of population proportions (p), Estimation of population means and totals (x), Estimation of differences in proportion among population segments (P-P2), and Estimation of differences in mean values among population segments (x,-82). EPA does not anticipate using multivariate methods such as analysis of variance or the various forms of regression to a significant extent. The planned estimates of and characterization of market response to ENERGY STAR efforts typically do not rely on these methods. Exhibit B-5-1 shows some example research questions to be addressed in the evaluation of the three programs in question, along with the statistical procedures likely to be employed. Exhibit B-5-1 Examples of Research Questions and Statistical Procedures to be Employed Program (Survey)/Example Research Questions Do levels of consumer recognition and understanding of the What percentage of customers have used the ENERGY STAR Is recognition and use of the ENERGY STAR label related to What percentage of copiers are installed with their power Do enabling rates vary depending upon the type or size of end-user establishment? What percentage of office and systems managers are aware of the operations and benefits of power management technology? What share of office and systems managers associate the ENERGY STAR label with the benefits of power management technology? Residential HVAC Program (Contractors Survey) What is the difference in the share of energy-efficient equipment sold by participating versus nonparticipating contractors? What are the differences between participating and nonparticipating contractors in regard to promotional practices for energy-efficient equipment? Are there systematic differences between participating and nonparticipating contractors in terms of size, brands carried, rural v. urban location? ✓ |