Spotlight is a survey conducted twice a month with at least 500 respondents per survey (for a total of 1,000 per month). The sampling process includes randomly drawing respondents from a base of survey takers who have previously agreed to participate in surveys (they “opt-in” to completing surveys) as part of an online panel/community. We ask respondents several questions about themselves, including their age, gender, income, marital status, family composition, ZIP code and education status. We gather this data so that we can accurately weight our results to be representative of the entire US population.
Once we identify a potential respondent, we email or text them a secure link (one time use; tied to their specific identity/details) that takes them to a survey hosted on our secure servers. After completing the survey, we inform them the survey is complete by showing them a “thank-you” page. Once our respondents complete a Spotlight survey, we do not invite them to participate again for another 12 months. That way there is no risk that any questions from prior surveys will unduly influence respondents’ answers.
Our respondents come from all 50 states and include respondents living in urban areas, suburbs and rural communities. All respondents have access to the Internet so they can complete our surveys. This access includes broadband, as well as mobile access.
All the surveys are written in English, though we occasionally translate into Spanish as necessary. Since our study often includes multiple topics, there are often many topics of interest to engage respondents.
To ensure our responses to reflect the US population, we weight data on a respondent level, as is a common practice in the survey industry. We use the US Census Bureau’s 2017 American Community Survey (ACS) to generate estimates that reflect the most recent demographic composition of the US. We use ACS data for age, gender, education ethnicity and geography as the basis of developing our weights. All respondents must answer these questions so we can weight to these parameters in our survey.
When producing national estimates for Spotlight, we perform multi-stage analysis to build our weights. We first determine the percent of the population that lives in each state. Each state has their own weight. We weigh data to the US population aged 18 or older. We then adjust these initial weights by gender, age, ethnicity and income to match results from the ACS.
The margin of error allows us to know if the differences we are observing are actual differences, or differences that could have been generated through a sampling error or other bias. We use an online panel with over 10 million members; our prior research shows this panel mirrors the normal distribution for age, gender, ethnicity and geography from the ACS (what we call a priori sampling). Based on this normal distribution and our number of completes (N500), we can generate a margin of error. The minimum margin of error for Spotlight is +4.38 points. This means is that if the difference between two data points is greater than five percentage points, we are confident that this indicates an actual difference within the given population and not just naturally-occurring variability or “noise” in the data. We perform all analysis at a 95% confidence level.