Large and complex population-based surveys have become an invaluable resource for many public health professionals.
However, data users are rarely involved in the design and implementation of the survey.
It is of paramount importance, then, that before any analysis occurs,
you take the time to understand how the survey was conducted.
Failure to account for sampling design in your analysis may lead to biased estimates and drawing flawed conclusions.
When we use data from one of the many large surveys that are available,
it is usually because we want to be able to draw some conclusion about the nation as a whole,
a state as a whole, or some subgroup of people across a state or the nation.
But, if we just head out to the streets with our survey and start asking people questions,
how do we know if the people who answer our questions are representative of the
typical person in the geographic area we are interested in?
Well, the most rudimentary way, of course, is just to ask our questions of every single person in our population of interest.
However, what if you want to draw conclusions about the United States as a whole.
It is currently literally impossible to gather data in this way.
Not only that, but it’s unnecessary to gather data this way when proper sampling methods are used.
Another way we can gather our data is with a simple random sample.
In a simple random sample, we basically have a list, or sampling frame, of everyone in our population of interest.
We then use a randomization technique, of which there are many, to select people out of the population for our sample.
The key here is that we know every member of our population,
and each member of the population has the exact same
probability of being selected for our sample as every other member of the population.
And once our sample is large enough, we expect the conclusions about our sample to be valid for our whole population.
However, is this method really any more feasible that counting every member of the population?
Where do you get a list of every single person in the United States, or even Texas?
And even if you have a list, for how long is it valid?
How many people are born or die every minute in the United States?
Clearly we need some other method
Any sampling design that deviates from a simple random sample is called a complex sample design, of which there are multiple types.
Common techniques used in complex sampling designs include:
Stratification: Or, classifying a whole population into strata based on
information available for one or more particular variables and randomly sampling within those groups.
Clustering: Or, sampling groups of units rather than individual units, often in several stages.
Oversampling: Or, intentionally selecting more people from a subgroup of interest than would be
selected if everyone in the sample had an equal chance of being selected.
This is typically done to increase the precision of estimates about some subgroup of the population.
Because, in complex survey designs, every member of the population does not have an
equal probability of being selected into the sample,
weights are used to compensate for the unequal selection probability.
Sample weights are also often modified to adjust for nonresponse.
For example:
In the first National Health And Nutrition and Examination Survey or NHANES,
the country is first divided into geographic areas called primary sampling units, or PSU’s.
PSU’s do not have to be geographically defined, and they are not unique to NHANES.
Anytime, multistage sampling is used, the unit in the first stage is referred to as the PSU.
And because not all PSU’s are exactly equal, your probability of being selected is different from PSU to PSU.
And, because NHANES uses multistage sampling, PSU is not the only factor that affects your probability of selection.
In fact, your final probability of being selected is equal to the probability of your PSU being selected
times the probability of your census enumeration district being selected
times the probability of your segment of the PSU being selected
times the probability of your household being selected
times the probability of you being selected from within your household,
times the probability of you being eligible.
As you can see, this can be quite complicated.
In Simple Random Sample, every unit in the population has an equal probability of being selected.
If big N is the population size and little n is the sample size,
then the selection probability, or sampling fraction, is n/N and the sample weight is N/n
For example, let’s say that we have 25 students in our class,
and let’s say that our class is the population of interest.
And let’s say that our sample size for some study is 5
Then the probability of selection for each student is 0.2
And their weight is 5.
This can be calculated by dividing N by n or by calculating the reciprocal of the sampling fraction.
In a simple random sample, the weight is the same for every person in the sample and is irrelevant in most analyses,
other than for estimates of population totals.
But, in complex sampling designs, every person has a known, but not necessarily equal, nonzero probability of selection.
This table provides another simple example of some steps that may be used to create a weight variable.
In this example, 150 people are drawn from a population of 1000 people of different degree programs.
If this was a simple random sample, where every person has an equal probability of selection,
we expect the sampling fraction to be the same for all individuals: 0.15,
and one would expect about 15% of members of each program to be present in any given sample.
An SRS sample would therefore yield relatively few DrPH or PhD students. Just 18 and 19 respectively.
If we wanted to conduct subpopulation analysis within DrPH or PhD students, we may find that these samples are not large enough.
An alternative sampling design would be to oversample DrPH and PhD students,
and under sample MPH students such that we would select an equal number of each group into the sample.
To achieve this, each group would be sampled with a different probability, or sampling fraction,
which is equal to the desired sample size (50) divided by the group specific population size.
The corresponding design weights are calculated by taking the Population size divided by the sample size, and are listed here.
Finally, applying the design weights to the target sample frequencies gives us the population totals.
Often because of random fluctuation or non-response,
the sample drawn does not match the expected sample.
In this case post-stratification weights are needed to make the sample match the known distribution in the population of interest.
Finally, nonresponse occurs when we are unable to contact selected individuals, or when selected individuals refuse to participate.
Because the response rate is less than 100% in most surveys, some type of nonresponse adjustment is almost always used.
Luckily for you, in large population-based surveys the weights are almost always calculated for us and included in the data set.
Let’s look at some examples.
Here I’m just importing the data, which is 2012 BRFSS data from Washington D.C. I’m also doing a little bit of data cleaning.
If you look at the codebook, you can see that for anyone coded as a 7 or a 9, we don’t know if they ever had asthma or not.
Whether it’s because they said they weren’t sure or because the refused to answer is of little consequence to us.
The bottom line is that we don’t know if they have asthma or not, so we need to recode those values to missing.
Also notice that a value of 1 currently means “yes” and a value of 2 currently means “no”.
While it’s completely optional, I recommend that you recode all dichotomous Yes/No variables to zeros and ones.
In this data step, I’m just creating a new data set and recoding my variables of interest.
Not that long ago, in order to analyze weighted survey data, a good deal of programming was necessary.
Fortunately we don’t have to worry about that.
SAS now has a whole series of procedures that take complex sampling design into account when estimating statistics.
In this class we have already used PROC FREQ and PROC MEANS.
The analogous procedures for weighted survey data are PROC SURVEYFREQ and PROC SURVEYMEANS.
You program them exactly as you would PROC FREQ and PROC MEANS,
except that you specify relevant complex design variables in the data.
For the 2012 BRFSS data, you must specify the strata,
the primary sampling unit, and the final variable weight using the strata, cluster, and weight statements respectively.
Then all you have to do is run the procedure as you normally would.
Here I have run PROC SURVEYFREQ and PROC FREQ together so that you can see the similarities and differences.
As you can see, the frequency of observations is given in both procedures, but SURVEYFREQ also gives you the weighted frequency.
This is the number of people in the population represented in the current analysis.
You can also see that the weighted and unweighted percents are slightly different.
If you are trying to draw conclusions about the population from which this data was drawn,
ONLY the weighted percents are valid.
In this example I’m just showing you the slightly differences between PROC SURVEYFREQ
and PROC FREQ when creating two-way frequency tables.
As you can see, the layout of the results table is slightly different when we use PROC SURVEYFREQ.
Reading from left to right across the first row we can interpret this table in the following way:
there are 1,929 women in our data who report never being told that they have asthma.
After applying survey weights, this represents 231,938 female resident of Washington, D.C.
Then we can interpret the 44.05 in the percent column to mean that roughly 44 percent of Washington, D.C.
residents are women who have never been told that they have asthma.
But what if we want a row percent or a column percent.
Here we have the SURVEYFREQ procedure again, but we add the “row” statement option to the tables statement.
As you can see we’ve added two new columns to our results table. Row percent and the standard error of the row percent.
So, how is the 82.55 in the row percent column interpreted?
Among female residents of Washington, D.C. in 2012, about 83% have never been told that they have asthma.
In this example we are comparing the SURVEYMEANS and MEANS procedures on the variable children,
which represents the number of children under age 18 in the respondents household.
Again, the only difference is the specification of the complex design variables.
And as before, you can see that there is a slight difference between the weighted and unweighted estimates.
And as before, only the weighted estimate is a valid estimate of the population mean.
And finally, just like with PROC MEANS, we can estimate mean values across levels of a second variable in PROC SURVEYMEANS.
However, there is one very important difference.
When using PROC SURVEYMEANS you should always use a DOMAIN statement for your categorical variable.
If you use a BY statement instead, it will give you a result, but the standard errors and 95% Confidence intervals will be biased.
And when you look at the log file, you can see that SAS tries to warn us about the biased
subpopulation estimates when we use a BY statement instead of a DOMAIN statement.
Large population-based data sets are a staple of information for public health.
They can be extremely useful, and give us remarkably good estimates of population-level characteristics.
However, their complex sampling designs make it necessary for you to have at least a
basic knowledge of why survey data is weighted and how it should be analyzed.
And, that is what we attempted to cover in this lecture.