August 25, 2011

How Accurate Are Counts of Same-Sex Couples?

The counts and characteristics of same-sex couples are among the most written-about data from the 2010 Census and American Community Survey. Yet, two decades after the Census Bureau began offering people the option to describe themselves as a same-sex “unmarried partner,” producing accurate numbers remains a challenge.

The quality of information about same-sex couples depends both on the questionnaire responses the Census Bureau receives, and the procedures the agency implements to collect and edit those responses. There have been changes to both during the years the bureau has been releasing same-sex couple data, and the agency has cautioned against comparing numbers from different years to produce trends.

The Census Bureau has just completed the release of state-by-state total counts of same-sex couples from the 2010 Census, which adds up to a national total of about 902,000. Later this year, the bureau will release a highly anticipated count of same-sex married couples from the 2010 Census, the first time it has done so from a decennial census. In conjunction with that release, the bureau will publish its own evaluation of data quality about same-sex couples. This posting describes what is known so far about data quality for both married and unmarried same-sex couples from past censuses, the American Community Survey and Census Bureau research.

The challenges of counting same-sex couples illustrate the difficulties of data collection and group identification in an era of rapid social change. The Census Bureau has a delicate balancing act as it tries to capture a demographic snapshot even as national norms are more akin to a motion picture.

Same-sex marriage licenses are currently issued in six states and the District of Columbia, according to the National Conference of State Legislatures. Massachusetts became the first state to license same-sex marriages in 2004. California had allowed same-sex marriages, but that has been challenged in court, and the state does not allow new marriages pending the outcome of that case. A number of additional states offer same-sex couples the option to register in civil unions or domestic partnerships that offer some or all rights within the state that married couples have. The federal government and more than three dozen states prohibit same-sex marriage via laws defining marriage as a union between a man and a woman.

The Census Bureau’s numbers for same-sex couples are derived mainly from answers to the question on the Census or American Community Survey form about how other people in a household are related to the householder. There are more than a dozen response categories, including “husband/wife,” “roomer, boarder” or “housemate, roommate.” The ACS (as well as the 1990 and 2000 Censuses) also asks a separate question about marital status. [Scroll down to see the wording of those questions from the questionnaires.]

The 1990 Census was the first to add the option for people to be counted as the “unmarried partner” of the householder. Counts of same-sex couples were produced by combining responses to that question and to the gender question on the form. The bureau’s American Community Survey, which replaced the census long form, has reported data for same-sex couples since 2005. (The bureau’s Current Population Survey also has data on same-sex couples—which differ from those in the ACS on some dimensions—but it is not as widely used as the ACS.)

Obstacles to Accuracy

The first potential hurdle to obtaining accurate data is the willingness of men and women in same-sex couples to indicate their status on a government form. Their readiness to do so may be influenced by rising public acceptance of homosexuality: In the 1990s, half of Americans said it should be discouraged by society; in a 2011 Pew Research Center survey, a third said so. The Census Bureau also has expanded its outreach to gay and lesbian adults over the years, urging them to fill out the form accurately and assuring them their information is confidential. On the other hand, Census Bureau research also indicates some people may describe themselves as married even though they are not, especially if they live in states that do not allow same-sex marriage.

One data quality issue is that people make unintentional errors when they fill out their census forms. For example, in the question that asks people how they are related to the householder, “husband/wife” is the first response and “unmarried partner” is the 13th item on a list. Due to something called “the primacy effect,” people are more apt to choose an earlier category if they are given a long list of them.

It also turns out that a notable share of same-sex couples counted in the 2000 Census were actually opposite-sex couples. The couples were misclassified because one partner checked the wrong gender box, according to analysis from the Census Bureau and work by other researchers. A Census Bureau analysis of data from a 2004 test census in Queens, N.Y., which looked at whether first names of respondents matched the gender checked on the form, implied that “perhaps 20% of same-sex couples are miscoded different-sex couples,” according to a Williams Institute methodology brief. That brief also points out that the New York census was a limited sample that may not apply to the total count. In addition, some opposite-sex couples counted in the census could actually be same-sex couples.

Gender misreporting is relatively uncommon, but can have a large impact on a small population. The 2000 Census counted 594,000 unmarried same-sex couples, compared with 4.9 million unmarried opposite-sex couples and 54.5 million married opposite-sex couples.

Another factor that influences the count is how the Census Bureau edits the responses it receives, and the editing may have magnified the 2000 misclassification problem.

In the 1990 Census, if census forms showed two people of the same sex married to each other, the Census Bureau changed the gender of one of the spouses on the assumption that an error had been made, resulting in an opposite-sex couple. In the 2000 Census, when people in more than 253,000 households said they were part of same-sex married couples, Census Bureau staff changed the relationship status in those households to that of unmarried partners. This was done in part because no state allowed same-sex marriage in 2000 and in part, according to bureau officials, because people tend to report their gender more accurately than they report their relationship. However, that change means that in the 2000 Census, any opposite-sex couple, including a married couple, could potentially have been misclassified as a same-sex couple, whereas in 1990 only unmarried opposite-sex couples could be so misclassified.

Census 2010 Data

In the 2010 Census, census-takers who made follow-up visits to people who did not mail back their forms used a questionnaire that was poorly designed, according to Gary Gates and Michael Steinberger of the Williams Institute at the UCLA School of Law. Gates and Steinberger estimate that perhaps 25% of same-sex couples counted in the 2010 Census could be misclassified opposite-sex couples, which would produce an overcount of same-sex couples.

However, Gates and Steinberger also say the 2010 Census did not count all same-sex couples. That is because some same-sex couples did not identify themselves as such, and because unmarried-partner couples are counted only if one partner is the head of the household.

They conclude that the overall count of same-sex couples in the 2010 Census  apparently was accurate, because the overcounts and undercounts offset each other. However, Gates and Steinberger theorize that more detailed information about these couples, such as their age or where they live, is less trustworthy.

American Community Survey Data

The quality of data about same-sex couples in the American Community Survey appears to have improved in 2008 compared with earlier years, according to Census Bureau and outside researchers. Beginning with 2008 data, the Census Bureau imposed more sophisticated data-editing rules and redesigned the survey questionnaire, in part to make it more consistent with the upcoming 2010 Census form.

Before 2008, for example, if someone marked multiple categories for gender or relationship-type on the survey form, bureau staff had resolved the conflict by choosing the first option shown on the questionnaire–specifically, “male” and “husband/wife.” This process “could have created erroneous same-sex households,” the Census Bureau concluded, and it changed the procedure in 2008 to use statistical methods that select a category based on a variety of factors.

The main effect of the changes was to reduce the number of same-sex married couples. In the 2007 ACS, according to bureau research, about 341,000 same-sex married couples were reported; in 2008, the number fell to about 150,000. The number of unmarried same-sex couples was about the same in both years.

Not only did the number of same-sex spouses decline in the ACS, but their reported geographic distribution and characteristics changed from 2007 to 2008. For example, according to the Census Bureau analysis, there had been “inexplicably high” reports of same-sex spouses in some Midwestern states in 2007, but not in 2008. A lower share of same-sex married couples reported children in the household and a lower share was ages 55-64 in 2008 compared with the previous year.

However, the Census Bureau analysis of the ACS data points to another problem—a mismatch between the number of legally married same-sex couples and the number counted in the survey. In 2008, there were about 32,000 legally married same-sex couples in the U.S. and about 87,000 couples registered in civil unions or domestic partnerships, according to government records cited by the Williams Institute. On the other hand, the 2008 ACS reported 150,000 married same-sex couples.

Question Wording

In part because of the mismatch between government records and survey data, the Census Bureau has begun looking into whether changing the wording of the questions people are asked about their relationship type and marital status could produce more accurate responses. Census Bureau researchers have co-authored two papers on this topic presented to the American Association for Public Opinion Research (AAPOR). They did not turn up evidence that current question wording led to a massive accuracy gap, but did unearth intriguing differences in how people perceive the response categories they are offered.

In a 2010 paper presented at the AAPOR conference, researchers described the findings of 14 focus groups of people in same-sex relationships and four groups with unmarried opposite-sex couples at seven locations around the country earlier that year. This year, bureau researchers reported on the results of a series of one-on-one interviews in which they tested alternative question wording.

 

In answering the relationship question (pictured right), researchers found that the same-sex couples they talked with (who may or may not be a representative sample) generally would say they were married if they had been legally married anywhere, even if their current state of residence did not allow legal same-sex marriage. In states where same-sex marriage is legal, and couples are not married, they tended to choose “unmarried partner” as an accurate reflection of their status.

However, the marital-status question (pictured below) “did not work well” for unmarried same-sex couples who are not registered partners and live in states that do not allow same-sex marriage, according to the research presented this year at AAPOR. They did not see their experience reflected in the options such as “never married,” or “divorced.” (Some opposite-sex unmarried couples said the same thing.) A small number of participants marked “now married” or left the question blank.

After testing a number of alternatives, Census Bureau researchers endorsed large-scale tests of a new relationship question that includes categories of opposite sex husband/wife/spouse, opposite-sex unmarried partner, same-sex husband/wife/spouse and same-sex unmarried partner. Providing those choices, they said, “would help reduce classification errors by providing a check against mismarks in the sex question.” However, they acknowledge that this change could “open the door for more same-sex couples without legal status to select ‘same-sex husband/wife.’”

The Census Bureau paper raises the issue that this new question wording could offend some social conservatives who disapprove of same-sex relationships and do not wish the government to acknowledge them. Researchers tested the wording in interviews in North Carolina, which was chosen because it does not recognize same-sex marriage or unmarried partnerships. Some respondents said they disapproved of adding same-sex categories, although the paper noted they were not so offended that they would have refused to fill out the survey.

Despite the criticism by people in unmarried partnerships that the marital-status categories did not reflect their experience, the Census Bureau researchers did not recommend adding a category in the marital-status question about whether people are in a “registered domestic partnership or civil union.” They said most people do not understand what this concept really means, and that including it could inflate the number of people who report incorrectly. It “may be safer to collect this information in a separate question,” they said.

They also recommended testing a question for each person that says: “Is this person currently living with a boyfriend/girlfriend or partner in this household?” The question would serve two purposes—it would let cohabiting couples acknowledge their relationship whether or not it had legal status, and it would give analysts a more complete count of unmarried couples by incorporating those that do not include the householder (so-called “subfamilies.”).

The research is being done for the Office of Management and Budget’s Interagency Working Group on Measuring Relationships in Federal Household Surveys, which is studying new ways to define and measure relationships in federal household surveys, especially for same-sex couples who report being married.

“It is clear that the Census Bureau and other statistical agencies must begin to rethink how these constructs are measured to keep up with legal and societal changes,” wrote the authors of a paper presented at the 2010 AAPOR conference. “With same-sex marriage laws in flux but looking to expand, we must closely examine the evolving definition of marriage to accurately reflect and include the living situations of same-sex partners.”