A Challenge.

I was reading this morning about how Finland is teaching their children how to spot fake news…

Another slide, featuring a diagram of a Twitter profile page, explained how to identify bots: look for stock photos, assess the volume of posts per day, check for inconsistent translations and a lack of personal information.

The lesson wrapped with a popular “deepfake” — highly realistic manipulated video or audio — of Barack Obama to highlight the challenges of the information war ahead.

And that got me to thinking about bogus research studies and I googled “how to spot bogus research studies.”

On the website Researchgate one of the people who commented on the question said,

• Don’t explain their methodology or avoid technical terminology.
• Don’t indicate any limitations on the conclusions of the research. (A study on mice cannot draw firm conclusions about humans.)
• Are based on research from a journal that nobody has heard of.

There are some things to keep in mind when reading about research studies.

First who sponsored the research? Almost all legitimate research they will report on who funded the research.

Then how did they find the research subjects? There was a study on what the study authors called “Rapid Onset of Gender Dysphoria” (ROGD), the study recruited participants from a website and the trans study “2015 U.S. Transgender Survey (USTS)” collected their participants from internet so how were they different?

Well in the ROGD study they recruited participants from only conservative websites while the USTS  recruited participants from a broad number of websites and from support groups.

The ROGD recruited parents for the study of trans youth, not the children who were the subject while the USTS recruited trans people for the trans study. The USTS reported,

The survey was produced and distributed in an online-only format after a determination that it would not be feasible to offer it in paper format due to the length and the complexity of the skip logic required to move through the questionnaire.

Meanwhile the ROGD study,

Restar’s study, published today in the Archives of Sexual Behavior, points to several methodological problems, such as relying on survey responses from parents who had visited sites promoting anti-trans views, and biasing their responses with the wording of the study’s consent forms. Littman’s approach, Restar contends, pathologizes trans people. “It’s important to use methods and terminologies that don’t further stigmatize an already disenfranchised community,” she said.

The way they recruited participants has a built-in bias.

Another clue to built in bias of the study is the demographic profile of the parental-respondents, the demographics of the RODG study was…

...The parental-respondents displayed very narrow demographic stratification despite being sampled from a very specific venue: 82.8% were female sex at birth, 91.4% were White, 99.2% were non-Hispanic, 66.1% were aged 46–60, and 70.9% had attended college. Notably, 76.5% believed that their child’s trans identification is not correct, and recruitment relied heavily on three particular Web sites known to be frequented by parents specifically voicing out and promoting the concept of “ROGD.” Thus, these are not just “worried parents,” but rather a sample of predominantly White mothers who have strong oppositional beliefs about their children’s trans identification and who harbor suspicions about their children having “ROGD...”

The USTS survey reported that,

The age profile of respondents differed widely by gender identity categories, with nearly half (47%) of transgender men and women being aged 25–44, compared to 35% of non-binary respondents, and 29% of crossdressers. Non-binary respondents were more likely to be younger, with nearly two-thirds (61%) being aged 18–24, in contrast to transgender men (43%), transgender women (24%), and crossdressers (8%). One in five (20%) crossdressers were aged 65 or older, compared to only 5% of transgender women, 1% of non-binary respondents, and less than 1% of transgender men. Researching marginalized community is difficult, many of them in hiding and there is no one way to find them, it takes a lot of legwork. 

At Friday’s meeting of the Lesbian, Gay, Bisexual, Transgender and Queer (LGBTQ) Health and Human Services Network we discussed a survey that the Department of Public Health will be doing and the questions that we asked were who is going to design the survey, how are they going to capture gender identity and expression. Asking just male, female, or transgender just doesn’t work.

Also we question how they are going to reach out to the LGBTQ+ community? How are they going to find us?

Surveys are important but it is true “garbage in, garbage out” we need to know how to spot fake research, just because a survey is in our favor doesn’t mean that it is not biased and vise-a-versa just because a survey finds negative results doesn’t mean it is flawed, you need to dig deeper.