/

When research is poorly done, and then that research is used to “prove” something in Christian circles, everything can go awry.

Over the next few weeks, we’ll be telling you some of the things we learned in our survey of 20,000 women last year, leading up to the release of our new book The Great Sex Rescue. (when you pre-order, you get some pretty amazing bonuses!)

One of the reasons that we did the survey in the first place is that we were dismayed at the quality of what often passed for research in evangelical resources–and even more dismayed by what was then done with that research.

This week we’re talking about the “gatekeeping” message, that boys will want to push girls’ sexual boundaries. On our podcast last week, Rebecca and I took a look at a section in the book For Young Women Only which told girls that boys had little ability/little responsibility to stop a sexual progression, and so the boys needed their help. Please listen to the podcast–it was an important one. And subscribe to the Bare Marriage podcast wherever you listen to podcasts, so you don’t miss one!

It was such an important point that I wanted to put it in blog form, too, to make it more easily shareable.

So let’s start with the question:

Feldhahn & Rice combine the answers like this: 82% say they have little ability/responsibility to stop.

They conclude: “With a guy, if you want to be able to stop it, it’s best to not even start.”

They use words like, “a sizeable minority feel no responsibility to stop.” “Be careful.” “Be cautious.” “Watch out.”

They say, “boys needed the girls’ help” to stop.

What is this telling girls? You are responsible for stopping in a make out situation, because he can’t/won’t. If you go too far, then, it’s your fault, because you know he can’t stop.

The real issue here is that they asked boys about a consensual situation, and then applied the data to non-consensual ones. That is not a valid use of the data.

We’d like to do an exercise today where we look at that survey question and fix the problems with it.

How to Ask Proper Survey Questions that Actually Answer the Relevant Questions

1. Define the question better

They surveyed 400 predominantly non-Christian boys, and said: (a) in a consensual situation where she wants to keep going; (b) what ability do you have to stop?

The problem is that (a) and (b) do not logically go together. What does his ability to stop have to do with her wanting to keep going?

Additionally, even though her question assumes that boys will want to stop, she never actually asked them. Giving boys a leading question (implying they should be wanting to stop) without ascertaining their sexual boundaries just isn’t good practice if you want to get meaningful data about boy’s desire to keep up their own sexual boundaries.

Ironically, the first answer they allow boys to have gets at this issue. They allow boys to answer: “why would I want to stop?”

A proper survey question is clear on what it is asking. In this case, that means splitting this question up into two: first, ascertaining what the boys’ sexual boundaries are, and then asking how they would feel about maintaining those in a consensual sexual encounter:

Regardless of any past sexual experiences, which of the following statements best describes you currently: 

  • I am saving sex for marriage 
  • I do not feel I will be ready to have sex until I’m older 
  • I am not sure if I am ready to have sex now or not 
  • I feel ready to have sex now
  • I am actively having sex/I am actively pursuing a sexual partner

If you were in a make-out situation with a willing partner who does not signal a desire to stop, how likely are you to want to stop that sexual progression before it leads to sex?

  • Very likely – we would not have sex
  • Likely 
  • Somewhat likely 
  • Somewhat unlikely 
  • Unlikely 
  • Very unlikely – we would have sex

These questions tell us whether boys have set sexual boundaries for themselves that they don’t want crossed and also how strong boys perceive their self-control in this area to be. This allows us to make conclusions about what responsibility boys feel to stop, as Feldhahn and Rice did without actually asking boys about if they felt a responsibility to abstain from sex.

2. Ask a separate question to address the non-consensual situation

They extrapolated the results from a consensual situation to imply that boys would not be able to stop if she asked them to (“it’s safer to not even start.”) However, we have absolutely no way of knowing this, because they never actually asked! So let’s add a second question to the survey that actually addresses this:

If you were in a make-out situation and your partner signalled she would like to stop, how likely are you to stop that sexual progression?

  • Very likely 
  • Somewhat likely 
  • Somewhat unlikely 
  • Very unlikely 

This will get at whether or not boys are likely to stop in non-consensual situations, and then we’d be able to make conclusions about boys’ responsibility to stop in those situations, too.

You may ask, “but who would admit it if they wouldn’t stop?” But actually it’s been shown that people tend to be fairly honest about their shortcomings, even major ones, on anonymous surveys, so this isn’t a huge worry–as long as the survey is truly anonymous.

Also, please note: this is the minimum that should be asked to allow the conclusions that Feldhahn and Rice made in their book. Personally, I don’t think they’re sufficient to truly measure what they’re trying to measure. I’d add at least one more question: If your partner signalled she wanted to stop, how likely are you to pressure her to keep going? And I would ask about what types of pressure they may use. But let’s keep going with the minimum that is necessary to make the conclusions they’re trying to make.

Note from Rebecca: You may have read “For Young Women Only” and not gotten the impression that Feldhahn and Rice were talking about non-consensual situations. We don’t think that they meant to talk to girls in date rape situations, but the problem is that they fail to emphasize that these are consensual situations. They fail to mention that these are situations where she wants it as much as he does. Instead, they tell girls to “watch out” and to be afraid of boys who have little ability and feel little responsibility to stop. If a girl were to be date raped, her boyfriend could easily use Shaunti’s own words to prove to her that it was, in fact, her fault without having to look too hard at all. The caveat in the book about rape means nothing when Feldhahn has written an entire chapter grooming girls for rapists and abusers to take advantage of them in the name of “it’s just too tough to stop the fun.”

3. Remove any language that justifies/enables abusive behaviour

Note that in the original question, Feldhahn asked about the boy’s ability to stop. What does that word imply in context? To imply that a boy may not have an ability to stop the sexual progression implies that some boys can’t help but rape (even in a consensual situation, one should still have an “ability” to stop). But she asked this in the same question she’s measuring boys’ desire to stop. In doing so, she conflates desire and ability to stop sex but then talks about them throughout the chapter as if they are one and the same.

We need to be careful in how we craft questions.

Rebecca here for a minute: frankly, it’s bizarre to ask boys who want to have sex (as many non-Christian boys do) if they would have an “ability” to stop consensual sex! Can you imagine if this question were asked of married, Christian men? I’m pretty sure that more than 90% would answer, “Why would I want to stop?” Does that mean that over 90% of Christian married men are marital rapists? Of course not. They’d most likely be answering based on the “want” in the question, not the “able.” Of course we feel out of control during really good sex, but we know that we’re able to stop (I mean, if you were having sex and a raccoon jumped through your window onto the bed, you could stop is all I’m saying.)

However, Shaunti’s use of the word “ability” implies to her female readers that it’s okay for a boy to think that he has no ability to stop once making out has started. But by conflating “wanting” to stop and “being able to” stop and then not emphasizing the consensual nature of the encounter, Feldhahn and her team handed a whole generation of young men a get-out-of-jail-free-date-rape card.

Again, it’s OK to feel out of control in consensual, safe settings where you actually are in control and do have an ability to stop. But it’s not OK to raise young girls to think that if she kisses him for too long it’s only a matter of time until he loses his ability to not push her further. Shaunti could have emphasized that boys do, in fact, have the ability to stop, regardless of what they say they feel. She could have emphasized personal responsibility, and respect. But she didn’t. Because of this, when Shaunti used the word ability, she taught the girls reading her book, “he can’t stop this like you can,” especially when you consider that later she goes on to say that boys NEED girls to help them stop. (OK, Rebecca out.)

Rape is not a matter of inability; rape is a conscious choice. To imply that boys’ crossing girls’ boundaries is a matter of “inability” as Feldhahn and Rice did rather than as a matter of sin and abuse is highly problematic, to put it lightly.

4. Take out emotional language in the response section

It seems as if the researchers were trying to be “cute”, but it’s very bad form to have emotional language in the response options. Note how Feldhahn says, in option 2, “it’s just too tough to stop the fun!”. That makes this response sound “cool” compared to the option in question 4–“I find it easy to stop the sexual progression.” Note the use of the word “fun” in option 2, and the words “sexual progression” in option 4, all refer to the same thing.

5. Make each response straightforward, and do not combine elements in the response section

Let’s take a look at option 3 in this survey. It says:

Some ability, but it would require a massive effort, and I might go further than intended.

Here we have three different elements to this response:

  1. I have some ability
  2. It would require a massive effort
  3. I might go further than intended

If someone picked this, which element are they reacting to? All of them? One of them? We have no way of knowing.

(Rebecca here again. Whenever possible, it’s best to have only one phrase or clause per question or response option. Often this rule needs to be broken, and that does not immediately invalidate the question or the response–not at all! But this response option is particularly problematic because (a) it’s a triple-barreled response option to an already convoluted question, (b) all the response options are incredibly different from one another so this has a higher risk of being a “catch all” for the “not sure” crowd, and (c) the three elements are all measuring completely different constructs. First, ability; second, effort; third, sexual actions.)

6. Include all options in the response section

Note how the responses jump from #3–I have some ability but it requires massive effort–and #4–it’s easy.

What if it’s not easy, but it doesn’t require massive effort? Isn’t there something in between?

And you know what? There’s an easy way around the the problems in #4-6, and it’s standard in surveys: Simply offer what is called a Likert Scale set of responses, like this:

  • Very Likely
  • Likely
  • Somewhat Likely
  • Somewhat Unlikely
  • Unlikely
  • Very Unlikely

You take out all the emotional language; you include all meaningful possiblities; and it’s very straightforward. There’s a reason this is standard practice.

7. Dichotomize the Answers Appropriately

Now let’s get into how we use the results. Feldhahn looked at those results and concluded that 82% of boys had little ability/little responsibility to stop. So she separated the results between options 3 and 4.

However, is this an appropriate way to dichotomize the data?

The question she asked was about the boy’s perceived ability to stop. Then, throughout her book, she talks about boys going too far or crossing girls’ sexual boundaries. It makes more sense, then, to dichotomize the data based on whether boys reported an ability to stop, and whether they didn’t.

In that case, the appropriate way to split the data is between options 2 and 3. Option 3 does say they have the ability to stop. This gives us, instead of an 82/18 difference, a 48/52 difference.

Using a likert scale makes dichotomizing even easier–you simply split it among whether they are at all likely or at all unlikely.

(Rebecca says: Of course, there are lots of reasons you may want to split up the data in different ways than just “yes” or “no.” In our survey, for example, we asked about orgasm frequency. It may make sense in some contexts to talk about all women who have ever orgasmed, regardless of frequency, and others where it makes sense to only talk about women who always orgasm. But it would be wholly inappropriate for us to combine the 88% of women who have ever had an orgasm, regardless of frequency, into one group and use that data to talk about highly orgasmic women when that includes women who orgasm less than half of the time they have sex. Similarly, using data about boys who were able to stop the sexual progression to bolster the claim that boys have little ability to stop sex was a gross misrepresentation of the data.)

8. Ask Girls the Same Questions

Feldhahn and Rice state in their book:

For a guy even more than a girl, making out often starts a physical drive towards sex that requires a major effort to override.

Shaunti Feldhahn and Lisa A. Rice

For Young Women Only, p. 148

They are drawing a conclusion from their question that boys find this harder than girls.

But just as we spoke about in our podcast in December about unconditional respect and the problem with Shaunti Feldhahn’s survey data on “men want respect and women want love”, you can’t make a gender comparison without actually asking both genders. And it’s not enough to ask similar questions. You have to ask the same question to both genders. In the love & respect case, they never asked women. When other researchers asked the same question of women, women answered the same as men. No gender difference. And yet a whole doctrine was born because of a faulty survey question.

If you haven’t listened to the Unconditional Respect Isn’t a Thing podcast, you really must!

They are assuming that girls can easily stop the sexual progression, but we have no data that this is true at all. This is what we call a presumption fallacy in the study of logic, and it’s the same fallacy that Shaunti used in her love & respect conclusions.

When we look at what she asked girls, this is the closest question we could find:

Many people joke that boys “only think about one thing.” This question is designed to determine whether girls think about and want that one thing as much as boys do. In your experience, if a girl and her boyfriend make out from time to time, but have not made the move to a sexual relationship, do you think the girl thinks about and physically wants sex with him as much as he probably does with her? {Choose One Answer}

  • [46%] Yes, I think in that situation she’s wanting to go to bed with him as much as he wants it (whether or not she actually does go to bed with him).
  • [42%] No, I think it’s probably the guy that most wants the relationship to progress to actual sex. Most girls would be fine with continuing to make out, without crossing that line.
  • [12%] No, I think it’s probably only the guy that wants the relationship to progress to actual sex, and she actively doesn’t want to cross that line (even if she enjoys making out).

(Note from Rebecca: I have… a lot of thoughts about this question.)

This is not a perfect comparison, but it’s the closest we’ve got. And in this question, 46% of girls say that sex would likely occur. So 46% of girls think that sex would happen, and 48% of boys think that sex would happen.

That’s not a gender difference. 

In an ideal world she would have asked the same question of girls at the same time she did her survey of guys. But she surveyed boys a year before she surveyed girls, wrote a book saying there was a gender difference she herself did not find, did not cite any other research supporting her claim, and then when she did ask girls a similar question it pointed to no gender difference. She had no basis for saying what she did.

To sum up: Here’s how to ask survey questions that would allow Shaunti Feldhahn and Lisa Rice to talk about the concepts that they did in their book:

Regardless of any past sexual experiences, which of the following statements best describes you currently:

  • I am saving sex for marriage
  • I do not feel I will be ready to have sex until I’m older
  • I am not sure if I am ready to have sex now or not
  • I feel ready to have sex now
  • I am actively having sex/I am actively pursuing a sexual partner

If you were in a make-out situation with a willing partner who does not signal a desire to stop, how likely are you to want to stop that sexual progression before it leads to sex?

  • Very likely – we would not have sex
  • Likely
  • Somewhat likely
  • Somewhat unlikely
  • Unlikely
  • Very unlikely – we would have sex

If you were in a make-out situation and your partner signaled she would like to stop, how likely are you to stop that sexual progression?

  • Very likely
  • Somewhat likely
  • Somewhat unlikely
  • Very unlikely

And then ask the exact same questions to girls. 

Again, in an ideal world I would ask additional questions about sexually coercive behaviour to flesh this out, but these are the minimum questions necessary to draw conclusions about boys and boundaries. 

(Rebecca note: also, all questions would have to be coded by operational definitions of the research constructs which would be developed during a thorough literature review, the survey would include multiple questions for each construct, we would use previously validated questions whenever possible, the whole thing would be pilot tested, and we would only ask the intended research group, or else make it very clear that our findings could be entirely inaccurate to the people reading the book).

We hope you see now how writing bad questions, writing bad response options, and then interpreting survey questions inappropriately is all too easy to do.

All of us who worked on The Great Sex Rescue did university training in survey development and analysis (Joanna and Rebecca far more than me, but even I remember my courses from 1990 where we learned everything in this post. It was actually quite basic).

But what wounds us and worries us the most is that Feldhahn and Rice used a bad question to tell girls that boys had little ability to stop the sexual progression, and that boys “needed” their help to stop.

If a girl who had been date raped read this book, would she know that she had been assaulted? Or would she blame herself? After all, if you want to stop, it’s better not to start! Guys feel little responsibility or ability to stop. You should watch out. Be cautious. Guys need you to do the right thing.

This is rape culture. It formed the basis for a whole chapter in her book. And it was based on extremely faulty research methods.

Church, we simply must do better. Please. What are we doing to our teenage girls? And who are we expecting our boys to turn into?

We’d like to set the record straight.

We want to set a much higher standard for what passes as research in the Christian world, and we think we’ve done that with The Great Sex Rescue. In that book, we identify what teachings harm women’s sexual satisfaction. If you’ve never been able to reach orgasm; if you have no libido; if your marriage is stale–it might not be your fault! It may simply be that you were taught stuff that has seriously messed you up. And seeing what that “stuff” is can be freeing, validating, liberating! It can help you have the marriage and sex life you’ve always wanted.

And when you pre-order The Great Sex Rescue, email us your receipt and we’ll give you our healthy sexuality rubric, our scorecard of how Christian books fared on our rubric, and a sneak peek at our data!

The Great Sex Rescue

Launches March 2!

What if you're NOT the problem with your sex life?

What if the things that you've been taught have messed things up--and what if there's a way to escape these messages?

Welcome to the Great Sex Rescue.

Pre-Order Now! (Helps us out a ton)

And if you email your receipt, we'll send you a special pre-order BONUS

Time to Pre-Order

Day(s)

:

Hour(s)

:

Minute(s)

:

Second(s)

If you’d rather watch than read, or if you want to see our passion for this, here’s the part of the podcast where Rebecca and I looked at this survey:

We really can do better.

What do you think? Did anything else stand out to you about that survey question? Have any thoughts about all of this? Let’s talk in the comments!

4d5d2dc667e7acd64221c42a103248a4?s=96&d=mm&r=g - Fixed It For You! We Fix a Survey Question So It Doesn't Enable Date Rape

Sheila Wray Gregoire

Founder of To Love, Honor and Vacuum

Sheila has been married to Keith for 28 years, and happily married for 25! (It took a while to adjust). She’s also an award-winning author of 8 books, including The Good Girl’s Guide to Great Sex, and a sought-after speaker. With her humorous, no-nonsense approach, Sheila is passionate about changing the evangelical conversation about sex and marriage to line up with kingdom principles. ENTJ, straight 8

Related Posts

The Church’s #MeToo Reckoning

What happens when the church, which is supposed to be your safe haven, becomes the place where you...

Tags: , ,