Justin has most famously been linked to Selena Gomez.

Before deciding to settle down and marry wife Hailey Baldwin, Justin Bieber had his fair share of past relationships, including his most notable on-again, off-again romance with ex-girlfriend Selena Gomez. And according to Hailey, her husband’s dating history has sometimes affected her self-esteem — especially when his fans are constantly rooting against her.

On Friday, Hailey spoke about the topic further during an episode of the couple’s Facebook Live show, Live with The Biebers, when a fan asked them, “How do you manage being a couple with everyone feeling like they have a say in your relationship?” Allowing his wife to take the lead, Bieber told Baldwin, “Oh, that’s good for you.” Without naming names, Hailey didn’t hold back her feelings about being compared to Justin’s exes.

“I’ve had a really, really hard go and a really hard time with a lot of the things people say and a lot of the ways that I feel like people have made comparisons and put me in a position where they’ve made me feel like less of a woman,” she said. “It’s not easy, I think especially when people have fans that are really passionate and they want to express that and they want to express their opinion.”

From the on-set of her and Justin’s reconciliation, the couple has been bombarded by negative comments from Jelena shippers, and Hailey explained that they don’t have the full picture.

“People think they know what happened in his life and his relationships and they think they know what happened between us and our relationship and how we got here and there’s no possible way they could,” the model revealed. “They invent details that they think are true that really aren’t, and I got to a point where I’m like, ‘Why am I even gonna try to go out of my way to set the record straight where it’s really nobody’s business and it doesn’t matter?'”

Hailey added, “There’s so much that people don’t know that went on behind the scenes.”

Elsewhere in the Q&A session, Hailey described how marrying Justin brought the level of public scrutiny to entirely new proportions. Despite her famous last name, the 23-year-old said she felt like she was still able to maintain a sense of normalcy, but when she got married and moved to Los Angeles, “all of that kind of went out the window.”

“To me there there has to be a balance,” she said. “I don’t think it’s healthy to try to be extremely isolated because you don’t want to go outside and you don’t want to leave your house because you know people are gonna follow you with cameras. But at the same time, you have to understand that that’s been a struggle for me.”