Most historical harems weren't because the man was such a Casanova, but due to obligation and horrible gender roles/ imbalance of power. The most realistic version of it nowadays are healthy poly relationships, and they are a lot more complicated than: You did XY for me, so please let me have your children!
The typical idea of a harem is build upon bad writing. It prerequisites shallow characters, mental gymnastic on the reader's part and a lot of lucky happenstances for it to exist. Maybe a great writer could still make it work, but I have yet to see it.
On a personal note, being in a relationship with one person can get exhausting at times. Being in one with four or more people without unfairly neglecting anyone? Good luck!
(typical being: its one to many, they don't have a romantic interest in each other or outside the harem.)
edit.: xianxia, which I also am not a fan of, is not as disruptive to the story. There is nothing quite like a handful of cardboard cutout characters that need to be conquered by the MC to put the plot on hold.
Same thing with lucky happenstance, when you get down to it. Ultimately, if the protagonists of many stories weren't insanely lucky, the story wouldn't happen at all. That a story is written about somebody at all, in many genres, implies an unrealistic base level of luck. For example, how unrealistic is it that Frodo just happens to be the favorite cousin of a famous adventurer, whose parents just happen to die, leading to a series of events which lead to his becoming that famous adventurer's heir, and inheriting the most powerful artifact in the world? Never even minding the fact that they share a birthday. Then, after seventeen years of nothing really happening, he is finally convinced to leave at the exact last possible moment, and so on and so on. An alternative story could be told about the insanely lucky person who rose to fight the evil after Frodo was killed and the ring stolen, and the land cast into darkness. Or, if that person died, some other lucky person. Whoever the story is about, by their nature as a protagonist, is in some sense lucky (or unlucky). The story being told requires somebody have that particular kind of luck.
A story about a person with a harem may likewise require a particular kind of luck; else, the story would be about somebody else, who did have that luck.
So luck itself isn't necessarily a problem, it's when the luck feels unnatural and out of place, when the suspension of disbelief is popped, that it becomes a problem, and that problem isn't unique to harem stories.
It comes back to "Harem tends to be written by people who, for a variety of reasons, don't actually write it very well." There are two takeaways here; first, if you think you can write it better, and you want to give it a try, feel free. Second, if you don't think you can write it well, and you want to give it a try anyways - again, feel free, because that's how you improve.
All that said - I'm more likely to skip over a story that has the harem tag on it. Because, again, it usually isn't written well; like a typo in the story's title, it may not guarantee the story is bad, but it does mean it is far more likely to be something I don't enjoy.