Ratings on my novel
In case it eases your concerns, I -- and several others, certainly -- don't care about the overall star rating when looking for new reading material, especially if it only has a single-digit rating count. The quality of the synopsis and cover (if applicable), and general trend in reviews is usually what I pay attention to to tell if a work is worth reading or not. Even if your story was rating 2 stars overall due to hate ratings, that wouldn't affect my judgment of your work.
Try not to let it get you down!