Re: On The Subject of Ratings

#1
Hello all, SenescentSoul here, author of Delve, one of the highest-rated stories on this site. Wait! Before you write this post off as me tooting my own horn or anything, read at least the premise. It is this: five-star rating systems are crap. I will tell you why.


Firstly, though, I’m done caring about my rating.


But you are writing this article, clearly you care.”



Yes, fine. Of course I still care, it’s really hard not to. I’m not here to whine, though. This is a problem that hits everyone, and not just on RoyalRoad. When was the last time you even considered buying something off Amazon that was rated less than 3 stars? Yeah, that’s what I thought.


I suppose what I really want to say is that there are inherent flaws in rating systems that ask users to rate something using a single rating on a scale of 1 to 10. It gets even worse when that something is, say, a web serial. A lot of authors who are just starting out have a rough time of it for their first few chapters. Mine were pretty bad before I revised them. I know that for sure. I’m afraid to go back and read them, even now. If I did, I’d be tempted to revise them even further, and then I’d never get the next chapter done. What happens when someone rates an entire story on just the first few chapters?


Let’s back up a bit. I’m getting a bit ahead of myself. Don’t worry, I’ll come back to the issue of judging a book by its cover. There’s an issue with the star rating system itself. Even if everyone waits to rate until they’ve read the whole thing, there’s a fundamental problem that makes it mathematically unsound.


What problem?”


As I see it, it comes down to weighting, differentiation, and honesty. Let’s deal with the last one first. Also, buckle up, this is going to be long.


How honest is a rating?


Let’s say I’m a reader. I love a book to bits. Absolutely the best book I’ve ever read. I rate it 5 stars because that is the max on Goodreads, RoyalRoad, Amazon, or wherever. Pretty honest, about as honest as you can get, really. Right?


Sure. There’s no problem with a five-star rating in terms of honesty alone.


Now, let’s say I’m another reader. I read the first page of the novel and hate it immediately. Something about it just rubs me the wrong way. The author has made an attack on everything that is wholesome in the world and deserves to be ostracized by the community. I give the story 0.5 stars to tell the world of this crime against literature.


Also not a problem.


What?”


You heard me. 0.5 reviews are fine. In terms of honesty, anyway. They are about as honest as it gets. More honest than 5-star reviews. A 0.5-star review shouts “I HATE THIS” from the mountaintop. That’s a perfectly valid and honest position to take.


Okay...I’m not sure I see where you’re going with this...”


Bear with me. If it helps, you can imagine me as an actual bear typing this with my claws. No, don’t do that, that’s silly.


To see the problem, we need to talk about the second issue a little bit. That’s the issue of differentiation.


The problem here is twofold. As a reader looking at average ratings, I want the rating to give me an indication of how good the story is. Let’s stick to the 0.5 to 5 point scale because that’s what RoyalRoad uses. I’m using my bear claws to type this into the RoyalRoad forums, am I not? Didn’t I tell you to stop imagining me as a bear? Bears can’t write.


That’s speciest. 0.5 stars.”





I should stay on topic.


If we’re talking math, and you know I love my math, the average story on this site should have a rating of 2.75 stars. That is the average of 0.5 and 5. Absolute middle of the pack. The story that epitomizes the sentiment: “Eh.”


Clearly, that’s not the case. No, I don’t know that for a fact. I don’t have the distribution to look at, but I don’t need to see it to know. That’s because of the second problem. The problem of differentiation.


Let’s take the case of the first reader from before. Let’s pick a name for this person. We’ll call them SuperFan19. SuperFan19 just gave story A a rating of 5 stars. To fill the gaping void in their soul while they wait for the next chapter, they need to find something else to read. Against all odds, they find a story that they somehow like MORE than the first one. Story B. How can SuperFan19 tell the world about this?


Story B: 5 stars.


Now we have a problem. SuperFan19 now has two stories that they’ve given the same rating, though they clearly like one more than the other. There is no differentiation between them. What can SuperFan19 do about this?


They could go back and decrease the rating on the other one to 4.5.”


True, they could. They won’t though. SuperFan19 reads hundreds of stories, and ranking them all individually is unrealistic. You can’t keep going back and changing your ranking every time you read something new. Besides, there is another problem. SuperFan19 still loves Story A. They still think Story A is amazing and everyone should read it. However, story A has an average rating of 4.6 stars. If they give it anything other than a 5, they will actually bring the rating DOWN. They can’t do that. Story A is amazing.


Is ‘Story A’ Delve?”


Stop looking at me like that. Yes, I am biased. Fine, let’s say it is. For the moment, Delve has a rating of greater than 4.5 stars (we’ll see about that after I post this. Anyone seen Nosedive? Great episode of Black Mirror. 5 stars (Yes, I am aware that I just used a star rating in an article about why star ratings are bad (By the way, Netflix doesn’t use star ratings anymore. I wonder why that is? (YouTube either… (Stop! You’ve used too many parentheses, any further, and–(((̸̧̧̑̔̌͜(̵̨̹̰̺̟̜͂̐̐̃̒͊Ẁ̸̢̧̥̗̹̝̃̒̐̓͝H̸̛͖͖̩͊̌͝O̷̠͎̘͇͂̈͆ ̸̡̫̻̻̺̜̂͊̂D̵͚̆̓̉́̕Ą̷̧̺͓̈́̇R̸͉̄͗́͛͝E̴̖̗͖̭̓S̶̢̱̋̂͝͝ ̴̪̲̬̬̗̀͆̃͝Ș̷͍̈̽Ư̵̝̺͑̂̐̽̔͜M̸̛̗̜̤̬̞͇̈́̒̒̿M̴͉̣̯̯̥͂̃Ȯ̸̪͕̮̝̑͊̽̋͠N̴̡̛͙̦̫̹̘ ̶̟̗̙̥͓̊̚X̸̗̪̟̞͆͌T̷͕̮̐Ḧ̶̲́͑͘U̴̫̒̓L̴͔͓̞̆̿͊̈́̓̂͜ͅͅU̵̩̫̍̽͝X̵̣̘̀́͐͗͂B̶̥͒͂̓̌́U̷̢̬̭̟͍̺͛͛͒̂̓̎Ẍ̴̨̛͍̺̤̠́̌͌D̴͖͍̰̘̻̻̋̐́̉̕̕,̷̧̗͊́̂̃ ̴̺͍͐͗̕Ḑ̶̩́͆̓̕Ę̷̫̗̮͇̮͌͗V̸͙̰̣̠̭͆͑̀̾̐̌Ŏ̵̩̙̬́͜U̵̖̬̟̣̟͓̇͗͋̊̓͝Ṟ̸̰͕͆̌Ẻ̶̯̝͚̾̀̽̕ͅR̶̠̤͖̟͚͎̾̒̋̉ ̷̰̋́̾O̵̠͝F̷̯͚͎̤̒͋ ̸̝̲̩̻͙̲͛G̷̢̡̢̩̩̹̅͗̕R̸̡̥͕͙̓̒̉͆Å̶͎̐̄͛̔M̸̧̹͖̱̻̯̾́̆M̸̺̰͓̜̘̻͛͛́̓͛A̸̲͗R̸̭͐͜?̵̭̫͎̀͌̚͝!̸̜̌̄͂?̷̧̞̠̹͖͖̉͗͊̚ ̶͇̺̳̌͒͑͘(̵̢͋̂́͗̔(̷͈͚̔͑͊͂̌͠&̵̭̣̙̠͒́̓͆͒*̵̣̦̍̔̚$̷̰̱̹̮̦͒̎̾̕(̵͍͎̳̭̯̦͛̐


Well, that happened.


Anyway, for a story with a rating greater than 4.5 stars, anything less than that hurts. SuperFan19 can’t do that to the author of Story A, so they don’t. They leave both stories at 5 stars. They can’t differentiate between them, and that is a little annoying, but they get on with their life.


Let’s look at the second reader now. They need a name too. How about EasilyTriggered123. Same problem. The latest story that they have read, Story C, sinks to depths of depravity hitherto unfathomable, even in the stinking cesspit of filth and corruption that is Story A. What can they do?


0.5-star rating for story C, plus a nasty review in which they wax eloquent about the many failings of the author and anyone ‘dumb’ enough to enjoy reading anything like this.


EasilyTriggered123 would like to remind us that it is always possible to sink lower. That’s another problem that I won’t get into right now, namely that the more extreme the viewpoint, the more likely that you will feel the need to tell people about it. The term for it is ‘selection bias’, and more qualified people than me have already written plenty about it. I’ll oversimplify: you are more likely to rate something if you feel strongly about it. Mid-range ratings are underrepresented in the dataset.


That’s exactly the problem. Let’s take a third reviewer. xXHonestManXx. Choice in screen name flair aside, xXHonestManXx is just that, honest. He thinks Story A deserves a rating of exactly 3.5 stars. Good, but not great. He rates it and goes about his day.


How is his rating counted? Not accurately is how. I’m not going to go into the math, I’ve spent enough time typing this as is. I really should be working on the next chapter of Delve…


YEAH, GET BACK TO WORK, WORD SLAVE!”





You people are insatiable.


As I was saying, it isn’t accurate. Because of all the dishonest reviews floating around, it is impossible to combine this honest rating with them without breaking statistics. You could try some elaborate system of weighting each user’s vote by their voting history, how many pages they have read, yadda yadda yadda, but it is a task doomed to failure.


That touches on the third problem, weighting. How do various ratings affect the average?


How many 5-star ratings does it take to offset one 0.5-star rating? It depends on how highly the story is rated. For Delve, because the rating is so high (...for now *nervous author noises*) a single 0.5-star is like a bombshell. It takes a lot of 5-star ratings to cancel it out.


That’s not a problem in itself, that’s just statistics. The problem is that the 5s and 0.5s aren’t honest.


A lot of the 5s are there because nice people like SuperFan19 are trying to help the story. The ratings are inflated. A lot (all – it is bimodal AF. This time, I have seen the distribution) of the 0.5-star ratings are there because of people like EasilyTriggered123 trying to bring the story down. If everyone was like xXHonestManXx, then there would be no problem, but people aren’t like that. Rating something is incredibly difficult. Tell me, of the movies that you have seen, which is your 127th favorite? Even if you have that list, can you honestly say that not a single thing on it is out of order? Humans are biased. I am one, I should know.


I thought you were a bear.”


I am ignoring you.


Now we get to the meat of the issue of weighting. SuperFan19 and xXHonestManXx read the whole story before making their decision. EasilyTriggered123 read one page, yeeted the 0.5 star button, dabbed, and ran away. A better person would have just not rated it. If a story is not for you, just don’t read it. I’m not saying you shouldn’t rate things down if you honestly think there are issues. There need to be both positive and negative ratings for the system to be worth anything. I’m saying that you shouldn’t rate things down just because it isn’t your genre, or it contains something that is personally offensive to you. In a star system, giving something the minimum rating based on the first chapter is pretty much the definition of judging a book by its cover. If you do read it through to the end on pure grit alone and still think it deserves 0.5, good on you, rate away. Also, maybe get a new hobby.


Yet, people will still rate based on the first chapter, or early-on when triggered. Your rating system shouldn’t weight this type of vote heavily, but in the case of a star system, it does. Because EasilyTriggered123’s vote has the same weight as everyone else’s, that 0.5 review has a devastating impact on the average rating, far greater than those who have been reading along for the past 50 chapters. Those people have no way to fight back. They’ve already rated 5, whether they think the story is a 5 or not. What more can they do? A nice, well-written review, sure, but that’s not nearly enough to counteract the devastation of the incomprehensible existence that is EasilyTriggered123. The math doesn’t lie. 0 is much further from 4.6 than 5 is, so the percentage change to the average due to the new rating is bigger.


One last point before I propose a solution.


Oh, thank heavens, he’s almost done.”


The star system is not statistically fair, true, but it is fair in that it applies to everyone equally. 0.5 stars are a problem that everyone has to live with.


Now to talk solutions. How can we fix this? I don’t think it can be fixed, at least not completely. It sure can be better, though.


Let’s get binary. No, actually, trinary.


Each chapter gets a thumbs up and a thumbs down. Three possible states: 1, 0, and -1.


How does it work?


Well, if it is time to rate a story, it is dead simple. You read a chapter, if you liked it, you press the thumbs up. You didn’t like it, you press the thumbs down. Of course, some people won’t press either, but like I said, no solution to this problem is perfect. Selection bias is still going to be there.


For the purpose of ranking stories, what does it do? How does it work?


Let’s say you have only one chapter. Let’s also say all chapters of all stories on the site are the exact same length. The rating shown on the front page is the ranking of the story based on the ratio of likes to dislikes. That gets you a number between 1 and -1. Stories with a rating of 0 are average. Positive ratings are “good” stories, and negative ratings are “not bad, just misunderstood”.


How do we deal with actual stories, not just spherical ones in a vacuum?”


Good question. Real stories have more than one chapter. Each chapter gets its own rating. This compensates for people who read three chapters, then rate the whole thing.


The overall rating is a number between -1 and 1. Call it R.


The overall rating is the weighted average of the chapter ratings for all users. Let’s make the list of all those averages into a vector and call it CR. The weights can be a similar vector, W.


Wait, vector? I don’t–“


What is W? The percentage of the story that the user has read. Based on word count, not chapters. Not all chapters are the same length, after all. We need to account for that, but for the purposes of this, you can just think of it on a chapter basis.


So R = average(CR * W)


Watch out, that’s a vector multiplication in there.


Why are you like this?”


I don’t understand the question.


Let’s have an example. There are only 3 users on the whole site. SuperFan19, EasilyTriggered123, and xXHonestManXx.


There are only two stories on the site, Story A and Story B. Story A has one chapter. Story B has two. Both chapters are exactly six words long.


These don’t sound like real stories. There’s a minimum wordcount, too.”


Quiet.


Story A, Chapter 1:
SuperFan19: read it, liked it → 1,1
EasilyTriggered123: read it, hated it → 1,-1
xXHonestManXx: read it, meh → 1,0


Story B, Chapter 1:
SuperFan19: 1,1
EasilyTriggered123: 1,-1
xXHonestManXx: 1,1


Story B, Chapter 2:
SuperFan19: 1,1
EasilyTriggered123: 0,0
xXHonestManXx: 1,0


What are CR and W?


Story A:


CR = [1, -1, 0]
W = [1, 1, 1]


Story B:


CR = [1, -1, 0.5]
W = [1, 0.5, 1]


Wait, where did those 0.5s come from?”


Each number in there is an average for each user.


For CR, the 0.5 means that xXHonestManXx has given 2 ratings for this story, a 1 and a 0. Average them, get 0.5.


For W, the 0.5 means that EasilyTriggered123 has only read 1 of the two chapters for that story. They still get a value of -1 for the rating itself. Of their 1 review(s), all 1 of them were negative. The average is -1.


How do the ratings shake out?


Story A:


R = average(CR * W) = (1*1 + -1*1 + 0*1) / 3 = 0


0 means the same number of people liked it as disliked it.


Story B:


R = average(CR * W) = (1*1 + -1*0.5 + 0.5*1) / 3 = 0.333...


More people liked it than disliked it.


You take all of those ratings for all the stories on the site and put them in order. That’s the new best-rated list. You can even normalize it to a 5-point scale if you want, even though it is all thumbs underneath with nary a star in sight.


Also, check my math. I’m not a real statistician, I just play one on TV. I’ve also haven’t had my coffee yet.


Wait, this is you without coffee?”


Think about the system a bit. Would it be better or worse on the issues I described? What do you think? Am I glossing over something? Am I just plain wrong?


If we were to move to a system like this, there is also the problem of what to do with all the existing ratings. It isn’t like you can just ignore all that. No matter what you do, you have to compromise somewhere. Maybe something like this could work alongside the existing system. It also has a nice fringe benefit in that it gets you a rating for each chapter individually. That way, you can see where you really started pissing people off with your comments about bears.


Anyway, that’s all I have to say about that for now. Keep on trucking.


Re: On The Subject of Ratings

#2
Alright, while I believe the idea holds some merit, it has some obvious downsides. Yes, you have solved the problem of certain stories being underrated, but you have now made the stories overrated, and here's why.

According to your math, a person needs to read the entire story to give it a fully weighted dislike. The problem with that is if a story is terrible enough that most people don't finish reading it, then it's going to lean heavily towards those who are super fans of the story, as they can somehow withstand what the rest of us cannot.


While I do agree that the rating system needs to be updated and needs a better system than it has now, I don't think this is the way to do. Your idea needs a little bit of modification, though I'm not entirely certain how.

Re: On The Subject of Ratings

#4
Sorry to say I agree with HURGMCGURG, your idea does have some merit though. Maybe make it so you have a limit on the weight of a rating? For example, make it so that someone who dislikes only the first chapter have a weight of 25%, where it ramps up to 100% over the course of four chapters. That way, negative reviewers don't have to read a crap story all the way through to call it crap, but people who only rate off of the first chapter still don't have as much weight. Of course, the latter could just rapid fire dislike the first four chapters, but that could be partially solved by keeping it as a hidden statistic. Just some food for thought, the topic of rating systems definitely deserves to get discussed.

Re: On The Subject of Ratings

#5
A more blunt solution would be to to have two ratings, active reading rating that works as OP says and a more simple like/dislike that doesn't factor in the amount you've read; or at least barely factors it in.

As in some whats the person who gives up in the first chapter or so has valued feedback as well.

Maybe something to show/take into account how many people have read some and dropped it and take into account how much they have read. But no clue how one could implement that. It would be complicated but I believe it could be good feed back.


Edit: Personally I really dislike the star system. I thought it was useful for netflix as it made you feel as if you were rating things so netflix could help you find stuff you like, not rating it to rate it.

I feel as though 1/10 systems are good for when you are teaching a system to recommend stuff to you but bad for things like RR, reddit, or well anything when being simple doesn't mean less accurate.

Re: On The Subject of Ratings

#6
I would say put the hard limit at about 5 chapters. If you hate a story after 5 chapters you are unlikely to change your mind.

I do agree with the 1, 0, -1 though. That is pretty clever. I do occationally give 4/5 stars to books I consider 'okay' but not fantastic, but I agree that a 0 would be a better response to that.

Re: On The Subject of Ratings

#7
Three suggestions for how to account for the haters not reading past chapter 1 while the fanboi's rate every chapter:

1) it affects every story equally so the rankings are fair in comparison to other stories being rated the same way.

2) make the minimum votes you can give a story 10 - if you rate only one chapter your vote gets counted 10 times - as you keep reading and rating the multiplier on your initial vote gets reduced until your 11th chapter when each vote just counts as one point.

3) right next to the -1 "did not like this chapter". Have a -10 "I hate this story so much I want it removed from my feed" - if you choose the blacklist option then story is hidden from you. If you deliberately seek it out and try to rate a second chapter, your initial -10 rating gets converted to a standard -1 rating.

Re: On The Subject of Ratings

#10
I do quite agree that the rating system as it currently is, is simply so flawed, though on the other hand, its also very hard to change something so critical to the site, when its just so much woven into what the site is, even if there will be changes, expect it to take months before they are even scripted, tested and ready.

I myself am also one of those who do that kind of ratings, in some cases not because i think that they are actually that good, but because giving a lower 'honest' rating would drive the overall rating lower, especially if its a story with just a few ratings, such can be quite hard on the author who recieves one.

Re: On The Subject of Ratings

#11
I like the concept, however I think we can always use more data.
The concept of up down and vector seems right. However, I'd stipulate that it should be done overall and on each tag.
Perhaps easilyTriggered just wants a smut and they see what you do with Amelia and thus move on.
Now if easilyTriggered were to go do a search by tags and find a 0.23 romance, they could rest easy.
Someone could rate it a chapter poorly on grammar, but give it a thumbs up on romance or something.
Romance, fantasy, isekai, etc.
People could short it out and just do an overall rating, but if they get in the weeds they rate each tag. If no element was present for romance in the story, it is left blank at "meh", but if the book series starts to get into romance and it is done in a cringeworthy fashion, that can help the entire community know.

If SenescentSoul started getting an average rating of -0.15 on inclusions of a female lead for the romance, he'd know perhaps what their reader base is starting to yearn for. Especially once you get up to more than 20+ chapters.

Having a stat for % of readers that kept with the book series could be valuable as well. This would be based on ratings of first chapter vs number of ratings on the last chapter - 30 days. The -30 days to give stragglers time to catch up. That way we know that a book that is rated as 0.76 and a retention rate of 0.1% we know that it could be a very niche story. Where a rating of 0.08 and a retention rate of 63.2% could tell us that people who pick up the series enjoy it in the long run and could help people "slog" through the first few chapters to get to see why so many people are sticking with it.

So, to summarize.

  • I like the up/down vote methodology

  • I believe it could positively benefit the site as a whole if done by tags

  • Also having a % picked up and continued reading. If you first chapter has 105,000 reads/ratings but chapter 50 only has 105. The book series retention rate is 0.1%. This could tell us of the number of people that gave this series a try how many kept with it. Never do this stat on the most recent chapter, but maybe based on the trailing 30 day chapter, thus giving stragglers to catch up on their reading to not negatively impact the stats.

Re: On The Subject of Ratings

#12
hi,
let me give you my point of view / make a comparison with other rating-systems(without reading all of your argument, i stopped at “I thought you were a bear.”, sorry but i still have a chapter to read)

how i see the royal road voting system is very simple,
3,8 is the point where I say: still going to give it a try
i give storys with a lower rating a try, as long as there are less than 10 ratings
this is the way i rate on royal roads

  • <4: well something went wrong there

  • 4: like it

  • 4,5: like it alot

  • 5: love it (yes, delve is here)

  • if a story i like has a slightly higher rating i don't vote

  • if i don't like it, not because it is bad, but because i don't like it, than i don't vote
    (i am planing to read LITRPG storys, if i stumble upon a yaoi school drama i don't rate it)

yes, that is annoying but that is the only way to keep it in relation to the other readers


the rating-system of amazon is better than the royalroad-system,
you can see how the rating is devided


the system kongregate (a free online gaming site) is very good
it gives you descritions to the stars (hate it || don't like it || like it || like it alot || love it)
it also gives you both "the games rating" and "your estimated rating" on a game, based on how you voted other games, and based on how others who voted similar to you vote this game [yes that is a bit of a bigbrother nightmare, if i understood how it works correctly, also it is very useful]
you don't vote only to give feedback, but also to have an easier time finding other stuff you like. similar to youtube and the video suggestions

Re: On The Subject of Ratings

#13
Yeah, this is a known problem with rating systems, with the core issue simply being a misalignment of the median story with the median voting option.

If you have a 10-option system where only one option out of the 10 (or maybe two of them) means 'I liked this', you're going to end up with a system best-suited for parsing out exactly how much disliked works are disliked, which isn't particularly useful.

The solution is as simple as it would be hated: Scale the results to match the spread you want.  After a site's been around long enough to establish enough data, change everyone's rating (which, in practical terms means drastically reducing everyone's rating), so that the median story on the site has a rating of 3.0 stars--or, if you're trying to be a bit more opaque, do this as part of a revision where you change the 0.5 to 5 system to a 1 to 10 system, but give everyone scaled values instead of just doubling them.

Lots of details involved (e.g., do you weight stories by number of chapters and/or number of votes when ordering them and finding the median?), but the essence is that if you want a system where 3 stars is average and 4 stars is good, then you need to forcibly deflate everyone's rating until that's actually true.

It goes without saying how horrific the howls of anguish would be if an author learned their 4.6 story was suddenly going to become 3.4 (and how easily it could lead to drama and to popular authors leaving the site over it), which is why these kinds of system have so much inertia.

I have actually seen this work, once.  Years back, StoriesOnline (NSFW) re-scaled their scores (so, e.g., the then-average rating of 8.62 became 6, 9 became 7.1 and so on).  The two other key changes they made were (a) the top 5% and bottom 5% of votes were no longer used to calculate the rating, and (b) pretty crucially, instead of selecting a number, the system now had you rank a story by selecting a descriptor from an ordered list (i.e., "Amazing", "Great", "Very Good", "Good", "Not Bad", etc.).  That was 10+ years ago, and the system managed to stay properly scaled after that.

Re: On The Subject of Ratings

#14
Having readers vote on each chapter seems like a pretty bad idea. It's hard enough to get readers to rate the story once, as a lot of readers on here aren't even logged in users. Making it something that has to be done repeatedly would be terrible, especially for stories that have higher ratios of unregistered readers:registered readers. Other sites, such as scribblehub and wattpad, have a vote per chapter button (or at least a like/chapter button), but I don't find it particularly useful as a metric.

I also think that any system that relies on weighting towards "how much has a reader read" is very questionable, since currently afaik RR only tracks the chapter that a person has most recently clicked on. It seems like it would be a pretty big burden to change that to track some other statistic. So currently, a person can click on the most recent chapter of a story (which they often do by just clicking on a story in the latest updates) and it will "look" like they've read the entire story.

Any change to the rating system would additionally involve taking the ratings of older/completed stories, as they no longer have the "traffic volume" of readers to upvote them, especially after they're kicked out of high visibility spots. Long, in progress stories would additionally be at a disadvantage, because older readers are unlikely to go back and upvote early chapters of a work. Maybe you see it as a benefit to shake up the top rated list and status quo, but it would be a big shakeup that would upset a lot of people.

Personally, I believe that the biggest way that ratings could be made more useful to the reader would be to show the chart of how many users rated a story a certain number of stars-- the same one that's shown on the stats page of the author dashboard. That way readers could more easily judge for themselves if a story has been review bombed with 0.5s or if it has many justifiable middle of the road reviews. I'm not here saying I love the current rating system, as it has a somewhat mysterious backend in terms of how it translates to actual ratings, but I don't think it's worth it to change dramatically.

Re: On The Subject of Ratings

#15
I can still see the mean score shifting to the right with this system.

That's because people who hate the story will be less likely to finish it, meaning they will only 'downvote' a fraction of the total chapters. In contrast, those who love it will probably read many more chapters, leaving upvotes on each of them. That systematically shifts the distribution to the right.

Even so, rightward shifts are probably inevitable for any practical solution here. Yours is good because it shifts the distribution closer to the center, but it'll never be perfectly aligned without implementing a negative comparative curve.

Edit: Basically what Tagmin said in the post below this one

Re: On The Subject of Ratings

#16
Overall this seems like a solid system, certainly more fair than a flat-out star rating system. The largest problem I see with this rating scale is a disregard for the value of a .5 star rating. Every author hates a .5 star rating, as most are seen as undeserved hate onto the work that was delivered by some troll. For the most part, this is true, but the .5 star rating isn't completely without justification. As you stated, the ideal star system has the "eh" works sitting at around 2.5 stars, straight in the middle. Looking at RR's rating scale, 2.5 stars (excepting a few outliers) is the absolute bottom line. If you take a reader who hasn't perused through the enormous amount of stories on this site and isn't familiar with the common rating standard, 2.5 stars for a clearly sub-standard story is way too high, and as such, a .5 star rating has a justified delivery to balance the scales more towards the ideal rating system. 

Your plan identifies that low ratings are necessary for some situations, so you assign a weight to everybody's score based on how much of the story they have read (hence the .5 weight for the guy who had only read 1 of the two chapters). The problem comes in when you have somebody who's a diehard fan of a story (whether for the quality or the concept) who reads every chapter and consistently delivers a thumbs up (1 avg and let's say he has read 30 out of 30 chapters). Then you take a guy who goes two chapters into the story, immediately dislikes it, thumbs both chapters down, and moves along his merry way (avg -1). 

The weight-adjusted ratings of these two readers are [(2/30)*-1+(30/30)*1]/2 = .933 according to your algorithm (I may have misread the formula, so let me know if I got this wrong). The "troll" who rated poorly but only read two chapters has been defeated by the weighting system right? Well, his opinion should be just as valuable as the diehard fan, so no. By an ideal system, the system rates the story exactly where the true public sentiment is at, 0. But not everybody delivers a poor rating because it is deserved (og .5 star trolls), so his score must be weighted down to account for the people who don't respect the sanctity of the rating system. We can't have the best of both worlds there, so some sort of negotiation between weighted and nonweighted must be created. Assuming that the majority of people who deliver low ratings aren't just bitter trolls, an exponential weighting system could help balance the ratings by making sure that trolls get mostly filtered out without destroying people who authentically don't like a story.

For example, Story A has 10 chapters and three readers; GoodReader12, CynStory52, and xxJoeMama6969.

GoodReader12 likes the story and has read all 10 chapters, with each getting a thumbs up, so his scoreboard is R1 = [1 1 1 1 1 1 1 1 1 1]

CynStory52 doesn't like the story as much, but he gave it a try and read three chapters, both getting thumbs down, so his scoreboard is R2 = [-1 -1 -1 0 0 0 0 0 0]

xxJoeMama6969 has a bitter resentment for the world and rates any story on trending as low as possible but doesn't care enough to rate each chapter, so he only rates the first chapter. His scoreboard is R3 = [-1 0 0 0 0 0 0 0 0 0]

If we were to linearly filter out scores by weights that depend on the percentage of the story read, then CynReader52's valid opinion would be destroyed. However, if we to weigh ratings based on some decaying logistic factor (see internet for an example) from percentage read, say w=k/[a+b*e^(c*n-d)]+f where n is the rating number (by chapter starting from 0) and k, a, b, c, d, and f are arbitrary variables, then the weighting plays out a little more fairly. An exponential decay with a y offset would do essentially the same trick. The point behind using an exponential/logistic decay rather than a linear one is it would take all of that extra weight off of regular readers (which would basically only rate thumbs up, or they would just drop the story). 

I'm way too lazy to do the number-crunching behind it, but it would play out with earlier ratings counting more towards the average than later ratings, up until a certain rating number f (can be tuned so that all of your ratings after a certain amount of ratings given would have the same nonzero weight value). This would ensure that the maximum amount of trolls are filtered out without giving excessive power to the diehard fans and quenching the opinions of the critics.  

TLDR; A linear weighting scale would give longtime readers disproportionate power over ratings (same problem as now with abnormally large ratings, then people review bombing to compensate). Instead, the weight of each review should decrease relative to the one before it, so people who stick with the novel still have more power, but not to the crazy extent that a linear scale would give. 



Re: On The Subject of Ratings

#17
Wait so in your system would the default be a score of 0 if you read a chapter and neither upvote nor downvote it? You never mentioned a third button for xXHonestManXx's meh ratings, so I assumed so. 

Though if that was the case, then any long series' ratings would drift close to 0 assuming many more people read it then bother rating it. Especially since anyone catching up on the story would contribute a bunch of 0's unless they experienced chapters they really liked/hated enough that they broke immersion to rate the individual chapter instead of continuing to read.

If that's not the case (there is a 'meh' button that has to be clicked to give a rating of 0), then I'm not sure how to extend the system to people who don't rate every chapter. Like if someone has read 100/100 chapters, but only bothered to upvote 2 of them, would their value in CR be +1 and in W be 1? Though that would make their contribution to the story's rating equal to that of a fervent fan who upvoted every single one of the 100 chapters. That seems like a major flaw of the rating system, especially if you consider the case where these two fans are following different stories which is what caused the difference in voting.


I personally don't give ratings on this site. Is that irresponsible of me as a reader, especially since it does influence what I read? *shrug*. The problem is that I would only bother to rate stories I like, and of those I would have trouble differentiating their scores. I'd also have trouble rating those stories that I followed more or less happily for tens or even hundreds of chapters before I found myself repeatedly frustrated or asking myself why I keep reading them (at which point I drop them because I have better things to do with my time, including reading other fictions I still emphatically enjoy).

I mean RoyalRoad has a sort of binary (possibly trinary/quaternary depending on how you view combining them) review system already in place in the form of follow/favorite. The number of followers shows up next to the story in all of the site's various ratings lists. It doesn't have the option of showing negative feedback or even a percentage for the positive feedback, but if lots of people are following a fiction I'm more likely to pick a story up and tough it out through bad beginnings. 

I pick up new fictions mostly from the trending section actually (where I found Delve among others). I figure it's more isolated from the ratings bs you talk about, but even if it's not it's still a good place to start looking as opposed to the entirety of this site. It also has the benefit of not being mostly stagnant like the top ratings section.

Re: On The Subject of Ratings

#18
jjetpack Wrote: I pick up new fictions mostly from the trending section actually (where I found Delve among others). I figure it's more isolated from the ratings bs you talk about, but even if it's not it's still a good place to start looking as opposed to the entirety of this site. It also has the benefit of not being mostly stagnant like the top ratings section.



I think that what really gets people going is that the problem is not one actually of "ratings" per se, but of discover-ability. Trending is good because it does allow fictions that are not necessarily high rated to get some time in the sun, but it's a bandaid at best-- one that's designed to give new fictions a brief shot at exposure and thus a chance to gain readers/climb rankings fast. There are vast numbers of good fictions that maybe got onto trending a while ago, but failed to climb the rankings list enough to be discoverable through that. If there were better ways for "midlist" fictions to be found, the rating/ranking number wouldn't necessarily matter as much and be so fraught. After all, it's not the number value that people actually want-- it's more readers to appreciate what they're putting out in the world.

That's just my reaction to the whole thing though, and maybe I've veered slightly off the original topic haha.

Re: On The Subject of Ratings

#19
I find it so amusing to see more math in this author's post, very fitting of him

And yes, the rating system is broken, the other day i saw a review that admitted to only have read half of the first chapter, to think people like that have as much weight as anybody else is just bullshit

And tis user touches many fair points

javert Wrote: Having readers vote on each chapter seems like a pretty bad idea. It's hard enough to get readers to rate the story once, as a lot of readers on here aren't even logged in users. Making it something that has to be done repeatedly would be terrible, especially for stories that have higher ratios of unregistered readers:registered readers. Other sites, such as scribblehub and wattpad, have a vote per chapter button (or at least a like/chapter button), but I don't find it particularly useful as a metric.

I also think that any system that relies on weighting towards "how much has a reader read" is very questionable, since currently afaik RR only tracks the chapter that a person has most recently clicked on. It seems like it would be a pretty big burden to change that to track some other statistic. So currently, a person can click on the most recent chapter of a story (which they often do by just clicking on a story in the latest updates) and it will "look" like they've read the entire story.

Any change to the rating system would additionally involve taking the ratings of older/completed stories, as they no longer have the "traffic volume" of readers to upvote them, especially after they're kicked out of high visibility spots. Long, in progress stories would additionally be at a disadvantage, because older readers are unlikely to go back and upvote early chapters of a work. Maybe you see it as a benefit to shake up the top rated list and status quo, but it would be a big shakeup that would upset a lot of people.

Personally, I believe that the biggest way that ratings could be made more useful to the reader would be to show the chart of how many users rated a story a certain number of stars-- the same one that's shown on the stats page of the author dashboard. That way readers could more easily judge for themselves if a story has been review bombed with 0.5s or if it has many justifiable middle of the road reviews. I'm not here saying I love the current rating system, as it has a somewhat mysterious backend in terms of how it translates to actual ratings, but I don't think it's worth it to change dramatically.


A system that allows to see how many people have voted each rating would be good, like in novelupdates, and that can still be combined with the "vote by percentage read" system, example:

Average % of the story read - rating number - number of people who gave that rating

20% Five Stars - 70 votes
40% Four Stars - 50 votes
10% Three Stars - 50 votes
10% Two Stars - 30 votes
5% One Star - 5 votes


The idea should be to give the reader a better overview of how the data is obtained and not a final single score to rule them all