Re: On The Subject of Ratings

#62
I have a lot of thoughts about ratings systems, but I'll mention two in particular.

I don't give much for the opinion of habitual trolls, nor relentlessly positive people. Thus, a user's vote should be weighted according to their spread of votes. A 5 from someone with a median vote of 3 is objectively better than a 5 from someone who always votes 5, AND better than a 5 from someone who always votes 1 or 5 because "Why wouldn't I maximize my impact?".

I also avoid voting for fairly nice novels I'd give the a 4, since I'd end up lowering the score. Solution:Change the votes to '5', '>3', '3', '<3' and '1'. Then let every '>3' negate one '1', and every '<3' negate one '5', before votes are averaged. Nicely cuts down on extreme scores.

Re: On The Subject of Ratings

#63
Hi! I'm a lurker, but I've worked with some recommendation systems so I thought I'd chime in on the 5-star vs binary rating systems discussion. I'm mostly responding to @kanadaj's first post. Thanks for all the work on the site! Hopefully I can contribute a little here.

kanadaj Wrote:

  1. Clearly, it's virtually impossible at this point to reliably substitute for the existing rating system without losing far too much data, so it's a fairly simply issue to see and very hard to work around.

I think this is a bit of a misconception. While you lose data converting 5-star ratings to up/down, it's not valuable data. If you do it well, you are actually improving the dataset.

If you look at Netflix, one of the companies with the most work in rating and recommendation systems, they switched from a 5-star system to a binary system several years ago. In the 5-star system, many of the machine learning algorithms end up normalizing ratings for each user. For example:

  • User A rates between 3-5 stars, with an average of 4 stars.

  • User B rates between 1-5 stars, with an average of 3 stars.


When making recommendation systems, generally you end up equating User A's 3-star rating with User B's 1-star rating. Thus, converting these ratings to a binary system is actually a more accurate representation of the dataset than displaying the simple averages in the 5-star rating system.

Regarding rating on a per-chapter basis, I agree that it's too much work for users. I would further add that a binary system makes it even easier cognitively for users to rate. Don't underestimate this effect: for Netflix, the up/down system was twice as popular as the 5-star system. More data is better.

A lot of webnovels have drops in quality, and generally users can figure that out from reviews. I think Steam's separation of "overall rating" and "recent rating" does a decent job of indicating some change over time of the rating while not requiring additional work on the part of the user.

kanadaj Wrote: Now before going through the rest of this thread to answer things, I will start by pointing out that a pure upvote/downvote based system is not any better than the existing system. Yes, a .5 rating statistically moves the average more if you are at 4.7 vs if you were at, say, 4.1.

But here is the thing: How would a downvote change the % of votes if you had 80% upvotes vs if you had 90% or 95%? 

I'll make it easy for you with some simple mathematical transformations:
A downvote is 0
An upvote is 1
80% = 0.8
90% = 0.9
95% = 0.95

Let's multiply everything by 5 (which we can, since it doesn't affect the outcome of the % as long as we divide by 5 at the end of it), shall we? :)

A downvote is 0
An upvote is 5
80% = 4
90% = 4.5
95% = 4.75

See what is wrong here?

Regarding the math at the end of the post, you have to include 1-4 star ratings to see the effects. If you include some 4-star ratings (assuming they convert to 1 in the binary system), you'll see that the overall rating is higher in the binary system. A way to interpret these effects is that previous 4-star ratings would be effectively forced into 5-star ratings. While this may seem like you're losing specificity, because everyone's mental rating scales are so different, the binary system is actually as accurate if not more accurate. This addresses the common author complaint that 4-star ratings should be good but feel terrible.

Regarding review bombing, that's an issue every open review site has, and it's difficult to address purely through the rating system. Steam has addressed it directly by spending time to detect review bombing.

Most people want the review system to answer the question: "Should I read this?" So I think basing the system around a direct question like: "Would you recommend this to others?" is most likely to collect data that matches what you're looking for.

I hope my input is helpful! Please let me know if anything I wrote isn't clear.

Re: On The Subject of Ratings

#64
Replacing the star system with a binary thumbs-up and thumbs-down system would be a terrible idea and I would fight against that with all my willpower. It is basically useless on Netflix where its recommendation systems now consistently overestimate how much you'll enjoy something, and Royal Road doesn't even HAVE a recommendation system. Why would we want the nuance of a ten-point system turned into a bludgeoning binary if there isn't even a benefit to the reader to have this?

The kind of unstated message in these last two pages of posts is, "I hate 0.5 ratings and wish my ratings were higher." It seems like a selfish wish by writers because they want to have a higher ranking and more views to try and eliminate the lowest scores. But it's only that way because everything below a 5-star rating basically hurts your story... due to the fact that so many people bomb everything with 5-star ratings.

I think the problem is not the review system at all, but just a simple community issue where people are far too willing to rate things up or down, sometimes just tactically. It's really tough for authors who get a new 4.5 rating--which is super high--and then see their ranking plummet. The system is not at fault--it's just that people rate and review way too high, and way too low. It's very polarized. 

And, I think I can speak for reviewers when I say that there is a problem with authors lambasting or harassing people who criticize their story. Because giving out anything below a 5 or maybe a 4.5 is harmful, it's basically a bomb set off anytime you give a story something even like a 3.5. I've actually been too afraid of porting bad reviews that I wrote on another website to certain Royal Road stories because I know that authors (or their fans) would likely retaliate with ratings bombs on my works.

So the community has created a strange culture around reviews needing to be very high, and then sneaky anonymous 0.5 ratings done exclusively to hurt others. How do you stop that, other than wait for the teenagers to grow up and be more mature about their ratings? I don't have any good answers, and I doubt there are any cleanly good ones. I like Elaborate's suggestion of weighting individual users' ratings based on their own average score, but I can imagine that going completely out of control, and people would be able to game that with fake accounts if they really wanted to (e.g. making a fake account rating most stories 1 or 0.5, then rating their own story with a 5 for a huge boost). I think that there should be more weight behind reviews and not ratings, especially with reviews that have a lot of thumbs-up, but then dissenting opinions could be lost thanks to mobs of downvoters. It really is a community thing.

Re: On The Subject of Ratings

#65
I went back and forth on if I wanted to weigh in on this and I decided that it was a worthwhile use of my time on the toilet to do so.

TL;DR:
Authors feel bad and gonna keep feeling bad because humans are bad people.
RR needs to do a better job of connecting readers to content because doing so will solve multiple problems.
SenescentSoul's system should be implemented as an additional tool for authors but not as a rating scale.
RR staff should be paid more.

---

So what is this argument actually about? I see a lot of people arguing for very different things without being clear about their goals and what actual change they want to see. From what I've read so far:

  • Authors want:
        ▵Readers to find their stuff
        ▵To feel good and not bad about their writing (author feels)
        ▵Some want feedback on their actual writing

  • Readers want:
        ▵To find good things to read
        ▵To vent their feelings (both positive and negative)
        ▵ <readers should imagine a pithy third thing here>

  • RR staff want:
        ▵less work
        ▵more pay
        ▵not have RR go bankrupt hiring specialists

The following relationships between groups seem obvious:
    ● The venting readers and author feels are at cross-purposes.
    ● The author feels and authors who want feedback on their writing are at cross purposes. Censorship and criticism do not mix.
    ● The less work and more pay are at odds with death and taxes.

The current rating system sucks for all of these groups, except for the readers who want to vent. Even for them, it will feel hollow after a certain point. The review system is actually helpful, I think. Good Job, RR staff.

Now, I'm not smart enough to create a solution that would make things perfect for these groups that have diametrically opposed goals. I am however, reasonably confident that things can be improved by focusing on the two groups that are not at odds: authors who want readers to find their stuff and readers who want to find things to read.

There is no way to find new works except the current rating system. The search system is bugged (try to sort by ascending!) and the underlying tags are near useless for fine searches. There have been novels on RR publishing weekly for years without a single review because works that are no longer on trending and have no reviews which means they are at the bottom of the ranking list. Even all negative ratings and 1 review counts as higher than no reviews, not that anyone will click through 60+ pages of novels to find those with 1 review. 

Were I an author on RR, I'd be utterly pissed that readers are dependent on ratings to find my work.
Yes, bold italics. This is the learning objective.


If there was a better way to find things you might like to read or just a better matching of content to users, I think a lot of the rating issue(s) could be put to bed or greatly reduced in severity. If readers got to see things that they would like to read, they're more likely to give a positive review. Authors will get more positive reviews, readers will have a more positive experience, everyone wins!

This won't completely solve the bad feels from low ratings but Authors will be able to develop a more healthily irrational neurosis about it instead of the very real concern it is now.

The next topic:
SenescentSoul's System / SSS Idea


This is a terrible idea for a rating system for all the reasons listed in this thread and more.
However, as a tool for authors it is brilliant. I think it should be implemented but shouldn't be connected to the rating system.

Why? It will help authors to identify if people are just haters or there are probelms with their writing.
More importantly, it enables authors to zero in on WHERE they done did bad or done gud.
Wondering where your writing has the most impact? Look at total number of votes by chapter.
Story suddenly has a chapter with a lot of positive votes? Immediately copy that success.   
Story getting lots of negative reviews? Look back at which chapter had the most negative votes, read comments, fix.
Got a majorly negative review but no upsurge in disliked chapters? Then no need to worry about it.

Not every reader needs to use it for it to work, either. In fact it is better if not all do.


Does this solve the problem that @AwakenedKingdoms pointed out, with the bad feels? No.
Could it mitigate some of the feels if explained to authors? Maybe. You'd have to sell it.
Does this solve the problem of there being trolls that will find a way to exploit this? No.
Does this solve the problem of the numerical ranking being a terrible way to find new things to read? No.
Does it solve the problem of the current format making it hell to convert works from here to a published format? No.
Does it solve the apathy and minuscule wages of RR staff? No.

However, this is a cheap and easy to code thing compared to solutions for all of those and might act as a band-aid until someone is bamboo... convinced to work for RR that can implement a more sweeping change.
This won't affect the ranking system at all and will give authors another metric to feel comfortable about.

I still think it should be implemented anyway. If nothing else, it'll placate authors and users until something is done to fix other things.

Re: On The Subject of Ratings

#66
And while I'm here...

Binary/Ternary rating systems (like/dislike/unvoted)


Google and Netflix didn't implement these because they are better for reviewing or rating or somehow are a better scale inherently.
They are implemented to simplify AI decision making  on what content to push to a user and the algorithms for such.

@aralbair pointed out some nice positives about them as rating systems. These are true, although I'd like to lodge protest about the unqualified phrase 'more accurate representation of the data.'

However, they should not be implemented under the assumption they will improve rating in accuracy or utility for humans.
The reason the binary/ternary systems are used is because a content provider like Netflix doesn't actually care about how the user feels about a given work. They don't truly care about the user's critiques.
They only care about if they want to see more content like it or not. So, instead of parsing all that extra information, might as well only ask for the information they need, which is if whatever is being shown incites enough emotion to make them click something. No more trying to normalize metrics inter- and intra-user, just, "user hates this" and "user does not hate this."

As was pointed out by kanadaj, you can convert the aggregate like/dislikes to 0-5 rating scale with mathematical wizardry and use this. While his implied argument that they are equivalent is wrong, he is right about it is a bad idea as doing so will result in a very misleading scale because it represents a completely different thing than the current rating system.

Currently the .5 to 5 rating gives a rough estimate (in theory) of "This work is <X> units of good to an 'average' reader."
Aggregating like/dislike into a 0-5 rating would be "This is a normalized percentage of the people who aren't indifferent to the work."
As a reader I find this is a much harder measure to interpret for the decision of "how much can I expect to enjoy this story?" 

It MIGHT be easier on the author to determine "my work sucks" vs "my work is doing OK" but then people will still bitch about how they don't know HOW good they're doing for their target audience and it gives equal weight to people who dislike it and people who LOVE it.
It MIGHT shift the argument from "A negative review tanked my ranking" to "I improved my writing so much but I have so many reviews I can't ever move from my spot."

So, I don't really see how switching to such a system could be expected to fix any of the problems mentioned in this thread.

So, should RR move to a more simple rating scale? Oh, probably, as part of a content delivery system to stay competitive. RR needs to stop thinking and acting like a forum host and start thinking like a content provider or their content creators will move elsewhere.

Authors and readers just shouldn't expect the change to be better than the current system or to provide obvious benefits to them.

Re: On The Subject of Ratings

#67
I actually disagree that this system would be inflated. The issue is the weighted value of CR. You're applying a weight twice, once for the a vote participation weight in CR, then a further word count weight in W. What this does is not only favor people who read consistently, but also people who don't bother voting (0). I'd argue most voters don't vote at all. So they'd contribute a 0 assuming they vote a single time somewhere in the story. Thus most stories will probably favor 0.

Re: On The Subject of Ratings

#68
I feel like a simpler alternative would just be to switch from displaying the rating average to using the rating median. The benefit of this is that it would be less susceptible to having your overall rating damaged by a single bad score while at the same time displaying roughly a more accurate representation of what people think of your work. It also means that the current rating system (along with all ratings that have already been given) wouldn't have to be tossed aside, it would just change the way they are presented.

Re: On The Subject of Ratings

#69
Ratings as well as up- and down-votes of reviews should be reserved to actual readers. I.e., it should be necessary to spend a reasonable amount of time in the first two to three chapters, while they have been visible on the user's screen (not in the background) before a rating or up-/down-vote is taken into account for the score shown to other readers. It should still be shown to the rater himself (herself?), but not to others. That way, many of the most blatant trolls and tactical down-bashers would have a much harder job. It's technically fairly easy to do, and unfortunately necessary because the purely tactical 0.5 and 1 star ratings and downvotes, especially for trending stories, have such a large negative influence. It hurts both writers and readers who otherwise would enjoy RR much more.

Re: On The Subject of Ratings

#70

Entitled Wrote: Ratings as well as up- and down-votes of reviews should be reserved to actual readers. I.e., it should be necessary to spend a reasonable amount of time in the first two to three chapters, while they have been visible on the user's screen (not in the background) before a rating or up-/down-vote is taken into account for the score shown to other readers. It should still be shown to the rater himself (herself?), but not to others. That way, many of the most blatant trolls and tactical down-bashers would have a much harder job. It's technically fairly easy to do, and unfortunately necessary because the purely tactical 0.5 and 1 star ratings and downvotes, especially for trending stories, have such a large negative influence. It hurts both writers and readers who otherwise would enjoy RR much more.


This is only easy to do until you have volume traffic, at which point this becomes an infrastructural nightmare, and extremely hard to enforce.

Re: On The Subject of Ratings

#71

kanadaj Wrote:
Entitled Wrote: Ratings as well as up- and down-votes of reviews should be reserved to actual readers. I.e., it should be necessary to spend a reasonable amount of time in the first two to three chapters, while they have been visible on the user's screen (not in the background) before a rating or up-/down-vote is taken into account for the score shown to other readers. It should still be shown to the rater himself (herself?), but not to others. That way, many of the most blatant trolls and tactical down-bashers would have a much harder job. It's technically fairly easy to do, and unfortunately necessary because the purely tactical 0.5 and 1 star ratings and downvotes, especially for trending stories, have such a large negative influence. It hurts both writers and readers who otherwise would enjoy RR much more.


This is only easy to do until you have volume traffic, at which point this becomes an infrastructural nightmare, and extremely hard to enforce.


Hmm. Maybe on the client side? And there's nothing to enforce, it's all code only.

Re: On The Subject of Ratings

#72

ProbablyATurnip Wrote: I feel like a simpler alternative would just be to switch from displaying the rating average to using the rating median. The benefit of this is that it would be less susceptible to having your overall rating damaged by a single bad score while at the same time displaying roughly a more accurate representation of what people think of your work. It also means that the current rating system (along with all ratings that have already been given) wouldn't have to be tossed aside, it would just change the way they are presented.
the problem with a median is the large number of novels that would have a 5 star rating, how do we then sort through the 18,000 novels that had over half 5 star ratings to find the best, there is considerably more 5 and .5 star ratings than any other type.

Re: On The Subject of Ratings

#73

Entitled Wrote:
kanadaj Wrote:
Entitled Wrote: . . . it should be necessary to spend a reasonable amount of time in the first two to three chapters, while they have been visible on the user's screen (not in the background) before a rating or up-/down-vote is taken into account for the score shown to other readers . . .


This is only easy to do until you have volume traffic, at which point this becomes an infrastructural nightmare, and extremely hard to enforce.


Hmm. Maybe on the client side? And there's nothing to enforce, it's all code only.
Doing this is actualy rather difficult, depending on how the site is being accessed. Not all browsers keep a running tally of how long a page has been open. Getting the users device to track that that would require more than a cookie which means it would be blocked by most antivirus protection. Keeping track of that on the server end would require a constant stream of data transfer back and forth instead of the usual bursts of data --> would put a big drain on the servers --> would cost Royal Road more with negligable increases in writer satifaction.

that is only to keep track of how long the page has been open, you actualy need a much more intrusive program to know if the page is being displayed on screen the whole time, requiring administartor permissions just to acess the novels is too much for all but the most dedicated fans.

Re: On The Subject of Ratings

#74

orangepeel Wrote:
Entitled Wrote:
kanadaj Wrote:
Entitled Wrote: . . . it should be necessary to spend a reasonable amount of time in the first two to three chapters, while they have been visible on the user's screen (not in the background) before a rating or up-/down-vote is taken into account for the score shown to other readers . . .


This is only easy to do until you have volume traffic, at which point this becomes an infrastructural nightmare, and extremely hard to enforce.


Hmm. Maybe on the client side? And there's nothing to enforce, it's all code only.
Doing this is actualy rather difficult, depending on how the site is being accessed. Not all browsers keep a running tally of how long a page has been open. Getting the users device to track that that would require more than a cookie which means it would be blocked by most antivirus protection. Keeping track of that on the server end would require a constant stream of data transfer back and forth instead of the usual bursts of data --> would put a big drain on the servers --> would cost Royal Road more with negligable increases in writer satifaction.

that is only to keep track of how long the page has been open, you actualy need a much more intrusive program to know if the page is being displayed on screen the whole time, requiring administartor permissions just to acess the novels is too
much for all but the most dedicated fans.

Ok, I understand. Thank you for the explanation!

Re: On The Subject of Ratings

#75

Thedude3445 Wrote: So the community has created a strange culture around reviews needing to be very high, and then sneaky anonymous 0.5 ratings done exclusively to hurt others. How do you stop that, other than wait for the teenagers to grow up and be more mature about their ratings? I don't have any good answers, and I doubt there are any cleanly good ones.


Two years later, I still don't have any good answers to how to combat this community rating issue (the rating system itself probably isn't the issue is all the insight I've gained). But my fury towards anonymous 0.5 drive-bys by people who never read the book still remains, because sadly the teenagers still haven't grown up.

Entitled Wrote: Ratings as well as up- and down-votes of reviews should be reserved to actual readers. I.e., it should be necessary to spend a reasonable amount of time in the first two to three chapters, while they have been visible on the user's screen (not in the background) before a rating or up-/down-vote is taken into account for the score shown to other readers. It should still be shown to the rater himself (herself?), but not to others. That way, many of the most blatant trolls and tactical down-bashers would have a much harder job. It's technically fairly easy to do, and unfortunately necessary because the purely tactical 0.5 and 1 star ratings and downvotes, especially for trending stories, have such a large negative influence. It hurts both writers and readers who otherwise would enjoy RR much more.

One issue with trying to take reading time into account is that not everyone reads while logged in, and many people read on other websites then rate on Royal Road. For a while on my newest book I actually had more favorites than ratings, because so many readers from SpaceBattles, I assume, came to follow and put it on their favorites list without rating it yet. Surely there's ways to lessen the impact of trolls or make it harder to do something against them, but I still have no idea what it is.

Re: On The Subject of Ratings

#76

Thedude3445 Wrote:
Thedude3445 Wrote: So the community has created a strange culture around reviews needing to be very high, and then sneaky anonymous 0.5 ratings done exclusively to hurt others. How do you stop that, other than wait for the teenagers to grow up and be more mature about their ratings? I don't have any good answers, and I doubt there are any cleanly good ones.


Two years later, I still don't have any good answers to how to combat this community rating issue (the rating system itself probably isn't the issue is all the insight I've gained). But my fury towards anonymous 0.5 drive-bys by people who never read the book still remains, because sadly the teenagers still haven't grown up.

my coping mechanism when i see ratings like those
is thinking that they happen to everyone, but who knows
maybe it's a level playing field with or without
it's like a cold everyone catches just by going out
DrakanSigh

Re: On The Subject of Ratings

#77

Thedude3445 Wrote:
Thedude3445 Wrote: So the community has created a strange culture around reviews needing to be very high, and then sneaky anonymous 0.5 ratings done exclusively to hurt others. How do you stop that, other than wait for the teenagers to grow up and be more mature about their ratings? I don't have any good answers, and I doubt there are any cleanly good ones.


Two years later, I still don't have any good answers to how to combat this community rating issue (the rating system itself probably isn't the issue is all the insight I've gained). But my fury towards anonymous 0.5 drive-bys by people who never read the book still remains, because sadly the teenagers still haven't grown up.


I also wonder. And I'm not sure these are all teenagers.

Why can't individual ratings simply be as public (associated with the raters' profiles pages) as reviews? This would make ratings much less carefree and maybe less malicious, since there would be some accountability. Of course there will always be ways around, but the hurdle will be slightly higher.

The current system is detrimental to readers and writers.

Re: On The Subject of Ratings

#79

eric_river Wrote:
Thedude3445 Wrote:
Thedude3445 Wrote: So the community has created a strange culture around reviews needing to be very high, and then sneaky anonymous 0.5 ratings done exclusively to hurt others. How do you stop that, other than wait for the teenagers to grow up and be more mature about their ratings? I don't have any good answers, and I doubt there are any cleanly good ones.


Two years later, I still don't have any good answers to how to combat this community rating issue (the rating system itself probably isn't the issue is all the insight I've gained). But my fury towards anonymous 0.5 drive-bys by people who never read the book still remains, because sadly the teenagers still haven't grown up.

my coping mechanism when i see ratings like those
is thinking that they happen to everyone, but who knows
maybe it's a level playing field with or without
it's like a cold everyone catches just by going out
DrakanSigh


It's not level, probably just random, but the problem is that small initial differences have huge effects. Once an author is established and has a large reader base, the bombing has no effect on his or her new stories, even when they get into the rising stars listing, because the existing fans are quick enough to easily outrate the trolls. But for new authors, it can be a random killing effect if they are not discovered early enough by a sufficient number of real readers. Once the score is low, it will almost never recover significantly, because potential real readers won't find it, unless the story is pushed in some other way, but that happens only very seldom.

Repairing this sorry state of affairs is RR's responsibility IMHO. It's really bad for budding new authors, and there's no resolve besides giving up at the moment.

Also, one more thing putting the "level-field" theory into question is evidence from imdb (and rotten tomatoes before they started restricting ratings to people who had actually bought tickets in 2019) that there are cliques that downrate movies and TV shows from female directors or with only strong female stars. Not saying that this is happening here, but we never know before looking at it more closely. RR could do something similar to what rotten tomatoes is doing, maybe by providing a second score from paying users only? 

Re: On The Subject of Ratings

#80

Entitled Wrote: Repairing this sorry state of affairs is RR's responsibility IMHO. It's really bad for budding new authors, and there's no resolve besides giving up at the moment.

i agree that it's not perfect, but i can't complain
i'm not paying for a service (though i feel your pain)
for a young and budding website, it's evolving well
will it conquer rating-culture? only time will tell

peoeating