Wikipedia talk:WikiCup/Archive/2025/1

Increase points for GA reviews

There's been some discussions over the past year about raising the score for GA reviews. For one, GA reviews require a large amount of time and effort from a single individual, not quite on the level of writing the article, but enough that such a low score fundamentally disincentivises WikiCup participants from reviewing GA nominations. This creates a problem for the GA process, which is already facing a constant struggle trying to incentivise reviews; the GA backlog regularly sits at hundreds of articles, which sometimes have to wait for up to a year for a review.

I think, given the strain that the GAN project has been under for a while, it would be very much appreciated if the WikiCup could encourage reviews more. As such, I would like to propose that the scoring for a completed GA review be raised to 10 or 15 points, with bonus points for longer and more in-depth reviews. Keeping it lower than half the score for a GA would prevent gaming of the system, as 2 reviews for every nomination helps reduce the backlog, while bonus points would encourage people to engage thoroughly with the review process. --Grnrchst (talk) 12:29, 14 December 2024 (UTC)

Recent discussions: Wikipedia_talk:WikiCup/Archive/2024/1#GAN reviewer points and Wikipedia talk:WikiCup/Archive/2024/3#Should reviews be worth more?. The main problem is that if we give many points, we need to ensure some review quality, especially with bonus points, but we should not ask judges to do significantly more work than they do now (or pay them twice as much as now). —Kusma (talk) 13:41, 14 December 2024 (UTC)

Personally I'm still all for a flat increase to 7 points for all types of reviews just to encourage and reward reviewing more than we currently do. Perhaps if we wanted to evaluate the quality of GA reviews we could add 2 additional coords next year and implement it then. Hey man im josh (talk) 14:25, 14 December 2024 (UTC)

In my personal experience, GA reviewing is usually more time effort than PR or FAC reviewing, so I would prefer it if GA reviews were worth more than other types of reviews. (Also, PR and FAC reviews can be done and then immediately claimed at any time, while with GA reviews you typically need to wait for the reviewer to respond, which slows down scoring). —Kusma (talk) 15:13, 14 December 2024 (UTC)

@Kusma, just as a quick note, GAN reviews and peer reviews are currently tabulated together; contestants submit PRs under the "Good article nomination review or peer review" section of their submissions page. There can be a separate discussion if we want to instead tabulate peer reviews with FAC reviews, or if we want to tabulate peer reviews on their own. – Epicgenius (talk) 16:47, 14 December 2024 (UTC)

If we have different scores for different types of reviews, I would suggest to put PR together with FAC reviews, as FAC/PR are more similar to each other than to GA reviews (in a FAC or PR you can just concentrate on parts of the article or parts of the criteria, while for GA you need to do everything, plus the difference in when you can score them). Having PR as a separate category would give another bonus under Tamzin's scoring, which may or may not be a good idea. —Kusma (talk) 18:49, 14 December 2024 (UTC)

A specific bonus for peer reviews might not be a bad thing if it encourages more activity there. Thebiguglyalien (talk) 19:43, 14 December 2024 (UTC)

@Kusma: That may be so in some cases, but that also depends on the quality of the reviews as we know. For instance, when I do source reviews at FLC I imagine it takes longer than some GA reviews.

Never the less, I believe that whether we want to make GANRs worth more or not, both deserve a bump, and that's a quick improvement that can encourage reviews. I don't think bumping GAs up to 10/15 points and doing nothing for featured content reviews is something we should do either. After all, our biggest bottleneck does seem to be having people actually review things, and I more or less viewed it as a bandaid suggestion for improvement while a more thorough approach for evaluating GANRs can be crafted.

I guess I'm just not confident we can get the GANR scoring system that's been suggested down pat for the upcoming drive, and I'd prefer we move the needle in the right direction at least somewhat as opposed to not at all for the upcoming cup. Hey man im josh (talk) 17:08, 14 December 2024 (UTC)

If we just uniformly bump all types of reviews, I would suggest not to bump by too much. I can live with 7, but I would oppose 15 as too high (even though I would probably benefit from this, as I usually review more than I nominate). Some PR and some FAC prose reviews can be done very quickly while thorough source spotchecks can take a very long time. —Kusma (talk) 18:53, 14 December 2024 (UTC)

I actually raised the exact same point here just today. A bump up to 7 points (or even higher) sounds good, but we should also incentivize harder reviews more with bonus points-maybe based on how vital or large the article being reviewed is- say 0.2x bonus for every 1000 words, capped at 1.5. As for evaluating GANR quality- I assume they are already given a cursory glance, so basing it on an objective criteria wouldn't make it harder. DoctorWhoFan91 (talk) 18:28, 14 December 2024 (UTC)

Agree that the overall amount for reviews, whether GANs or overall, should be raised. Seven to ten points sounds like a reasonable range. I'd also be okay with stricter enforcement of GAN review quality. Reviews that are just a few bullet points or solely focus on style/grammar issues rather than the full criteria might be worth scrutinizing. Thebiguglyalien (talk) 19:43, 14 December 2024 (UTC)

I think most GANRs are good- usually bad reviews happen bcs someone is trying to game the system. That can be done with a cursory glance I think, on the size of the review page and a general glance on what it's checking. Even GAN backlog drives do not give points for reviews under 1000 bytes, so maybe something like can that can used. DoctorWhoFan91 (talk) 20:14, 14 December 2024 (UTC)

IMO 10 points for GANRs and 7 points for PRs/FLCRs/FACRs sounds like a reasonable proposal. I think a GANR is about equal in difficulty and time investment to writing a 10 pt DYK worthy article (albeit a different skill set). Generalissima (talk) (it/she) 20:54, 14 December 2024 (UTC)

I second Generalissima's opinion here. λ NegativeMP1 22:35, 14 December 2024 (UTC)

At the upper limit of effort for either, I agree. But the lower limit for GANRs is much lower than for a 10-point DYK, IMO. I received credit for Talk:Fernanda Farias de Albuquerque/GA1. The rules allow scoring detailed quickfails like that one, and I think that's a good thing, but I also don't think it took me anywhere near the effort of Ray cat, probably my easiest 10-point DYK. One solution, then, might be making the review categories "5 or 10" or "5 or 7" based on length. -- Tamzin^{[cetacean needed]} (they|xe|🤷) 22:52, 14 December 2024 (UTC)

Length of the article or length of the review? Because if it's the latter, that would probably just encourage unhelpful comments like "place a comma here" or "add ISSNs to your references". Thebiguglyalien (talk) 23:46, 14 December 2024 (UTC)

Length of the review. I see your point, but we do already have As a rough guide, no review shorter than 1000 bytes will be considered, though the judges reserve the right to decline other short reviews, so that incentive already exists with a 5-point differential, just 0-to-5 not 5-to-10. There's also the standing warning about disqualifications for gaming GANR in particular. Maybe making 1,000 to 2,000B of substantive review the 5-point range would work? For context, my Farias review was 2,016B, of which 1,769 was substantive review (i.e. not the salutation or valediction). -- Tamzin^{[cetacean needed]} (they|xe|🤷) 00:04, 15 December 2024 (UTC)

I think length of the article would be better- there are ways to game a review: copy-editing is by far the biggest section, and if the reviewer decides to just do it themself- can reduce the review by like half. A full review for any article would always take time and effort, and the hardness might not be seen in the length of the review: a 5000+ word article with no typos might get a shorter review than a 1000+ one with many typos, especially if the nom for the latter one is more combative, even though it takes around 4-5 times more work to review the bigger one. DoctorWhoFan91 (talk) 06:46, 15 December 2024 (UTC)

Perhaps we could count lists of typos against the word count so reviewers will stop creating additional work for themselves and for the nominator by listing them out and describing them one at a time. Only joking. Mostly. Thebiguglyalien (talk) 07:16, 15 December 2024 (UTC)

I'm not at all sure that additional points are necessary given that we've significantly reviewed more GANs that we've submitted for many years now, often by large numbers. This year it was by a nearly 40% margin. The backlog at GAN is systemic due to people's desire to collect all the shinies and disinclination to review as many as they nominate. I appreciate editors' desires to help reduce the backlog, but that's not necessarily the task of the Cup. Just pointing this out, I'm not opposed to increasing the points; it's a laudable goal.--Sturmvogel 66 (talk) 09:20, 15 December 2024 (UTC)

I think the Cup does have a responsibility to be considerate of its effects on various Wikipedia processes. I guess the place where we are most contributing to the backlog but not helping to reduce it is Did You Know. More volunteer prep builders and queuers would be most welcome there. —Kusma (talk) 14:11, 15 December 2024 (UTC)

I've toyed with suggesting a way for people to "sacrifice" DYK QPQs for points, something like you get 1 point for committing to not use 5 otherwise eligible QPQs toward any future DYK review. -- Tamzin^{[cetacean needed]} (they|xe|🤷) 16:35, 15 December 2024 (UTC)

Unfortunately that would still not help with the bottleneck at WP:DYKQ. —Kusma (talk) 17:17, 15 December 2024 (UTC)

Unless two active judges can be found before the cup starts (see below), I wouldn't be in favour of any changes that require judges to adjudicate on each of 693 reviews. ~~ AirshipJungleman29 (talk) 18:36, 15 December 2024 (UTC)

I would just like clarification on a point: don't the judges also give a cursory glance to the reviews already, and if they don't, will that need to change with this? A points bump definitely doesn't, and if we associate any points with say the size of the article, then it's 2-3 quick clicks to check the size of an article- more work, but only a min per review at most. I mean I would assume they are already failing quickfails and <1000 bytes GANR like the scoring page says. DoctorWhoFan91 (talk) 20:05, 15 December 2024 (UTC)

They do give cursory reviews, and they actually made a point of contacting me last year (my first year in the cup) and letting me know that I basically had to show my work for some of my featured list reviews. They may recognize that I likely do appropriate work in said reviews, but I was not able to collect points on a couple of reviews because of that. Hey man im josh (talk) 13:25, 16 December 2024 (UTC)

So, any changes to the review points that do not greatly increase the work already been done by the judges should be fine, right? DoctorWhoFan91 (talk) 13:42, 16 December 2024 (UTC)

Theoretically, yes, that's my take as simply a participant and non-judge. I think the idea of multipliers for GANRs is a good one, but I think it needs to be ironed out and needs the appropriate support from judges in order to function properly. That's why I'm in favor of a band-aid bump to 7 points for all reviews for now, with a reevaluation and implementation of a more thorough system with the proper support for the 2026 cup. Hey man im josh (talk) 13:51, 16 December 2024 (UTC)

don't the judges also give a cursory glance to the reviews already

@DoctorWhoFan91, yes, we do. Any review that isn't long enough and doesn't address the WP:GACR, WP:FACR, or WP:FLCR criteria is ineligible for points. GAN quick fails are almost always ineligible because they don't fully address the criteria. However, in some cases, there have been "quick" GAN fails that are long enough and detailed enough to count as a full review (and are not really quick fails); they're probably only counted as quick fails because these GANs were failed immediately. – Epicgenius (talk) 14:23, 16 December 2024 (UTC)

Good to know, I already assumed that would happening. Question- if a consensus gets formed on giving bonus points for reviews based on say the length of the article (as that is probably a good criteria a GANR was, having to read through it all), would that greatly increase the work done by the judges, (as word count is easily available through the article history) or would that be miniscule? Also, thank you to you and the other judges, for your work handling the WikiCup. DoctorWhoFan91 (talk) 16:27, 16 December 2024 (UTC)

The numbers support your point @Sturmvogel 66, the cup does result in more reviews than successful GA promotions. I do suspect it might be a tad bit skewed based on the most active reviewers, which I don't have the numbers for. I will however note that, in 2021, The Rambling Man had 324 of the 650 GANR/PRs. With that said, I do still think that, in the interest of improving content overall on the site, and based on the time it takes to perform various reviews, an increase of some sort does make sense. Hey man im josh (talk) 13:23, 16 December 2024 (UTC)

Call for new judges

If anyone is interested, I'd like to bring on one or two more judges for the 2025 contest. Given the above discussion about changing the points system, it looks like a rough consensus is forming for option C (permanent link), which entails the following:

1. Allow people to sign up for the Cup on a rolling basis, rather than only allowing people to sign up from November to January
2. Discontinue the elimination of the lowest-scoring competitors in each round; instead, all competitors will be allowed to compete in rounds 1-5, unless they withdraw.
3. Introduce a "Tournament Points" system, separate from the current round-points system, that will be used to determine the highest-scoring contestants at the end of round 5. The top 16 in each round get a number of tournament points as outlined above by @Tamzin. The round points will still be cleared at the end of each round (the same as the current system), but the tournament points will carry over between rounds.

This will likely result in a substantial increase in submissions during later rounds, compared with the current system. During the past year, I've handled perhaps two-thirds of the work of running the Cup. However, I won't be able to do this as much in 2025, so additional help would be appreciated.

I'm also pinging my current co-judges, @Cwmhiraeth and @Frostly, for their thoughts. – Epicgenius (talk) 01:47, 15 December 2024 (UTC)

Worth noting that Frostly and Cwmhiraeth are semi-inactive—their last 250 edits go back to July and October 2023 respectively, whereas Epicgenius's go back to three or four days ago. One active judge is not exactly what you want when the role they play is expected to greatly expand. ~~ AirshipJungleman29 (talk) 18:33, 15 December 2024 (UTC)

Thanks for crunching the numbers, Airship. I actually forgot to mention this earlier, but I most likely won't have any access to my computer for a few weeks in mid-2025 (if not longer), as I'll be on vacation during that time. I won't be able to review submissions, add new contestants, review/clear scores, or send out the round 3 newsletter during these weeks. This is not to mention that I will have less time on WP in 2025 due to real-life commitments.

Unless we add another judge or two, I'm not sure that the existing judges can review all these submissions in a timely manner. If no one responds in the next two weeks, I will also send out a call for judges in the next newsletter. – Epicgenius (talk) 19:08, 15 December 2024 (UTC)

Is there a good way to crowdsource the reviews of the submissions to the judges only have to intervene in cases of conflict? —Kusma (talk) 19:20, 15 December 2024 (UTC)

On paper, I think the community should always be double checking reviews. The problems arising from this are: one, any sort of challenge will cause drama and a stigma against doing so, and two, I have particular ideas about what a review should look like. I used to try to catch quickpasses at GAN when I saw them, but it becomes such a problem for the reporter that now I just pretend I don't see them. Thebiguglyalien (talk) 19:41, 15 December 2024 (UTC)

I have no plans on joining next year's competition, and would be happy to help out with judging the competition. Lee Vilenski ^{(talk • contribs)} 19:38, 15 December 2024 (UTC)

@Lee Vilenski, given your experience, I'd love having you on as a judge. I have to do some house cleaning right now, but tomorrow, I'll send you an email about what the judges' duties entail. – Epicgenius (talk) 19:56, 15 December 2024 (UTC)

Sure thing. Given I wouldn't enter, I'd love to be able to support this competition. Lee Vilenski ^{(talk • contribs)} 21:11, 15 December 2024 (UTC)

I was hoping some former competitors would step up and lend a hand. A 2x winner and 2x runner up is an excellent person to have lending a hand with judging in any capacity they're available for. Hey man im josh (talk) 13:27, 16 December 2024 (UTC)

I can help -- Guerillero ^{Parlez Moi} 15:11, 16 December 2024 (UTC)

@Guerillero, I'd love to have you on as well. After I get off work, I can send you an email about what the judges' duties entail, similar to what I told Lee yesterday. – Epicgenius (talk) 16:00, 16 December 2024 (UTC)

It looks like we have our two new judges. Welcome to the team, @Lee Vilenski and @Guerillero, and thank you very much for graciously offering to help out. – Epicgenius (talk) 21:18, 16 December 2024 (UTC)

Welcome @Lee Vilenski and @Guerillero, and thank you to @Epicgenius for stepping up to take this on! While I’ve had some real-life commitments in 2024, I'm looking forward to contributing much more to the Cup in 2025. Best, — Frostly (talk) 23:41, 16 December 2024 (UTC)