Rejected Because a Reviewer's LLM Could Do Better? What Peer Review Really Tests

A computer-science PhD student brought one sentence from their advisor to Academia Stack Exchange: in the era of LLMs, your submission will be rejected if a reviewer can generate a better paper by using LLMs. The student had spent a year writing a paper. The fear was simple. A reviewer could spend five minutes with a chatbot, get something better, and sink a year of work by comparison.

The fear has teeth because generative AI has made it almost free to produce something that looks like a paper. Once plausible text costs nothing, peer review has to decide what it is actually rewarding. Two answers to the question, one with 38 votes and one with 12, give competing and partly overlapping accounts of where that bar now sits.

The disagreement is not over whether LLMs matter. Both answerers accept that a paper an LLM can trivially beat is in trouble. They split on what “beat” means, who gets to run the comparison, and what your paper is measured against.

What the student objected to

The question is worth reading in full because it lists the objections most researchers would raise. They fall into four groups:

• The irony. If five minutes of prompting beats a year of work, why not generate a paper every day and collect more than 300 publications a year?
• Apples and oranges. As far as the student knows, many publishers forbid AI-generated manuscripts. A human paper rejected for losing to a machine paper the publisher would never print is being measured against something outside the competition.
• Conflict of interest. A reviewer who reads your paper and then generates a similar one now holds a rival manuscript.
• Grading their own homework. A reviewer who writes the comparison paper and then finds it better has judged their own work.

Under all four sits one principle. A review should rest on the merit of the submitted paper, not on the merit of some other article that nobody has peer-reviewed. That is the student's strongest instinct, and both answers take it up, though from different directions.

The case that the advisor is stating the obvious

The top-voted answer sees no red flag at all. Its test case is deliberately extreme. If typing “prove the Riemann hypothesis” into ChatGPT gives a better output than your paper, then your paper has no value and should be rejected. Nobody would defend a submission that loses to a one-line prompt on a problem that famous.

The answer then removes LLMs from the picture. Suppose you write a paper giving a closed-form solution to a gnarly integral, and a reviewer types the integral into Mathematica and gets the same result in five minutes. That paper, the answer says, probably shouldn't be published. Symbolic algebra tools made a whole class of results routine long before chatbots existed, and referees had already learned to discount work a tool could reproduce on demand.

Read this way, the advisor's sentence is an old norm with a new tool plugged in. A referee has always asked whether someone could get the result cheaply without the author. Generative AI simply enlarges the set of results where the answer is yes.

Where the comparison breaks: your ideas in the reviewer's prompt

The same answer names what it sees as the student's strongest objection, and it is neither the irony nor publisher policy. It is information leakage. If the reviewer uses insights from your paper to prompt the AI, the apples-to-apples comparison falls apart. The LLM is no longer competing with you from a blank start. It is finishing a path you already mapped.

Again, the answer points out that this is not specific to LLMs. In some areas of mathematics it is very hard to find integers with certain properties but easy to check that a given candidate has them. Once a paper hands over the candidate, checking it, or regenerating a version of it, is cheap. The five minutes the reviewer spent say nothing about how hard the work was the first time.

There is a darker reading too. If the advisor meant that a reviewer might steal an idea and publish it after using an LLM to reproduce the results quickly, the top answer calls that obviously a red flag. Taken out of context, though, it would not jump to that conclusion.

A stricter test before rejection

The second answer expects the situation to be more nuanced than the advisor suggests. Its author says plainly that they would not reject a paper merely because a frontier LLM can prove a stronger result, even with a simple prompt and thinking time measured in minutes.

What would justify rejection, for this reviewer, is a set of observations that all hold together:

• the result is not unexpected;
• the question is one anyone could ask;
• an LLM gives a satisfactory answer within minutes;
• the proof in the paper is essentially the LLM's argument;
• if that argument contains a surprising step, the paper does not bring that central idea to the front.

Work that meets every one of these conditions may still deserve a place in the record. The answer suggests it belongs in preprints, on GitHub or in a dedicated repository rather than in a journal. That is a different verdict from “worthless”. It is a judgement about which venue a result needs, not about whether it should be kept at all.

The amateur astronomer problem

The second answer's strongest argument is an analogy. An amateur astronomer notices something genuinely interesting, say a strange new variability in the output of a star like Tabby's star, and writes it up well. That paper may well deserve publication, even though professional instruments could gather better observations straight away and someone could then write a better paper.

The point is that low resources, and even fairly low effort, do not make work scientifically uninteresting. And if they do not, then the fact that the work could be repeated with even less effort counts for even less. As the answer puts it, someone still had to point their telescope at the right patch of sky at the right time.

Applied to LLMs, this splits the advisor's sentence in two. A reviewer who can produce a better paper after reading yours is the professional observatory turning up after the amateur's alert. A reviewer who could produce it without ever seeing yours is a different case: evidence that no telescope needed pointing.

What counts as a contribution

Neither answer pretends the bar has gone away. The second is explicit that a paper in a good journal must contribute knowledge the community values. What it does is widen the list of things that qualify:

• a good question;
• a surprising answer;
• an answer reached through perseverance;
• a particularly enlightening write-up;
• a new method;
• the luck of being first to point the telescope at the right patch of sky, or to notice that patch of mould crowding out the bacteria.

Several items on that list are things a bare LLM prompt cannot supply. A model can answer a question once someone asks it, but it does not decide that the question was worth asking. A write-up can be machine-assisted. An answer reached through perseverance, or a first observation, is a fact about how the work came to be. A paper that leads with one of these stands on ground that five minutes of generation does not reach.

The competition that is actually changing

Where the second answer agrees with the advisor is on competition. However a paper came into existence, it will compete for attention with LLM-generated and LLM-aided papers. If it loses to them on quality, that is a barrier to publication. And on this point it does not matter whether the underlying question can be answered by a single prompt.

In this view, acceptance at a good journal depends on convincing reviewers that your work adds to knowledge more meaningfully than most other submissions to the same journal. The comparison is not your paper against a reviewer's private chatbot output. It is your paper, as a whole piece of work, against everything else in the pile. If that is what the advisor meant, the answer's author says they would agree.

Set side by side, the two answers are less opposed than their tone suggests. Both reject a paper that a bare prompt can match. They differ on how quickly a reviewer should reach that verdict, and on whether effort and timing count for anything once a machine can follow.

What is still unsettled

The thread leaves real gaps. The student's claim that many publishers forbid AI-generated manuscripts is offered as their own understanding, not a checked policy. Anyone relying on it should read the specific publisher's and venue's rules. The conflict-of-interest point, about a reviewer who now holds a similar generated paper, is raised but not settled by either answer beyond the warning about stolen ideas.

The deeper open question is how a reviewer could run the comparison honestly at all. Because of leakage, the only clean test starts from the bare question. But by the time a reviewer is holding your paper, they already know what to ask. For computer-science publications, watch whether venues' reviewer guidance tackles this directly or leaves it to individual judgement.

This article draws on “Your paper will be rejected if a reviewer can generate a better paper by using LLMs.” and answers to it (1, 2), on the Stack Exchange network, licensed under CC BY-SA 4.0.