What the September 2026 creative-writing benchmark means for a book
Vulsar's benchmark ranks 24 AI models against human writers on short stories. Here is what it measured, what it did not, and how to test a model on your own book.
By ScribistIQ ·
Prepared with AI assistance and automated editorial checks. Examples labeled illustrative are invented to explain the method.
The short answer: the benchmark is useful evidence about short stories, not about books. On 17 September 2026 Vulsar published Creative Writing Bench V1, which ranks 24 AI models and a group of human writers by how likely readers are to prefer a single short story. It says nothing about whether a model can keep 30 chapters consistent, hold a voice across a manuscript or follow your story bible. Treat it as one input to a choice, then test on your own material.[1]
What the benchmark actually measured
Each model wrote one short story for each of 475 prompts. A reward model, a scoring system trained on human preference data for creative writing, gave every story a single score, and the differences between scores were turned into a predicted win rate. The page says no AI judge or rubric decides the ranking, refusals and empty replies are left out rather than counted as losses, and reasoning was switched off where a model supports it.[1]
Human writers are included as a reference. Their overall predicted win rate is 86.6%. The page also lists a separate professional-writer group at 99.9% and an amateur group at 79.7%, and notes that these groups used different prompt sets, so the numbers are not a like-for-like race.[1]
The results that matter to a book writer
| Entry | Predicted win rate | Average reply length (tokens) |
|---|---|---|
| GPT 6 Astra (rank 1) | 87.8% | 1,537 |
| Human writers, overall (rank 2) | 86.6% | 2,592 |
| GPT 5.6 Sol (rank 3) | 77.6% | 1,779 |
| Claude Fable 5.1 (rank 5) | 70.3% | 2,114 |
| Claude Opus 5 (rank 6) | 63.4% | 2,330 |
| Grok 4.6 (rank 10) | 59.3% | 496 |
Read the table carefully. A predicted win rate is a prediction of reader preference by one trained scoring model, not a count of real readers voting. The lead over the overall human group is 1.2 percentage points, and the professional group scores far higher than both. Reply length also varies a lot, from about 500 to about 2,600 tokens on average, so the models were not all producing stories of the same size.[1]
What the benchmark does not tell you
- Book-length coherence. The page does not address long-form work; each story stands alone, one per prompt.
- Consistency of names, facts and timeline across chapters, which is what readers notice first in a long manuscript.
- How a model follows a story bible, an outline or your voice sample. The prompts are single requests.
- Whether the story is original. The page itself says a high score indicates predicted reader preference and does not establish that a story is novel or unique.
- What happens when reasoning is on, since it was disabled where supported.
- Which product you can use. A model name on a leaderboard does not mean any writing tool offers it.
Scale matters too. A short benchmark reply is roughly 500 to 2,600 tokens. As a rough rule of thumb (an approximation, not a measured figure), English text runs about three-quarters of a word per token, so an 80,000-word novel is on the order of 100,000 tokens, dozens of times the length the benchmark scored. A model that writes one good short story has not shown it can sustain a book.
Does this change what tool you should pick?
Only a little. It is a reason to look at the underlying model when a tool names it, and a reminder that product design decides a lot once a project gets long. ScribistIQ's own guide explains that its generator writes one chapter at a time against a story bible, summaries of recent chapters and the end of the previous chapter, and that facts the AI invented were more likely to drift than facts stated up front.[3]
On model availability: ScribistIQ's features page does not name the text model behind its books. Paid plans in the Pro or Ultra Max generator offer a Writing model control with Standard and experimental options, and the page says available options can differ between generators and change over time. Nothing in the benchmark, or on that page, shows that any model on this leaderboard is used in ScribistIQ, so do not assume a result here carries over to it.[2]
Other tools differ in how much choice you get. One comparison guide notes that Novelcrafter lets you connect your own provider and pay that provider directly, while Sudowrite offers a choice of models, which changes the credit cost of a book. That choice is a design difference, not a quality ranking.[4]
A five-step check on your own book
This is an untested editorial method, not a validated protocol. Its point is to measure the thing the benchmark leaves out. Use a premise you already know well, so you can spot errors.
- Write down ten fixed facts about your story: names, ages, a place, a date, a rule of your world. Keep the list.
- Give the tool the premise and those facts, and ask for chapter 1 and then a later chapter, such as chapter 6, using the same inputs.
- Read both chapters against your list and mark each fact as kept, changed or dropped.
- Compare the voice of the two chapters to your own sample paragraph: sentence length, word choice, how much feeling is stated outright versus shown.
- Repeat with a second tool or model setting if you are choosing between them, with identical inputs, and keep the results side by side.
Premise: A lighthouse keeper's daughter, Mara, returns to the island in 1962 to sell the house. Fixed facts: 1) Mara is 34. 2) The lighthouse has a green door. 3) Her brother Teo died in 1951. 4) The ferry runs only on Fridays. 5) Mara has not spoken to her mother in nine years. Check after chapter 6: Is Mara still 34? Is the door still green? Is Teo still dead, and in 1951? Is the ferry still Fridays only?
How to judge the result
Count kept, changed and dropped facts out of the ten. A tool that is lovely in chapter 1 but changes four facts by chapter 6 is a worse fit for a novel than a plainer one that keeps them. ScribistIQ's guide reports the same kind of check on its own test novel: of 12 tracked details, 7 held, 4 were contradicted and 1 was dropped. That is one novel, one run and the vendor's own test, so use it as an example of the method, not as a benchmark.[3]
Limits of this article
The Vulsar page was read directly on 5 November 2026. The article does not compare prose from any model, and no model was run for it. Leaderboards change, so recheck the live page before relying on a ranking.[1]
Sources
- Vulsar Creative Writing Bench V1 (leaderboard and method)Checked 2026-11-05
- ScribistIQ features pageChecked 2026-11-05
- ScribistIQ guide: AI novel continuityChecked 2026-11-05
- ScribistIQ guide: best AI book generatorsChecked 2026-11-05