How the review notes scored
Every note in a game review makes claims about the board, so the writer is tested on a fixed set: fifteen real Lichess games, three from each 200-point band from 1200 to 2200. A review teaches from up to four moments of its game; in these games that made sixty notes, and every one was marked by hand: right, misleading or wrong.
The grader
Sixty notes is too many to mark by hand for every version, so an AI grader marks them, handed the same board facts and engine lines as the writer. It’s stricter than our first pass was: it flagged 47 notes we had passed, and on a second look we agreed with all 47. Against the corrected hand marks, its right-or-not call matches on 97–98% of notes.
The scores
| Notes | Right | Misleading | Wrong |
|---|---|---|---|
| Where we started: mid-October’s writer, 60 notes, marked by hand | 28 | 17 | 15 |
| Today’s writer: 58 notes, marked by the grader | 51 | 6 | 1 |
Two of the sixty were moves in a game already lost by four pawns or more; reviews no longer teach from those.
The old notes mostly misread the lines: an engine line told as if it had been played, a skewer called a hit. Today’s writer is handed the facts of each line, worked out from the moves, down to whether taking an attacked piece actually wins anything.
What’s still wrong
Trades and material. A knight “lost” that was really traded; “wins the rook outright” where it wins the exchange; “the queen had to retreat” where it stayed three more moves.
Keep in mind
- 58 notes is a small test: 51 right puts the real rate somewhere between about 77% and 94%.
- Today’s notes carry the grader’s marks; we haven’t marked them by hand yet.
- The 1400–1600 band is one player’s games.
- In the app, the engine check on offered pieces runs as time allows, so it covers fewer lines than it did here.
Next
When a piece can’t be taken because taking it loses, the note should say what it loses. Today’s writer is handed that line and doesn’t use it yet.