TwiddleLab / Artificial Intelligence / Generative AI: Making New Things
Can We Detect Generated Content?
What you will learn
- An AI detector gives a score, and honest and AI work overlap. Catching more fakes always means accusing more honest people, as with precision and recall.
- Small error rates add up: 5% of 600 honest essays is 30 wrongly accused students.
- A watermark put in by the generator, or a signature on the file, is much stronger evidence, but only if the tool adds it.
- A detector score alone is not proof. Pair it with a human check.
How the lesson goes
A teacher's AI detector flags your essay as AI-written. You wrote it yourself. How could that happen?
First, a guess.
Set the detector's threshold: catch most AI essays without accusing honest students.
Every essay at or above the line gets flagged. Orange dots are honest students accused.
Fill in the detector's score card at threshold 60.
Drag each number into its box. Use the dots to count.
At threshold 60 the detector catches 79 of the 100 AI essays and misses 21. It wrongly flags 2 of the 100 honest essays and clears the other 98.
Detectors guess. Watermarks are put in on purpose.
That difference decides how much a flag is worth.
Strong or weak evidence that a work was AI-made?
Drag each signal into a box.
Smooth writing, rare words and high detector scores also happen with honest work. A watermark or a signed credential is added on purpose by the tool, so finding it is much stronger evidence.
A detector wrongly flags 5% of honest essays. A school checks 600 of them. How many students are accused?
Tap your answer, then Check.
5% of 600 = 0.05 × 600 = 30. A small error rate becomes a lot of real students when many essays are checked.
One text came from a chatbot that adds a watermark. Which?
Scan the texts, then tap This one under your pick and press Check.
A text watermark tilts the generator toward a secret list of green words. Normal writing hits the list about half the time; the watermarked text is far above that. Heavy rewording can wash it out.
Code: a signed file shows whether anyone changed it. Break the signature, then fix it.
Change one pixel and watch the hash. Then put it back.
A detector says your essay is "90% AI". What can you conclude?
Honest and AI essays overlap on the score line, so a high score can happen to an honest essay. A fair decision needs other evidence, like drafts, a talk with the student, or a watermark.
Write a fairer school policy.
Set the threshold and choose the extra checks. Reach the goal and compare with the old policy.
Previous lesson: Copyright and Consent · Next lesson: Project: Design a Critter Generator · Part of the course: Generative AI: Making New Things
Lessons are interactive and need JavaScript. Turn it on to play them.
