llms.txt is a proposed standard: a Markdown file at the root of your site (/llms.txt) that gives large language models a concise, curated map of your most important pages with short descriptions. It is meant to help AI assistants find and understand your content quickly, in a format that is cheaper to parse than rendering full HTML pages.

What is the difference between llms.txt and robots.txt?

robots.txt controls access — it tells crawlers which paths they may or may not fetch. llms.txt does the opposite job: it does not grant or deny anything, it simply offers a clean, human-curated index of your best content for AI models. You need correct robots.txt rules for bots to reach you at all; llms.txt is an optional summary on top.

How do I add llms.txt to a Next.js app?

Create a route handler at app/llms.txt/route.ts that returns Markdown with a text/plain content type. Generate the link list from the same source as your sitemap — your pages and blog posts — so it stays current automatically. Mark it force-static so it is built once and served from the edge.

Which AI crawlers should I allow to get cited?

If you want to appear in AI answers, allow the retrieval bots: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google's standard crawler. You can still block training-only bots if you prefer. Blocking these retrieval crawlers is the single most common reason a site never gets cited.

llms.txt Explained: How to Add One to Next.js

Q: Does llms.txt actually work?

Evidence is mixed in 2026. Some AI tools read it, others ignore it, and Google has said it gives no special AI-search advantage. It is low effort and harmless, and it signals that you care about machine-readability, but it is not a magic switch. The fundamentals — crawlable HTML, structured data, and not blocking AI bots — matter far more.

If you want AI assistants like ChatGPT, Claude, and Perplexity to find and cite your site, you have probably seen llms.txt mentioned. This guide explains what llms.txt is, how it differs from robots.txt, how to add one to a Next.js app in a few lines, and — honestly — how much it really helps in 2026.

I added an llms.txt to this very site, so the implementation below is the exact pattern I ship, not a copy-paste from the spec.

What is llms.txt?

llms.txt is a proposed convention: a single Markdown file at /llms.txt that gives large language models a concise, curated map of your most valuable pages. Instead of forcing a model to crawl and render every HTML page, you hand it a clean, link-rich summary it can read cheaply.

A minimal llms.txt looks like this:

# Your Name

> One-line summary of who you are and what the site covers.

## Pages
- [About](https://example.com/about): Background and expertise
- [Blog](https://example.com/blog): Guides on web development and AI search

## Blog posts
- [Post title](https://example.com/blog/slug): Short description

The format is deliberately simple: an H1 title, an optional blockquote summary, then H2 sections of annotated links. That is the whole idea — a machine-friendly table of contents.

llms.txt vs robots.txt — they solve different problems

People confuse these two files constantly, so here is the clean distinction:

File	Job	Controls access?
`robots.txt`	Tell crawlers which paths they may fetch	Yes
`llms.txt`	Offer a curated index of your best content	No

In other words, robots.txt is the gate and llms.txt is the welcome map. If your robots.txt blocks the AI crawlers, no llms.txt will save you — the bots never get in. So fix access first. To go deeper on AI visibility overall, see my guide on what GEO is and why it matters and the broader AI search optimization playbook.

How to add llms.txt to Next.js

In the App Router, a route handler is the cleanest way because you can generate the file from the same data as your sitemap, so it never goes stale. Create app/llms.txt/route.ts:

import { getAllPosts } from "@/lib/blog";
import { profile, SITE_URL } from "@/data/site";

export const dynamic = "force-static";

export function GET() {
  const posts = getAllPosts();
  const lines = [
    `# ${profile.fullName}`,
    "",
    `> ${profile.bioShort}`,
    "",
    "## Blog posts",
    ...posts.map(
      (p) => `- [${p.title}](${SITE_URL}/blog/${p.slug}): ${p.description}`,
    ),
    "",
  ];
  return new Response(lines.join("\n"), {
    headers: { "content-type": "text/plain; charset=utf-8" },
  });
}

force-static builds the file once and serves it from the edge, and pulling from getAllPosts() means new articles appear automatically. That is the entire implementation — no extra dependencies.

Does llms.txt actually work?

Here is the honest part most posts skip. As of 2026 the evidence is mixed. Some AI tools fetch llms.txt, others ignore it, and Google has publicly said it provides no special advantage for AI search. It is genuinely useful as a low-cost signal that you care about machine-readability, but it is not a ranking switch.

What moves the needle far more:

Do not block AI crawlers. Allow GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot in robots.txt. Blocking them is the number-one reason sites get zero AI citations.
Ship structured data. FAQ and Article JSON-LD map directly to the question-and-answer shape AI engines prefer.
Lead with the answer. Put a 40–60 word direct answer near the top of each section.
Earn authority. Most AI citations come from pages that already rank and show real expertise.

For the ranking side of this, my guide on how to rank higher on Google in 2026 covers the fundamentals that AI engines build on.

The bottom line

Add llms.txt — it takes ten minutes in Next.js and signals good machine-readability. But spend the real effort on crawler access, structured data, and answer-first writing. According to the llms.txt proposal, the file is meant to complement, not replace, the rest of your site's machine-readable surface. Treat it exactly that way: a useful map, not the territory.