IndexNow in Practice: 6,000 URLs in One Request

Bing Webmaster Tools has a sitemap form, but it needs a browser session that is logged in, and mine often was not. I wanted new posts to reach Bing automatically, from a cron job, with no login at all. That is exactly what IndexNow is for.

What IndexNow is

IndexNow is a simple protocol. When a URL on your site is added, changed or deleted, you send an HTTP request saying so. Participating search engines then recrawl that URL soon, instead of waiting to rediscover it.

Bing, Yandex, Seznam and Naver participate, and a submission to one endpoint is shared with the rest. Google does not use IndexNow, so you still need a sitemap in Google Search Console.

Step 1: create and publish a key

Ownership is proven with a key: any string of 8 to 128 letters, digits or dashes. I generated a 32-character hexadecimal key and published it as a text file at the root of the site:

https://sandeepkumarchaudhary.com/<key>.txt

The file contains only the key itself. In a Next.js app, that means a file in public/. Remember that Next.js's standalone build copies public/ into the image at build time, so a new key file needs a rebuild before it is served.

Check it returns 200 before going further:

curl -s -o /dev/null -w "%{http_code}\n" https://sandeepkumarchaudhary.com/<key>.txt

Step 2: submit the whole sitemap at once

One request can carry up to 10,000 URLs. This Python script reads the sitemap and submits every URL:

import json, urllib.request
import xml.etree.ElementTree as ET

SITE = "https://sandeepkumarchaudhary.com"
KEY = "<key>"

xml = urllib.request.urlopen(SITE + "/sitemap.xml").read()
ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
urls = [u.findtext("s:loc", "", ns) for u in ET.fromstring(xml).findall("s:url", ns)]

for i in range(0, len(urls), 10000):
    body = json.dumps({
        "host": "sandeepkumarchaudhary.com",
        "key": KEY,
        "keyLocation": f"{SITE}/{KEY}.txt",
        "urlList": urls[i:i + 10000],
    }).encode()
    req = urllib.request.Request("https://api.indexnow.org/indexnow", data=body,
                                 method="POST",
                                 headers={"Content-Type": "application/json; charset=utf-8"})
    print(urllib.request.urlopen(req).status)

My first full submission sent 6,263 URLs in a single request and got HTTP 200.

The 403 you will probably see first

My very first request did not succeed. It returned HTTP 403 with SiteVerificationNotCompleted. IndexNow checks the key file asynchronously the first time it sees your site, and the request arrived before that check finished.

The fix was simply to wait a few minutes and send the same request again. It returned 200 and has worked ever since. If the 403 persists, check that the key file returns 200 and contains exactly the key, with no extra characters.

Step 3: ping each new post automatically

Bulk submission handles the backlog. For new posts, the publishing cron job pings the single new URL right after writing it:

url="$SITE/blog/$newest"
code=$(curl -s -m 15 -o /dev/null -w '%{http_code}' -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json" \
  --data "{\"host\":\"sandeepkumarchaudhary.com\",\"key\":\"$INDEXNOW_KEY\",\"keyLocation\":\"$SITE/$INDEXNOW_KEY.txt\",\"urlList\":[\"$url\"]}")
echo "$(date -u +%FT%TZ) | indexnow: HTTP $code for $url" >> "$LOG"

Logging the status code matters. Over one stretch of about three weeks, the log showed 870 successful pings, two timeouts and one 503 from the IndexNow service. Those three posts were simply re-sent later, and Bing's IndexNow report in Webmaster Tools showed them arriving.

Use it for removals too

IndexNow is not only for new content. When I later marked thousands of low-value pages as noindex and removed hundreds of thin listing pages, I submitted all 6,899 changed URLs through IndexNow. That prompts Bing to recrawl them and notice the change, rather than waiting weeks to stumble across it.

Check that it is working

Bing Webmaster Tools has an IndexNow page listing recently received URLs and their source. After a bulk submission, the "URLs submitted in last 4 hours" counter should match what you sent.

The takeaway

IndexNow takes about ten minutes to set up and removes the need to log in anywhere for Bing. Publish a key file, submit your sitemap once, ping new and changed URLs from your publishing job, and log the responses. Just remember it does nothing for Google.