SEO for Developers: Crawling, Indexing, and Content That Ranks

You shipped the page three weeks ago. It's live, it looks great, and it appears in exactly zero search results. Not ranked low. Absent.

That's usually not a ranking problem. Ranking is the last step, and a page has to survive three earlier ones to get there.

Four Steps, Four Ways to Fail

Work the list in order. Don't jump straight to keywords.

Getting Discovered

Links. Crawlers follow <a href>. A <div onClick={navigate}> is not a link and is not followed. It is the most common way a "fully working" navigation ends up invisible to search engines.

A sitemap. Generate it from the same data your navigation uses, so it can't drift:

1export default function sitemap(): MetadataRoute.Sitemap {
2  return getAllCategoryFiles().map((file) => ({
3    url: `${BASE_URL}${file.href}`,
4    lastModified: new Date(file.updatedAt ?? file.publishedAt),
5    changeFrequency: 'monthly',
6  }));
7}

Making Sure the Content Is There

The crawler fetches your HTML first. If the content is in it, it's read immediately. If not, the URL joins a render queue to be processed later, sometimes days later, sometimes never if your JavaScript errors.

1# What the crawler sees on the first pass
2curl -s https://example.com/pricing | grep -i "starting at"

If your headline isn't in that output, it isn't in the initial HTML.

Telling the Index What This Page Is

The title is the highest-leverage tag on the page: 50 to 60 characters, specific, and unique per page. Duplicate titles are a strong signal of duplicate content. Use a template so the brand suffix is never hand-typed, and set metadataBase, without which relative Open Graph paths and canonicals silently break in production.

1export const metadata: Metadata = {
2  metadataBase: new URL('https://example.com'),
3  title: {
4    template: '%s - Example',
5    default: 'Example: building things on the web',
6  },
7};

The description is not a ranking factor but very much a click factor. 150 to 160 characters, written for a human. Leave it out and the engine writes one by grabbing a paragraph, usually badly.

Canonicals name the one true URL when the same content is reachable at several of them: ?ref=twitter, trailing-slash variants, http vs https. Every page should have one pointing at itself, even with no duplicates today; query parameters appear later and you won't notice.

Structured data (JSON-LD) makes you eligible for rich results: FAQ dropdowns, review stars, recipe cards. It doesn't raise your ranking, it changes how your result looks, and a result with an FAQ accordion under it earns more clicks than a plain link in the same position. Escape < when injecting it, the same XSS guard you'd use anywhere.

The Content Half

Technical work gets you indexed. It doesn't get you ranked.

Write for a question, not a keyword. Keyword stuffing has been dead for a decade. Engines match intent. They know "why is my postgres query slow" and "how to read a query plan" are the same need. Answer it completely enough that the reader doesn't hit back and click the next result. That behavior is measurable, and pages that produce it slide down.

Match the intent type. Serving a product page to an informational query loses to a blog post every time, regardless of domain authority. Look at what currently ranks for your target query. The format that is winning tells you what the engine believes the intent is.

Structure is content. Headings are an outline the crawler reads. One <h1>, <h2> for real sections, no skipped levels, and descriptive rather than clever. "Getting Discovered" tells a crawler and a scanning human the same thing; "The Journey Begins" tells neither anything. Descriptive headings are also what an AI summarizer quotes when it answers on your behalf.

Depth beats volume. Ten thin posts that each half-answer a question rank worse than one that answers it properly, and thin pages dilute the whole site. If you already have a shallow page on a topic, expand it rather than publishing a second one beside it. Two pages competing for the same query split their own signals, and neither wins.

Link internally with real anchor text. Anchor text tells the engine what the destination is about, so click here wastes the signal. Link new posts to related older ones, then go back and link the old ones forward.

Speed and Layout Count

Core Web Vitals are a modest ranking factor and a large factor in whether people stay: LCP under 2.5s, INP under 200ms, CLS under 0.1. CLS is the cheapest to fix and the most visible when broken. A user tapping a button that moves out from under them will leave.

Verifying Any of It Worked

Search Console is the only place with the engine's actual opinion. URL Inspection settles "crawl problem or content problem" definitively. The Page Indexing report groups the reasons URLs aren't indexed. The Performance report shows real queries. Filter to where you rank 8 to 20, since those pages are one good edit away from page one, and a better use of an afternoon than starting something new.

The Takeaway

Search is a pipeline, and you debug it in order:

1. Discoverable: real links plus a sitemap generated from your own data.
2. Crawlable: not blocked, fast, returning 200.
3. Renderable: content in the server HTML.
4. Indexable: unique title, self-referencing canonical, enough substance to be worth storing.
5. Rankable: genuinely the best answer to a question someone asks.

Steps one through four are engineering. They are mostly one-time work, and getting them wrong makes step five irrelevant. Step five never ends, and it is the only one that decides whether you win.

Do the plumbing once. Then spend your time answering the question better than anyone else.