Essential Website Settings for SEO & GEO in the AI Era
TL;DR. Website settings for SEO and GEO start with crawlable pages, selective robots.txt, sitemap.xml, accurate structured data, and static or SSR HTML. llms.txt is optional—not a replacement for SEO or a citation promise.
Table of Contents
- Understanding GEO (Generative Engine Optimization) and how it differs from traditional SEO
- 1. Selective robots.txt: allow AI Search and limit training scrapers
- 2. Add /llms.txt and /llms-full.txt at the domain root
- 3. Strengthen entity context with JSON-LD Schema markup
- 4. Semantic HTML architecture: SSG or SSR exposes primary content in the first response
- FAQ: website settings for SEO and GEO
- Technical references
Understanding GEO (Generative Engine Optimization) and how it differs from traditional SEO
Search no longer always ends at a list of ten blue links. Answer systems can retrieve several pages, select the most relevant passages, and compose a summary with supporting links. This retrieval-augmented generation (RAG) flow makes information structure, evidence, and readability matter. But GEO and AEO are working terms, not independent ranking systems that can be enabled with one file. For Google AI Overviews and AI Mode, the foundation is still SEO: a page must be indexable, eligible for a snippet, and useful to people.
| Focus | Main purpose | Testable implementation |
|---|---|---|
| Classic SEO | Be discovered, indexed, and understood by search engines | Canonical URLs, sitemap.xml, original content, performance, and internal links |
| AEO | Answer a user's question quickly | Direct answers, question-led headings, clear lists, and tables |
| GEO | Make page context easier for generative answer systems to select | Verified facts, consistent entities, structured data, and crawlable pages |
1. Selective robots.txt: allow AI Search and limit training scrapers
robots.txt is a crawl instruction at the domain root, not a firewall. It does not protect private pages; dashboards, APIs, and sensitive data still need authentication or server-side access controls. For public content, separate the purpose of a bot. OpenAI documents OAI-SearchBot for ChatGPT search results and GPTBot for content that may be used to train foundation models. ChatGPT-User is different again: it can be triggered by a user action, and robots rules may not apply.
Start with a policy that matches the rights you actually hold. The example below allows selected search crawling, keeps operational paths out of scope, and declines OpenAI training. Add PerplexityBot, ClaudeBot, Applebot-Extended, or other tokens only after checking their current official documentation and verification mechanism; a user-agent name alone is not proof of bot identity.
CODE
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
Sitemap: https://yourname.aiclub.ai.id/sitemap.xmlThis is a policy example, not a universal recipe. Blocking GPTBot is not the same as blocking OAI-SearchBot; OpenAI treats them independently. Google-Extended is likewise a separate Google product control, not a replacement for Googlebot. Do not disallow Googlebot or Bingbot if you still want ordinary SEO. When a site appears unreadable to ChatGPT, test the public /robots.txt, /sitemap.xml, canonical page, and HTTP status before changing bot rules.
2. Add /llms.txt and /llms-full.txt at the domain root
llms.txt can be a concise Markdown index for people and tools that choose to use it: describe the project, primary pages, documentation, and information limits. llms-full.txt can provide a longer companion if you can keep it accurate. Neither file is a Google requirement, a replacement for sitemap.xml, nor a guarantee that ChatGPT, Perplexity, Claude, Gemini, or another system will cite the site. Google specifically cautions site owners not to create unnecessary AI text files solely to chase visibility.
CODE
# Raka Pratama's Developer Portfolio
> Indonesian backend developer building Go APIs, automation, and technical documentation.
## Primary pages
- [About](https://raka.aiclub.ai.id/about/): profile and skills.
- [Projects](https://raka.aiclub.ai.id/projects/): case studies with source code.
- [API notes](https://raka.aiclub.ai.id/docs/): public endpoint documentation.
## Limits
- This site contains no paid service or customer data.
- Use the contact page for security questions.On GitHub Pages, place the file in the root public artifact so it is available at /llms.txt. On Vercel or a modern framework, put it in the public folder mapped to the root. On Cloudflare Pages, place it in the publish directory. After deployment, open the final URL and confirm a 200 response with readable plain text. Do not rely on a local link that works only on a developer machine.
3. Strengthen entity context with JSON-LD Schema markup
JSON-LD Schema.org gives readers and machines explicit clues about what a page contains. Google recommends JSON-LD because it is generally easier to maintain, but markup must describe content that is genuinely visible. Schema is not a shortcut to AI Overviews and must not invent ratings, FAQs, organizations, or products. Choose a fitting type: Person for a portfolio, SoftwareApplication for an app, Article for a guide, and FAQPage only when the questions and answers are on the page.
CODE
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Person",
"name": "Raka Pratama",
"url": "https://raka.is-a.cv/",
"jobTitle": "Backend Developer",
"sameAs": ["https://github.com/rakapratama"],
"knowsAbout": ["Go", "REST API", "PostgreSQL"]
}
</script>4. Semantic HTML architecture: SSG or SSR exposes primary content in the first response
A pure React CSR site often fails not because React is bad, but because the initial HTML contains an empty root and content appears only after JavaScript, APIs, and hydration finish. Google can process JavaScript when it is not blocked, but JavaScript SEO is still more complex. Other bots may have different capabilities, time budgets, or policies. Static Site Generation (SSG) and Server-Side Rendering (SSR) reduce that dependency by sending the title, paragraphs, links, and schema in the initial HTML.
CODE
<main>
<article>
<h1>How to deploy a Go API</h1>
<p>This guide explains how to deploy a Go API to a VPS with HTTPS.</p>
<h2>Prerequisites</h2>
<ul><li>A Linux server</li><li>A public hostname</li></ul>
</article>
</main>Use one h1, ordered headings, main, article, valid tables, descriptive image alt text, and clear links. GitHub Pages, Vercel, and Cloudflare Pages fit this pattern. Connect a project through CNAME from AICLUB.AI.ID; its 13 themed options include .aiclub.ai.id, .std.ai.id, .is-a.cv, .promptku.web.id, .dataset.web.id, and .kripto.web.id. Choose an extension that fits the project, and keep prioritizing clear content and reliable site operations.
With this foundation, you are not forcing a model to cite your site. You are making the site easier to discover, read, and evaluate. Browse the AICLUB domain catalog, use the AICLUB Badge Builder to add a project badge and receive an additional slot when eligible, then share your experiment in the AICLUB Telegram community.
FAQ: website settings for SEO and GEO
What is the difference between robots.txt, sitemap.xml, and llms.txt?
robots.txt gives crawl instructions to specific user agents. sitemap.xml lists canonical URLs you want search engines to discover. llms.txt is an optional Markdown summary; it is not a Google indexing standard and cannot replace a sitemap, canonical tags, or useful page content.
Will blocking GPTBot remove a site from ChatGPT Search?
Not necessarily. OpenAI separates GPTBot, for content that may train foundation models, from OAI-SearchBot, used for ChatGPT search results. If you want Search eligibility, evaluate OAI-SearchBot separately and check OpenAI’s current policy before changing a rule.
Can a static site on a free subdomain appear in Google or be cited by AI?
It can be eligible for crawling and indexing when it meets technical requirements, offers useful content, and is not blocked from access. A free subdomain or static host is not a ranking or citation guarantee. Quality, relevance, and accessibility of the page still determine outcomes.
How do I add /llms.txt to GitHub Pages, Vercel, or Cloudflare Pages?
Place the text file at the root of the public artifact: the GitHub Pages output root, your Vercel or framework public folder, or the Cloudflare Pages publish directory. Deploy it, then open /llms.txt at the final domain. Keep it public, readable, and synchronized with canonical pages.
Technical references
- OpenAI: Overview of OpenAI Crawlers
- Google Search Central: Optimizing for generative AI features
- Google Search Central: Structured data and JSON-LD
- Google Search Central: JavaScript SEO basics
Is the website itself in good order?
The code-level settings are done. Next, make sure a firewall or challenge does not block the crawlers you explicitly allow.
Continue with Part 2: limit harmful scraping without blocking approved search crawlers.