astronit
/
oakazanin
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: дополнил robots.txt

This commit is contained in:
Astronit 2026-02-27 14:19:52 +00:00
parent dc532ca592
commit 1cbbb645fb
1 changed files with 16 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
static/robots.txt
View File

@ -1,9 +1,4 @@
# ============================================= # --- Yandex ---
# robots.txt для oakazanin.ru
# IT-блог на Hugo | /posts/ | RU
# =============================================
# --- Яндекс ---
User-agent: Yandex User-agent: Yandex
Allow: / Allow: /
Allow: /posts/ Allow: /posts/
@ -20,7 +15,7 @@ Disallow: /tags/
Disallow: /categories/ Disallow: /categories/
Disallow: /page/ Disallow: /page/
# --- AI-боты --- # --- AI-bots ---
User-agent: GPTBot User-agent: GPTBot
Allow: / Allow: /
@ -33,7 +28,20 @@ Allow: /
User-agent: Claude-Web User-agent: Claude-Web
Allow: / Allow: /
# --- Все остальные боты --- User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
# --- Google AI Overview ---
User-agent: Google-Extended
Allow: /
User-agent: ClaudeBot
Allow: /
# --- All other bots ---
User-agent: * User-agent: *
Allow: / Allow: /
Allow: /posts/ Allow: /posts/