{"id":602,"date":"2026-11-20T08:47:00","date_gmt":"2026-11-19T21:47:00","guid":{"rendered":"https:\/\/onlinewebdirectory.net\/blog\/research\/yapay-zeka-tarayici-listesi\/"},"modified":"2026-11-20T08:47:30","modified_gmt":"2026-11-19T21:47:30","slug":"yapay-zeka-tarayici-listesi","status":"publish","type":"post","link":"https:\/\/onlinewebdirectory.net\/blog\/tr\/ai-crawlers-tr\/crawler-list-tr\/yapay-zeka-tarayici-listesi\/","title":{"rendered":"Yapay Zek\u00e2 Taray\u0131c\u0131 Listesi 2026: Her AI Botu, User Agent&#8217;\u0131 ve G\u00f6revi"},"content":{"rendered":"<p>Bulaca\u011f\u0131n\u0131z &#8220;AI taray\u0131c\u0131 listelerinin&#8221; \u00e7o\u011fu birbirinden kopyalanm\u0131\u015ft\u0131r ve bir\u00e7o\u011fu g\u00fcncel de\u011fildir. H\u00e2l\u00e2 <code>Claude-Web<\/code>&#8216;i Anthropic&#8217;in taray\u0131c\u0131s\u0131 olarak g\u00f6sterir, &#8220;Googlebot-Extended&#8221; diye bir bot uydurur ya da Google-Extended&#8217;in sayfalar\u0131n\u0131z\u0131 tarad\u0131\u011f\u0131n\u0131 s\u00f6ylerler. Taramaz. Google-Extended&#8217;in kendine ait bir user agent&#8217;\u0131 yoktur; yaln\u0131zca bir robots.txt token&#8217;\u0131d\u0131r.<\/p>\n<p>Bu liste iki a\u00e7\u0131dan farkl\u0131. A\u015fa\u011f\u0131daki her user agent, 20 Kas\u0131m 2026 itibar\u0131yla operat\u00f6r\u00fcn kendi belgelerine g\u00f6re do\u011fruland\u0131. Ayr\u0131ca ger\u00e7ek sitelerin bu botlara nas\u0131l davrand\u0131\u011f\u0131n\u0131 \u00f6l\u00e7t\u00fck: Online Web Directory kayd\u0131ndan rastgele se\u00e7ilen 3.000 sitenin robots.txt dosyas\u0131n\u0131 indirdik ve hangi AI taray\u0131c\u0131lar\u0131n\u0131 and\u0131klar\u0131n\u0131 ve engellediklerini sayd\u0131k. Ayr\u0131ca robots.txt, Nginx ve Apache i\u00e7in test edilmi\u015f kurallar ve bir &#8220;GPTBot&#8221; ziyaret\u00e7isinin ger\u00e7ekten OpenAI&#8217;dan gelip gelmedi\u011fini kontrol eden bir betik de bulacaks\u0131n\u0131z.<\/p>\n<div class=\"owd-definition\">\n<p><strong>AI taray\u0131c\u0131 listesi nedir?<\/strong> AI taray\u0131c\u0131 listesi, yapay zek\u00e2 \u015firketlerinin web sayfalar\u0131n\u0131 almak i\u00e7in kulland\u0131\u011f\u0131 botlar\u0131n bir ba\u015fvuru listesidir; her botun robots.txt token&#8217;\u0131n\u0131, user agent dizesini ve g\u00f6revini g\u00f6sterir: e\u011fitim verisi toplamak, AI aramas\u0131 i\u00e7in sayfalar\u0131 dizine eklemek ya da bir kullan\u0131c\u0131 istedi\u011fi i\u00e7in bir sayfay\u0131 an\u0131nda getirmek. robots.txt kurallar\u0131 yazmak, sunucu g\u00fcnl\u00fcklerini okumak ve g\u00fcvenlik duvar\u0131 politikalar\u0131 belirlemek i\u00e7in kullan\u0131l\u0131r.<\/p>\n<\/div>\n<h2 id=\"at-a-glance\">Bir bak\u0131\u015fta AI taray\u0131c\u0131lar\u0131<\/h2>\n<figure class=\"wp-block-table owd-table\">\n<table>\n<thead>\n<tr>\n<th>Soru<\/th>\n<th>K\u0131sa cevap<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Ka\u00e7 AI taray\u0131c\u0131s\u0131 \u00f6nemli?<\/td>\n<td>9 operat\u00f6rden (OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Common Crawl, Mistral) yakla\u015f\u0131k 20 belgelenmi\u015f user agent, ayr\u0131ca DuckDuckGo ve uzun bir k\u00fc\u00e7\u00fck bot kuyru\u011fu.<\/td>\n<\/tr>\n<tr>\n<td>Ana t\u00fcrler<\/td>\n<td>E\u011fitim taray\u0131c\u0131lar\u0131, AI arama taray\u0131c\u0131lar\u0131 ve kullan\u0131c\u0131 taraf\u0131ndan tetiklenen ajanlar veya getiriciler. Google-Extended ve Applebot-Extended taray\u0131c\u0131 de\u011fil, kontrol token&#8217;lar\u0131d\u0131r.<\/td>\n<\/tr>\n<tr>\n<td>robots.txt&#8217;ye uyarlar m\u0131?<\/td>\n<td>Belgelenmi\u015f e\u011fitim ve arama taray\u0131c\u0131lar\u0131 uyduklar\u0131n\u0131 s\u00f6yl\u00fcyor. Kullan\u0131c\u0131 taraf\u0131ndan tetiklenen getiriciler (ChatGPT-User, Perplexity-User, Google&#8217;\u0131n kullan\u0131c\u0131 tetiklemeli getiricileri, Meta-ExternalFetcher) robots.txt&#8217;nin ge\u00e7erli olmayabilece\u011fini s\u00f6yl\u00fcyor.<\/td>\n<\/tr>\n<tr>\n<td>robots.txt 403 hatas\u0131 d\u00f6nd\u00fcrebilir mi?<\/td>\n<td>Hay\u0131r. robots.txt bir kilit de\u011fil, bir ricad\u0131r. 403 yan\u0131t\u0131 sunucunuzdan, CDN&#8217;nizden veya g\u00fcvenlik duvar\u0131n\u0131zdan gelir.<\/td>\n<\/tr>\n<tr>\n<td>Siteler onlar\u0131 ne s\u0131kl\u0131kla engelliyor?<\/td>\n<td>20 Kas\u0131m 2026 \u00f6rneklemimizde robots.txt dosyalar\u0131n\u0131n %16,9&#8217;u GPTBot&#8217;u tamamen engelliyor, ancak yaln\u0131zca %2,8&#8217;i OAI-SearchBot&#8217;u tamamen engelliyor.<\/td>\n<\/tr>\n<tr>\n<td>Bir botu nas\u0131l do\u011frulars\u0131n\u0131z?<\/td>\n<td>\u0130ste\u011fin IP&#8217;sini operat\u00f6r\u00fcn yay\u0131mlad\u0131\u011f\u0131 IP listesiyle kar\u015f\u0131la\u015ft\u0131r\u0131n (\u00f6rne\u011fin <code>https:\/\/openai.com\/gptbot.json<\/code>). User agent kolayca taklit edilebilir.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<h2 id=\"the-list\">AI taray\u0131c\u0131 listesi (Kas\u0131m 2026&#8217;da do\u011fruland\u0131)<\/h2>\n<p>Tablo her robots.txt token&#8217;\u0131n\u0131, kimin i\u015fletti\u011fini, operat\u00f6re g\u00f6re ne yapt\u0131\u011f\u0131n\u0131 ve operat\u00f6r\u00fcn robots.txt&#8217;ye uydu\u011funu s\u00f6yleyip s\u00f6ylemedi\u011fini g\u00f6steriyor. &#8220;T\u00fcr&#8221; s\u00fctununda \u00fc\u00e7 grup var: <strong>e\u011fitim<\/strong> (modelleri e\u011fitebilecek i\u00e7erik toplar), <strong>arama<\/strong> (bir AI \u00fcr\u00fcn\u00fc daha sonra al\u0131nt\u0131layabilsin diye sayfalar\u0131 dizine ekler) ve <strong>kullan\u0131c\u0131<\/strong> (bir ki\u015fi istedi\u011fi i\u00e7in sayfay\u0131 ger\u00e7ek zamanl\u0131 getirir). Operat\u00f6r\u00fc do\u011frulanabilir belge yay\u0131mlamayan botlar\u0131 listeye almad\u0131k.<\/p>\n<figure class=\"wp-block-table owd-table\">\n<table>\n<thead>\n<tr>\n<th>robots.txt token&#8217;\u0131<\/th>\n<th>Operat\u00f6r<\/th>\n<th>T\u00fcr<\/th>\n<th>Ne yapar<\/th>\n<th>robots.txt&#8217;ye uyar m\u0131?<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>GPTBot<\/code><\/td>\n<td>OpenAI<\/td>\n<td>E\u011fitim<\/td>\n<td>OpenAI&#8217;\u0131n temel modellerini e\u011fitmek i\u00e7in kullan\u0131labilecek i\u00e7erikleri tarar.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>OAI-SearchBot<\/code><\/td>\n<td>OpenAI<\/td>\n<td>Arama<\/td>\n<td>Sayfalar\u0131 ChatGPT arama sonu\u00e7lar\u0131nda g\u00f6r\u00fcnebilmeleri i\u00e7in dizine ekler.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>ChatGPT-User<\/code><\/td>\n<td>OpenAI<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Bir ChatGPT veya Custom GPT kullan\u0131c\u0131s\u0131 soru sordu\u011funda sayfay\u0131 ziyaret eder.<\/td>\n<td>&#8220;Ge\u00e7erli olmayabilir&#8221;<\/td>\n<\/tr>\n<tr>\n<td><code>OAI-AdsBot<\/code><\/td>\n<td>OpenAI<\/td>\n<td>Reklam kontrol\u00fc<\/td>\n<td>ChatGPT reklam\u0131 olarak g\u00f6nderilen a\u00e7\u0131l\u0131\u015f sayfalar\u0131n\u0131 kontrol eder; e\u011fitim i\u00e7in kullan\u0131lmaz.<\/td>\n<td>Yaln\u0131zca g\u00f6nderilen sayfalar\u0131 ziyaret eder<\/td>\n<\/tr>\n<tr>\n<td><code>ClaudeBot<\/code><\/td>\n<td>Anthropic<\/td>\n<td>E\u011fitim<\/td>\n<td>Claude modellerinin e\u011fitimine katk\u0131da bulunabilecek web i\u00e7eri\u011fi toplar.<\/td>\n<td>Evet, <code>Crawl-delay<\/code>&#8216;e de uyar<\/td>\n<\/tr>\n<tr>\n<td><code>Claude-SearchBot<\/code><\/td>\n<td>Anthropic<\/td>\n<td>Arama<\/td>\n<td>Claude&#8217;un arama sonu\u00e7lar\u0131n\u0131 iyile\u015ftirmek i\u00e7in i\u00e7erikleri dizine ekler.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>Claude-User<\/code><\/td>\n<td>Anthropic<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Bir Claude kullan\u0131c\u0131s\u0131 soru sordu\u011funda sayfalar\u0131 getirir.<\/td>\n<td>Evet, Anthropic&#8217;e g\u00f6re<\/td>\n<\/tr>\n<tr>\n<td><code>Google-Extended<\/code><\/td>\n<td>Google<\/td>\n<td>Kontrol token&#8217;\u0131<\/td>\n<td>Google&#8217;\u0131n tarad\u0131\u011f\u0131 i\u00e7eri\u011fin Gemini&#8217;yi e\u011fitip e\u011fitemeyece\u011fini veya yan\u0131tlar\u0131na dayanak olup olamayaca\u011f\u0131n\u0131 belirler. Ayr\u0131 bir taray\u0131c\u0131 de\u011fildir.<\/td>\n<td>Yaln\u0131zca token<\/td>\n<\/tr>\n<tr>\n<td><code>Google-CloudVertexBot<\/code><\/td>\n<td>Google<\/td>\n<td>Site sahibinin talebiyle<\/td>\n<td>Sahipleri Vertex AI Agents olu\u015fturan siteleri tarar; Google Arama&#8217;ya etkisi yoktur.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>Google-Agent<\/code><\/td>\n<td>Google<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Bir kullan\u0131c\u0131n\u0131n iste\u011fiyle gezinen ve i\u015flem yapan, Google taraf\u0131ndan bar\u0131nd\u0131r\u0131lan ajanlar.<\/td>\n<td>Kullan\u0131c\u0131 tetiklemeli: genelde hay\u0131r<\/td>\n<\/tr>\n<tr>\n<td><code>PerplexityBot<\/code><\/td>\n<td>Perplexity<\/td>\n<td>Arama<\/td>\n<td>Perplexity arama sonu\u00e7lar\u0131 i\u00e7in sayfalar\u0131 dizine ekler; e\u011fitim i\u00e7in kullan\u0131lmaz.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>Perplexity-User<\/code><\/td>\n<td>Perplexity<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Bir kullan\u0131c\u0131n\u0131n Perplexity oturumu s\u0131ras\u0131nda sayfa getirir.<\/td>\n<td>&#8220;Genelde yok sayar&#8221;<\/td>\n<\/tr>\n<tr>\n<td><code>Applebot-Extended<\/code><\/td>\n<td>Apple<\/td>\n<td>Kontrol token&#8217;\u0131<\/td>\n<td>Applebot&#8217;un tarad\u0131\u011f\u0131 sayfalar\u0131n Apple&#8217;\u0131n temel modellerini e\u011fitip e\u011fitemeyece\u011fini belirler. Kendisi hi\u00e7bir \u015fey taramaz.<\/td>\n<td>Yaln\u0131zca token<\/td>\n<\/tr>\n<tr>\n<td><code>meta-externalagent<\/code><\/td>\n<td>Meta<\/td>\n<td>E\u011fitim<\/td>\n<td>AI modellerini e\u011fitmek veya i\u00e7erik dizinlemek gibi ama\u00e7larla tarar.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>meta-externalfetcher<\/code><\/td>\n<td>Meta<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Ajan g\u00f6revleri dahil, kullan\u0131c\u0131n\u0131n iste\u011fiyle belirli ba\u011flant\u0131lar\u0131 getirir.<\/td>\n<td>&#8220;Atlayabilir&#8221;<\/td>\n<\/tr>\n<tr>\n<td><code>Amazonbot<\/code><\/td>\n<td>Amazon<\/td>\n<td>E\u011fitim<\/td>\n<td>Amazon \u00fcr\u00fcn ve hizmetlerini geli\u015ftirir; Amazon&#8217;un AI modellerini e\u011fitmek i\u00e7in kullan\u0131labilir.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>Amzn-SearchBot<\/code><\/td>\n<td>Amazon<\/td>\n<td>Arama<\/td>\n<td>Alexa gibi arama deneyimleri i\u00e7in sayfalar\u0131 dizine ekler; e\u011fitim i\u00e7in kullan\u0131lmaz.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>Amzn-User<\/code><\/td>\n<td>Amazon<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Bir kullan\u0131c\u0131y\u0131 yan\u0131tlamak i\u00e7in anl\u0131k bilgi getirir, \u00f6rne\u011fin bir Alexa iste\u011fi.<\/td>\n<td>&#8220;Hepsine uymayabilir&#8221;<\/td>\n<\/tr>\n<tr>\n<td><code>CCBot<\/code><\/td>\n<td>Common Crawl<\/td>\n<td>E\u011fitim verisi<\/td>\n<td>Yapay zek\u00e2 e\u011fitim verisinin yayg\u0131n bir kayna\u011f\u0131 olan Common Crawl a\u00e7\u0131k web ar\u015fivini olu\u015fturur.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>MistralAI-Training<\/code><\/td>\n<td>Mistral AI<\/td>\n<td>E\u011fitim<\/td>\n<td>Mistral modelleri i\u00e7in e\u011fitim veri k\u00fcmeleri olu\u015fturmak amac\u0131yla i\u00e7erik tarar.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>MistralAI-Index<\/code><\/td>\n<td>Mistral AI<\/td>\n<td>Arama<\/td>\n<td>Mistral aramas\u0131 i\u00e7in i\u00e7erikleri dizine ekler; e\u011fitim i\u00e7in kullan\u0131lmaz.<\/td>\n<td>Evet<\/td>\n<\/tr>\n<tr>\n<td><code>MistralAI-User<\/code><\/td>\n<td>Mistral AI<\/td>\n<td>Kullan\u0131c\u0131<\/td>\n<td>Bir kullan\u0131c\u0131 Mistral&#8217;in asistan\u0131 Vibe&#8217;a soru sordu\u011funda sayfay\u0131 ziyaret eder.<\/td>\n<td>Evet, kullan\u0131c\u0131 istekleri i\u00e7in<\/td>\n<\/tr>\n<tr>\n<td><code>DuckAssistBot<\/code><\/td>\n<td>DuckDuckGo<\/td>\n<td>Arama<\/td>\n<td>DuckDuckGo&#8217;nun AI destekli yan\u0131tlar\u0131 i\u00e7in sayfalar\u0131 ger\u00e7ek zamanl\u0131 tarar.<\/td>\n<td>Evet (72 saat sonra ge\u00e7erli olur)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>Her sat\u0131r\u0131n kaynaklar\u0131 makalenin sonunda ba\u011flant\u0131l\u0131. Son s\u00fctundaki al\u0131nt\u0131lar do\u011frudan operat\u00f6r belgelerinden al\u0131nd\u0131. Bytespider (ByteDance) bilerek yok: \u00f6rneklemimizdeki robots.txt dosyalar\u0131n\u0131n %18,5&#8217;i onu ansa da ByteDance&#8217;in onu belgeleyen bir sayfas\u0131n\u0131 bulamad\u0131k. Ayn\u0131 nedenle <code>cohere-ai<\/code>, <code>Diffbot<\/code> veya <code>omgili<\/code> gibi uzun token kuyru\u011funu a\u00e7\u0131klam\u0131yoruz; ne s\u0131kl\u0131kla ge\u00e7tikleri veri b\u00f6l\u00fcm\u00fcnde.<\/p>\n<h3>G\u00fcnl\u00fcklerinizde g\u00f6rece\u011finiz tam user agent dizeleri<\/h3>\n<p>robots.txt k\u0131sa token ile \u00e7al\u0131\u015f\u0131r, ancak eri\u015fim g\u00fcnl\u00fckleriniz tam dizeyi g\u00f6sterir. Bunlar operat\u00f6rlerin yay\u0131mlad\u0131\u011f\u0131 \u00f6rneklerdir. S\u00fcr\u00fcm numaralar\u0131 de\u011fi\u015fir, bu y\u00fczden tam dizeye de\u011fil token&#8217;a g\u00f6re filtreleyin.<\/p>\n<pre class=\"wp-block-code\"><code># GPTBot\nMozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko); compatible; GPTBot\/1.4; +https:\/\/openai.com\/gptbot\n\n# OAI-SearchBot\nMozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/131.0.0.0 Safari\/537.36; compatible; OAI-SearchBot\/1.4; +https:\/\/openai.com\/searchbot\n\n# ChatGPT-User\nMozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko); compatible; ChatGPT-User\/1.0; +https:\/\/openai.com\/bot\n\n# PerplexityBot\nMozilla\/5.0 (compatible; PerplexityBot\/1.0; +https:\/\/perplexity.ai\/perplexitybot)\n\n# Perplexity-User\nMozilla\/5.0 (compatible; Perplexity-User\/1.0; +https:\/\/perplexity.ai\/perplexity-user)\n\n# Amazonbot\nMozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; compatible; Amazonbot\/0.1) Chrome\/W.X.Y.Z Safari\/537.36\n\n# CCBot\nCCBot\/2.0 (https:\/\/commoncrawl.org\/faq\/)\n\n# meta-externalagent\nmeta-externalagent\/1.1 (+https:\/\/developers.facebook.com\/docs\/sharing\/webmasters\/crawler)\n\n# MistralAI-User\nMozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; compatible; MistralAI-User\/1.0; +https:\/\/docs.mistral.ai\/robots)\n\n# DuckAssistBot\nDuckAssistBot\/1.2; (+http:\/\/duckduckgo.com\/duckassistbot.html)<\/code><\/pre>\n<p>OpenAI ayr\u0131ca GPTBot ve OAI-SearchBot&#8217;un robots.txt dosyan\u0131z\u0131 al\u0131rken user agent&#8217;a bir <code>robots.txt<\/code> i\u015fareti ekleyebilece\u011fini belirtiyor; b\u00f6ylece yol i\u00e7ermeyen g\u00fcnl\u00fcklerde bu istekleri ay\u0131rt edebilirsiniz.<\/p>\n<h2 id=\"types\">\u00dc\u00e7 t\u00fcr AI taray\u0131c\u0131s\u0131 ve fark\u0131n neden \u00f6nemli oldu\u011fu<\/h2>\n<p>&#8220;AI botlar\u0131n\u0131&#8221; tek grup olarak engellemek, g\u00f6rd\u00fc\u011f\u00fcm\u00fcz en yayg\u0131n hata. \u00dc\u00e7 t\u00fcr\u00fcn maliyeti ve faydas\u0131 farkl\u0131d\u0131r ve operat\u00f6rler robots.txt&#8217;yi her biri i\u00e7in farkl\u0131 uygular.<\/p>\n<h3>E\u011fitim taray\u0131c\u0131lar\u0131<\/h3>\n<p>GPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot ve MistralAI-Training, e\u011fitim verisine girebilecek sayfalar\u0131 toplar. Onlar\u0131 engellemek sizi ayn\u0131 \u015firketin AI arama sonu\u00e7lar\u0131ndan \u00e7\u0131karmaz: OpenAI her robots.txt ayar\u0131n\u0131n ba\u011f\u0131ms\u0131z oldu\u011funu s\u00f6yl\u00fcyor, yani GPTBot&#8217;u engelleyip OAI-SearchBot&#8217;a izin verebilirsiniz. \u00d6rneklemimizde en \u00e7ok engellenen grup e\u011fitim taray\u0131c\u0131lar\u0131. robots.txt dosyalar\u0131n\u0131n %17,3&#8217;\u00fc CCBot&#8217;u, %16,9&#8217;u GPTBot&#8217;u ve %15,9&#8217;u ClaudeBot&#8217;u tamamen engelliyor.<\/p>\n<h3>AI arama taray\u0131c\u0131lar\u0131<\/h3>\n<p>OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index ve DuckAssistBot, bir asistan\u0131n daha sonra bulup al\u0131nt\u0131layabilmesi i\u00e7in sayfalar\u0131 dizine ekler. Onlar\u0131 engellemek sizi bu yan\u0131tlar\u0131n d\u0131\u015f\u0131nda b\u0131rakabilir. Site sahipleri bunu biliyor gibi: GPTBot i\u00e7in %16,9&#8217;a kar\u015f\u0131l\u0131k dosyalar\u0131n yaln\u0131zca %2,8&#8217;i OAI-SearchBot&#8217;u, %5,8&#8217;i PerplexityBot&#8217;u tamamen engelliyor.<\/p>\n<h3>Kullan\u0131c\u0131 taraf\u0131ndan tetiklenen getiriciler ve ajanlar<\/h3>\n<p>ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User ve Google-Agent, t\u0131pk\u0131 bir taray\u0131c\u0131 gibi her seferinde tek bir ki\u015fi ad\u0131na hareket eder. Sayfay\u0131 bir insan istedi\u011fi i\u00e7in bir\u00e7ok operat\u00f6r robots.txt&#8217;nin ge\u00e7erli olmayabilece\u011fini s\u00f6yl\u00fcyor. Al\u0131\u015fveri\u015f ve rezervasyon ajanlar\u0131n\u0131n kulland\u0131\u011f\u0131 botlar da bunlar; onlar\u0131 engellemek ger\u00e7ek m\u00fc\u015fterileri uzakla\u015ft\u0131rabilir. \u00d6rneklemimizdeki dosyalar\u0131n %5,9&#8217;u ChatGPT-User&#8217;\u0131 tamamen engelliyor.<\/p>\n<h3>Hi\u00e7bir \u015fey taramayan kontrol token&#8217;lar\u0131<\/h3>\n<p>Google-Extended ve Applebot-Extended sitenizi asla ziyaret etmez. Google ve Apple her zamanki gibi Googlebot ve Applebot ile tarar, ard\u0131ndan i\u00e7eri\u011fin modellerini e\u011fitip e\u011fitemeyece\u011fine karar vermek i\u00e7in bu token&#8217;lara bakar. Google, Google-Extended&#8217;i engellemenin Google Arama&#8217;ya dahil olmay\u0131 veya s\u0131ralamay\u0131 etkilemedi\u011fini s\u00f6yl\u00fcyor. \u0130ncelenen dosyalar\u0131n %14,1&#8217;i Google-Extended&#8217;i, %13,5&#8217;i Applebot-Extended&#8217;i tamamen engelliyor. Google ayr\u0131ca ortak AI e\u011fitim sinyalini (<code>user-agent: __usage_ai-training<\/code>) 2027 ba\u015f\u0131nda desteklemeyi planlad\u0131\u011f\u0131n\u0131 a\u00e7\u0131klad\u0131; bunu <a href=\"\/blog\/tr\/robots-txt-tr\/training-opt-out-tr\/robots-txt-yapay-zeka-egitimi-reddi\/\">robots.txt ile AI e\u011fitiminden \u00e7\u0131kma rehberimizde<\/a> ele al\u0131yoruz.<\/p>\n<h3>robots.txt dosyalar\u0131n\u0131 h\u00e2l\u00e2 dolduran eski token&#8217;lar<\/h3>\n<p>Anthropic&#8217;in g\u00fcncel yard\u0131m sayfas\u0131 yaln\u0131zca ClaudeBot, Claude-User ve Claude-SearchBot&#8217;u an\u0131yor. Yine de incelenen dosyalar\u0131n %15,6&#8217;s\u0131 h\u00e2l\u00e2 <code>anthropic-ai<\/code>&#8216;yi, %13,8&#8217;i <code>Claude-Web<\/code>&#8216;i an\u0131yor. Bu adlar i\u00e7in kurallar zarar vermez, ancak Anthropic i\u00e7in tek kurallar\u0131n\u0131z bunlarsa Anthropic&#8217;in bug\u00fcn belgeledi\u011fi botlar\u0131 kapsamayabilirler. <code>ClaudeBot<\/code>&#8216;un a\u00e7\u0131k\u00e7a an\u0131ld\u0131\u011f\u0131ndan emin olun.<\/p>\n<h2 id=\"what-it-looks-like\">AI taray\u0131c\u0131 eri\u015fimi pratikte nas\u0131l g\u00f6r\u00fcn\u00fcyor<\/h2>\n<p>User agent&#8217;lar i\u00e7in esas kaynak operat\u00f6r belgeleridir. \u00d6rne\u011fin OpenAI&#8217;\u0131n sayfas\u0131 her botu \u00f6rnek bir user agent dizesi, robots.txt i\u015faretiyle ilgili notlar ve yay\u0131mlanan IP dosyas\u0131na bir ba\u011flant\u0131yla listeliyor.<\/p>\n<figure class=\"wp-block-image size-full\"><img fetchpriority=\"high\" class=\"wp-image-576\" srcset=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/openai-bots-v3-768w.webp 768w, https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/openai-bots-v3.webp 1200w\" sizes=\"(max-width: 848px) calc(100vw - 48px), 800px\" src=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/openai-bots-v3.webp\" alt=\"OAI-SearchBot i\u00e7in OpenAI belgeleri; \u00f6rnek user agent dizesi, robots.txt i\u015fareti ve yay\u0131mlanan IP dosyas\u0131\" width=\"1200\" height=\"900\" loading=\"eager\" decoding=\"async\" \/><figcaption>OpenAI&#8217;\u0131n OAI-SearchBot belgeleri, 20 Kas\u0131m 2026&#8217;da kaydedildi.<\/figcaption><\/figure>\n<p>Bir sitenin bu botlara ger\u00e7ekte ne yapt\u0131\u011f\u0131 ba\u015fka bir konu. Sunucumuzdan alt\u0131 ana sayfay\u0131 bir Chrome user agent&#8217;\u0131, OpenAI&#8217;\u0131n GPTBot user agent&#8217;\u0131 ve bir ClaudeBot user agent&#8217;\u0131 ile istedik. Sunucumuz, bir\u00e7ok bot korumas\u0131n\u0131n \u015f\u00fcpheyle bakt\u0131\u011f\u0131 bir veri merkezi IP&#8217;si kullan\u0131yor; ev ba\u011flant\u0131s\u0131ndan sonu\u00e7lar farkl\u0131 olabilir.<\/p>\n<figure class=\"wp-block-image size-full\"><img class=\"wp-image-570\" srcset=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/curl-tests-v2-768w.webp 768w, https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/curl-tests-v2.webp 1200w\" sizes=\"auto, (max-width: 848px) calc(100vw - 48px), 800px\" src=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/curl-tests-v2.webp\" alt=\"20 Kas\u0131m 2026'da Chrome, GPTBot ve ClaudeBot user agent'lar\u0131yla istenen alt\u0131 site i\u00e7in curl sonu\u00e7lar\u0131\" width=\"1200\" height=\"991\" loading=\"lazy\" decoding=\"async\" \/><figcaption>Ayn\u0131 URL, \u00fc\u00e7 user agent. Veri merkezi IP&#8217;sinden sonu\u00e7lar, 20 Kas\u0131m 2026, 08:03 AEDT.<\/figcaption><\/figure>\n<p>\u00dc\u00e7 \u00f6r\u00fcnt\u00fc ortaya \u00e7\u0131k\u0131yor. Reddit Chrome&#8217;a sayfay\u0131 verdi ama GPTBot&#8217;a 403, ClaudeBot&#8217;a 429 d\u00f6nd\u00fcrd\u00fc: sunucuda veya CDN&#8217;de user agent filtrelemesi. The New York Times \u00fc\u00e7\u00fcne de 403 d\u00f6nd\u00fcrd\u00fc; bu, user agent&#8217;a de\u011fil IP&#8217;mize veya istek parmak izine dayal\u0131 bir engellemeye i\u015faret ediyor. bbc.com ise robots.txt dosyas\u0131 GPTBot, ClaudeBot, CCBot, PerplexityBot ve en az be\u015f ba\u015fka AI token&#8217;\u0131n\u0131 engelledi\u011fi h\u00e2lde GPTBot&#8217;a 200 d\u00f6nd\u00fcrd\u00fc. Bu, robots.txt&#8217;nin rica etti\u011finin ama zorlamad\u0131\u011f\u0131n\u0131n en a\u00e7\u0131k kan\u0131t\u0131. Amazon \u00fc\u00e7\u00fcne de bo\u015f g\u00f6vdeli bir 202 ile, ger\u00e7ek sayfa yerine ara bir do\u011frulama sayfas\u0131yla yan\u0131t verdi.<\/p>\n<p>Online Web Directory&#8217;deki her site kayd\u0131 bu iki katman\u0131 ay\u0131r\u0131r: ajanlar\u0131n siteye ula\u015f\u0131p ula\u015famad\u0131\u011f\u0131 ve robots.txt&#8217;nin ne dedi\u011fi.<\/p>\n<figure class=\"wp-block-image size-full\"><img class=\"wp-image-572\" srcset=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/owd-site-bbc-v2-768w.webp 768w, https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/owd-site-bbc-v2.webp 1200w\" sizes=\"auto, (max-width: 848px) calc(100vw - 48px), 800px\" src=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/owd-site-bbc-v2.webp\" alt=\"bbc.com i\u00e7in Online Web Directory kayd\u0131: ajanlara a\u00e7\u0131k, bot duvar\u0131 yok, robots.txt baz\u0131 yollar\u0131 k\u0131s\u0131tl\u0131yor\" width=\"1200\" height=\"1097\" loading=\"lazy\" decoding=\"async\" \/><figcaption>bbc.com&#8217;un OWD kayd\u0131: ajanlara a\u00e7\u0131k, baz\u0131 yollar\u0131 k\u0131s\u0131tlayan bir robots.txt ile.<\/figcaption><\/figure>\n<h2 id=\"how-to-check\">Sitenize hangi AI taray\u0131c\u0131lar\u0131n\u0131n ula\u015fabildi\u011fini nas\u0131l kontrol edersiniz<\/h2>\n<h3>1. robots.txt gruplar\u0131n\u0131z\u0131 AI token&#8217;lar\u0131 i\u00e7in okuyun<\/h3>\n<p>Taray\u0131c\u0131lardan ne istedi\u011finizle ba\u015flay\u0131n. Bu komut, yayg\u0131n bir AI token&#8217;\u0131n\u0131 anan her grubu sonraki sat\u0131r\u0131yla birlikte g\u00f6sterir:<\/p>\n<pre class=\"wp-block-code\"><code>curl -s https:\/\/www.bbc.com\/robots.txt | grep -i -A1 -E '^user-agent: *(gptbot|oai-searchbot|chatgpt-user|claudebot|google-extended|perplexitybot|ccbot|bytespider|applebot-extended)'<\/code><\/pre>\n<p>bbc.com i\u00e7in 20 Kas\u0131m 2026, 08:09 AEDT&#8217;deki ger\u00e7ek \u00e7\u0131kt\u0131 (dokuz gruptan d\u00f6rd\u00fcne k\u0131salt\u0131ld\u0131):<\/p>\n<pre class=\"wp-block-code\"><code>User-agent: CCBot\nDisallow: \/\n--\nUser-agent: ClaudeBot\nDisallow: \/\n--\nUser-agent: GPTBot\nDisallow: \/\n--\nUser-agent: OAI-SearchBot\nDisallow: \/<\/code><\/pre>\n<p>Bir taray\u0131c\u0131n\u0131n yaln\u0131zca token&#8217;\u0131yla e\u015fle\u015fen en \u00f6zel gruba uydu\u011funu unutmay\u0131n. <code>GPTBot<\/code>&#8216;u kendi grubunda anarsan\u0131z <code>User-agent: *<\/code> kurallar\u0131n\u0131z\u0131 tamamen yok sayar; ortak kurallar\u0131 GPTBot grubunda tekrarlay\u0131n.<\/p>\n<h3>2. Sunucunuzun AI user agent&#8217;lar\u0131na yan\u0131t\u0131n\u0131 test edin<\/h3>\n<p>robots.txt &#8220;hay\u0131r&#8221; derken sunucunuz h\u00e2l\u00e2 &#8220;evet&#8221; diyebilir ya da tersi. Ayn\u0131 sayfay\u0131 farkl\u0131 user agent&#8217;larla isteyin ve durum kodlar\u0131n\u0131 kar\u015f\u0131la\u015ft\u0131r\u0131n:<\/p>\n<pre class=\"wp-block-code\"><code>GPT='Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko); compatible; GPTBot\/1.4; +https:\/\/openai.com\/gptbot'\nCHROME='Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/130.0.0.0 Safari\/537.36'\nfor ua in \"$CHROME\" \"$GPT\"; do\n  curl -s -o \/dev\/null -A \"$ua\" -w \"%{http_code} %{size_download}B\\n\" https:\/\/www.example-shop.com\/\ndone<\/code><\/pre>\n<p>URL&#8217;yi kendinizinkiyle de\u011fi\u015ftirin. Yaln\u0131zca GPTBot i\u00e7in 403 veya 429, user agent filtrelemesine i\u015faret eder. \u0130kisi i\u00e7in de 403 genellikle IP itibar\u0131 veya bot do\u011frulamas\u0131 demektir. Hem ev ba\u011flant\u0131s\u0131ndan hem sunucudan test edin: ger\u00e7ek taray\u0131c\u0131lar veri merkezi IP&#8217;lerinden gelir, bu y\u00fczden sunucu sonucu onlar\u0131n g\u00f6rd\u00fc\u011f\u00fcne daha yak\u0131nd\u0131r.<\/p>\n<h3>3. Eri\u015fim g\u00fcnl\u00fcklerinizde AI taray\u0131c\u0131lar\u0131n\u0131 bulun<\/h3>\n<p>Bot ba\u015f\u0131na istekleri say\u0131n, ard\u0131ndan her botun hangi durum kodlar\u0131n\u0131 ald\u0131\u011f\u0131n\u0131 kontrol edin. Bu komutlar\u0131 20 Kas\u0131m 2026&#8217;da bir Nginx test sunucusunun eri\u015fim g\u00fcnl\u00fc\u011f\u00fcnde \u00e7al\u0131\u015ft\u0131rd\u0131k:<\/p>\n<pre class=\"wp-block-code\"><code>$ grep -o -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|meta-external[a-z]+' access.log | sort | uniq -c | sort -rn\n      2 OAI-SearchBot\n      2 GPTBot\n$ awk -F'\"' '\/GPTBot\/ {split($3,a,\" \"); print a[1]}' access.log | sort | uniq -c\n      1 200\n      1 403<\/code><\/pre>\n<p>\u0130kinci komut, durum kodunun t\u0131rnak i\u00e7indeki istek sat\u0131r\u0131ndan sonra geldi\u011fi standart &#8220;combined&#8221; g\u00fcnl\u00fck bi\u00e7imiyle \u00e7al\u0131\u015f\u0131r. E\u011fitim taray\u0131c\u0131lar\u0131n\u0131 sunucuda engellerken istedi\u011finiz tam olarak \u015fudur: GPTBot i\u00e7in <code>\/robots.txt<\/code>&#8216;de 200, ba\u015fka her yerde 403.<\/p>\n<h3>4. Bir &#8220;GPTBot&#8221; ziyaret\u00e7isinin ger\u00e7ekten OpenAI&#8217;dan geldi\u011fini do\u011frulay\u0131n<\/h3>\n<p>Herkes GPTBot user agent&#8217;\u0131 g\u00f6nderebilir. OpenAI, Perplexity, DuckDuckGo, Mistral ve Google IP listeleri yay\u0131ml\u0131yor, bu y\u00fczden kaynak IP&#8217;yi kontrol edin. Bu betik bir IP&#8217;yi OpenAI&#8217;\u0131n \u00fc\u00e7 dosyas\u0131yla kar\u015f\u0131la\u015ft\u0131r\u0131r:<\/p>\n<pre class=\"wp-block-code\"><code># verify_ip.py - usage: python3 verify_ip.py 132.196.86.1\nimport sys, json, ipaddress, urllib.request\n\nip = ipaddress.ip_address(sys.argv[1])\nfor name in [\"gptbot\", \"searchbot\", \"chatgpt-user\"]:\n    data = json.load(urllib.request.urlopen(f\"https:\/\/openai.com\/{name}.json\", timeout=20))\n    nets = [ipaddress.ip_network(p.get(\"ipv4Prefix\") or p.get(\"ipv6Prefix\")) for p in data[\"prefixes\"]]\n    hit = [str(n) for n in nets if ip.version == n.version and ip in n]\n    print(f\"{name:13} {len(nets):3} ranges  {'MATCH ' + hit[0] if hit else 'no match'}\")<\/code><\/pre>\n<p>20 Kas\u0131m 2026, 08:12 AEDT&#8217;deki \u00e7\u0131kt\u0131:<\/p>\n<pre class=\"wp-block-code\"><code>$ python3 verify_ip.py 132.196.86.1\ngptbot         18 ranges  MATCH 132.196.86.0\/24\nsearchbot      39 ranges  no match\nchatgpt-user  237 ranges  no match\n$ python3 verify_ip.py 1.1.1.1\ngptbot         18 ranges  no match\nsearchbot      39 ranges  no match\nchatgpt-user  237 ranges  no match<\/code><\/pre>\n<p>Testimiz s\u0131ras\u0131nda OpenAI&#8217;\u0131n <code>gptbot.json<\/code> dosyas\u0131, olu\u015fturulma tarihi 22 Eyl\u00fcl 2026 olan 18 IPv4 aral\u0131\u011f\u0131 i\u00e7eriyordu. Dosyalar\u0131 koda sabitlemek yerine her seferinde indirin.<\/p>\n<h3>5. Sitenizi kay\u0131tta aray\u0131n<\/h3>\n<p>Online Web Directory, sitelerin basit HTTP istemcilerine nas\u0131l yan\u0131t verdi\u011fini ve robots.txt&#8217;lerinin ne dedi\u011fini kontrol eder. Alan ad\u0131n\u0131z\u0131 aray\u0131n, \u00f6rne\u011fin <a href=\"https:\/\/onlinewebdirectory.net\/sites\/reddit.com\">reddit.com<\/a> veya <a href=\"https:\/\/onlinewebdirectory.net\/sites\/nytimes.com\">nytimes.com<\/a>, ya da \u00fccretsiz <a href=\"https:\/\/onlinewebdirectory.net\/tr\/check\">haz\u0131rl\u0131k kontrol\u00fcn\u00fc<\/a> \u00e7al\u0131\u015ft\u0131r\u0131n. <a href=\"https:\/\/onlinewebdirectory.net\/lists\/blocks-ai-agents\">AI ajanlar\u0131n\u0131 engelleyen siteler<\/a> listesi, hangi alan adlar\u0131n\u0131n basit istemcileri reddetti\u011fini g\u00f6sterir.<\/p>\n<figure class=\"wp-block-image size-full\"><img class=\"wp-image-574\" srcset=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/owd-blocks-list-v2-768w.webp 768w, https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/owd-blocks-list-v2.webp 1200w\" sizes=\"auto, (max-width: 848px) calc(100vw - 48px), 800px\" src=\"https:\/\/onlinewebdirectory.net\/blog\/wp-content\/uploads\/2026\/11\/owd-blocks-list-v2.webp\" alt=\"Online Web Directory'nin AI ajanlar\u0131n\u0131 engelleyen siteler listesi: kontrol edilen 55.144 siteden 725'i basit HTTP istemcilerini engelliyor\" width=\"1200\" height=\"960\" loading=\"lazy\" decoding=\"async\" \/><figcaption>Online Web Directory&#8217;nin AI ajanlar\u0131n\u0131 engelleyen siteler listesi, 20 Kas\u0131m 2026.<\/figcaption><\/figure>\n<h2 id=\"how-to-control\">AI taray\u0131c\u0131lar\u0131n\u0131 kontrol etmek: alt\u0131 y\u00f6ntem<\/h2>\n<p>Hedefinize g\u00f6re se\u00e7in. \u00c7o\u011fu site AI arama trafi\u011fini korumak, e\u011fitim konusunda duruma g\u00f6re karar vermek ve kullan\u0131c\u0131 taraf\u0131ndan tetiklenen ajanlara dokunmamak ister.<\/p>\n<h3>Y\u00f6ntem 1: e\u011fitim taray\u0131c\u0131lar\u0131n\u0131 engelleyin, AI aramas\u0131n\u0131 koruyun (robots.txt)<\/h3>\n<p>Bu, \u00f6rneklemimizdeki siteler aras\u0131nda en yayg\u0131n politika. Bunu sundu\u011funuz her ana makinenin k\u00f6k\u00fcndeki <code>\/robots.txt<\/code> dosyas\u0131na ekleyin:<\/p>\n<pre class=\"wp-block-code\"><code># Block AI training crawlers\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nUser-agent: Applebot-Extended\nUser-agent: meta-externalagent\nUser-agent: Amazonbot\nUser-agent: MistralAI-Training\nUser-agent: Bytespider\nDisallow: \/\n\n# Allow AI search crawlers (so you can be cited in AI answers)\nUser-agent: OAI-SearchBot\nUser-agent: Claude-SearchBot\nUser-agent: PerplexityBot\nUser-agent: Amzn-SearchBot\nUser-agent: DuckAssistBot\nUser-agent: MistralAI-Index\nAllow: \/\n\n# Everyone else\nUser-agent: *\nDisallow: \/wp-admin\/<\/code><\/pre>\n<p>Bu dosyay\u0131 RFC 9309 uyumlu bir ayr\u0131\u015ft\u0131r\u0131c\u0131 olan Protego ile test ettik. <code>\/blog\/<\/code> yolu i\u00e7in sonu\u00e7lar:<\/p>\n<pre class=\"wp-block-code\"><code>GPTBot             \/blog\/  blocked\nClaudeBot          \/blog\/  blocked\nCCBot              \/blog\/  blocked\nGoogle-Extended    \/blog\/  blocked\nApplebot-Extended  \/blog\/  blocked\nOAI-SearchBot      \/blog\/  allowed\nClaude-SearchBot   \/blog\/  allowed\nPerplexityBot      \/blog\/  allowed\nGooglebot          \/blog\/  allowed\nBingbot            \/blog\/  allowed<\/code><\/pre>\n<p>Tek bir kural blo\u011funun \u00fcst\u00fcne birden \u00e7ok <code>User-agent<\/code> sat\u0131r\u0131 y\u0131\u011fmak RFC 9309&#8217;a g\u00f6re ge\u00e7erlidir. AI arama grubunun <code>*<\/code> grubundaki <code>\/wp-admin\/<\/code> kural\u0131n\u0131 devralmad\u0131\u011f\u0131na dikkat edin; gerekiyorsa oraya da ekleyin.<\/p>\n<h3>Y\u00f6ntem 2: t\u00fcm belgelenmi\u015f AI taray\u0131c\u0131lar\u0131n\u0131 engelleyin (robots.txt)<\/h3>\n<p>Hi\u00e7bir AI kullan\u0131m\u0131 istemiyorsan\u0131z arama ve kullan\u0131c\u0131 token&#8217;lar\u0131n\u0131 engellenen gruba ekleyin: <code>OAI-SearchBot<\/code>, <code>ChatGPT-User<\/code>, <code>Claude-SearchBot<\/code>, <code>Claude-User<\/code>, <code>PerplexityBot<\/code>, <code>Perplexity-User<\/code>, <code>meta-externalfetcher<\/code>, <code>Amzn-SearchBot<\/code>, <code>Amzn-User<\/code>, <code>MistralAI-Index<\/code>, <code>MistralAI-User<\/code> ve <code>DuckAssistBot<\/code>. AI yan\u0131tlar\u0131ndan kaybolmay\u0131 bekleyin. Ve birka\u00e7 kullan\u0131c\u0131 tetiklemeli getiricinin robots.txt&#8217;nin kendileri i\u00e7in ge\u00e7erli olmayabilece\u011fini s\u00f6yledi\u011fini unutmay\u0131n: bu bir duvar de\u011fil, belirtilmi\u015f bir tercihtir.<\/p>\n<h3>Y\u00f6ntem 3: engeli Nginx&#8217;te zorunlu k\u0131l\u0131n<\/h3>\n<p>\u0130stekleri ger\u00e7ekten reddetmek i\u00e7in sunucuda filtreleyin. Bu yap\u0131land\u0131rma, e\u011fitim taray\u0131c\u0131lar\u0131na robots.txt d\u0131\u015f\u0131ndaki her yolda 403 d\u00f6nd\u00fcr\u00fcr; b\u00f6ylece kurallar\u0131n\u0131z\u0131 h\u00e2l\u00e2 okuyabilirler. <code>map<\/code> blo\u011fu <code>http<\/code> ba\u011flam\u0131na yerle\u015ftirilir:<\/p>\n<pre class=\"wp-block-code\"><code># Block AI training crawlers by user agent, but still let them read robots.txt\nmap $http_user_agent $block_ai_crawler {\n    default 0;\n    \"~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent|Amazonbot|MistralAI-Training)\" 1;\n}\nserver {\n    listen 80;\n    root \/var\/www\/html;\n    location = \/robots.txt { }\n    location \/ {\n        if ($block_ai_crawler) { return 403; }\n    }\n}<\/code><\/pre>\n<p>20 Kas\u0131m 2026&#8217;da Nginx 1.26.3 ile test edildi: <code>nginx -t<\/code> ba\u015far\u0131l\u0131 oldu, GPTBot <code>\/<\/code> i\u00e7in 403, <code>\/robots.txt<\/code> i\u00e7in 200 ald\u0131; Chrome ve OAI-SearchBot ise ikisinde de 200 ald\u0131. Bir <code>if<\/code> i\u00e7indeki basit bir <code>return<\/code>, Nginx&#8217;in g\u00fcvenli olarak belgeledi\u011fi kullan\u0131mlardand\u0131r.<\/p>\n<h3>Y\u00f6ntem 4: engeli Apache&#8217;de zorunlu k\u0131l\u0131n (.htaccess)<\/h3>\n<p>Ayn\u0131 kural, <code>mod_rewrite<\/code> ile Apache i\u00e7in:<\/p>\n<pre class=\"wp-block-code\"><code># Block AI training crawlers by user agent, but still let them read robots.txt\n&lt;IfModule mod_rewrite.c&gt;\nRewriteEngine On\nRewriteCond %{REQUEST_URI} !^\/robots\\.txt$\nRewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent|Amazonbot|MistralAI-Training) [NC]\nRewriteRule ^ - [F,L]\n&lt;\/IfModule&gt;<\/code><\/pre>\n<p><code>AllowOverride All<\/code> ile Apache 2.4.71&#8217;de test edildi: GPTBot ana sayfada 403, robots.txt&#8217;de 200 ald\u0131; Chrome ve OAI-SearchBot 200 ald\u0131. <code>[F]<\/code> bayra\u011f\u0131 403 Forbidden d\u00f6nd\u00fcr\u00fcr.<\/p>\n<h3>Y\u00f6ntem 5: CDN&#8217;nizin AI taray\u0131c\u0131 kurallar\u0131n\u0131 kullan\u0131n<\/h3>\n<p>Cloudflare&#8217;de bir liste tutman\u0131za gerek yok. Security, Settings b\u00f6l\u00fcm\u00fcnde &#8220;Bot traffic&#8221; ile filtreleyin ve <strong>AI Crawler and Agent access policies<\/strong>&#8216;i a\u00e7\u0131n. Cloudflare, Search, Agent ve Training trafi\u011fi i\u00e7in ayr\u0131 eylemler belirlemenize izin verir. Belgelerine g\u00f6re yeni alan adlar\u0131 \u00fc\u00e7\u00fc i\u00e7in de varsay\u0131lan olarak &#8220;Allow (do not block)&#8221; ayar\u0131ndad\u0131r ve Training i\u00e7in &#8220;Disallow&#8221;, Bot Preference Sync&#8217;i a\u00e7ana kadar yaln\u0131zca robots.txt&#8217;yi de\u011fi\u015ftirir. Cloudflare&#8217;in AI Crawl Control panosu, hangi taray\u0131c\u0131lar\u0131n geldi\u011fini ve robots.txt&#8217;nize uyup uymad\u0131klar\u0131n\u0131 da g\u00f6sterir. Kayd\u0131m\u0131zdaki 62.088 sitenin 6.966&#8217;s\u0131 (%11,2) denetleyicimize bir Cloudflare do\u011frulamas\u0131 g\u00f6sterdi.<\/p>\n<h3>Y\u00f6ntem 6: \u00f6nce do\u011frulay\u0131n, sonra izin verin<\/h3>\n<p>User agent kurallar\u0131 d\u00fcr\u00fcst botlar\u0131 engeller, d\u00fcr\u00fcst olmayanlar\u0131 ge\u00e7irir. U\u00e7ta engelliyorsan\u0131z kural\u0131 IP do\u011frulamas\u0131yla e\u015fle\u015ftirin (4. ad\u0131ma bak\u0131n) ve do\u011frulanm\u0131\u015f arama ve kullan\u0131c\u0131 ajanlar\u0131n\u0131 ge\u00e7irin. Anthropic, IP adreslerini engellemenin ClaudeBot&#8217;un robots.txt dosyan\u0131z\u0131 okumas\u0131n\u0131 engelledi\u011fi i\u00e7in g\u00fcvenilir bir \u00e7\u0131k\u0131\u015f y\u00f6ntemi olmayabilece\u011fi konusunda uyar\u0131yor. Politika i\u00e7in robots.txt&#8217;yi, taklit\u00e7ileri yakalamak i\u00e7in IP kontrollerini kullan\u0131n.<\/p>\n<h2 id=\"owd-data\">2.124 robots.txt dosyas\u0131 AI taray\u0131c\u0131lar\u0131 hakk\u0131nda ne s\u00f6yl\u00fcyor<\/h2>\n<p>20 Kas\u0131m 2026&#8217;da 08:01 ile 08:03 AEDT aras\u0131nda Online Web Directory&#8217;nin <a href=\"https:\/\/onlinewebdirectory.net\/exports\/sites.csv\">a\u00e7\u0131k veri k\u00fcmesindeki<\/a> 62.088 siteden rastgele 3.000 ana makine se\u00e7tik ve sunucumuzdan <code>\/robots.txt<\/code> istedik. 2.124&#8217;\u00fc ayr\u0131\u015ft\u0131r\u0131labilir bir dosya d\u00f6nd\u00fcrd\u00fc. Bir token kendi <code>User-agent<\/code> sat\u0131r\u0131na sahipse &#8220;an\u0131lm\u0131\u015f&#8221;, o grup <code>Allow: \/<\/code> olmadan <code>Disallow: \/<\/code> i\u00e7eriyorsa &#8220;tamamen engellenmi\u015f&#8221; say\u0131l\u0131r.<\/p>\n<figure class=\"wp-block-table owd-table\">\n<table>\n<thead>\n<tr>\n<th>robots.txt token&#8217;\u0131<\/th>\n<th>An\u0131lan<\/th>\n<th>Tamamen engellenen<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>GPTBot<\/code><\/td>\n<td>23.6%<\/td>\n<td>16.9%<\/td>\n<\/tr>\n<tr>\n<td><code>ClaudeBot<\/code><\/td>\n<td>22.0%<\/td>\n<td>15.9%<\/td>\n<\/tr>\n<tr>\n<td><code>CCBot<\/code><\/td>\n<td>20.6%<\/td>\n<td>17.3%<\/td>\n<\/tr>\n<tr>\n<td><code>Google-Extended<\/code><\/td>\n<td>19.4%<\/td>\n<td>14.1%<\/td>\n<\/tr>\n<tr>\n<td><code>Bytespider<\/code><\/td>\n<td>18.5%<\/td>\n<td>16.6%<\/td>\n<\/tr>\n<tr>\n<td><code>Amazonbot<\/code><\/td>\n<td>17.4%<\/td>\n<td>14.3%<\/td>\n<\/tr>\n<tr>\n<td><code>meta-externalagent<\/code><\/td>\n<td>16.7%<\/td>\n<td>14.0%<\/td>\n<\/tr>\n<tr>\n<td><code>Applebot-Extended<\/code><\/td>\n<td>16.6%<\/td>\n<td>13.5%<\/td>\n<\/tr>\n<tr>\n<td><code>anthropic-ai<\/code> (eski)<\/td>\n<td>15.6%<\/td>\n<td>13.3%<\/td>\n<\/tr>\n<tr>\n<td><code>Claude-Web<\/code> (eski)<\/td>\n<td>13.8%<\/td>\n<td>12.2%<\/td>\n<\/tr>\n<tr>\n<td><code>PerplexityBot<\/code><\/td>\n<td>12.4%<\/td>\n<td>5.8%<\/td>\n<\/tr>\n<tr>\n<td><code>ChatGPT-User<\/code><\/td>\n<td>12.2%<\/td>\n<td>5.9%<\/td>\n<\/tr>\n<tr>\n<td><code>OAI-SearchBot<\/code><\/td>\n<td>9.3%<\/td>\n<td>2.8%<\/td>\n<\/tr>\n<tr>\n<td><code>Claude-User<\/code><\/td>\n<td>6.5%<\/td>\n<td>3.2%<\/td>\n<\/tr>\n<tr>\n<td><code>Perplexity-User<\/code><\/td>\n<td>6.4%<\/td>\n<td>3.2%<\/td>\n<\/tr>\n<tr>\n<td><code>Claude-SearchBot<\/code><\/td>\n<td>6.2%<\/td>\n<td>2.8%<\/td>\n<\/tr>\n<tr>\n<td><code>DuckAssistBot<\/code><\/td>\n<td>4.2%<\/td>\n<td>2.7%<\/td>\n<\/tr>\n<tr>\n<td><code>MistralAI-User<\/code><\/td>\n<td>3.7%<\/td>\n<td>2.6%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>D\u00f6rt \u015fey \u00f6ne \u00e7\u0131k\u0131yor:<\/p>\n<ul>\n<li><strong>\u00c7o\u011fu site AI taray\u0131c\u0131lar\u0131ndan hi\u00e7 s\u00f6z etmiyor.<\/strong> 2.124 dosyadan 629&#8217;u (%29,6) en az bir AI&#8217;ye \u00f6zg\u00fc token an\u0131yor. Anan dosyalar \u00e7o\u011fu zaman pek \u00e7o\u011funu an\u0131yor: medyan 12 token ve 357 dosya 10 veya daha fazlas\u0131n\u0131 an\u0131yor; bu, kopyalanm\u0131\u015f engelleme listelerinin i\u015fareti.<\/li>\n<li><strong>E\u011fitim engelleniyor, arama engellenmiyor.<\/strong> GPTBot, OAI-SearchBot&#8217;tan alt\u0131 kat daha s\u0131k tamamen engelleniyor (%16,9&#8217;a kar\u015f\u0131 %2,8).<\/li>\n<li><strong>robots.txt ile ger\u00e7ek eri\u015fim \u00f6rt\u00fc\u015fm\u00fcyor.<\/strong> robots.txt&#8217;de GPTBot&#8217;u tamamen engelleyen 358 siteden 284&#8217;\u00fc (%79) OWD eri\u015fim kontrol\u00fcnde &#8220;a\u00e7\u0131k&#8221;t\u0131: sunucuda basit bir istemciyi durduran hi\u00e7bir \u015fey yoktu.<\/li>\n<li><strong>Tam yasaklar nadir.<\/strong> Yaln\u0131zca 13 dosya (%0,6) t\u00fcm user agent&#8217;lara her \u015feyi yasakl\u0131yor.<\/li>\n<\/ul>\n<p>Tam kay\u0131t ayn\u0131 hik\u00e2yeyi di\u011fer taraftan anlat\u0131yor. \u0130\u015fte en son 11 Ekim ile 19 Kas\u0131m 2026 aras\u0131nda kontrol edilen 62.088 sitenin t\u00fcm\u00fcn\u00fcn eri\u015fim sonu\u00e7lar\u0131:<\/p>\n<figure class=\"wp-block-table owd-table\">\n<table>\n<thead>\n<tr>\n<th>Basit bir HTTP istemcisinin kar\u015f\u0131la\u015ft\u0131\u011f\u0131<\/th>\n<th>Siteler<\/th>\n<th>Pay<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>A\u00e7\u0131k<\/td>\n<td>50,844<\/td>\n<td>81.9%<\/td>\n<\/tr>\n<tr>\n<td>Bot do\u011frulamas\u0131<\/td>\n<td>7,821<\/td>\n<td>12.6%<\/td>\n<\/tr>\n<tr>\n<td>Engelli<\/td>\n<td>3,060<\/td>\n<td>4.9%<\/td>\n<\/tr>\n<tr>\n<td>H\u0131z s\u0131n\u0131rl\u0131<\/td>\n<td>199<\/td>\n<td>0.3%<\/td>\n<\/tr>\n<tr>\n<td>Giri\u015f gerekli<\/td>\n<td>150<\/td>\n<td>0.2%<\/td>\n<\/tr>\n<tr>\n<td>\u00d6deme duvar\u0131<\/td>\n<td>14<\/td>\n<td>0.0%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>Tam veri k\u00fcmesini (CC BY 4.0) <a href=\"https:\/\/onlinewebdirectory.net\/exports\/sites.csv\">onlinewebdirectory.net\/exports\/sites.csv<\/a> adresinden indirin veya <a href=\"https:\/\/onlinewebdirectory.net\/reports\/ai-agent-access\">AI ajan eri\u015fimi raporuna<\/a> bak\u0131n.<\/p>\n<h2 id=\"related-errors\">\u0130lgili hatalar ve yan\u0131tlar<\/h2>\n<ul>\n<li><strong>403 Forbidden.<\/strong> Sunucunuz, CDN&#8217;niz veya g\u00fcvenlik duvar\u0131n\u0131z iste\u011fi reddetti. Bunun nedeni asla robots.txt&#8217;nin kendisi de\u011fildir ve kal\u0131c\u0131 da de\u011fildir: kural\u0131 de\u011fi\u015ftirin, bir sonraki istek ba\u015far\u0131l\u0131 olabilir.<\/li>\n<li><strong>429 Too Many Requests.<\/strong> H\u0131z s\u0131n\u0131rlamas\u0131. Reddit bunu ClaudeBot iste\u011fimize d\u00f6nd\u00fcrd\u00fc. Anthropic, ClaudeBot i\u00e7in daha yumu\u015fak bir se\u00e7enek olan <code>Crawl-delay<\/code>&#8216;i destekliyor.<\/li>\n<li><strong>Bo\u015f g\u00f6vdeli 202 veya 503 do\u011frulama sayfas\u0131.<\/strong> Genellikle bir JavaScript veya CAPTCHA do\u011frulamas\u0131. Taray\u0131c\u0131lar bunu \u00e7\u00f6zemez, bu y\u00fczden 403 olmasa bile sayfa onlar i\u00e7in engellidir.<\/li>\n<li><strong>robots.txt 4xx d\u00f6nd\u00fcr\u00fcyor.<\/strong> RFC 9309&#8217;a g\u00f6re 400 ile 499 aras\u0131 bir yan\u0131t dosyan\u0131n &#8220;kullan\u0131lamaz&#8221; oldu\u011fu ve taray\u0131c\u0131lar\u0131n her \u015feyi tarayabilece\u011fi anlam\u0131na gelir. \u00d6rneklemimizde 3.000 siteden 314&#8217;\u00fc (%10,5) robots.txt iste\u011fimize 403 ile yan\u0131t verdi; standarda uyan bir taray\u0131c\u0131 bunu kural yok diye okur.<\/li>\n<li><strong>robots.txt 5xx d\u00f6nd\u00fcr\u00fcyor veya zaman a\u015f\u0131m\u0131na u\u011fruyor.<\/strong> RFC 9309 bu durumda taray\u0131c\u0131lar\u0131n tam yasak varsaymas\u0131n\u0131 ister; yani bir robots.txt kesintisi Googlebot dahil t\u00fcm taray\u0131c\u0131lar\u0131 sessizce engelleyebilir.<\/li>\n<li><strong>500 KiB&#8217;den b\u00fcy\u00fck robots.txt.<\/strong> RFC 9309 taray\u0131c\u0131lar\u0131n yaln\u0131zca ilk 500 KiB&#8217;yi ayr\u0131\u015ft\u0131rmas\u0131n\u0131 \u015fart ko\u015far; \u00f6tesindeki kurallar yok say\u0131labilir.<\/li>\n<\/ul>\n<p>AI ajanlar\u0131 i\u00e7in bir llms.txt dosyas\u0131 yay\u0131ml\u0131yorsan\u0131z izin verilen botlar\u0131n onu ger\u00e7ekten alabildi\u011finden emin olun. <a href=\"\/blog\/tr\/llms-txt-tr\/basics-tr\/llms-txt-nedir\/\">llms.txt rehberimiz<\/a> bunun nas\u0131l test edilece\u011fini anlat\u0131yor.<\/p>\n<h2 id=\"summary\">\u00d6zet<\/h2>\n<p>G\u00fcvenilir bir AI taray\u0131c\u0131 listesi ba\u015fka listelerden de\u011fil, operat\u00f6r belgelerinden ba\u015flar. Bug\u00fcn bu, 9 \u015firketten yakla\u015f\u0131k 20 user agent anlam\u0131na geliyor ve \u00fc\u00e7 t\u00fcre ayr\u0131l\u0131yor: e\u011fitim taray\u0131c\u0131lar\u0131, AI arama taray\u0131c\u0131lar\u0131 ve kullan\u0131c\u0131 taraf\u0131ndan tetiklenen ajanlar; buna hi\u00e7bir \u015fey taramayan iki kontrol token&#8217;\u0131 (Google-Extended ve Applebot-Extended) ekleniyor. Harekete ge\u00e7en sitelerin \u00e7o\u011fu e\u011fitimi engelleyip aramay\u0131 koruyor ve robots.txt yaln\u0131zca bir rica: ger\u00e7ek bir engele ihtiyac\u0131n\u0131z varsa bunu sunucuda veya CDN&#8217;de zorunlu k\u0131l\u0131n ve IP&#8217;leri do\u011frulay\u0131n.<\/p>\n<p>AI ajanlar\u0131 siteleri ziyaret etmeden \u00f6nce Online Web Directory&#8217;de ar\u0131yor. Sitenizin kayd\u0131 yanl\u0131\u015fsa veya ajanlar\u0131n API&#8217;nizi, MCP sunucunuzu ya da AI izinlerinizi bilmesini istiyorsan\u0131z <a href=\"https:\/\/onlinewebdirectory.net\/tr\/claim\">sitenizi 25 $ kar\u015f\u0131l\u0131\u011f\u0131nda sahiplenin<\/a>. \u00d6nce \u00fccretsiz <a href=\"https:\/\/onlinewebdirectory.net\/tr\/check\">haz\u0131rl\u0131k kontrol\u00fcn\u00fc<\/a> \u00e7al\u0131\u015ft\u0131rabilirsiniz.<\/p>\n<h2 id=\"faq\">S\u0131k sorulan sorular<\/h2>\n<h3>ChatGPT&#8217;nin user agent&#8217;\u0131 nedir?<\/h3>\n<p>ChatGPT \u00fc\u00e7 tane kullan\u0131r. GPTBot e\u011fitim verisi toplar, OAI-SearchBot ChatGPT aramas\u0131 i\u00e7in sayfalar\u0131 dizine ekler ve ChatGPT-User bir kullan\u0131c\u0131 istedi\u011finde sayfay\u0131 getirir. robots.txt token&#8217;lar\u0131 <code>GPTBot<\/code>, <code>OAI-SearchBot<\/code> ve <code>ChatGPT-User<\/code>&#8216;d\u0131r.<\/p>\n<h3>GPTBot&#8217;u engellemek sitemi ChatGPT aramas\u0131ndan \u00e7\u0131kar\u0131r m\u0131?<\/h3>\n<p>Hay\u0131r. OpenAI aramay\u0131 OAI-SearchBot \u00fczerinden y\u00fcr\u00fct\u00fcr. E\u011fitimi reddetmek i\u00e7in GPTBot&#8217;u engelleyebilir, sayfalar\u0131n\u0131z\u0131n ChatGPT aramas\u0131nda g\u00f6r\u00fcnmesi ve al\u0131nt\u0131lanmas\u0131 i\u00e7in OAI-SearchBot&#8217;a izin verebilirsiniz.<\/p>\n<h3>Google-Extended bir taray\u0131c\u0131 m\u0131?<\/h3>\n<p>Hay\u0131r. Google, Google-Extended&#8217;in ayr\u0131 bir user agent&#8217;\u0131 olmad\u0131\u011f\u0131n\u0131 s\u00f6yl\u00fcyor. Taramay\u0131 Googlebot yapar; Google-Extended token&#8217;\u0131 yaln\u0131zca bu i\u00e7eri\u011fin Gemini&#8217;yi e\u011fitmek ve yan\u0131tlar\u0131na dayanak olmak i\u00e7in kullan\u0131l\u0131p kullan\u0131lamayaca\u011f\u0131n\u0131 belirler. Google Arama s\u0131ralamas\u0131n\u0131 etkilemez.<\/p>\n<h3>AI taray\u0131c\u0131lar\u0131 robots.txt&#8217;ye uyar m\u0131?<\/h3>\n<p>Bu listedeki belgelenmi\u015f e\u011fitim ve arama taray\u0131c\u0131lar\u0131 uyduklar\u0131n\u0131 s\u00f6yl\u00fcyor. ChatGPT-User, Perplexity-User ve Meta-ExternalFetcher gibi kullan\u0131c\u0131 taraf\u0131ndan tetiklenen getiriciler, sayfay\u0131 bir ki\u015fi istedi\u011fi i\u00e7in robots.txt&#8217;nin ge\u00e7erli olmayabilece\u011fini s\u00f6yl\u00fcyor. Belgelenmemi\u015f kaz\u0131y\u0131c\u0131lar onu tamamen yok sayabilir.<\/p>\n<h3>Bir botun ger\u00e7ekten GPTBot veya ClaudeBot oldu\u011funu nas\u0131l anlar\u0131m?<\/h3>\n<p>Kaynak IP&#8217;yi kontrol edin. OpenAI aral\u0131klar\u0131n\u0131 <code>gptbot.json<\/code>, <code>searchbot.json<\/code> ve <code>chatgpt-user.json<\/code> dosyalar\u0131nda yay\u0131mlar; Perplexity, DuckDuckGo, Mistral ve Google da kendilerininkini yay\u0131mlar. Listede olmayan bir IP&#8217;den gelen e\u015fle\u015fen user agent bir taklit\u00e7idir.<\/p>\n<h2 id=\"sources\">Kaynaklar<\/h2>\n<ol>\n<li>OpenAI, <a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">Overview of OpenAI crawlers<\/a> (eri\u015fim: 20 Kas\u0131m 2026)<\/li>\n<li>Anthropic, <a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Does Anthropic crawl data from the web, and how can site owners block the crawler?<\/a><\/li>\n<li>Google, <a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\" target=\"_blank\" rel=\"noopener\">Google&#8217;s common crawlers<\/a> ve <a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-user-triggered-fetchers\" target=\"_blank\" rel=\"noopener\">user-triggered fetchers<\/a><\/li>\n<li>Perplexity, <a href=\"https:\/\/docs.perplexity.ai\/docs\/resources\/perplexity-crawlers\" target=\"_blank\" rel=\"noopener\">Perplexity crawlers<\/a><\/li>\n<li>Apple, <a href=\"https:\/\/support.apple.com\/en-us\/119829\" target=\"_blank\" rel=\"noopener\">About Applebot<\/a><\/li>\n<li>Meta, <a href=\"https:\/\/developers.facebook.com\/docs\/sharing\/webmasters\/web-crawlers\/\" target=\"_blank\" rel=\"noopener\">Meta web crawlers<\/a><\/li>\n<li>Amazon, <a href=\"https:\/\/developer.amazon.com\/docs\/amazonbot\/amazonbot.html\" target=\"_blank\" rel=\"noopener\">Amazonbot and Amazon crawlers<\/a><\/li>\n<li>Common Crawl, <a href=\"https:\/\/commoncrawl.org\/ccbot\" target=\"_blank\" rel=\"noopener\">CCBot<\/a><\/li>\n<li>Mistral AI, <a href=\"https:\/\/docs.mistral.ai\/robots\" target=\"_blank\" rel=\"noopener\">Mistral AI robots<\/a><\/li>\n<li>DuckDuckGo, <a href=\"https:\/\/duckduckgo.com\/duckduckgo-help-pages\/results\/duckassistbot\" target=\"_blank\" rel=\"noopener\">About DuckAssistBot<\/a><\/li>\n<li>IETF, <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.html\" target=\"_blank\" rel=\"noopener\">RFC 9309: Robots Exclusion Protocol<\/a><\/li>\n<li>Cloudflare, <a href=\"https:\/\/developers.cloudflare.com\/bots\/additional-configurations\/block-ai-bots\/\" target=\"_blank\" rel=\"noopener\">AI Crawler and Agent access policies<\/a> ve <a href=\"https:\/\/developers.cloudflare.com\/ai-crawl-control\/\" target=\"_blank\" rel=\"noopener\">AI Crawl Control<\/a><\/li>\n<li>Online Web Directory, <a href=\"https:\/\/onlinewebdirectory.net\/exports\/sites.csv\">a\u00e7\u0131k site veri k\u00fcmesi<\/a> (62.088 site, 20 Kas\u0131m 2026&#8217;da d\u0131\u015fa aktar\u0131ld\u0131) ve 3.000 ana makinelik robots.txt \u00f6rneklemimiz<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>2026&#8217;da do\u011frulanm\u0131\u015f yapay zek\u00e2 taray\u0131c\u0131lar\u0131 ve user agent listesi (GPTBot, ClaudeBot, PerplexityBot ve dahas\u0131): her birinin i\u015fi, test edilmi\u015f engelleme kurallar\u0131 ve 2.124 sitenin robots.txt verisi.<\/p>\n","protected":false},"author":2,"featured_media":557,"comment_status":"closed","ping_status":"open","sticky":false,"template":"owd-single-tr","format":"standard","meta":{"footnotes":""},"categories":[1288],"tags":[],"class_list":["post-602","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-crawler-list-tr"],"lang":"tr","translations":{"tr":602,"en":567,"de":580,"es":582,"pt":584,"fr":586,"it":588,"nl":590,"sv":592,"pl":594,"cs":596,"ru":598,"uk":600,"ja":604,"ar":606,"zh":608,"ko":610,"hi":612,"id":614,"vi":616,"th":618},"_links":{"self":[{"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/posts\/602","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/comments?post=602"}],"version-history":[{"count":1,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/posts\/602\/revisions"}],"predecessor-version":[{"id":603,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/posts\/602\/revisions\/603"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/media\/557"}],"wp:attachment":[{"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/media?parent=602"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/categories?post=602"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onlinewebdirectory.net\/blog\/wp-json\/wp\/v2\/tags?post=602"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}