Policies
Content & AI use policy
How search engines, AI assistants and automated tools may and may not use HiredPK content.
Last updated
Summary
- Search indexing: permitted.
- Quoting in AI answers with a link (retrieval, citation): permitted.
- Training AI models on the site: permitted.
- Crawling, including by SEO and research tools: permitted at a polite rate.
- Republishing our listings as your own, in bulk: not permitted. Link to the listing instead.
These terms are also published machine-readably in our robots.txt
(including a Content-Signal line) and in llms.txt.
Crawlers we welcome
- Googlebot (Google): Search, AI Overviews, AI Mode
- Bingbot (Microsoft): Bing search and Copilot answers
- DuckDuckBot (DuckDuckGo): Search
- Applebot (Apple): Siri and Spotlight search
- OAI-SearchBot (OpenAI): ChatGPT search index
- ChatGPT-User (OpenAI): Fetches a page a ChatGPT user asked about
- Claude-SearchBot (Anthropic): Claude search index
- Claude-User (Anthropic): Fetches a page a Claude user asked about
- PerplexityBot (Perplexity): Perplexity search index
- Perplexity-User (Perplexity): Fetches a page a Perplexity user asked about
- Google-Extended (Google): Gemini answers (grounding) and Gemini training
- GPTBot (OpenAI): Model training
- ClaudeBot (Anthropic): Model training
- anthropic-ai (Anthropic): Model training (legacy)
- Claude-Web (Anthropic): Legacy crawler
- CCBot (Common Crawl): Open dataset used for training
- Applebot-Extended (Apple): Apple Intelligence training
- Bytespider (ByteDance): Model training
- meta-externalagent (Meta): Model training
- Meta-ExternalFetcher (Meta): Meta AI fetches
- FacebookBot (Meta): Model training
- Amazonbot (Amazon): Model training and Alexa
- cohere-ai (Cohere): Model training
- cohere-training-data-crawler (Cohere): Model training
- AI2Bot (Allen Institute): Model training
- Diffbot (Diffbot): Knowledge graph / training data
- PanguBot (Huawei): Model training
- PetalBot (Huawei): Petal search
- DeepSeekBot (DeepSeek): Model training
- Timpibot (Timpi): Index / training data
- ImagesiftBot (ImageSift): Image dataset
- omgili (Webz.io): Data resale
- Webzio-Extended (Webz.io): Training data
- Kangaroo Bot (Kangaroo LLM): Model training
- img2dataset (Open source): Image dataset tool
- Scrapy (Open source): Scraping framework
- Baiduspider (Baidu): Chinese search
- Sogou web spider (Sogou): Chinese search
- 360Spider (Qihoo 360): Chinese search
- YisouSpider (Alibaba): Chinese search
- YandexBot (Yandex): Russian search
- AhrefsBot (Ahrefs): SEO data
- SemrushBot (Semrush): SEO data
- MJ12bot (Majestic): SEO data
- DotBot (Moz): SEO data
- BLEXBot (WebMeUp): SEO data
- DataForSeoBot (DataForSEO): SEO data
- serpstatbot (Serpstat): SEO data
- Barkrowler (Babbar): SEO data
Automated access
Any client that sends more than 60 requests in 10 seconds from one address is paused briefly; verified search and AI crawlers are exempt.
Data or partnership requests: legal@hiredpk.com