Matthew Widdop
Matty is a Digital SEO Executive at Intelligency, helping our clients to improve their digital performance. Matty is currently studying for his Level 3 Multi-Channel Marketer apprenticeship and has completed a Sport Journalism Degree at the University of Huddersfield.
With the advances in AI search over the last few years, being cited in AI chatbots is now more paramount than ever to a business’s presence in this ever-evolving digital age. Robots.txt is a plain text file generated by a website that lets search engine crawlers know which of your content it can and can’t index for users to view. LLMs.txt is the same concept but for chatbots, pointing Chatbots to your content to allow them to know which of your content it can rank. Or at least this is how people in the industry are currently using it. In this article, we’ll explain all about LLMs.txt and if you really need to implement it on your site.
What is LLMs.txt
As we’ve mentioned, LLMs.txt is a text file that lives on your web server and is being used to try and make content more discoverable for AI chatbots. However, this isn’t really why it was designed. Unlike a traditional Robots.txt file, according to Google’s John Mueller, LLMs.txt doesn’t actually help you to rank in search engines at all, but was rather designed to help search engines who already know about your site to find out what else is here,
“So I talked with, I think, one of the people who created that proposal a while back. And the idea was really not to create something that makes it easier for search engines or LLM systems to discover all of your content, but almost more that if an LLM already knows about your site and wants to find out what else is here, then that might be an approach.
And I think the aspect of using this as a way to optimise for Discovery by AI systems or Discovery by search systems, that doesn’t make any sense at all.“
Many users at the minute think that using LLMs.txt will help them rank more on search engines, and this simply just isn’t the case.
How can you help your website rank in AI?
While using an LLMs.txt file won’t help you rank in AI, there are still steps you can take to make sure your website is more likely to rank in LLMs, which is going to be important moving forward as the amount of people using them is going to grow. Some steps you can take to help improve AI performance include:
- Question-based queries with direct answers
- Implement structured data
- Build brand authority
- Allow crawlers in robots.txt file
This last one is crucial: while LLMs.txt is largely irrelevant for ranking, Robots.txt is not. If, for whatever reason, you don’t permit LLM crawlers in your robots.txt file, then they won’t be able to crawl your site. Make sure you explicitly state that they can in your robots.txt file.
What this means for Marketers
An LLMs.txt file isn’t going to help improve your rankings in Chatbots, so if this is what your aim is in creating one, you don’t need to. Creating clear, structured, trustworthy content that is crawlable is going to do you way more favours in terms of improving your rankings.
