Matthew Widdop
Matty is a Digital SEO Executive at Intelligency, helping our clients to improve their digital performance. Matty is currently studying for his Level 3 Multi-Channel Marketer apprenticeship and has completed a Sport Journalism Degree at the University of Huddersfield.
Google users search engine crawlers or spiders to follow internal links on your website to crawl your content to discover, understand and index your content in its search results. In this article, we will talk about the dos and don’ts when it comes to crawling and indexability.
What is crawling?
Google’s crawlers, known as Googlebot, crawl your site via links. The bots travel from one page to the next, collecting information on the way. Googlebot looks at links from both internal and external links to find more content to crawl.
What is Indexing?
Once the page has been crawled, it doesn’t automatically appear in search results, Google
Analyses the content of the page, including the heading, text and images to determine how relevant your content is to specific search results and whether it is worthy of being displayed, this is where making sure your site is optimised for SEO becomes important.
How to help Google Crawl and Index your Content
Submit a sitemap – Users can submit a sitemap to Google to speed up the process of crawling and indexability. A sitemap is a list of URLs that you can submit to Google via Google Search Console. This provides Google with a structured list of your pages and helps guarantee visibility for all pages on your site you want indexing. If you use a CMS System, such as WordPress, there are plug-ins that will automatically generate a sitemap, such as Yoast SEO.
Optimised Internal Links – As stated, Google uses internal links to crawl the content on your site and understand the structure and hierarchy. Making sure you use descriptive anchor text for your internal links is important, using relevant keywords explains the content that is being linked to.
Use Breadcrumbs – Breadcrumbs are links on a webpage that display site structure and page hierarchy, for example, Home > Digital Marketing > SEO. This helps Google further understand the structure of your site and how pages are related to each other, further improving SEO. You can also add schema to your breadcrumbs, which is structured data that tells Google to display your page in search engines as Home > Digital Marketing > SEO and helps Google understand the hierarchical nature of these pages.
