Google users search engine crawlers or spiders to follow internal links on your website to crawl your content to discover, understand and index your content in its search results. In this article, we will talk about the dos and don’ts when it comes to crawling and indexability.

What is crawling?

Google’s crawlers, known as Googlebot, crawl your site via links. The bots travel from one page to the next, collecting information on the way. Googlebot looks at links from both internal and external links to find more content to crawl.

What is Indexing?

Once the page has been crawled, it doesn’t automatically appear in search results, Google

Analyses the content of the page, including the heading, text and images to determine how relevant your content is to specific search results and whether it is worthy of being displayed, this is where making sure your site is optimised for SEO becomes important.

How to help Google Crawl and Index your Content

Submit a sitemap – Users can submit a sitemap to Google to speed up the process of crawling and indexability. A sitemap is a list of URLs that you can submit to Google via Google Search Console. This provides Google with a structured list of your pages and helps guarantee visibility for all pages on your site you want indexing. If you use a CMS System, such as WordPress, there are plug-ins that will automatically generate a sitemap, such as Yoast SEO.

Optimised Internal Links – As stated, Google uses internal links to crawl the content on your site and understand the structure and hierarchy. Making sure you use descriptive anchor text for your internal links is important, using relevant keywords explains the content that is being linked to.

Use Breadcrumbs – Breadcrumbs are links on a webpage that display site structure and page hierarchy, for example, Home > Digital Marketing > SEO. This helps Google further understand the structure of your site and how pages are related to each other, further improving SEO. You can also add schema to your breadcrumbs, which is structured data that tells Google to display your page in search engines as Home > Digital Marketing > SEO and helps Google understand the hierarchical nature of these pages.

Google Search Labs, Google’s AI experimental testing ground, has announced plans to test out Web Guide, an AI-organised search engine results page experience.

How does Web Guide work?

Web Guide is a search engine results page (SERP) that uses AI to group content and reorganise the SERP making it easier to find information or links based on specific aspects of users queries.

Here’s what Google had to say,

“Web Guide groups web links in helpful ways — like pages related to specific aspects of your query. Under the hood, Web Guide uses a custom version of Gemini to better understand both a search query and content on the web, creating more powerful search capabilities that better surface web pages you may not have previously discovered. Similar to AI Mode, Web Guide uses a query fan-out technique, concurrently issuing multiple related searches to identify the most relevant results.”

You can see examples of how Web Guide will look in Google’s latest announcement.

What does this mean for Marketers

As for now, Web Guides is only an experiment, but should it one day be live on the SERP. It could play a dramatic role in SEO and potentially see a renewed surge in site visits, which have been gradually declining since the introduction of AI Overviews. A recent study from Pew found that users clicked on a traditional result in just 8% of searches with an AI summary which is down from 15% on traditional SERP pages.

However, if Google starts to begin grouping together multiple different content types to target different aspects of search queries, this could see an opportunity for more sites to appear higher up the SERP; appearing for queries that they wouldn’t typically have appeared for on traditional or AI Overview SERP pages, seeing a rise in click through rate (CTR).

Alternatively, some users may be reluctant to embrace more AI adaptations on the SERP, as AI Overviews has already seen a decline in site visits for many businesses. One thing is for certain, the Google landscape is rapidly changing and those sites adopting the latest SEO techniques, creating informative content and staying abreast of all the latest changes on the SERP, will be increasingly likely to reap the benefits. 

Google has finished rolling out its second major algorithm update of the year, the June 2025 core update. It began on June 30 and officially wrapped up on July 17, taking just over 16 days to complete.

What is a core update, and why is it important?

Google’s core updates are broad changes to the way it ranks websites in search results. These updates happen a few times a year and are designed to improve the relevance and quality of the content that appears when someone uses Google Search. Rather than fixing a specific issue, a core update looks at the bigger picture to make sure the most useful and helpful content is being shown.

For marketers, this matters because a core update can significantly affect how visible your website is in Google’s search results. Some sites may see a boost in traffic, others might drop, and many will stay the same. The changes are not personal and do not target specific websites, but they reflect Google’s ongoing effort to surface better content for its users.

What happened in the June 2025 update?

The update began at the end of June, with noticeable effects showing up around July 2. During the following two weeks, websites across many industries saw their search rankings shift. Some saw large improvements in visibility, while others experienced drops. Some sites that had been negatively impacted by earlier updates, such as the September 2023 helpful content update, showed signs of recovery.

There was a spike in volatility between July 11 and July 14, when many websites reported sudden changes in rankings. As with most core updates, the impact varied depending on the quality, relevance, and usefulness of a site’s content.

What to do if your website was affected

If your site’s rankings or traffic dropped after this update, it does not necessarily mean something is wrong. Google has stated that a decline is not a penalty. These updates simply reassess content across the web to decide what deserves to rank highest.

Instead of reacting with quick fixes, the best response is to review your content. Ask yourself whether it is original, helpful, and created with people in mind. Make sure your pages answer user questions clearly, show your expertise, and offer real value.

There is no guaranteed way to recover immediately. However, by focusing on creating helpful, high-quality content, your website may improve gradually or during future updates. Google continues to recommend its guide on producing helpful, reliable, people-first content.

Why marketers should care

Core updates directly affect how your site appears in Google Search, which can have a major impact on traffic, leads, and sales. Staying up to date with these changes helps you stay ahead in SEO and avoid sudden losses in visibility.

Even if your site was not affected this time, it is a good reminder to review your content and user experience regularly. That includes making sure your site is fast, mobile-friendly, trustworthy, and built to help users genuinely.

Long-term SEO success comes from consistency. Keep your audience in mind, continue improving your content, and you will be better prepared for whatever Google changes next.

Search advertising is one of the most effective tools for reaching potential customers with high intent. Appearing at the exact moment someone is searching for what you offer is incredibly powerful. But recent findings show that a lack of transparency in Google Ads could be quietly draining your budget, and small brands are likely paying the highest price.

The Unseen Drain on Performance

An independent analysis of over $20 million in Google Ads spend has uncovered a major issue: hidden search terms. These are user queries that trigger ads but aren’t reported back to advertisers. The performance gap is significant:

• $0.85 in waste per $1 spent on hidden terms

• 52% higher CPCs
• 44% lower CTRs

Despite claims that these terms are excluded due to low volume, there’s minimal structural difference between hidden and visible queries. This suggests the exclusion isn’t just about rarity — it may reflect lower-quality traffic that advertisers cannot assess or control.

Impact on Strategic Execution

When performance data is obscured, campaign optimisation becomes guesswork. Budgets are allocated without clarity. ROI tracking becomes fragmented. And the strategy shifts from precision to reactive adjustments.

Brands operating with tight budgets and narrow margins are particularly vulnerable. Without access to full search term data, they lose the ability to fine-tune bidding, refine targeting, and eliminate inefficiencies. In short, they lose control over performance.

Why Visibility Matters More Than Ever

Transparency in performance data isn’t a bonus feature — it’s the foundation of intelligent strategy. The ability to analyse, test, and refine campaigns depends entirely on access to real, actionable information.

Without it, even the best media plan becomes reactive. Marketers deserve better than hidden inefficiencies. They need data they can trust, tools they can control, and platforms that prioritise outcomes, not just impressions.

New data shows that around 13% of the answers in Google’s “People Also Ask” boxes are written by Google’s AI, not pulled from websites. This change could affect how much traffic your content gets from search, but it also points to new opportunities.

What is the People Also Ask box?

When you search on Google, you often see a drop-down box with related questions such as:
“People also ask: What is X? How does it work?”

This feature helps users dig deeper into a topic without doing new searches. When someone clicks a question, Google shows an answer, usually taken from a third-party website, with a link to that page.

Why it matters for marketers:

• It’s a valuable source of organic traffic, especially for informational content (like blog posts or service pages).
• Being featured in these answers helps build visibility and authority for your brand in search results.
• It often appears high on the page, sometimes even before standard links, so ranking here gives you prime real estate.

What are Google AI Overviews?

AI Overviews are summaries that Google generates using artificial intelligence. Instead of linking to a website, Google’s AI tries to answer the question directly, based on what it has learned from scanning the web.

You’ll often see these appear at the very top of a results page, especially for more complex or multi-part questions.

• These AI answers can replace or push down traditional links in search results.
• If Google answers a user’s question directly, they may never click through to your website, which can hurt your traffic.
• At the same time, they give clues about what information Google thinks is missing, and that’s a signal you can use to shape your content.

What’s happening now?

An analysis of over 8.4 million English-language “People Also Ask” results found that 12.6% of answers are now AI-generated, rather than pulled from existing web pages. That’s according to Mark Williams-Cook, founder of SEO tool Also Asked.

This shift impacts both SEO (Search Engine Optimisation) and content marketing.

• Less visibility: If Google’s AI answers a question, fewer people may click through to your site even if you’ve written great content.
• New opportunities: If AI is stepping in, it likely means no page fully answered the question. This shows where you could step in and create a better, more complete piece of content.

When did this start?

Google began inserting AI-generated answers into People Also Ask boxes in November 2024. This is the first large-scale data showing how common it has become. What should you do about it?

• Review your content: Are you answering key questions clearly and completely?
• Fill content gaps: Use tools like Also Asked or Answer the Public to find what users want to know, then answer it better than AI can.

Google is changing how it delivers answers using AI to fill in where it sees gaps. For marketers, this is both a warning sign and an opportunity. If you can produce the most helpful, complete content, you still have a chance to earn visibility even in an AI-first search world.

Yoast SEO is launching an add-on to Google Docs to provide users with accessibility to enhance their writing with SEO recommendations in real time.

What is Yoast?

Yoast is a plug-in developed to add SEO functionality to websites, improving the way they are seen on search engines. Yoast has a number of features to help SEOs, including:

SEO Analysis – When you add written content to your site, Yoast can analyse it to make sure that it meets SEO specifications, giving recommendations to improve your content so it drives new traffic.

Crawling and Indexing – Yoast allows you to adjust which pages on your site are crawled and indexed. You may not want search engine crawlers to see all of your pages in search engines, such as a paginated page or a page with similar content to another, for example. You might want Google to see the content on a page and the links.

Sitemaps – Sitemaps are used to allow search engines to understand and crawl and index websites more easily. Yoast automatically generates a sitemap for your site to help with your pages’ search engine rankings.

Redirections – Sometimes internal and external links to pages expire as the pages that were being linked to no longer exist. Yoast allows you to redirect to the links to new relevant links so users aren’t sent to an error page (404).

Yoast and Google Docs

Yoast has recently announced their latest feature, which will mean their SEO analysis and readability feature will be available with an add-on within Google Docs. The SEO analysis tracks how well your content is optimised for a specific keyword phrase, including the presence and frequency of the keyword and making sure there is no keyword stuffing. The readability tracks how well written your content is, including sentence and paragraph length and structure.

This latest feature will allow SEOs to make improvements to their content in real time and speed up the process of creating optimised content for their sites.

In Google Ads, choosing between broad match and exact match keywords can make or break your campaign performance. Exact match means your ads show only when people search for your keywords exactly as you’ve set them—great for precision and control. Broad match, on the other hand, lets your ads appear for searches that include variations, related terms, or even synonyms of your keywords—giving you a wider reach but sometimes less relevance. 

Knowing which match type drives the best results is key, and that’s where experiments come in.

Simplified Experiment Setup

Google is rolling out a new way to test broad match keywords directly within existing campaigns, aiming to deliver faster results with fewer mistakes. Instead of duplicating your entire campaign to run an experiment, you can now split traffic and budget inside one search campaign. 

Google says this new setup avoids common experimentation errors and speeds up insights by eliminating the need to manage separate campaign copies.

Built-In Control and Treatment

The updated system allows you to assign a portion of your existing campaign’s traffic as the control group—using your original keywords—and another portion as the trial group, where broad match keywords are tested alongside your originals. 

This feature is available only for campaigns using Smart Bidding, though portfolio bid strategies aren’t supported. The new approach ensures experiments are simpler, more accurate, and easier to manage.

Easier Analysis and Next Steps

Once the experiment finishes, results appear in the “Experiments” tab, now with an expanded “Experiment summary” page for a clearer comparison of control and trial performance. If you’re happy with the results, you can click “apply the experiment” to add broad match versions to your live campaign seamlessly. 

This streamlined process makes it easier for advertisers to find the right keyword strategy without the hassle of duplicating campaigns or juggling multiple setups.

What is keyword cannibalisation?

Keyword cannibalisation occurs when multiple pages on your website target the same keywords, causing them to compete against each other in search engine results. Rather than boosting your visibility, this overlap can dilute your SEO efforts and hurt your rankings.

Why is keyword cannibalisation such a problem?

Keyword cannibalisation directly impacts your content. Here’s how it directly impacts your SEO efforts:

Impacting your rankings – the greatest issue with keyword cannibalisation is the impact it can have on your rankings. Negative Impact on ranking decreases the chance of content being displayed, clicked on and engaged with.

Decreasing page authority – when your content is directly competing with more of your content, it can negatively impact your authority. It dilutes the authority you have potentially established on one page by redirecting your audience to another.

The spending of unnecessary crawl budget –if you have multiple pages targeting the same keywords, it can impact the chances of other important pieces of content being crawled. This is because Google determines the crawl budget, meaning other pieces of important content can be directly impacted because of keyword cannibalisation.

Negatively impacting conversion rates – if two pieces of content are competing with each other and one ranks higher than the other, it is going to have a higher click-through rate. You never want to compete with your content, as the goal is to create unique pieces of content that perform consistently.

Preventing and fixing keyword cannibalisation

Prevention

The key is to define the intent of your content; your intent will directly influence the way your content is ranked on search engines. Using long-tailed keywords allows you to directly answer and cater to different search intents, which would be one of the key factors in content not being ranked against other pieces of your content.

Avoid repeating the same keyword sets, diversify the words you target to make sure your content isn’t competing against each other. Coupling this with pinpointing your intent will ensure there’s no content battling to be ranked.

Monitor your content after it’s been created and look at how your content ranks for your keywords. If it’s ranking poorly, the content may need to be optimised to add more value and demonstrate to search engines it’s worth displaying.

Fixing keyword cannibalisation

Once you’re certain keyword cannibalisation is your problem, you can begin to strategise how to move forward with your different pieces of content. 

The best approach is to consolidate your content. Merging your content into one comprehensive piece can be beneficial for preserving content. You can do this by implementing 301 redirects, which will allow the original link to still exist but funnel it back into the new consolidated content.

If you think your content serves different audience intents, then optimise the content to cater to that intent. You can do this by adapting the copy and integrating more long-tailed keywords.

If you decide you need to keep the content as well as needing to keep the pages separate, then canonical tags may be the best approach for you. A canonical tag lets the search engine know which page to prioritise for ranking and indexing.

What not to do…

Whilst some scenarios might call for you to delete the page to fix a cannibalisation, its not an ideal solution. Not only can you lose the value that piece of content added to your site, but any links embedded on the page are lost.

In Summary
Keyword cannibalisation can quietly damage your SEO performance if left unchecked. By planning strategically, optimising with intent, and auditing your content regularly, you can ensure each page has a distinct purpose, helping search engines and users alike navigate your site more effectively.

Google has begun rolling out its June 2025 core update, the second major algorithm refresh of the year. It launched globally on the morning of 30 June, and the rollout is expected to continue for around three weeks, affecting websites in all regions and languages.

This is what Google calls a “regular update” to its core search systems. It doesn’t introduce anything dramatically new, but it is still significant, with the potential to cause noticeable changes in how sites perform in search results.

What is a core update?

Google’s core updates are part of its ongoing efforts to improve the quality of search results. They are not designed to punish sites but to reassess which pages best match user intent based on what people are searching for today.

Rather than targeting specific websites or content types, this update looks at all types of content across the web. It includes changes to how Google ranks and surfaces content not only in standard search results but also across features like Google Discover and featured snippets.

The company has described it as an update that helps surface “relevant, satisfying content for searchers from all types of sites.” In other words, it aims to reward web pages that deliver strong user value and elevate them in the rankings.

If your rankings change, here’s what it means

If your website experiences a dip in visibility during this period, it’s important not to assume you’ve done something wrong. Core updates are not penalties. Instead, they are an opportunity for Google to highlight content it considers more useful or authoritative based on the latest data from real users.

However, if you’ve seen a significant drop, especially in key landing pages or traffic-driving content, now is a good time to take stock. Google’s advice remains the same: review your content with a people-first mindset. Does it demonstrate clear value? Is it original, well-written, and useful to your audience? Are you genuinely meeting the needs of the people landing on that page?

While Google has said there’s nothing specific you need to “fix” after a core update, the most successful sites are those that invest in content improvements with long-term users in mind. Thin, outdated, or SEO-driven content that lacks substance may be at a disadvantage when compared to pages that focus on clarity, depth, and authority.

How long should you wait before reacting?

Google’s core updates take time to fully roll out. In this case, it will take up to three weeks, so it’s best to wait until mid to late July before making any firm conclusions. Search visibility can fluctuate during the rollout, and traffic shifts may not settle until the update is complete.

Once the update has finished, review your data using tools like Google Search Console or GA4. If particular pages have seen a decline, assess them in terms of quality, usefulness, and clarity, not just keyword usage.

Google’s long-term approach

Google has also confirmed that it will continue to refresh this update periodically, meaning its systems will keep learning and evolving in the background. However, future refreshes may not always be announced, so some fluctuations may occur quietly over time.

John Mueller, Search Analyst at Google, described the June 2025 core update as a “bigger” search update in a comment on Bluesky. That said, it still falls within the regular rhythm of Google’s ongoing efforts to improve how it ranks content across the web.

Recent update history

This is the second major update of the year. The previous one ran from 13 March to 27 March 2025. In 2024, core updates occurred in August, November, and December, with the March 2024 update standing out as the most impactful in recent memory.

If your site has been affected, don’t panic. Avoid making surface-level changes in a hurry. Focus instead on making your content more useful, more focused, and more relevant to real users. Google continues to emphasise helpfulness, trustworthiness, and user satisfaction above all else.

Treat this update as a reminder to audit your most important content. Look for opportunities to answer searchers’ questions more clearly, add depth to existing articles, and remove or consolidate outdated material. Improvements made today may not yield immediate results but can position your content more competitively for future updates.

Google introduced a new algorithm that they say speeds up retrieval, ranking and improves accuracy. The Multi Vector Retrieval Algorithm (MUVERA) can be used for search as well as YouTube, and will have an effect on the way content appears if implemented as the new algorithm on search.

What Google have said

Google’s current system uses a system called RankEmbed to embed the content on the SERP. Rank Embed is a model that is extremely useful for common queries but often falls short when users have long-tail queries (more detailed, longer questions). MUVERA is said to improve upon RankEmbed and as a multi-vector system, is much better at answering long-tail queries. The Google announcement had this to say about the new MUVERA system:

“Unlike single-vector embeddings, multi-vector models represent each data point with a set of embeddings and leverage more sophisticated similarity functions that can capture richer relationships between data points.”

While this multi-vector approach boosts accuracy and enables the retrieval of more relevant documents, it introduces substantial computational challenges. In particular, the increased number of embeddings and the complexity of multi-vector similarity scoring make retrieval significantly more expensive.

In ‘MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings’, we introduce a novel multi-vector retrieval algorithm designed to bridge the efficiency gap between single- and multi-vector retrieval.

…This new approach allows us to leverage the highly optimised MIPS algorithms to retrieve an initial set of candidates that can then be re-ranked with the exact multi-vector similarity, thereby enabling efficient multi-vector retrieval without sacrificing accuracy.“

What does this mean for SEOs

Modern algorithms such as MUREVA are more sophisticated and can create more sophisticated leaps between queries and their intent. This means that targeting specific keywords that SEOs have previously been focused on might be a thing of the past, with content being focused more on the overall context and creating it with answering the intent of specific queries in mind.