Why Your Pages Are Stuck In Crawled-Currently Not Indexed & What To Do About It

I’ve had plenty of web site homeowners attain out to ask for assist with indexing points recently. Generally, I’m discovering that Google has categorised its pages as “crawled-not presently listed” within the web page indexing report in Google Search Console.

In virtually each case, I’ve examined these pages have high quality points. They’re often “commodity content material” – primarily rehashing what many others have already written on a subject with out providing something new or extra useful than what presently exists on-line.

On this article, I’ll share how I take a look at the crawled-currently not listed report in GSC. I’ll provide you with a software that can assist you discover the pages on this report that you need to analyze additional. And I’ll provide you with some ideas for enhancing so you possibly can probably get well. I have to give truthful warning, although. For many websites, you probably have a lot of pages you need listed, however they’re caught in crawled-currently not listed, restoration can be troublesome.

What Google Mentioned About Crawled-At present Not Listed At The Google Search Central Occasion In Toronto

I attended the Google Search Central occasion in April of 2026. The organizers requested us to not attribute quotes on to any Googler, however they did give us permission to share what was stated.

One presenter shared about how Search works. He stated that when Google crawls a web page it primarily means they obtain it. Then, “If we expect it’s helpful we would put it in a database,” or in different phrases, in Google’s index.

Then he talked about what sorts of issues Google desires to place within the index. He stated that AI has made the brink for creating issues decrease. If anybody can create content material on something, then the kind of content material Google desires so as to add to their index is content material that provides two issues: private expertise, and data nobody else has.

He stated that if Google has crawled your web page and has determined to not index it, there might be two causes:

1. There Might Be A Technical Challenge

I’ve discovered this to be uncommon. Nonetheless, simply final week I reviewed a web site that had undergone a migration and all of their pages have been caught in crawled-currently not listed. Word: This isn’t the identical as “Found-not presently listed” which signifies that Google is aware of the pages, however has not but crawled them.

My first step was to research whether or not Google might see the content material on pages. I used the web page inspection software in GSC by clicking the magnifying glass subsequent to the url within the crawled-currently not listed checklist and clicked, “Take a look at Reside URL” Surprisingly, once I considered the reside examined web page, all it confirmed was a heading, just a few boilerplate phrases and no content material in anyway.

On this case, the positioning proprietor did certainly have a technical problem. Their robots.txt had this line, Disallow: /*?*. The thought was to dam crawling of urls with parameters like ?replytocom or ?utm_source. However, their new theme relied on these parameters for his or her CSS information and javascript so that they have been primarily blocking Google and all different search engines like google from seeing most of their content material.

We’ve since eliminated this block and really slowly, pages are beginning to seem again within the index once more.

In case your reside check reveals Google can certainly see the content material in your pages, it’s most unlikely to be a technical problem that’s inflicting crawled-currently not listed issues.

I must also point out that some pages ought to be in your crawled-currently not listed checklist if they aren’t the canonical model. In the event you see /feed/ pages or pagination or pages with url parameters, that is regular.

2. High quality

The Googler in Toronto went on to clarify one other trigger for Google to crawl a web page and never index it. He stated it might be as a result of “we checked out it and located it to not be good.” He stated that if hundreds have coated the very same matter they could resolve that your web page is unlikely to be helpful in Search. It may be that there are different choices which can be extra in style or of higher high quality.

He additionally stated that typically Google experiments by permitting your web page to be listed for some time to see if customers prefer it, “We’re experimenting with seeing which one produces happier customers.” That could be a fairly wild assertion!

My wager is that you probably have pages that you really want listed, however Google has them within the crawled-not presently listed bucket, then your primary problem is expounded to commodity content material.

Commodity Content material Is The Most Possible Trigger

Google talked loads about commodity content material at this occasion.

Commodity content material is content material that just about anybody might write a few topic. It’s typically repeating what already exists on-line on different websites. Non-commodity content material brings a singular viewpoint or has content material that others lack or can’t simply replicate. It often demonstrates first-hand data or expertise.

Take this text you might be studying proper now. Anybody might use AI to jot down a useful article defining crawled-currently not listed pages. My article, nevertheless, talks about my expertise as knowledgeable who’s paid to offer my opinion on this topic. I’ve shared the real-world technical instance above, I’ve shared first-hand info I realized from attending a Google occasion, and I’m about to share my observations on pages which were deemed unfit of indexing.

My Observations Of Pages Caught In Crawled-At present Not Listed

These pages are often not junk. They’re good, respectable articles – pretty much as good because the pages that Google is rating. And that’s simply the purpose. The pages aren’t particular or any extra priceless than what presently exists.

Right here is the method I exploit to research these pages.

To search out the checklist, click on on “Pages” beneath Indexing in GSC. Then click on on crawled-currently not listed:

Beneath this, you’ll see an inventory of URLs to analyze. (Beneath, I’ll share extra a few software I’ve created that can assist you filter this list to see the URLs that really matter.)

I’ll discover a URL on this checklist that basically is one which we wish listed.

First, I’ll seek for some queries that you’d count on the web page to rank for. On the SERP, there’s often an AI reply that may be very useful. Typically, a person will discover the reply to their query there. If that is so, then why would they need to click through to your website to learn the very same factor?

I’ll develop the AI overview after which open up Gemini within the Chrome sidebar. Then I maintain down CTRL/Cmd and click on on the highest web sites linked to from throughout the AIO. In the event you do that when you’ve got Gemini within the Chrome sidebar opened, you’ll discover these tabs get added to your Gemini dialog.

Then I kind “/” which opens up the talents I’ve saved at chrome://expertise/ and select my Non-commodity verify. (In the event you’re a member of my paid community, you’ll find this full ability here.)

This ability is a really lengthy immediate that appears at a few of the issues Google tells us its algorithms purpose to reward in its documentation on creating helpful content, together with, however not restricted to:

• Does the content material present authentic info, reporting, analysis, or evaluation?
• Does the content material present insightful evaluation or fascinating info that’s past the apparent?
• If the content material attracts on different sources, does it keep away from merely copying or rewriting these sources, and as an alternative present substantial extra worth and originality?
• Does the content material present substantial worth when in comparison with different pages in search outcomes?

And Gemini provides me a few of the explanation why the pages linked to supply worth to the reader. Word: Generally pages are rating not due to their non-commodity worth however as a result of they’re an authoritative supply. If you’re a identified authority, you may get away with a bit extra “commodity-ness.”

Now, we have to acknowledge that Gemini doesn’t have inside perception into Google’s rating techniques. It doesn’t know why sure pages are rating. What we are attempting to be taught here’s what varieties of issues might be serving to a web page be worthy of presenting to searchers.

Then, I open up my shopper’s web page and immediate this, “Now analyze this web page in accordance with the identical standards.. This web page shouldn’t be rating effectively. It’s our shopper. Please share the place you suppose it’s missing. No must recommend enhancements at this level.”

Right here is the outcome for one crawled-currently not listed web page I used this immediate on.

John Mueller And Martin Splitt Mentioned Crawled-At present Not Listed In A Latest Podcast

As I used to be about to publish this, Google printed a Search Off the Report Podcast on “How to read the Indexing Report.” There’s loads in right here, so I bolded the elements that I assumed have been vital.

This dialogue begins at 20:32 within the video

Chapter 9: Found vs. Crawled Not Listed: Is it a technical or web site high quality problem?

“And in addition, when you add or change your web site or in case your web site may be very new, then you possibly can truly additionally use this report back to see a little bit bit how your web site goes via the totally different levels, as a result of sooner or later, you’re going to see pages in Found presently not listed. Which tells you we all know they exist, however we haven’t truly visited them. And if we haven’t visited them, we are able to’t put them within the index. Crawled-currently not listed, which suggests we visited them and we didn’t put them within the index. And that may have all kinds of various causes. Would you say that’s usually or solely typically an indication of a top quality problem?

So, it’s undoubtedly the case if our techniques are critically fearful concerning the high quality of an internet site, that they may scale back the variety of pages that they index. As a result of if now we have robust issues concerning the general high quality, then it doesn’t make a lot sense for our techniques to spend so much of time on the web site.

So, we’ll in all probability crawl loads much less, we’ll index loads much less, after which you’ll see issues like crawled, not listed or found, not listed, which from our viewpoint is principally our system saying, we find out about this, we checked out it, and as soon as we’re completely satisfied, we are going to take one other look and see if we are able to index it. It’s not a lot that I might say you need to take these conditions and attempt to repair them. From a technical viewpoint, it’s not that it’s good to repair this technical problem that Google shouldn’t be indexing this web page for the time being, however reasonably you virtually must whenever you acknowledge a much bigger sample like this, that Google shouldn’t be indexing a variety of your pages, and there’s no technical purpose, you virtually must take a step again and take into consideration the standard general.

And occupied with high quality is actually difficult as a result of a variety of instances, it’s your web site, and it’s your child. And naturally, it’s the perfect child ever. However taking a step again and making an attempt to take a look at it with the eyes of somebody who shouldn’t be straight concerned along with your web site. Generally that opens up some concepts for areas the place you possibly can enhance, the place perhaps if most of your web site is AI-generated and it labored for some time, it may be that individuals take a look at this AI-generated web site, they usually’re like, effectively, I can inform that is AI-generated. There’s nothing distinctive or priceless that’s out there right here for me. That’s to not say that each one AI-generated content material is unhealthy, however typically you simply run throughout web sites the place you’re like, anybody might have written this. This tells me nothing. Yeah, that’s true. And I believe what makes this troublesome shouldn’t be solely the truth that clearly the best way you wrote it’s the method you thought was greatest, and that’s why you suppose it’s prime quality, after all. In order that’s actually, actually exhausting to step out of your personal perspective. However typically, there’s additionally a lot different stuff that’s simply pretty much as good. So, why would we add it to the index.

After which that may let you know, like, perhaps this content material isn’t as priceless as I assumed it was as a result of different individuals are protecting the identical factor. After which what’s the worth of this model of it being within the index? Yeah that’s true. I really feel we might have a complete podcast about high quality. I believe perhaps one different factor that’s value mentioning with reference to high quality is it’s not simply the textual content. So a variety of instances folks will say, effectively, my textual content is exclusive, or my articles are good, they usually’re packaged in a web page that’s horrible to entry, the place anybody who, after they attempt to load it like their pc fan spins up they usually’re like, “Oh my gosh, I’ve to run away to ensure my pc doesn’t explode.” So perhaps that’s an excessive case, however you’ve all seen these pages the place principally the textual content is there, however it’s virtually hidden away, hidden behind advertisements, hidden behind interstitials, hidden behind different issues which can be shifting and coming and going, perhaps hidden beneath a bunch of filler content material, which we typically see, for instance, with recipes the place there’s this actually lengthy story on high that perhaps most individuals don’t actually care about. After which the recipe comes. These are all of the sorts of issues the place the general high quality is way more than simply that piece of textual content that you just say, that is my primary content material. That is what Google must be counting for my web site. And from our viewpoint, we virtually should take note of the total expertise on a web page, as a result of that’s what customers see. It’s not that customers go to an internet web page and activate some magic mode that simply pulls out the textual content, however reasonably they’ve the total expertise of this web site with all the 3D, 4D animations, and every little thing. I agree very a lot. Agree, oh my God.”

How Can You Repair This Challenge?

Oh boy, that is the robust a part of this text, as a result of in a variety of instances, I really feel that this can be very troublesome to get pages out of crawled-currently not listed. I imply, if extreme advertisements and filler are responsible, there are apparent issues to enhance on there. If there’s a technical problem, repair it and request reindexing through GSC – or simply be affected person and wait until Google tries to crawl your pages once more.

If it’s a top quality problem, although, you’re probably going to should put vital effort into enhancing these pages.

For a lot of websites that I analyze, their superpower previously was the flexibility to cowl a subject completely. Currently, there’s a pattern to not solely cowl a subject, however to anticipate all the fan-out queries and canopy these as effectively. This was talked about within the Google Search Central occasion just a few instances. In case you are creating a great deal of content material primarily based on this technique, you run the danger of going through a scaled content penalty. I can not show this but, however I believe that the June 2026 spam replace impacted plenty of websites that have been creating commodity content material at scale. If that is true, you received’t see a handbook motion in GSC. You’ll simply see a drop in natural visitors with no clarification.

I worry for lots of website positioning companies as a result of for a lot of, your primary software in your toolbox is content material creation. AI has made it a lot simpler to cowl content material on any topic. I’m not in opposition to utilizing AI to assist with content material creation. However, in case your website positioning firm can use AI to create content material in your subjects, then it’s probably not authentic, insightful, and considerably extra useful than what presently exists. There are exceptions. I do know of some companies that use intelligent AI pipelines to interview a enterprise, extract its related expertise, and switch that into good, authentic content material.

Though I don’t suggest utilizing AI to jot down your content material for you with none human enter, I do suppose you possibly can brainstorm with AI to assist enhance it. The issue, although, is that the options would require effort. The phrase “effort” is used 120 instances in Google’s Quality Rater Guidelines. It would be best to discover methods to attract out of your expertise to create content material that provides to the physique of data that presently exists in your subjects.

Do this easy immediate. Give your content material to an LLM or open up Gemini within the sidebar and ask this: “Is that this content material prone to be thought-about commodity content material?”

I simply opened up Gemini in Google Docs and requested about this very article you might be studying now:

Subsequent, do that for some concepts.

“Give me 20 concepts that assist me draw from my first-hand expertise to make this text much more useful, and considerably higher than anything that exists on this matter on the net.”

Rattling, there are some good concepts in right here.

Some Instruments To Assist You Assess Your Crawled-Not At present Listed Pages

I created a few instruments utilizing Google’s Antigravity. Yow will discover them at tools.mariehaynes.com.

There are two new instruments:

1. Filter your crawled-not currently indexed URLs. Export your crawled-not presently listed URLs from GSC. In the event you export as CSV, open the zip file and discover the desk.csv file. You’ll be able to add it to this software, and it’ll strip out /feed/ pages and others so as to see and click on on the URLs that you just need to examine.

2. GSC Index Checker. You have to to log in to your Google account to make use of this software, however know that I don’t see any of your knowledge. It’s going to verify an inventory of URLs to see what their indexing standing is. You’ll be able to select from the latest pages in your sitemap, paste an inventory of URLs in manually, or have the software seize your top-trafficked pages from GSC.

What you’re in search of right here is whether or not these pages that matter to you might be certainly listed, or whether or not they’re caught in crawled-currently not listed.

I hope this text helps! Google does appear to be getting more strict on what it is indexing nowadays.

Extra Sources:

Read Marie’s newsletter, AI News You Can Use. Subscribe now.

Featured Image: Tetiana Yurchenko/Shutterstock