Search engine optimization, in its the majority of fundamental sense, relies upon something above all others: Online search engine spiders crawling and indexing your website.
But almost every site is going to have pages that you do not want to include in this exploration.
In a best-case scenario, these are not doing anything to drive traffic to your website actively, and in a worst-case, they could be diverting traffic from more vital pages.
Luckily, Google enables webmasters to inform online search engine bots what pages and material to crawl and what to disregard. There are several ways to do this, the most typical being utilizing a robots.txt file or the meta robotics tag.
We have an excellent and comprehensive description of the ins and outs of robots.txt, which you must certainly read.
However in high-level terms, it’s a plain text file that lives in your site’s root and follows the Robots Exclusion Protocol (REP).
Robots.txt supplies spiders with guidelines about the site as an entire, while meta robots tags consist of directions for specific pages.
Some meta robots tags you may use include index, which tells online search engine to include the page to their index; noindex, which tells it not to add a page to the index or include it in search engine result; follow, which instructs an online search engine to follow the links on a page; nofollow, which informs it not to follow links, and an entire host of others.
Both robots.txt and meta robotics tags are useful tools to keep in your toolbox, however there’s likewise another way to instruct search engine bots to noindex or nofollow: the X-Robots-Tag.
What Is The X-Robots-Tag?
The X-Robots-Tag is another way for you to control how your webpages are crawled and indexed by spiders. As part of the HTTP header response to a URL, it manages indexing for an entire page, in addition to the particular elements on that page.
And whereas using meta robots tags is relatively straightforward, the X-Robots-Tag is a bit more complex.
But this, obviously, raises the question:
When Should You Use The X-Robots-Tag?
According to Google, “Any regulation that can be used in a robots meta tag can also be specified as an X-Robots-Tag.”
While you can set robots.txt-related instructions in the headers of an HTTP response with both the meta robots tag and X-Robots Tag, there are certain situations where you would wish to use the X-Robots-Tag– the two most typical being when:
- You want to control how your non-HTML files are being crawled and indexed.
- You want to serve instructions site-wide rather of on a page level.
For instance, if you wish to obstruct a specific image or video from being crawled– the HTTP response method makes this simple.
The X-Robots-Tag header is likewise helpful since it allows you to integrate numerous tags within an HTTP action or utilize a comma-separated list of instructions to specify instructions.
Perhaps you don’t want a specific page to be cached and desire it to be not available after a certain date. You can utilize a combination of “noarchive” and “unavailable_after” tags to instruct search engine bots to follow these instructions.
Essentially, the power of the X-Robots-Tag is that it is a lot more versatile than the meta robots tag.
The advantage of using an X-Robots-Tag with HTTP actions is that it permits you to use regular expressions to carry out crawl instructions on non-HTML, along with use specifications on a bigger, international level.
To assist you understand the difference in between these instructions, it’s handy to classify them by type. That is, are they crawler instructions or indexer instructions?
Here’s a helpful cheat sheet to explain:
|Spider Directives||Indexer Directives|
|Robots.txt– uses the user representative, allow, disallow, and sitemap instructions to specify where on-site search engine bots are enabled to crawl and not permitted to crawl.||Meta Robotics tag– permits you to specify and avoid search engines from showing specific pages on a site in search results.
Nofollow– permits you to define links that should not pass on authority or PageRank.
X-Robots-tag– permits you to control how specified file types are indexed.
Where Do You Put The X-Robots-Tag?
Let’s say you want to obstruct specific file types. An ideal technique would be to add the X-Robots-Tag to an Apache setup or a.htaccess file.
The X-Robots-Tag can be contributed to a site’s HTTP reactions in an Apache server setup via.htaccess file.
Real-World Examples And Uses Of The X-Robots-Tag
So that sounds great in theory, but what does it look like in the real world? Let’s have a look.
Let’s state we wanted search engines not to index.pdf file types. This configuration on Apache servers would look something like the below:
In Nginx, it would appear like the listed below:
place ~ *. pdf$ add_header X-Robots-Tag “noindex, nofollow”;
Now, let’s look at a different circumstance. Let’s say we wish to utilize the X-Robots-Tag to obstruct image files, such as.jpg,. gif,. png, and so on, from being indexed. You might do this with an X-Robots-Tag that would appear like the below:
Please note that comprehending how these directives work and the impact they have on one another is vital.
For instance, what happens if both the X-Robots-Tag and a meta robots tag lie when spider bots discover a URL?
If that URL is blocked from robots.txt, then particular indexing and serving instructions can not be discovered and will not be followed.
If instructions are to be followed, then the URLs consisting of those can not be disallowed from crawling.
Look for An X-Robots-Tag
There are a few different methods that can be utilized to look for an X-Robots-Tag on the site.
The easiest way to check is to set up a browser extension that will inform you X-Robots-Tag information about the URL.
Screenshot of Robots Exclusion Checker, December 2022
Another plugin you can use to identify whether an X-Robots-Tag is being utilized, for example, is the Web Designer plugin.
By clicking on the plugin in your internet browser and navigating to “View Response Headers,” you can see the various HTTP headers being utilized.
Another method that can be utilized for scaling in order to identify problems on websites with a million pages is Screaming Frog
. After running a site through Yelling Frog, you can browse to the “X-Robots-Tag” column.
This will show you which sections of the website are utilizing the tag, together with which particular regulations.
Screenshot of Screaming Frog Report. X-Robot-Tag, December 2022 Using X-Robots-Tags On Your Website Understanding and controlling how search engines connect with your site is
the foundation of seo. And the X-Robots-Tag is an effective tool you can utilize to do just that. Just know: It’s not without its risks. It is extremely easy to slip up
and deindex your entire website. That stated, if you read this piece, you’re most likely not an SEO novice.
So long as you utilize it wisely, take your time and check your work, you’ll discover the X-Robots-Tag to be a helpful addition to your toolbox. More Resources: Featured Image: Song_about_summer/ SMM Panel