Introducing Crawl Cleanup and Importing URLs from a CSV File to Your Sitemap

As our users, we want to ensure you get the most value out of our product. We believe that when you grow, we grow together.

That’s why we’re excited to announce two new features in AIOSEO 4.2.1:

• Crawl Cleanup
• Import additional sitemap URLs from CSV

These are two advanced features designed to give users more flexibility over their technical SEO. 

Improve Rankings and Save Crawl Budget with Crawl Cleanup

One of the much-requested features we’ve added in 4.2.1 is Crawl Cleanup, a powerful feature that allows you to remove Query Args from your URLs. You can find Crawl Cleanup under Search Appearance.

Query args, also known as URL parameters, are any part of a web address after a question mark (?), ampersand (&), percentage sign (%), equals (=) or other attributes added to a URL. Here’s what a URL with a query arg would look like: 

https://example.com/10-keys-to-driving-organic-traffic/?page=2

In this example, “?page=2” is the query arg.

 Common uses of URL parameters (and simplified examples) include:

• Tracking session IDs: example.com/?sessionid=123
• Filter product category pages: example.com/?product=small-blue-widget 
• Pagination: example.com/?page=2
• Onsite search: example.com/?q=users-query 

The problem with query args is that they result in multiple URLs for the same page. For example, an e-commerce site can have the following URLs pointing to the same page:

• https://example.com/store/products/ 
• https://example.com/store/products/?category=hiking&color=tan
• https://example.com/store/products/?category=hiking&type=womens&color=brown

This can hurt your SEO. For example, it can dilute search signals as search engines view the URLs as different pages. 

However, the biggest impact URL parameters have on SEO is that the multiple pages created can result in wasted crawl budget. This can lead to low-value pages being crawled and indexed, leaving high-value pages at a disadvantage. As a result, the low-value pages will rank higher than your important pages, costing you traffic.

This is why we developed the Crawl Cleanup feature.

To use this feature, head over to your AIOSEO dashboard and select the Search Appearance » Advanced. Scroll down the page until you find the Crawl Cleanup section. Turn the toggle on to activate the feature.

Once Crawl Cleanup is activated, all query args will be automatically stripped from your URLs, with the exception of any important query args that are used by WordPress Core (e.g. search terms for search results pages) or any query args you want to whitelist.

With Crawl Cleanup, you can remove unnecessary URL parameters. This will give you more flexibility and control over your URLs, thereby ensuring the right pages rank better.

Have More Control Over Your RSS Feeds

Another fantastic feature you’ll find under the Crawl Cleanup section is the ability to control your RSS feeds. 

By default, all RSS feeds are enabled in WordPress. However, once you enable Crawl Cleanup, you can decide which RSS feeds should be activated for your site.

This will help you fine-tune your RSS marketing to help you target the right audience, resulting in an improvement in the quality of traffic that comes through your RSS feeds. 

Import URLs from a CSV File to Add More Sitemap Pages

Sitemaps are an important element of SEO as they help search engines discover the essential pages on your site much faster. That’s why we make it easy for you to create optimized sitemaps. 

With AIOSEO 4.2.1, we’ve added a few improvements to our sitemap generator. One of the first changes is the improved user interface for adding additional sitemap pages. It’s cleaner and easier to use. We’ve also made some significant improvements to our multilingual sitemaps.

However, the most significant improvement we’ve made, which we’re extremely excited about, is the ability to import URLs to your sitemap from a CSV file. Manually adding URLs to a sitemap can take forever for large sites with many pages that aren’t contained within WordPress. That’s why we’ve added bulk import support. This will enable you to create a list of pages in a CSV file that you can then upload to your sitemap. This will ensure that all your important pages, even those outside WordPress, can be included in your sitemap.

You can find this feature by going to Sitemaps » General Sitemaps. Then, scroll down the page, find the Additional Pages section, and enable it.

Here, you can add URLs individually by entering them in the URL field. To add multiple pages, you can click the Import from CSV button. This will open a modal that enables you to upload your CSV file.

Once the CSV file is uploaded, the selected URLs will be added to your sitemap.

With a few clicks, you can easily and quickly add multiple URLs to your sitemap, making it easier for search bots to find and crawl your site’s important pages.

Want more details on the perfect URL structure for WordPress slugs? You can read our blog post on the best permalink structure here. 

That’s all we have for now. However, keep your eyes peeled because we’ve got a lot more updates planned for the future.

If you have any questions, feel free to reach our customer support team here, and they’ll be happy to help you out.

Don’t have AIOSEO yet? 

Go ahead and take advantage of the additional power of AIOSEO 4.2.1 by upgrading to a Pro or Elite plan. 

As always, we value your feedback and input. That’s why we encourage you to keep the suggestions and feature requests coming. Together, we’ll build the world’s best WordPress SEO plugin and toolkit.

Disclosure: Our content is reader-supported. This means if you click on some of our links, then we may earn a commission. We only recommend products that we believe will add value to our readers.

Ben Rojas President of AIOSEO

Ben Rojas is an expert WordPress developer and the President of All in One SEO (AIOSEO). With a robust foundation in the IT sector spanning over 25 years, Ben has developed a profound expertise in technology and digital landscapes.

See Full Bio

SEO WordPress E-commerce Content Marketing Analytics SaaS