Crawl AJAX Pages and All Links of Your Sitemap with Site Audit

We are proud to announce that we’ve implemented two new crawling features in the Site Audit tool: AJAX crawling and Sitemap links crawling.

You can now crawl your entire website without missing a piece of dynamically generated content or a URL, and immediately take action on technical issues.

This is a huge step in our tool development, since only a few tools on the market provide this functionality and now the SEMrush Site Audit tool is one of them.

Let’s get straight to the details.

AJAX crawling has always been a struggle for SEO specialists as search engines didn’t understand the pages that use JavaScript to show content to users until 2009. This is when Google offered its AJAX crawling scheme and thus put extra duty on SEOs to ensure their JS-content is indexed by search engines.

Fortunately since 2015 Google started to render and understand the pages with dynamic content like modern browsers and eventually deprecated the old AJAX crawling scheme.

However, it’s still crucial for SEO professionals to conduct the full-scale analysis to check if Google sees these pages correctly. Now with the new AJAX crawling feature in Site Audit, you can track your dynamic content and your internal linking health without extra efforts!

If you don’t have a project with Site Audit setup, head to the Projects dashboard, create a new project and set up the Site Audit tool. From now on, your AJAX pages will be crawled automatically.

With this update, you can now send our crawler directly to your sitemap file to make it crawl your website exactly in accordance with your sitemap the same way as you recommend Google to do it.

In order to crawl all links of your sitemap, first go to your project with Site Audit setup, or create a new one. Click on the gear wheel and select ‘Crawl sources’.

Now you can change your crawling settings — click on ‘Crawl source’ and select:

‘Sitemaps on site’ to make SEMrush find it automatically

‘Enter sitemap URL’ to enter the URL manually

Finally, click on the ‘Save’ button and re-run your campaign. Your website is now crawled in accordance with your sitemap.

Here at SEMrush, we are sure that this new functionality will help you conduct the most complete SEO analysis of your website and not a single page will be left unnoticed by our crawlers. And this is one more step in making the Site Audit tool the best site crawler on market.

What do you think about this update? Share your thoughts and ideas at
site-audit-feedback@semrush.com, and stay tuned!

Semrush is a leading online visibility management SaaS platform that enables businesses globally to run search engine optimization, pay-per-click, content, social media and competitive research campaigns and get measurable results from online marketing. Semrush offers insights and solutions for companies to build, manage, and measure campaigns across various marketing channels. Semrush, with over 94,000 paying customers, is headquartered in Boston and has offices in Philadelphia, Trevose, Austin, Dallas, Amsterdam, Barcelona, Belgrade, Berlin, Limassol, Prague, Warsaw, and Yerevan.