Technical SEO Tips & Tricks Through the Use of Screaming Frog

There are several tools used for search engine optimization (SEO). Screaming Frog is one of the best out there for technical and on-page audits and reporting. It is used for keyword research, link building, site audits, and competitive analysis. The Screaming Frog SEO Spider is a desktop app used to crawl website links, apps, images, and CSS with SEO in mind. If you happen to get the hang of using it, you can use it to audit your website for common SEO issues, improve onsite SEO and obtain relevant data.

With the free version, this site crawler can crawl up to 500 URLs. The paid version costs £149.00 per year and allows you to crawl an unlimited number of URLs while allowing you access to advanced features. 
Screaming Frog can efficiently crawl small and very large websites alike. Another advantage is that you can get real-time analysis of the results obtained.

Using this program- either with your PC or Mac- you can make well-informed search engine decisions.
If you are having trouble using Screaming Frog or optimizing your website, it’s recommended that you contact an SEO professional to help get you started and make sure no mistakes are made.

Setting Up Screaming Frog

Are you new to Screaming Frog? Most newbies often have trouble setting up and navigating Screaming Frog. We will show you how to set up your device and make configurations for a smooth site crawling experience.

Some sites are typically larger than others. Larger websites require more of your device's memory to store the site data, and in turn to process the data that Screaming Frog obtains from the site. 

Before you start crawling websites, Do not dedicate the entire RAM to Screaming Frog since your system may crash. The recommendation made by Screaming Frog is that you use 2GB less than your total RAM. 

Once you have allocated more RAM to the program, restart your software for the changes to be refreshed before you start crawling.

Set Appropriate Custom Configurations

For Screaming Frog to crawl efficiently, you have to accordingly adjust your configurations. Screaming Frog's default configuration settings apply the 'Basic' mode. 

You can set custom configurations that are specific to the crawl you want to work on by clicking on the 'Configuration' tab then the 'Spider' option.

This can be useful for especially large sites with tens of thousands of URLs. Carefully configure your SEO spider so that it’s only crawling the files and URLs that you need.

This will help you save time and space on your hard drive, as larger sites can take hours to complete if you let it crawl every single URL on the site.

For example, we were once working on a large eCommerce site that had nearly a hundred subdomains with the site replicated across them to optimize for local cities in the area.

This proved to be rather troublesome when the company decided to move away from this strategy and completely restructure their URLs at the same time.

When I was originally running my first set of scans using Screaming Frog, some of my reports were reaching the 150GB mark on my SSD, and we’re taking over a day to complete.

Configuring my SEO spider to properly scan specific URLs under my parameters, dramatically decreased the time to completion and reduced my report size by 10x.

Things like removing image files, removing subdomains, and creating expressions to look at only specific URLs with a certain structure can help immensely when spidering enormous sites.

So remember to always configure your spider to your needs when working with a large website.

Important Screaming Frog Features You Should Know About

These features are not easy to catch on the Screaming Frog program. Since they are hidden away, users often miss them and eventually do not use them as they crawl websites. It’s important to note that some of these features are only accessible with the paid SEO Spider subscription. They include:

1. Crawl Path Report

The fastest way to have the SEO Spider discover a URL or view 'in links' to a website is by clicking on the subject URL in the top window then the 'in links' tab at the bottom. 

In some instances, however, this is not as simple to do as it sounds. A good example is a case where there is a relative linking issue that causes infinite URLs to be crawled. In such a case, you are compelled to view the 'in links' of 'in links' of 'in links' (infinitely) for you to get the source.

Crawl path report comes in handy at this point. Just right-click on the URL then click on 'report' then 'crawl path report'. 

By so doing, you will be able to view how a URL was discovered during the crawling process and the shortest path to the URL.

2. List Mode

You can upload a list of URLs and have them crawled. List mode is limited to 500 URLs for the free version, and unlimited if you're using the premium version. To switch to list mode, click 'Mode>List' at the top bar just next to the 'Configuration' tab.

The two most notable differences between the 'List' mode and the regular 'Spider' mode are:

1. The Screaming Frog interface changes to show an upload button instead of an address bar as in 'Spider' mode.
2. The 'Limit Crawl Depth' is enabled and set to zero (0). This simply means that SEO Spider will only crawl the URLs you upload in 'List' mode.

This can be especially useful for looking at a specific set of URLs that you would like to analyze. You can dive deep into how each specific URL is optimized and how they can be improved upon by fixing the on-page and technical SEO aspects of each one.

You can find broken links, 301 or 302 redirect codes, page load speeds, and any technical SEO aspect of each URL.

You can then export the list in Excel. CSV, or Google Sheets for the individual aspects that you are looking for.

This is a very useful tool and we hope that you can utilize it in your technical SEO audits and help find new ways to improve your website.

3.SERP Mode

By switching to 'SERP' mode, Screaming Frog allows you to upload meta descriptions and page titles for the SEO Spider to calculate pixel widths and character lengths.

With this mode, there is no crawling. You are simply taking a look at some of the on-page SEO aspects of your site and determining if any need some updates or changes made to them.

This can be especially useful for large eCommerce sites that need some on-page work done on them and identifying which URLs have improper metadata that needs to be changed.

4. Auditing Redirects as You Migrate

It is cumbersome to manually audit redirects during a site or domain migration. You must have set up permanent 301 redirects from the old URLs to the new ones.

Screaming Frog makes it easier to audit redirects by providing a feature that lets you upload a list of the old URLs, crawl these URLs and follow the redirect chains. There is an 'always follow redirects' tick-box that you have to check throughout the process until you reach the target URL.

This can help you determine missing or broken redirects across your site migration that need to be changed immediately to avoid ranking losses and penalties for Google’s algorithm.

5. Crawling of AJAX Websites

Screaming Frog does not accommodate the crawling of JavaScript frameworks such as Angular JS that are still set up with the old crawling scheme. However, you can use it to crawl websites that adhere to Google AJAX crawling. 

Screaming Frog will crawl it just like any other site. The 'AJAX' tab shows both ugly and pretty URLs, picks the ugly version, and maps it to the pretty URLs.

Under the 'AJAX' tab, there are 'with hash fragment' and 'without hash fragment' filters. These are useful in identifying AJAX pages that only use the meta fragment tag that requires Google to double crawl.

Conclusion

Screaming Frog is a popular SEO tool used for link building, keyword search, and competitive analysis, among other functions. Setting up this program has over time proven to be one of the biggest challenges faced by new users. 

It is important to first set aside more of your device's RAM before you start crawling to enable the SEO Spider to operate efficiently. There are also certain functionalities that, even though often missed, are vital. 

They include the Crawling of AJAX websites, the 'List' mode, and the 'SERP' mode. If you manage to grasp these functionalities, you will be able to make the most out of your Screaming Frog experience.