As a web designer or administrator, you may find yourself wondering how to best balance UX with other factors like site crawlability to extend your site’s reach while maintaining a great design. The user experience, or UX, of your website describes how users feel about your site and content while browsing it. Easy navigation, creative menus and content, and animations and other effects can all improve your site’s UX. Too many types of scripting, however, along with other unique UX features, can sometimes interfere with the crawlability of your content. In order to crawl through and index your content, search engines like Google need to be able to read through your site’s structure and information. For example, Javascript and other scripting used to improve UX can often hide content and links from search engine crawlers, potentially lowering your position in search engine listings. Follow these tips to perfectly balance UX and site crawlability on your own website.

Dividing Your Focus

Every step of the process of designing, building, and maintaining your website requires attention to both, SEO and UX factors. Sometimes theses areas can seem in conflict with one another, making for some difficult decisions. In order to find the best balance between SEO and UX, you’ll need to divide your focus equally and make sure to cover the basics for each area.

SEO optimization, programming process and web analytics elements. Web design concept. Computer monitor with the screen of the program for design and architecture in flat design

The Basics of Website Crawlability

In order to become listed in search engine results, search engine crawlers need to be able to read through your website. These automated search engine crawler bots continuously scour the web for information to provide their users. Since they are just coded bots, they won’t be evaluating your website for aesthetic or superficial factors. Instead, crawlers look for functional links, coding and scripting errors, buggy redirects and other erroneous content, and any text or media found on your site. They then compute your site’s overall quality through secretive, proprietary algorithms and rank your site in association with a list of search terms.

How to Improve Your Site’s Crawlability

There are a few things you can do to improve your website’s crawlability. The first step is to make sure you’re using a fast, reliable web host. If you have a WordPress website or blog then make sure you have optimized WordPress hosting. By getting web hosting that’s tailored for your website’s platform, you are ensuring your content displays faster and with fewer errors than it otherwise would. Make sure your chosen WordPress hosting company has a strongly guaranteed uptime so that your site isn’t down when the crawlers visit it for indexing.

The next step is to make sure you have a well-written¬†robots.txt¬†file in place. By including a robots.txt file on your server, you will be able to tell search engines which parts of your website to index. If any particular part of your site would be bad for SEO purposes, such as example or demo content you’ve not deleted yet, then disallow search engines from viewing it.

If any part of your website is inaccessible to search engine crawlers then you’re reducing the site’s overall crawlability and most likely harming your chances at showing up in search results. Make sure your site’s main content is listed as allowable in your robots.txt file, and that you’re not using too many scripts that crawler bots can’t understand such as JavaScript. This is also a good reason to have text headers and logos, rather than using image files with embedded text.

Website UX

While it may seem that a fully accessible, boring, text-only website would be the best theoretical candidate for maximal site crawlability, users would be turned off by such a dull experience. A balance between crawlability and UX therefore, must be struck. It’s typically considered acceptable in terms of SEO to embellish your site with images, media, animations and special effects. Having lots of efficiently compressed images is actually thought to have a positive impact on search rankings now. To make sure your on-site images and video don’t interfere with search indexers, use meta-tags and hover descriptions to describe each media file being displayed. This will help current generation search engine crawlers identify your media.

Improving UX Without Harming Crawlability

Creative UX designs sometimes require the use of Javascript and other content that isn’t ideal for crawlability. In order to combat the negative impact this content could otherwise have, make sure to use your robots.txt to stop search engines from crawling their containing pages. You can also offer alternative content for devices that aren’t able to view your site’s primary display, such as for browsers and devices incapable of installing flash. Even if you’re using images and animated navigation menus, which crawlers may not be able to read, make sure to include a text-based link structure on the bottom of your website for easier indexing. This will ensure you’re able to provide a great UX and still get your content indexed.

By following these tips and dividing your focus between the UX and crawlability of your site, you will find your content being reached by more visitors without sacrificing creativity or aesthetics. Make use of robots.txt files and always include text descriptions for images and media to conform to search engine crawler standards while maintaining good design practices.