Serr.biz - CT SEO Web Designer

Back in the days when I started building blogs, I made a big (yet common) mistake of modifying WordPress core files. WordPress core files are essentially all files included in WordPress installation except the ones located in the wp-content folder (and its sub folders). As an open source CMS, WordPress gives you full access to the core files, which has many upsides of-course (I will not discuss those here though). These files contain built in functions and processes that your blog needs in order to work properly. But what's the big mistake? and why would you even want to modify these files in the first place?

You would want to (but shouldn't...) modify core files when you want to change to some built in functions to meet your needs. For example, you might want to change the way WordPress organizes the comments to your posts. If you know your way in PHP code, you can understand the temptation here. But if you decide to change these files as I once did, you will not be able to upgrade WordPress safely because at least some of these files will be replaced during the upgrade. This means that all your changes will be lost. Even if you manage to keep track of all of your changes and restore them after the upgrade, they may not be suitable for the new version of WordPress and can harm your blog functionality badly. There are much smarter ways to customize your blog - take a look in this article.

While there are some nice plugins to enhance WordPress search like Search Everything and Search Unlished, Google custom search engine (AKA Google CSE) seems to have the best algorithm yet.

Before integrating the custom search engine with your blog, I recommend testing it on a local WordPress installation (if you don't have one, read this), before implementing it on your live blog.

Let's see how to integrate Google CSE in WordPress:

Preparing your blog for Google custom search integration

First, we need to prepare two files for the integration:

1. Go to your WordPress theme folder and look for a file called searchform.php. if it doesn't exist, create it.
2. Create a new page template named searchResults and save it as a new file called searchResults.php.
Your searchResults.php should look more or less like this:

 <?php /* Template Name: searchResults*/?>  <?php get_header();?>       <div class="post">         <h2><?php the_title();?></h2>         <div class="postContent"> /* leave space here to paste the search results code later on*/         </div>   <?php get_sidebar();get_footer();?>

3. Login to your blog dashboard (www.yourblog.com/wp-admin).
4. Create a new page (Pages -> Add New) and name it Search Results (or any other name you would like).
5. As your page template select searchResults
6. Click on Publish

Creating a custom search engine for your blog

The next thing to do is create and customize your new search engine. If you don't have an account in Google (I assume you do, but just in case), create one here. Go to Google CSE, and click on Create a Custom Search Engine. On the next screen fill out the following fields:

1. Enter the desired name for the search engine (can be your domain name).
2. Enter the search engine description (can be the same as your blog's meta description tag).
3. Make sure Only Sites I Select is selected.
4. Type your domain name without http (e.g. www.yourblog.com).
5. Make sure the Standard Edition option is selected.
6. Go over the terms of service (if you know anyone who actually read the entire document, I'd like to know them too) and tick that box.
7. Click Next.
8. On the following screen click on Finish.

Setting keywords

You should now see your new search engine listed. Click on Control Panel.

1. Click on Basics on the left pane.
2. Enter Keywords for your search engine (can be the same as the keywords in your blog's keywords meta tag).
3. Click Save Changes.

Customizing the look and feel for the new search engine

After setting the keywords, it's time to customize the look and feel of the search engine:

1. Click on Look and feel on the left pane.
2. Select iframe. This means that the search results will be displayed on a separate page (which you already created and named it searchResults.php, remember?)
3. Select an option from the list and click on the customize button. However, you can also have your own design for the search box as I did in this blog; in this case, just select the Default option (in addition, check out these tips).
4. Click on Save .
5. If you want to, you can make money by people clicking the ads in your search results page. To that click on Make Money on the left pane and follow the instructions.
6. Click on Get Code .

Copying the code for the search box and the search results to your blog

Alright, now we can copy the generated code to the files we created in WordPress.

1. Make sure Get Code is selected in the left pane.
2. Enter the URL (permalink) of the page you created in WordPress for your search results and select Top and Bottom for the ads location (refer to these the second tip for the reason to this location).
3. Copy the code in the first box and paste it to the searchform.php file in your theme folder (if this file has some code in it already, delete it first and then paste the new code).
4. Copy the code in the second box and paste it to the searchResults.php file, where you left some space for pasting (should be line 7 according to that example).

We are done with the integration. However, you can modify those settings at any time and then recopy that code to your files again.

Finally, a few tips:

• The code for your search box can be customized according to your CSS needs (add IDs or classes to the Form and Input tags).
• If you want the search results iframe to have a width less than 795, you need to select Top and Bottom for the ads location.
• If you do the integration locally before going live (recommended), the height of the search results iframe will always have white space below the results. This space will disappear once you implement the CSE in your live blog.
• If your blog's background isn't white (like in this blog), wrap the code for the search results with a div. than give that div some right and left padding, so the search results will not appear that close to the edge of the white area. Search this blog to see an example of it.

Feel free to speak your mind or ask questions in the comment area!

Additional resources

• Google CSE for Thesis theme
• How to set up google custom search and make money

Robots.txt file is a way to tell search engines and other crawlers which parts of your website should be crawled and indexed. WordPress uses a virtual robots.txt file by default.
This means that if you open your blog's root directory, you will not find a file named robots.txt unless you created one manually. However, you can view the WordPress generated virtual robots.txt by adding "/robots.txt" to your blog's URL (e.g. yourblog.com/robots.txt).

The virtual robots file includes the following 2 lines (unless it is modified by some kind of plugin or by your blog's privacy settings): User-agent: * Disallow: These lines tell all crawlers (called user agents) that all the pages and directories of your site can be indexed, including your admin pages (such as yourblog.com/wp-admin/ ).

As you may have guessed, indexing admin pages in search engines is not recommended, so you should definitely do some tweaking to these settings in order to improve SEO and prevent irrelevant pages from being indexed. I recommend to add the following rules:

• If you use both categories and tags in your blog, do not index both (reason: duplicate content).
• Prevent search result indexation (if you have a search feature in your blog. reason: duplicate content).
• Consider preventing author and date archives (depends on whether you index categories and tags. reason: duplicate content).
• Prevent indexation of all admin pages.
• Advanced - consider preventing indexation of URLs that include arguments (depending of whether you use SEF premalinks). example: yourblog.com?s=q

There are a couple of options to set these rules:

• Install a plugin like Yoast's Robots meta. This plugin adds meta tags to the head section of the pages and tell tells the search engine whether or not to index them. It also allows you to control search engine indexing for individual posts or pages.
• Create a robots.txt file. This is very simple, you can use your notepad for this task and save the file as robots.txt.
  Alternately, you can generate this file using an online generator such as this.
  Once you have finished, you should upload the file to your blog's root directory. I use the following rules in my robots.txt file. It covers the steps above plus a few more.
  Note that I block search engines from crawling my category pages because I decided to use tags in this blog.

   User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /wp-login.php Disallow: /*wp-login.php* Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /author Disallow: /contact/ Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /z/j/ Disallow: /z/c/ Disallow: /stats/ Disallow: /dh_ Disallow: /category/* Disallow: /category/ Disallow: /login/ Disallow: /wget/ Disallow: /httpd/ Disallow: /i/ Disallow: /f/ Disallow: /t/ Disallow: /c/ Disallow: /j/ Disallow: /*.php$ Disallow: /*?* Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.cgi$ Disallow: /*.xhtml$ Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads  # alexa archiver User-agent: ia_archiver Disallow: /   # disable duggmirror by Digg User-agent: duggmirror Disallow: /   # allow google image bot to search all images User-agent: Googlebot-Image Disallow: /wp-includes/ Allow: /*   # allow adsense bot on entire site User-agent: Mediapartners-Google* Disallow: Allow: /* 

Once you have set a robots.txt file for your blog, you can test it to see if it does what it should (blocking certain pages). To test it, you can use the Crawler Access tool in Google Webmaster Tools:

• In GWT, go to Site Configuration -> Crawler Access.
• In this page, make sure the text area of the robots.txt file has been downloaded recently by Google and that it reflects the most recent changes you have made.
• In the URLs box, type different URLs to test against (for example, yourblog.com/wp-admin/) and click on the Test button.
  The result displays something like "Blocked by line 3: Disallow: /wp-admin".
  If it doesn't, you missed something when creating the file.

Another useful method to test the effectiveness of your robots.txt file, is to use Robots.txt Analyzer.