Implementing Solr4 in WordPress - No Plugin

December 2, 2012 by Remco | 1 Comment

I’m currently working on a WordPress based travel site….

I left some space here where the experts can chuckle, LOL, ROFL and whatever else they think is appropriate. To those cynics I say good luck with you’re home made platform. WordPress is so much more than a blog platform, it’s the perfect framework for almost every project. It’s known for it’s near perfect and solid database design, it’s equipped with a great admin interface, offers great caching plugins and many more. I at first thought I had to migrate to another platform but along the way I found out that at least for me the possibilities are unlimited. And scalability? Well, read this post and make up your mind again.

When and why Solr in WordPress?

For me there were 3 solid reasons besides that using text-based search is just really cool, which could be a reason when you run a simple blog like this one. But in that case it’s definitely not a necessity, caching plugins and CDN will do the trick for you.

Scalability

If you plan to become a high traffic site your database might become a bottle neck. In my case, my theme has a lot of custom fields. This resulted in that a 10 post-per page category page resulted in >100 database reads. Now with the current traffic and server capacity that’s not a problem but you don’t need to be an expert to see that one day this will become a serious problem.

Flexibility

In my opinion in principle MySQL is for writing and text-based search for searching and reading. Especially when using a lot of custom fields you’re MySQL based search will become complicated and resource intensive either resulting in extremely complex code to generate the right queries or you deciding to drop some functionality. Both I don’t opt for. Solr and other text based search engines allow you to “flatten” you data and in a very easy way do any kind of query you want all way much faster than MySQL.

Speed

It’s a bit derivative from the two points above, but Solr performs just way much faster than MySQL in querying your data, period.

Why not use a standard Solr plugin?

First of all, if you use WordPress as a framework for a travel site or anything else for which you could say WordPress is not intended, I would advice to stay away from all plugins that deliver front-end functionality. Nothing wrong with using a plugin for caching or SEO but for most others I would rely on my own code for many reasons. You don’t know what you get, how well it’s been coded and what the implications are when you update a plugin. Besides that, building your own code for widgets makes things much more flexible and you’re not limited by the functionality the plugin offers.

Second, I tried some of those plugins and wasn’t impressed or pleased. The output of text was distorted and again, it didn’t allow for much customisation because any update of the plugin would instantly kill all custom code.

And last, it’s really not that difficult to code it yourself.

Designing you schema

Before you can add a new core to Solr you need to design a schema defining the fields of your core. Here’s a short example:

<?xml version="1.0" encoding="UTF-8" ?>
<schema name="indah_base" version="1.0">
  <fields>
    <field name="uid" type="string" indexed="true" stored="true" required="true" multiValued="false" /> 
    <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" /> 
    <field name="acctype" type="string" indexed="true" stored="true" required="false" multiValued="true" /> 
    <field name="name" type="string" indexed="true" stored="true" required="true" multiValued="false" />
    <field name="city" type="string" indexed="true" stored="true" required="false" multiValued="false" />
    .....
    <field name="lastupdated" type="date" indexed="true" stored="true" required="false" multiValued="false" />
    <dynamicField name="prop_*" type="string" indexed="true" stored="true" multiValued="false" />
    <field name="_version_" type="long" indexed="true" stored="true" multiValued="false" />
    <field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>
  </fields>
  <uniqueKey>uid</uniqueKey>
  <types>
    fieldtype definitions
    ....
  </types>
</schema>

<?xml version="1.0" encoding="UTF-8" ?>

.....

</fields>

<types>

fieldtype definitions

....

</types>

</schema>

Once you’ve created your schema you need to save it in the conf directroy of your new core. Assuming that you’ve completed all steps in my previous post on how to install Solr 4, your next step is filling your core with data.

Adding data to your core

The first method of doing this I was thinking of was using the integrated Data Import Handler and using the MySQL handler. But the problem I ran into was that the queries would be extremely complex because of the many custom fields and other info related to a post I wanted to add to Solr. Basically it can not be done this way in my situation and any attempt would result in slow queries and the database to choke.

So I had to come up with another method. After doing some reading I realised that the Data Import Handler is not the way to go and that I would go for the update handler. The update handler allows for Json input and I figured I could easily and quickly create the Json input require for my schema.

Building the API

I chose to build an API that generates Json output, not on screen but in a file which I instantly move outside of the web accessible environment, I don’t want people to be able to download my complete or fragments of my database.

In this API I leverage on all that WordPress has to offer and all functions I already created to get data in an efficient way. In order to do that you need to tell the API that it’s running in a WordPress environment by adding the following lines:

require_once('../wp-config.php');
require_once( '../wp-load.php' );
require_once('../wp-includes/wp-db.php');
global $wpdb;

require_once('../wp-config.php');

require_once( '../wp-load.php' );

require_once('../wp-includes/wp-db.php');

global $wpdb;

Now your API is connected to your WordPress environment. Next I queried my post table:

$querystr = "select id, post_title as title, left(post_content,900) as content, post_modified from wp_posts as p where p.post_type='post' and p.post_status='publish'";
$pageposts = $wpdb->get_results($querystr);
foreach ($pageposts as $post) {
  assign values to the variables you want to write to your json file
}
// Next create an array that hold all data and in which the field names array keys correspond with the field names used in your schema:
$rows[] = array(
'uid' => $uid,
'id' =>'h-' . $id,
'acctype' => $property_type,
...... 
)
//And last you need to create the json file, copy it so a save location and delete the one in the web accesible location:
$jsonstring= json_encode($rows);
$jsonstring=str_replace("u2029","",$jsonstring);
$jsonstring=str_replace("u00a0"," ",$jsonstring);
$jsonstring=indent($jsonstring);
//echo $jsonstring;  //for quick debugging purposes
//exit;   //for quick debugging purposes
$myFile = "indah.json";
$fh = fopen($myFile, 'w') or die("can't open file");
fwrite($fh, $jsonstring);
fclose($fh);
copy("indah.json","/new location/indah.json"); //the new location is in my case a folder "dataimport" created in the folder of the core
unlink($myFile); // deletes the original file

So now you’ve created a proper Json file that has all your WordPress data. Obviously you can make it more lean and mean with incorporating “last_modified” in your query so it only holds new and changed records.

If Solr and WordPress run on different servers I guess you have to copy the file using ftp or so to the other server, Google that shit yourself!

Committing your data to Solr

The last step is pretty simple and I’ve created a simple script for that so I can easily assign it to a cron job:

#!/usr/bin/env bash
cd ~/webapps/solr/example/solr/<core name>/importdata
curl 'http://localhost:31326/solr/indah_base/update?commit=true' --data-binary @indah.json -H 'Content-type:application/json'

#!/usr/bin/env bash

cd ~/webapps/solr/example/solr/<core name>/importdata

curl 'http://localhost:31326/solr/indah_base/update?commit=true' --data-binary @indah.json -H 'Content-type:application/json'

The first line puts me in the folder where the json file is stored, the second line commits the Json file to the specified core in your Solr server. If all goes well the response would be something like this:

{"responseHeader":{"status":0,"QTime":242}}

1	{"responseHeader":{"status":0,"QTime":242}}

If you get an error check the field type and whether you’ve defined a field as required but has not value in your Json file.

Using Json in WordPress

While I’m writing this post I’ve only integrated it in my category page, but all other pages will be quite similar and especially for a custom search Solr will make your life so much easier.

Here is the beginning of the code of your category.php in your template:

<?php get_header(); ?>
<?php 
global $wp_query;
global $solr_base;
global $postperpage;
global $paged;
$current_cat=get_query_var('cat');
//trust nothing, solr can run behind on mysql
//get the current page
$paged=get_current_page($_SERVER["REQUEST_URI"]);
if ($paged==0) $paged=1;
$start= ($paged-1)*$postperpage;
$themeta="";
// dividing it in two sections, one for cat for accommodations and one for cat with posts/pages
$url = $solr_base . "query?q=catid:" . $current_cat . "&start=".$start."&rows=".$postperpage."&order=uid%20desc";
//echo $url . "<br />";
$JSONString = file_get_contents($url);
// now parse the json content
$obj = json_decode($JSONString);
$obj= $obj->{'response'};
Get postcount and start from the Solr response
$postcount=$obj->{'numFound'};
$poststart=intval($obj->{'start'})+1;
$last_record=$poststart+$postperpage-1;
if ($last_record>$postcount) $last_record=$postcount;
if ($last_record>($poststart + $postperpage -1)) $last_record=$poststart + $postperpage -1;
Now you can start looping the records Solr returned:
$docs=$obj->{'docs'};
foreach ($docs as &$document) {
  //set the variables you need for your category page
  $postid=intval($document->{'uid'});
  $doctitle = $document->{'name'};
  $permalink = $document->{'permalink'};
  $image=$document->{'image'};
  $content=$document->{'content'};
   .......
  <output the data to screen>
}

<?php get_header(); ?>

<?php

global $wp_query;

global $solr_base;

global $postperpage;

global $paged;

$current_cat=get_query_var('cat');

//trust nothing, solr can run behind on mysql

//get the current page

$paged=get_current_page($_SERVER["REQUEST_URI"]);

if ($paged==0) $paged=1;

$start= ($paged-1)*$postperpage;

$themeta="";

// dividing it in two sections, one for cat for accommodations and one for cat with posts/pages

$url = $solr_base . "query?q=catid:" . $current_cat . "&start=".$start."&rows=".$postperpage."&order=uid%20desc";

//echo $url . "<br />";

$JSONString = file_get_contents($url);

// now parse the json content

$obj = json_decode($JSONString);

$obj= $obj->{'response'};

Get postcount and start from the Solr response

$postcount=$obj->{'numFound'};

$poststart=intval($obj->{'start'})+1;

$last_record=$poststart+$postperpage-1;

if ($last_record>$postcount) $last_record=$postcount;

if ($last_record>($poststart + $postperpage -1)) $last_record=$poststart + $postperpage -1;

Now you can start looping the records Solr returned:

$docs=$obj->{'docs'};

foreach ($docs as &$document) {

//set the variables you need for your category page

$postid=intval($document->{'uid'});

$doctitle = $document->{'name'};

$permalink = $document->{'permalink'};

$image=$document->{'image'};

$content=$document->{'content'};

.......

}

Basically you have everything you need for setting up paging now, you could stick with the default wordpress paging but Solr might run behind a bit on the MySQL database due to synchronisation delays so I would suggest to make your own function for the links to the previous and next page which can be something like this:

function next_page_link($text="Next Page »",$language, $numresults){
  //first check if there's a next page
  global $paged;
  global $postperpage;
  $shown=$postperpage*$paged;
  if ($numresults>$shown) {
    $currenturl=$_SERVER["REQUEST_URI"];
    if ($paged==1) {
      $newurl= $currenturl . "page/2/";
    }
    else if ($paged>1) {
      $newurl= str_replace("page/".$paged ."/","page/".($paged+1) ."/" ,$currenturl);
    }
    $returnvalue="<a href="". $newurl . "" class="styled-button" >" . __($text,$language) . "</a>";
    echo $returnvalue;
  }
}
function previous_page_link($text="« Previous Page",$language){
  //first check if there's a next page
  global $paged;
  $returnvalue="";
  $currenturl=$_SERVER["REQUEST_URI"];
  if ($paged==2) {
    $newurl= str_replace("page/".$paged ."/","" ,$currenturl);
    $returnvalue="<a href="". $newurl . "" class="styled-button" >" . __($text,$language) . "</a>";
  }
  else if ($paged>2) {
    $newurl= str_replace("page/".$paged ."/","page/".($paged-1) ."/" ,$currenturl);
    $returnvalue="<a href="". $newurl . "" class="styled-button" >" . __($text,$language) . "</a>";
  }
  echo $returnvalue;
}

function next_page_link($text="Next Page »",$language, $numresults){

//first check if there's a next page

global $paged;

global $postperpage;

$shown=$postperpage*$paged;

if ($numresults>$shown) {

$currenturl=$_SERVER["REQUEST_URI"];

if ($paged==1) {

$newurl= $currenturl . "page/2/";

}

else if ($paged>1) {

$newurl= str_replace("page/".$paged ."/","page/".($paged+1) ."/" ,$currenturl);

}

$returnvalue="<a href="". $newurl . "" class="styled-button" >" . __($text,$language) . "</a>";

echo $returnvalue;

}

function previous_page_link($text="« Previous Page",$language){

//first check if there's a next page

global $paged;

$returnvalue="";

$currenturl=$_SERVER["REQUEST_URI"];

if ($paged==2) {

$newurl= str_replace("page/".$paged ."/","" ,$currenturl);

$returnvalue="<a href="". $newurl . "" class="styled-button" >" . __($text,$language) . "</a>";

}

else if ($paged>2) {

$newurl= str_replace("page/".$paged ."/","page/".($paged-1) ."/" ,$currenturl);

$returnvalue="<a href="". $newurl . "" class="styled-button" >" . __($text,$language) . "</a>";

}

echo $returnvalue;

}

And then replace the standard links in your category page with the new ones:

<div class="navdiv" style="float:right;"><?php next_page_link("Next Page »","language",$postcount) ?></div>
<div class="navdiv"><?php previous_page_link("« Previous Page","language") ?></div>

1 2	<div class="navdiv" style="float:right;"><?php next_page_link("Next Page »","language",$postcount) ?></div> <div class="navdiv"><?php previous_page_link("« Previous Page","language") ?></div>

There you go, you just turbo charged WordPress with Solr, no plugin needed and super fast.