Sunspot 1.2 with Spatial Solr Plugin 2.0

Currently I am porting an application from Rails 2.x to 3.x. The old version uses a plugin called acts_as_solr but is no longer actively maintained and has been stale for a while. Fortunately I found Sunspot, an elegant and clean solution to adding SOLR to Rails applications.

For geo-location based searches, Sunpot uses a really cool technique called Geohashing as Apache Solr does not provide built in support for location searches (a feature expected in the 4.0 release). However, in my case, I needed the ability to bound searches with finer granularity then the 10 or so precision bounds provided by geohashing. I opted to use Spatial Solr Plugin to provide the spatial component for search.

1. Download SSP and add the jar file to your Solr lib directory. Sunspot should have created this directory in your project root project_root/solr/lib.

2. Modify Rails.root/solr/conf/solrconfig.xml and add the following

...
<!-- Configuration for using Spatial Solr Plugin (SSP) 2.0 -->
<queryParser name="spatial" class="nl.jteam.search.solrext.spatial.SpatialQParserPlugin">
  <str name="latField">lat</str>
  <str name="lngField">lng</str>
</queryParser>
 
<searchComponent name="geoDistance" class="nl.jteam.search.solrext.spatial.GeoDistanceComponent"/>

<requestHandler name="standard" class="solr.SearchHandler" default="true">
  <!-- default values for query parameters -->
  <lst name="defaults">
    <str name="echoParams">explicit</str>
    <str name="distanceField">distance</str>
  </lst>
  <arr name="last-components">
    <str>geoDistance</str>
  </arr>
</requestHandler>
...

Note: The requestHandler with the name attribute as “standard” already exists so you just need to add the new fields to your current definition.

The two fields “latField” and “lngField” in the queryParser element defines the field name that SSP will use for its spatial search. For simplicity, I used the default names lat and lng.

3. Modify project_root/solr/conf/schema.xml and add the field definitions for lat and lng. In Sunspot’s configuration, it seems to already come with a lat and lng field that isn’t being used anymore so I decided to reuse them and just modified the type attribute to “tdouble”, a Trie Double.

...
<!-- *** This field is used by Sunspot! *** -->
<field name="lat" stored="true" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This field is used by Sunspot! *** -->
<field name="lng" stored="true" type="tdouble" multiValued="false" indexed="true"/>
...

4. In your ActiveRecord model, modify your searchable block to add indexing for lat and lng. For example, if your model attributes were latitude and longitude…

searchable do
...
  double :latitude, :as => "lat"
  double :longitude, :as => "lng"
...
end

5. Restart Solr server and reindex using

rake sunspot:solr:stop
rake sunspot:solr:start
rake sunspot:solr:reindex

If you browse to your solr schema browser (default: 127.0.0.1:8982/admin/solr/schema.jsp), you should see the lat and lng fields are indexed.

6. Next we need to add support to get the “distance” field from the results. To do this, I added the following into a file in project_root/conf/initializers

module Sunspot
  
  class Search::Hit
    def distance
      @stored_values['distance'] # distance_field_name
    end
  end
  
  class Query::Sort::DistanceSort < Query::Sort::Abstract
    def to_param
      "distance #{direction_for_solr}" # distance_field_name
    end
  end
  
end

Edit: Sunspot requires that a valid class name be defined for sorting on fields that are not explicitly defined in the searchable clause. I’ve added an additional class into the module Sunspot::Query::Sort. Koodos to Craig for pointing this out.

This defines a new method in the Hit object that returns the field ‘distance’ from the solr results. If you named the distanceField in solrconfig.xml differently, you will need to modify this accordingly.

7. Now you can define a search using the adjust_solr_params. Use the order_by with :distance to sort on the distance.

@results = Model.search do
...
  order_by(:distance, :desc) # Order by distance from farthest to closest

  adjust_solr_params do |params|
    params[:q] = "{!spatial qtype=dismax boost=#{some_boost_val} circles=#{search_lat},#{search_lng},#{search_radius}}" + "#{params[:q]}"
  end
end

The adjust_solr_params adjusts the solr query parameter right before its sent off to solr. We prepend the spatial search syntax to the beginning and leave the built query intact. That way, we can still make use of the other DSL defined search parameters. Since Sunspot uses dismax by default, i’ve set the qtype parameter so that the text portion of the search is passed to the dismax parser.

8. Access the distance field like

@results.hits do |hit, result|
  puts hit.distance
end

Note: I haven’t tested this with complex queries using faceting but I think it should work. The only thing is that I’m not sure what will happen if you use this alongside geohashing, not that you would need to.

About these ads

22 thoughts on “Sunspot 1.2 with Spatial Solr Plugin 2.0

    • Hi Craig,

      I just tested it out with order_by and it seems Sunspot does a sanity check on the attributes. I’ve added some code to allow sorting by distance. Please read the edits I made in step 6 and 7.

      Thanks!

      • Hi Joel,

        The edits from steps 6/7 will prevent the app from starting with the following error:

        uninitialized constant Sunspot::Abstract (NameError)

      • Ahh sorry, I forgot to add the modules when I cleaned it up to make it easier to read.

        It should read:
        class Query::Sort::DistanceSort < Query::Sort::Abstract

  1. Many thanks for this. I’d spent the whole day yesterday trying – and failing – to get geohash working on Sunspot for a pretty complex search with a proximity option. Was just about to give up when I saw this post added to the Sunspot wiki list. Your method works perfectly – and importantly for my site, gives a degree of precision for a 30 km+ search that doesn’t seem possible with geohash.

    One small thing. Because, unlike you, I’d only ever used Sunspot (and also perhaps because I’m still in development using its pre-packaged version of Solr), your step 3 wasn’t necessary – in fact, when I tried to add new lat/lng definitions to the Schema, reindex-ing was a horror story. But thanks to your very clear explanation of what was going on under the hood, I managed to get there in the end.

    • Yup, since I only started using Sunspot after they took out solr-spatial-light, even though I had the lat, lng field definitions, they weren’t being indexed. I’ll make a note in the blog in case anyone else has the same problem.

      I’m glad you were able to get it to work!

  2. It’s not a 1.2 issue – that’s what I’m using too. But my solr/conf/schema.xml only has a Sunspot schema – that’s probably how it will be for others just starting out with Sunspot, won’t it?

    • Hmm, that’s strange because I started fresh with the schema file provided by Sunspot 1.2 as well and I didn’t seem to have any errors reindexing. Was the problem changing the field from an sdouble to a tdouble?

  3. No. When you said ‘re-use’, I assumed that I maybe needed to create a non-Sunspot, Solr schema, so I copied the lines – exactly as your example (and with the trie), and started trying to create new schema. In fact, I didn’t need to do anything – when I reverted to the original, everything was fine. Because you had previous with Solr, I guess you may have had other configurations in the file ?? Anyway, not a problem. I’ve been delighted with the results.

  4. I followed your directions and I’m getting this when I try to run rake sunspot:solr:start etc..

    uninitialized constant Sunspot::Query::AbstractFieldFacet::RSolr

    If I comment out the stuff in the initializer then I can run the rake tasks. Then if I put the initializer code back in to run rails it all seems to work fine. Any ideas?

    I’m still trying to trace it myself.. seems like Rails can trace everything back to here /gems/sunspot-1.2.1/lib/sunspot/query/abstract_field_facet.rb:4 but then can’t resolve RSolr..

  5. Yeah it’s this line in the initializer

    @class Query::Sort::DistanceSort < Query::Sort::Abstract@

    When it's loading Query::Sort::Abstract, it's tracing back the dependencies then it fails on trying to load RSolr.

    I haven't managed to figure this one out, it's on the todo list but for now I'm just commenting it out every time I want to run a rake task. The onerosity of the whole thing is working to demand attention though. Oh, and I just invented a word by the way. ;-)

  6. Thanks for this great post, I am trying to follow the steps but it is failing when I call search…here is what I am trying to do

    @search = self.search() do
    order_by(:distance, :desc)
    adjust_solr_params do |params|
    key = “{!spatial qtype=dismax circles=100.0,-100.0,5}” + “#{keyword}”
    end
    end

    The above block is wrapped inside a class method of the model, and when I call that method from the console, i’m getting the following error

    RSolr::RequestError: Solr Response: can_not_sort_on_undefined_field_distance

    I am on jruby-1.6.1 rails 3.0.x

    Thanks
    -D

    • Hi D,

      I don’t have experience with JRuby so I’m not sure if I ‘ll be able to help you here.
      When you send the raw query to your solr server in the browser, is the distance field available?

    • Try with params[:q] instead of key :-)
      e.g. params[:q] = “{!spatial qtype=dismax circles=100.0,-100.0,5}” + “#{keyword}”

      Hope this helps
      luc

  7. Great post! I was already using the geohash version of the Sunspot gem and would love to use the Spatial Solr plugin. For Step 7, I’m assuming “Model” is the object that is to be indexed in Solr and that code should be placed in a method in the controller?

  8. Using ssp-4-solr-2.2-RC1.jar and sunspot_rails 1.3.0rc4 (and tried 1.2.1).

    Doing all that I get the following exception:
    INFO: [] webapp=/solr path=/update params={wt=ruby} status=0 QTime=228
    Oct 25, 2011 10:47:42 PM org.apache.solr.common.SolrException log
    SEVERE: java.lang.NoSuchFieldError: rsp
    at nl.jteam.search.spatial.GeoDistanceComponent.process(GeoDistanceComponent.java:62)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
    at org.mortbay.jetty.Server.handle(Server.java:285)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
    at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s