Adding a search engine

Another thing that almost every site needs is some sort of search functionality. By default we install this search engine in via the default site generator, but it's good to know what's happening there:

Prerequisites

To add search functionality you will have to install ElasticSearch first, since that's currently the only search engine we support at this moment (we might add others as well, but haven't felt the need to yet, ElasticSearch rocks!). For the PHP side of things we rely heavily on the Elastica library.

Installing and configuring ElasticSearch is a bit out of scope here, so just have a look at the excellent ElasticSearch Guide and install it.

For Linux users there are DEB and RPM packages available on the ElasticSearch downloads page, so installing it can be as easy as :

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.3.deb
sudo dpkg -i elasticsearch-0.90.3.deb
sudo service elasticsearch start

At least, if you're running Debian or Ubuntu :p.

For OS X users we suggest installing it using Homebrew, so you can simply run :

brew install elasticsearch
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plist

And you should be good to go!

To test if it actually is running you can run :

curl -X GET http://localhost:9200/

Which should return something like this :

{
  "ok" : true,
  "status" : 200,
  "name" : "Smythe, Spencer",
  "version" : {
    "number" : "0.90.3",
    "build_hash" : "5c38d6076448b899d758f29443329571e2522410",
    "build_timestamp" : "2013-08-06T13:18:31Z",
    "build_snapshot" : false,
    "lucene_version" : "4.4"
  },
  "tagline" : "You Know, for Search"
}

Indexing nodes

So, you've got ElasticSearch up and running, now you're ready to create the search page skeleton code, so just run :

app/console kuma:generate:search

This will first ask for the bundle namespace (you can accept the default - MyProject/WebsiteBundle), and finally it will ask for the table name prefix, so enter myprojectwebsitebundle as before.

The basic code skeleton should now be generated, so go ahead and create (and apply) a migration for the database changes :

app/console doctrine:migrations:diff && app/console doctrine:migrations:migrate

This should make sure the necessary table (which will store the search pages) is created.

As before, we would like to be able to add the search page as a subpage of the homepage, so we need to add it to the getPossibleChildTypes in the HomePage class, so open src/MyProject/WebsiteBundle/Entity/Pages/HomePage.php, and add it :

    /**
     * @return array
     */
    public function getPossibleChildTypes()
    {
    return array(
        ...
        ),
        array(
        'name' => 'Search Page',
        'class'=> 'MyProject\WebsiteBundle\Entity\Pages\SearchPage'
        )
    );
    }

After adding this snippet, you should be able to add a search page on the homepage in the backend, so go ahead and do that (use Search as title) - and make sure you publish it after it is created.

If you now go to /app_dev.php/en/search in your browser you should see the search page, but currently it's not yet indexing anything, so let's set up the index first, go ahead and run :

app/console kuma:search:setup
app/console kuma:search:populate full

This should create the search index (if it does not exist yet) and populate it as well.

If you would like to take a quick look at what is indexed, you could run the following :

curl -X GET http://localhost:9200/myprojectnodeindex/page/_search

This will list all pages that were indexed in the index that was created.

If you return to the search page, and enter Styles in the search box (or open /app_dev.php/en/search?query=Styles&search=Search in your browser), you should see the Styles page (unless you already deleted it of course, just enter any page title or some content).

How it works

Indexing

All the root nodes will be loaded from the NodeRepository, after which it will recursively load all child Nodes. The configuration will then index each Node's NodeTranslations. From each NodeTranslation, the public NodeVersion will be loaded. Which means only online available pages will be indexed.

From each public NodeVersion, the following information will be indexed :

$doc = array(
    'root_id'            => rootNode->getId(),
    'node_id'            => $node->getId(),
    'nodetranslation_id' => $nodeTranslation->getId(),
    'nodeversion_id'     => $publicNodeVersion->getId(),
    'title'              => $nodeTranslation->getTitle(),
    'lang'               => $nodeTranslation->getLang(),
    'slug'               => $nodeTranslation->getFullSlug(),
    'type'               => ClassLookup::getClassName($page),
);

Type

By default the type will be the class name of the object. You can overridde this by implementing the SearchTypeInterface interface. This will allow you to bundle multiple classes together under the same type.

Parent and Ancestors

In "parent" you will find the ID of the parent Node. In "ancestors" you will find a list of keys consisting of all the parent Nodes ID's (parent of the parent of the parent ...).

Content

The field "content" will contain the bulk text content from the page. By default we check if the page implements the HasPagePartsInterface. If it does, it will iterate over all PageParts and load its template. The content of that template will be rendered and stripped from all tags.

Extra fields

In case you want to index extra content or index addition information, create an EventListener on "kunstmaan_node_search.onIndexNode" which will allow you to manipulate the document before its being indexed.

ID

The document for your page will be given a unique idea which will allow the document to be able to be updated.

Updating

When a page is being updated, an event will be triggered to update the index with the latest information. If your page has been unpublished or deleted, it will also be deleted from the index.

Searching

Extend the AbstractSearchPage and add your new class as a possible child to a page in your website :

    /**
     * @return array
     */
    public function getPossibleChildTypes()
    {
        return array(
            ...
            ),
            array(
                'name'  => 'Search Page',
                'class' => 'MyProject\WebsiteBundle\Entity\Pages\SearchPage'
            )
        );
    }

Modifying the search templates

You can find the template in the normal place where you find your other templates from generated pages.

Search configurations

Adding a search configuration

If you want to index and search your own objects, you will need to create a SearchConfiguration.

Create a new class and implement the SearchConfigurationInterface. Implement the three methods from the interface.

Implement methods

createIndex

This method is expected to create one or more indexes. Elastica has a Mapping class to help create mappings for your index.

    public function createIndex()
    {
    // build new index
    $index = $this->searchProvider->createIndex($this->indexName);

    // create mapping
    foreach ($this->locales as $locale) {
        $this->setMapping($index, $locale);
    }
    }

    /**
     * @param \Elastica\Index $index
     * @param string          $lang
     */
    private function setMapping(\Elastica\Index $index, $lang = 'en')
    {
    $mapping = $this->getMapping($index, $lang);
    $mapping->send();
    $index->refresh();
    }

Refer to the Elastica documentation for more information regarding the Mapping.

populateIndex

The index method will be called upon to populate your index with documents. With the '$indexName' and '$indexType' parameter you can control where this document will be stored.

    public function populateIndex()
    {
    $doc = array(
        "title" => "Test Title",
        "content" => "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur nec lacus tortor, ut ultricies libero. Donec dapibus erat a nisi condimentum viverra."
    );
    $uid = "a_unique_doc_id";
    $this->searchProvider->createDocument($doc, $uid, $indexName, $indexType);
    }

When processing the objects, please be aware there's an interface "IndexableInterface" which provides the "isIndexable()" method. This method should return false when the object in question should not be indexed.

deleteIndex

Delete the index(es) in this method.

    public function deleteIndex()
    {
    $this->searchProvider->deleteIndex($indexName);
    }

Tagged service

Add the SearchConfiguration class as a tagged service to services.yml.

Here's an example from the NodeSearchConfiguration, used to index nodes from the KunstmaanNodeBundle

parameters:
    kunstmaan_node_search.search_configuration.node.class: Kunstmaan\NodeSearchBundle\Configuration\NodePagesConfiguration
    kunstmaan_node_search.indexname: "nodeindex"
    kunstmaan_node_search.indextype: "page"
    kunstmaan_node_search.node_index_update.listener.class: Kunstmaan\NodeSearchBundle\EventListener\NodeIndexUpdateEventListener

services:
    kunstmaan_node_search.search_configuration.node:
    class: %kunstmaan_node_search.search_configuration.node.class%
    arguments: ["@service_container", "@kunstmaan_search.search", "%kunstmaan_node_search.indexname%", "%kunstmaan_node_search.indextype%"]
    calls:
        - [ setAclProvider, ["@security.acl.provider"]]
    tags:
        - { name: kunstmaan_search.search_configuration, alias: Node }

Using the tag "kunstmaan_search.search_configuration", the SearchConfiguration will be added to the SearchConfigurationChain and in turn be called upon when creating, deleting and populating the indexes.

Adding a custom search provider

Want to trade in Elastica for another ElasticSearch library? It can be done by creating a new SearchProvider.

Create a new class and implement the SearchProviderInterface.

Implement methods

createIndex

Create the index

addDocument

Add the document to the index

deleteDocument

Delete the document from the index

deleteIndex

Delete the index

search

The search method allows 2 ways of searching. A standard search which is expected to search the 'title' and 'content' field for the $querystring. When the $json parameter is set to true, the $querystring will contain the full JSON request for ElasticSearch.

Commands

Create indexes by performing the following command. It will iterate over all SearchConfigurations and call the createIndex() method.

kuma:search:setup

Use the following command to populate the indexes. Use the 'full' argument to delete and create the indexes again. This command will iterate over all SearchConfigurations and call the populateIndex() method.

kuma:search:populate
kuma:search:populate full

Next command is used to delete the indexes. It will iterate over all SearchConfigurations and call the deleteIndex() method.

kuma:search:delete

Extras

In standard use cases, indexing the page parts from the main region will be enough. If you want index multiple regions you can add the following in your config.yml.

kunstmaan_node_search:
    contexts: [main, header, footer]