Installing A Single Instance of SolrCloud

After quite a bit of searching I can’t seem to find a simple example of setting up a single instance of SolrCloud on a local machine. I work with Solr every day. I do most of my development on my laptop and when everything is good I commit it and deploy. We recently hired someone who will be working with Solr and I wanted to get his laptop set up to run SolrCloud locally, too. However, I found it difficult to locate a document that I could just point him to. This is that document.

Note that I understand the installation described below is likely to be useful only when doing development. What’s the point of having a distributed SolrCloud when it’s only running on one machine?

See https://cwiki.apache.org/confluence/display/solr/SolrCloud+with+Legacy+Configuration+Files for a list of required SolrCloud configurations.

 ZooKeeper

ZooKeeper (https://zookeeper.apache.org/) is centralized service that is used to coordinate configuration information. You will be telling ZooKeeper where to find SolrCloud configuration files.

SolrCloud comes with an embedded ZooKeeper. However, our production configuration uses ZooKeeper as a stand-alone system and I want to mimic production.

Installation

  • Download ZooKeeper from Apache’s site https://zookeeper.apache.org/
  • Extract the downloaded file.
  • Follow the steps outlined in the getting started guide https://zookeeper.apache.org/doc/r3.3.4/zookeeperStarted.html.  Here are the basics.  Be aware that this may change with future versions of ZooKeeper.
    • Copy ZOOKEEPER_DIR/conf/zoo_sample.cfg to ZOOKEEPER_DIR/conf/zoo.cfg
    • I changed the value of dataDir in ZOOKEEPER_DIR/conf/zoo.cfg to an existing empty directory
    • Start zookeeper: ZOOKEEPER_DIR/bin/zkServer.sh start

Verification

Verify that ZooKeeper is running:

ZOOKEEPER_DIR/bin/zkCli.sh -server 127.0.0.1:2181

You should see a command prompt that looks something like this:

[zk: 127.0.0.1:2181(CONNECTED) 0]

Enter quit to exit the client

[zk: 127.0.0.1:2181(CONNECTED) 0] quit

If you get a java.net.ConnectException: Connection refused error you know the server is not running.

Note: Remember that ZooKeeper port (default is 2181). You’ll need it later.

ZooKeeper and Solr’s Configuration Files

Using the SOLR_DIR/example/scripts/cloud-scripts/zkcli.sh script upload Solr configuration files to ZooKeeper:

SOLR_DIR/example/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd upconfig -confdir SOLR_DIR/example/solr/collection1/conf -confname collection1-config

Remember the value of “-confname” and the name of the collection configured in the path value of “-confdir”.

From this you should see a bunch of output including a list of the configuration files.

Next link the collection to the configuration you just uploaded:

SOLR_DIR/example/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd linkconfig -collection collection1 -confname collection1-config

Again you’ll see a bunch of output but not as much as when you uploaded the configuration.

Be sure to use the same value for the collection name (-collection) and the configuration name (-confname) match the values provided with the ZooKeeper upconfig command above.

Start SolrCloud

We are now ready to start up this instance of SolrCloud.

java -jar SOLR_DIR/example/start.jar -Djetty.port=8983 -DzkHost=localhost:2181 -Dsolr.solr.home=SOLR_DIR/example/solr

Toward the end of the scrolling Solr log messages you should see something that looks like:

63111 [Thread-13] INFO org.apache.solr.cloud.Overseer – Update state numShards=null message={
 "operation":"state",
 "state":"active",
 "base_url":"http://127.0.0.1:8983/solr",
 "core":"collection1",
 "roles":null,
 "node_name":"127.0.0.1:8983_solr",
 "shard":"shard1",
 "collection":"collection1",
 "numShards":null,
 "core_node_name":"core_node1"}

You may now look at the SolrCloud admin page found here: http://localhost:8983/solr/#/

Using Hadoop to Create SOLR Indexes

One of the most challenging projects I faced at work recently was to create a Apache SOLR index consisting of approx 15 million records. This index had been created once in the history of the company using a MySQL database and SOLR’s Data Import Handler (DIH). It had not been attempted since then because the original indexing process was time consuming (12-14 hours), required human supervision, and on failure had to be restarted from the very beginning.
Continue reading

Android apps on my phone

I was the first one at work to get an Android phone. As word got around I have become the guy to go to when it comes to Android. I get asked questions about Android before they buy. I also get questions about Android after they’ve bought their new phone. One thing that people seem to appreciate is when I provide them with a list of apps to get them started.

Without further ado, here is a list of most of the apps I have on my phone. I’ve not included some of the apps (e.g., OEM pre-installed crap).
Continue reading

How I managed to install the CyanogenMod 6.0.0-Droid-RC2 on my Motorola Droid from a Mac Book Pro (OSX v.10.6.4)

First I read the wiki entry http://wiki.cyanogenmod.com/index.php?title=Full_Update_Guide_-_Motorola_Droid As I was reading I noticed that there were no directions for OSX; only for Windows and Linux. It turns out that the flash recovery tools are not available for OSX. That means that I would have to do this from a virtual machine.
Continue reading

r cannot be resolved

In the Google group Android Beginners I frequently see messages that ask what the error "r cannot be resolved" means in Eclipse.

Eclipse generates the file R.java for you using the aapt tool (Official Guide to the Android Asset Packaging Tool). R.java contains a mapping to all the resources your application will use. Note that you should have the Build Automatically option under the Project menu checked on.
Continue reading

To Android or to iPhone?

I’ve wanted an iPhone for at least a year. Six to eight months ago I decided I was going to make the switch from Verizon to AT&T – just for the iPhone. Our Verizon contract expires next month. Now that December 8th is almost here I have begun counting the days. How exciting it will be to walk into the AT&T store and pick up my new iPhone.

I appreciate the quality of Apple products. From the beauty and elegance of the design to the simplicity of the user interface. It is obvious that they put great thought into the product as a whole. I wouldn’t call myself a “fanboy” but I’m pretty close. This is why it has come as a surprise that I have decided to go with the new Droid that Verizon just made available (http://phones.verizonwireless.com/motorola/droid/).
Continue reading