Membase
Set up the basic N-node Membase cluster. This part is trivial. You'll end up with N servers, all running their local Moxi proxy server on port 11211. You'll be able to telnet into any of the servers on that port and set an object. Telnet to another of the servers on port 11211 and you can get the object back out.
memcached-session-manager
The tricky part is that memcached-session-manager does not want to know about the topology of the replica data buckets in the Membase cluster. Leave that job to the Moxi memcached proxy. You get Moxi when you install Membase, or it's available as a separate installer for Linux.
Run moxi on each of your Tomcat hosts (do it for HA, though you will now need to monitor the moxi process as well as your tomcat process from the load balancer - if either dies, the server should be pulled from the pool). We use port 11311 for the moxi to separate it conceptually from the moxi's running in "gateway" mode on the Membase servers on port 11211:
/opt/moxi/bin/moxi -Z port_listen=11311 \
http://192.168.0.1:8091/pools/default/bucketsStreaming/default,\
http://192.168.0.2:8091/pools/default/bucketsStreaming/default
Then in your Tomcat <Context> configuration, you use the local moxi details in the session manager:
<Manager className="..." memcachedNodes="n1:localhost:11311" ... />
What you end up with is...
Tomcat1 -> LocalMoxi --\
\ /--> Membase 1
Tomcat2 -> LocalMoxi ----\ /
>--<
Tomcat3 -> LocalMoxi ----/ \
/ \--> Membase 2
Tomcat4 -> LocalMoxi --/
And each local moxi is aware of the topology of the Membase cluster, and is aware of whcich Membase servers are running and which are down. You will still get errors in your Tomcat logs from the spymemcached library if you kill a Membase server. That will continue until you take a manual step of hitting "remove" on the dead server from the remaining server's console (and then you should rebalance the cluster).
Summary
It's possible this is overkill, but until we introduced the local moxi proxy processes, our Tomcat servers would not handle a Membase server being killed.
