firstly I would take a look at the HornetQ user manual, it will explain all in there.

1. are 1-3 above possible? can someone share documentation?

do you mean automated? starting, stopping servers is a manual task (or something you write yourself) but yes we support replication via live/backup servers, again see the manual.

for questions 3,4 and 5 you havent given any context so i dont really know what you are asking

Not clear from the user manual if you can add as many nodes as you want or not.

you can add as many as your network.OS resources can handle. we dont limit.

regarding 3-5 - I need to address a concern where all clients (let them be producers or consumers of any kind) to be able to 'talk' to all servers. AFAIK, from reading current manual, it is only possible to setup a 1:1 relationship between a client and a server. So I am asking how can I do that.

Im not sure what you are asking makes any sense. A connection is a physical connection to a node in a cluster, coould you explain what you mean by 'talk to' or what you are trying to achieve.

What I need is basically the ability to prevent HornetQ from being a bottleneck on very high loads. For this I need an option for a large cluster (and not only a master backup pair). Is that supported? From the user manual it looks like only HA is supported?

we support clustering and HA via active/passive pairs. The user manual explains what the difference is.

Second thing is the ability for clients to connect to such a cluster where any producer may send to any server and any consumer can receive from any server. If this is possible we can start as many nodes as needed and clients can roundrobin between them.

again that really makes no sense. a client will only send or receive messages to/from the server it is connected to. how messages are handled within a cluster depends on your topology.

It looks like for producers this is easy to achieve using a 'RoundRobinConnectionLoadBalancingPolicy', but would we support this for consumers?

again there is no connection between producers/consumers and client side load balancing, client side load balancing is done at the connection level.

If you could explain exactly what you are trying to achieve maybe i can provide better advice?

ok. so for starters there is not limit on active/passive pairs? and each pair has an optional replication between active and passive, but no replication between pairs. is that correct?

like i said before, we dont limit the size of a cluster. regarding replication being optional that is not true, a live/backup (active/passive) pair must be configured to use either replication or shared store.

for clients - let's assume we've found clients can consume 1000 messages per second. let's assume producers send 50000 messages per second. assuming clients are durable subscribers - how do we react to this load?

what do you mean 'how do we react", how many clients a node can cope with depends on what resources it has, i.e. disc speed, network speed, memory etc and how it is configured.

to be honest Im not really sure what you are asking, but here are a few main pointers

1) a cluster is a set of live servers that bridge with each other to support message distribution within that cluster.

2) if you want High Availability you need to configure backup servers to take over the work of a crashed live server.

3) clients connections are load balanced across the cluster round robin (best efforts).

4) messages are distributed (or redistributed) to nodes within the cluster so all consumers receive messages.

by optional replication I means you can use shared storage which is not replication.

ok thats fine.

I don't understand how a durable subscriber mode works in a cluster? Assume producer publishes message A to all subscribers. I can't afford to have only one client application processing these messages. The reason is I have 3 subscriber applications that all need to read the message data. In classic durable subscriber topology, I would have 3 processes each with its unique id - each application getting one copy of the published message A.

and thats what happens, it doesnt matter which server a message is routed to or which node a subscription is on they will all receive the message.

Problem: my application can only handle 1000 messages per second, but the publisher, publishes 50000 messages per second. Solution in HTTP world: run 50 instances of your client application and they will handle together the 50000 load. Problem is with durable subscribers - I can't start more than one process registered with the same unique id.

well (pre jms 2) you can only have 1 actual subscriber which means that that 1 consumer would be a bottle neck since it a) is on one single node and b) there is only 1. this was resolved in JMS2 where you can have shared subscribers (or use the core API in HQ 2.4 and above).

So - assuming I have a cluster of 20 HQ servers and 50000 messages are distributed between these server nodes, how do I start 50 client processes to handle this load and prevent the queue from fulling up?

This is really an application problem, how you create consumers and how many you have is completely up to you. Altho if you were using an application server you could deploy MDB's in each new node.

JMS 2 support is news to me. Can you share some links so I can read about it? Additionally can you point me on to how core API can be used to achieve this?

just google the JMS 2 API, look for createSharedSubscriber on session. for the core API we just create a queue as there are is no such thing as topis in teh core API. we just have addresses and queues bound to addresses and consumers on queues so you can create any topology you want.

So - you're saying I can do something like the following:

 

setup cluster with 20 nodes - 10 servers with 10 replicas.

 

producer can roundrobin the 10 servers whilst publishing messages and all messages will be evenly distributed (assume roundrobin policy) between the server nodes.

 

consumers (even durable) can consume messages from different server nodes.

 

in case a server fails, HQ will failover to the backup server seamlessly to the client.

 

is this description correct?

yes, note that how we deal with distributing messages depends certain settings such as forwardwhennoconsumers, take alook at the user manual