You should cross post this in the ISPN community, they might be able to help some more. One possible cause is attempting to read some existing data stored with an earlier version of ISPN with a newer version.

Is it by the way sensible to have both distributed modeshape nodes pointing to the same network filestore? It sounds like it could lead to data corruption problems, that might cause my issue - but my client prefers to set it up that way.

I'm not aware of anyone having tried this configuration yet with ModeShape. However, the far more common configuration is to use a JDBC shared cache store. So in theory, sharing a form of NFS *should work*, as long as the cachestore is marked as "shared=true".

As a side note, I don't think a distributed topology with ModeShape offers any benefits over a replicated one. In fact, I'd recommend 99% of the time a replicated topology (see Clustering - ModeShape 4 - Project Documentation Editor)

This confuses me because the Modeshape docs seem to encourage it:

ModeShape doesn't recommend any type of store in particular. What we do recommend is that whatever store you choose for the cluster, *it is configured/used as a shared store*. In other words, your cluster should never attempt to persist data in multiple locations, but should only use a central location. This is because transactions (which are required by ModeShape) aren't reliable across multiple stores (for the reason mentioned in the documentation).

Does this advice only apply to JDBC datastores? What is the recommended approach when using a File Store (given that sharing a filestore is apparently not recommended)?

The most commonly used store in a ModeShape cluster AFAIK is the JDBC store (database backed). You could try LevelDB (I don't know if ISPN recommends it in a cluster) but otherwise it seems JDBC is the only option.

Does it make a difference if the topology is distributed or replicated?

In my view, there are 2 cases when you should choose distributed over replicated (in all other cases replicated is the best choice):

* if the amount of data (i.e. JCR nodes) is so big that having it replicated in-memory on each cluster node is not an option (i.e. there is a memory constraint)

* you're running a large data-grid type cluster where everything is held in-memory and there is no persistent store. This means however that your environment has to make sure enough cluster nodes are always up & running at any given time so that no data is lost in case of unexpected failures.

The 2nd case is IMO a purely theoretical one because if you're planning on doing that, JCR is not a good choice. JCR requires strong consistency and such a large distributed system will perform very poor because of all the required synchronization.

As a side note - it wasn't (at least to me) very clear in the docs that the recommended architecture is a JDBC cache store - most of the config examples use a FileStore (or occasionally LevelDB), even on one of the Clustering pages

Our clustering tests are unit tests which run in the same process (JVM). As such, the easiest and fastest option was to use a separate FS location for each cluster node. A production system is a completely different topic....

I suggest we add a ticket to improve the docs to focus more on JDBC cache store config.

The fact that ISPN doesn't recommend sharing a NFS file store or using a shared LevelDB store is something new to me (none of their docs mention this). Also, I've never seen any official ISPN recommendation as to what shared store is preferred in a cluster. This something that the ISPN documentation should contain really (and I agree that the ModeShape docs should cross-link that). The fact of the matter is that the only ModeShape requirement (which is documented) is that the cache store be configured as shared (again due to ISPN's limitations)

This is in fact another good example of the complications of using Infinispan (if you haven't already done so, I recommend reading this ModeShape 5 and beyond, which essentially means that our focus on Infinispan stops with ModeShape 4).