Here at SLUB, we're seeing a few issues with the way indexing is implemented in Rosetta.
Currently, all Rosetta servers that are in the "Indexer" role hold a single part of the whole index with no redundancy between the indexing nodes. If there's an outage in one of the servers, there are two effects.
N°1: One third of the index entries will not be available and the missing server will not take part in indexing any new IEs, obviously. The IE that was just being processed by this server will not be in the index and information about it will not be available, even though it has already reached the permanent repository and is in status "module PERMANENT, stage FINISHED, status FINISHED".
N°2: As the remaining Rosetta servers continue to work like nothing happened, they might mistake an AIP Update coming from the Submission Application for a new ingest, thus generating duplicate IEs. This is due to the fact that Submission Applications can only rely on the data returned by the web service API, which in turn queries the index. Incomplete index data will hence lead to wrong behaviour in the Submission Application. This can result in duplicates of IEs with the same descriptive MD and external identifiers that represent different versions of the same package coming from the production workflow, some of which may even be incomplete due to differential AIP updates.

Reliable index data is extremely important for controlling fully automated workflows. SRU is the only way to find the IE PID for a given external identifier. The IE PID, in turn, is needed to call the updateRepresentation() and exportIE() APIs. As long as there's no API call that returns the IE PID for a given external ID by querying the database and not relying on the index, the Solr servers need to be highly available in order for any automated workflow to function flawlessly. Apart from the problems with ingest/AIP update mentioned earlier, incomplete index contents will also lead to wrong reporting results, which in turn leads to wrong billing and status information being delivered to tenants/3rd parties. This is even more important when you take the announced changes in Rosetta 5.3 into consideration, where indexing will be expanded to include SIP MD.

To date, there's no easy way of knowing that one of the indexing services has crashed or is unavailable. There's neither a warning or an UI notification given to the user like on other errors, nor (and this is even more relevant for highly automated workflows like ours) via SNMP where we could use established monitoring tools to monitor service health. An inoperative or incomplete index is not usable for automatic ingest workflows and should make the Submission Application pause all ingests immediately.

Also, there's no redundancy between the local indexes held on the indexing servers. This is highly undesirable from an operations point of view, as redundancy is key when working to increase availability. Luckily, Solr already has all the prerequisites for failover built into it, so ExLibris should be able to implement an appropriate solution without any negative impact and with minimum effort. More information can be found at https://wiki.apache.org/solr/SolrReplication.

We are aware that the Index Status can be seen at "Rosetta Administration → Repository → Index Status", but it's rather buried there and critical failures are not visible at first sight on the Dashboard. During the last outage, only our own homebrew server.log error monitoring script gave a warning the day after by sending us the relevant error message from the log.

This idea is based on Support case #00374004, Title: "Solr server down [Rosetta 5.0.1.1]" and #00173827 "indexing jobs not balanced [Rosetta 4.2.0.2]".