Rosetta offers an IE level export task that can be used with IE level sets and exports the whole IE/Representation. What we need in addition, is a file level export task. It should be possible to export sets of files (itemized sets based on a list of FL IDs, or logical sets based on search criteria, e.g. IE PID and file extension) to a given export path. Subdirectory name could be the set ID.

To date, the only options are file download – one at time – or IE export, which means unpacking the Rep.tar containers and selecting the required files. Both options are rather tedious.

Use Cases: A user requests a set of high-resolution images for publication, e.g. several pages from a digitized book or a collection of images from different sources.

Since version 4, Rosetta includes a feature called AIP update. This feature is used mainly by libraries to alter data in the permanent repository, which might be necessary for a number of reasons. Metadata might need updating due to a typo or due to new research results, new files that weren't included in the original ingest might be added, or existing files might be altered with higher resolution versions or to repair defective files. Libraries tend to use this feature, because they regularly ingest SIPs that include a minimal set of metadata. Later, when full metadata have been allotted or OCR processing has been done, that data is added and Rosetta creates a new AIP version while preserving the original version.

In the status that AIP update has in current Rosetta versions, we see various design flaws that endanger stability, robustness and safety of our data. We'd like to outline the issues that we see and suggest a number of changes to improve on those issues and make AIP update more secure and usable for the Rosetta user community. Please note that we use AIP update exclusively via API, which is necessary for running automated ingest/update/access workflows. SLUB has met all of these issues in real world production.

One key requirement for successfully processing AIP updates in an automated workflow is the ability to distinguish between three alternatives. Any package that is ingested into Rosetta can be: 1. a new ingest, 2. an AIP update of an existing unlocked and finished IE in the permanent repository, 3. undecidable (internal processing in Rosetta hasn't finished yet, so the ingest has to wait for the moment). The basis for this decision is a combination of the data returned from querying the system's APIs, namely the number of SIPs that contain the external identifier (as given in the SIP MD), the number of SIPs that contain the external identifier (as returned by the SRU search through the indexed IE-DC MD) and the processing status of each of these SIPs. To do so, preservation repository software needs to provide gapless tracking of the processing status for each IE right from the ingest. IE metadata contain these identifies, but they are currently only indexed at the very end of the processing after moving the IE to the permanent repository. Until that happens, we cannot use the SRU search (which relies in the Solr index) to find out if an IE with a given external identifier is already in the system, regardless of its current processing status. Due to this limitation, we only ingest exactly one IE with each SIP, so there's always a distinct mapping from SIP ID to IE PID and vice versa. That way, we could use the SIP metadata that is stored in the database to find out if there's an IE with a given external identifier in the system even before it reaches the permanent repository. Until recently, we used a custom built workaround that is no longer supported in Rosetta 5.2, including a BIRT report and adventurous authentication methods to cover most of the gap between LOADING stage and indexing. However, there was a remaining gap between the moment when the SIP status was set to "module PERMANENT, stage FINISHED, status FINISHED" and the moment when the index processing has been completed. This gap was supposed to be closed in Rosetta 5.1, when a new API was created to replace the BIRT report and create a stable and supported solution. However, the new API does not cover the functionality of the BIRT report, making it unusable for controlling automated AIP updates. The legacy BIRT report solution returned all SIPs in the system for a given external ID, including legitimate SIPs as well as rejected and declined SIPs. The new API, however, only returns a SIP's status if exactly one SIP ID for a given external ID was found. If there's more than one SIP for a given external ID, Rosetta just returns an error instead of returning a status list for all SIPs that were found.
The solution for this situation would be as follows:
- Move the indexing of all metadata as far to the beginning to the SIP processing as possible. Performance considerations, which were used to justify keeping the indexing of the IE metadata at the very end of the processing and even running it after the SIP is in "module PERMANENT, stage FINISHED, status FINISHED" (so, practically lying to the user and external software), are not valid on recent hardware and do not justify neither the need for workarounds nor the danger imposed on the preservation data.
- Index SIP MD as well. As far as I saw in the Rosetta 5.3 announcements, this is planned for the next version and will solve some of hte problems that we're having.
- Enhance the getSIPStatusInfoByExternalId() API (documented here https://developers.exlibrisgroup.com/rosetta/apis/Sipweb services) to provide the functionality needed and described above.

Any information about exactly which files are supposed to be updated with which replacements are currently kept in a CSV file stored below /operational_shared/. In in our environment, this is an NFSv3 share that is mounted with nolock option set due to ExLibris requirements (which, in itself, would be dangerous enough). However, as writing to files on network shares is a non-atomic operation, the CSV files can get corrupted if there's a network glitch. This will result in an AIP update going wrong with no indication whatsoever, as we had to experience first hand. From our point of view, the correct way to do this would be by keeping all AIP update information in the database, where operations are transactional and can be recovered a lot easier even if there's an issue with the database (i.e. via Oracle FlashBack Recovery or something similar). This approach would offer multiple advantages:
1. The AIP update information is kept safe in the database.
2. The information is kept in just one place instead of splitting it between the database and the file system.
3. Network load on the NFS shares is reduced.
4. The database handles concurrent access a lot better than NFS shares do.
Taking the importance of the data in Rosetta installations around the world and the perspective preservation durations into consideration, its bordering irresponsible to prefer file operations over database operations for control mechanisms.

After calling the updateRepresentation() web service, Rosetta returns a RIP ID that can be used to see the status of the process. While this is a good idea in general, there are some issues with the actual implementation. As of Rosetta 5.1.0.2, the RIP ID is deleted from Rosetta immediately after the AIP update process completes. Small AIP updates tend to finish really fast, so if the submission application queries Rosetta about that RIP ID for the first time, it might already be gone and Rosetta returns an error indicating that the RIP ID is unknown. We have contemplated using this behaviour as an indicator that the AIP update has finished successfully, but according to ExLibris, this wouldn't be acceptable even for a workaround (their words). Anyway, the correct behaviour for a well-designed API would be to:
- issue a RIP ID on updateRepresentation() call
- return a "running" status for that RIP ID as long as the process hasn't finished
- return an "error" status for that RIP ID if anything unexpected occurs
- return a "finished" status for that RIP ID as soon as the AIP update process has finished successfully
- return any other status messages that might be needed
- EITHER delete only after at least 24h after successfully finishing the AIP update OR (preferred) never delete RIP IDs and use suitably large data type (i.e. long unsigned integer) for these IDs to make sure the system never runs out of IDs
- return an "unknown RIP ID" message only if the RIP ID actually has never existed or doesn't exist any more
SupportCase 00366236 "Error in RIP Loading, IE (FileNotFoundException: Unable to read file: Import Descriptor.csv)" describes some more details about how we found out about this issue.
In any case, software using the API must be 100% confident about the process status of AIP updates the manipulate the actual repository master data.
From what we see in the Rosetta 5.2 changelog, ExLibris has implemented some changes in this area. We have yet to test the extent and reliability of the implementation.

From what we understand about the inner workings of Rosetta, some of Rosetta's internal tasks are handled via web services as well. However, these web services are undocumented. We learned about one of these web services when we encountered a failed AIP update (SupportCase 00376523 "failed AIP update on IE3761638 [Rosetta 5.0.1.1]", error message: 'SocketException from an internal web service "PermanentManagerWS": java.net.SocketException: SocketException invoking http://langzeitarchiv.slub-dresden.de/dpsws/permanent/PermanentManagerWS: Connection reset'). The search for the root cause is ongoing, but our assumption is that an internal timeout for this web service hit us before the process had finished. We only noticed this error because we regularly grep the server.log for error messages; Rosetta itself didn't give any indication about something being wrong with our AIP update whatsoever. This means that we nearly missed a failed process that is supposed to alter master production data with the AIP still being locked in mid-process. We suggest to document all web services and improve API error notification to make sure cases like this one are caught even in fully automated production scenarios. For us, this is especially critical because of our specific production workflow where multiple AIP updates are needed to represent one change that came for image production. If one of those updates fails, the AIP ends up in a semantically inconsistent state where medatada might point to files that don't exist yet and were supposed to be added in the subsequent AIP update.

[continued in comments]

we are using the Rosetta-Aleph-Integration in both directions:
1. creating the CMS-record in Rosetta and
2. setting the Rosetta-flag in Aleph as well as updating the Aleph record with the URN created in Rosetta.
In the past there was only one target field for the URN and the assignCMS task with updateURNTask worked without problems.

Due to changes in the cataloging rules - we are using now the international RDA-rules - we have to deal now with different target fields in Aleph for the URN depending on the type of material: e-only e-books, digital reproductions of printed books, online versions of printed books, etc.

The target field is configurable in the Rosetta-Aleph-Sync, but this is not sufficient with the new situation. We want to be able to configure profiles for the updateURNTask depending on the material type: to specify the target MARC-field, to populate subfields with fixed information depending on the material type and with the URN.
(It is not possible at the moment to update the URN in a subfield of an already existing field, a new field is created (case 00510676)).

Astrid Schoger on behalf of the Bavarian State Library

Right now when you use the Create Derivative Copy task you have to select a plug-in type (either Transformation Profile or Stream Handler) even in cases where you are not transforming anything - we create derivative copy representations that are exact copies of the preservation master representation with no transformations. I would like to have a 3rd radio button - something like "None" - added. Since we're not selecting any file formats and we are choosing "Copy other extensions" the system is doing what we need it to do and we think it's odd to have to choose a plugin type when it's not being used. Low priority, but worth mentioning I thought. Thanks!

For compliance purposes and as a lesson learned from SC 00560356, I suggest improving the documentation and configurability of passwords throughout subsystems of Rosetta installations.

I suggest creating and maintaining a KB article on how and where to change passwords for:
- System user "dps"
- System user "oracle"
- Oracle user "sys"
- default Rosetta user "admin1" (John Smith)
- Cantaloupe Image Server user "admin" (changeable by configuring ".../system.dir/thirdparty/tomcat/rosetta-webapps/cantaloupe.war/cantaloupe.properties")
- Solr user
- Rosetta console UI user

The passwords for Solr and the Rosetta console UI cannot be changed at the moment (Rosetta 5.5.0.0); I suggest considering changes to enable institutions to set their own passwords.

Furthermore, I suggest that ExLibris set up suitable infrastructure for exchanging passwords securely. GPG-encrypted mail or a public write-only (!) storage would be suitable candidates, SupportCase comments (that might be publicly readable) and encrypted mobile instant messaging services like Signal (that often involve personal accounts/devices) are certainly not.

Increasingly libraries and consortia are using counter / sushi to measure usage of digital and electronic resources across disparate parts of their holdings.

If holdings being held in Rosetta are to be valued by the broader institutions funding and maintaining those resources, we need to be able to report usage using the counter and sushi protocols (which ExLibris has already demonstrated knowledge of in their support in ALMA and 265 services).

Key features would be basic counter and sushi reporting; use of IP ranges and user groups already defined in Rosetta for reporting; and grouping or content by already defined sets and/or collections for reporting.

submitted on behalf of Michele Lindlar

Sometimes it's neccessary to update our cms metadata. if a librarian changes a record in the external cms-system, we want to update the record in rosetta too, but there's no task that works accordingly. we'd be really happy if there would be a task chain or a process that re-harvest existing cms-records.

We currently integrate two catalogue systems via SRU in Rosetta: Aleph and Pica. Both systems are integrated uni-directoinal, as in a Rosetta flag is not available in either cataloguing system. As we are integrating the union catalogues which are held by third parties, we are not at liberty of including any Rosetta-specific fields in the catalogue.

The initial CMS enrichment works without a problem. However, at times it becomes necessary to update the CMS information stored in Rosetta. This can be the case when either the catalogue was updated or when we have detected an error in our mapping.

While two tasks are available which suggest a working update - namely, "Re-Assign CMS" and "CMS Metadata Record Update" - these tasks will not re-harvest from the source CMS included in the SRU integration but will instead always pull from the physical MID copy already stored in the /metadata storage of Rosetta.
Actual re-harvesting is only possible by first manually deleting the MID record in the file storage and then re-harvesting.

ExL has provided us with a script which can be run across the file system. The script requires a list of CMS Ids first and then deletes the records from the storage. The script needs to be run by the administrator from the shell.

We do not consider the script solution sufficient. Instead, we require a functionality of re-enriching with updating CMS records from the integrated CMS system from within Rosetta.

Rosetta knows two CMS integration options: unidirectional and bi-directional. The bidirectional integration requires a Rosetta flag to be set in the CMS system. Via the flag, all Rosetta-relevant records from the CMS can be gathered and the metadata pool within Rosetta updated in bulk on a regular basis. However, institutions may be using a CMS for enrichment which belongs to a different / external institution, making the addition of a Rosetta-flag within the CMS system impossible.
This is where the unidirectional enrichment comes to play. The CMS is queried via SRU integration and records are pulled into Rosetta. The pre-requisite is that a CMS section is present within the ie.xml, containing a system name and identifier to automatically query the CMS. This works well exactly once - duirng the first enrichment of the IE from within Rosetta. While all required information to re-query the CMS is present within Rosetta, any re-enrichment attempts via Task Chains such as "Re-assign CMS" or via manual "remove" / "link" will not requery the CMS system but always pull the information from the rosetta-side shared metadata directoy containing the initial harvest. Not only is this behavior counter-intuitive, it also prohibits the update of CMS data for those institutions who do not have control over the source CMS they are enriching from.
Ex Libris has provided a script via which the re-enrichment of these records is possible. The script takes a list of CMS identifiers as an input parameter and returns newly harvested CMS records into the shared metadata folder. However, running the script requires shell access to the application servers and can therefore only be done by system administrators.
It seems that all the basic functionatily for re-enrichment is there, it's just not available in the UI. The users who typically conduct re-enrichment of CMS data, however, do not work via the shell, they work in the UI and expect the functionality to be included there.

Our idea proposed here is to make the re-enrichment mechamism avaiable via the script actionable via the UI. We urgently require this functionalilty as the experience of the past years has shown that CMS records DO change and that these changes are monitored by Rosetta staff users such as participating institutions in a consortia, who do not have sufficient rights to run shell scripts for updates.

We do not wish to have all of our institutional users in Rosetta, leaving that for administrators and publishers.

When a user has signed in successfully through SSO (in our case ADFS) we know that they are from our institution, so we should be able to do access rights rules based on this.

We are also able to pass through extra parameters through SAML to identify the type of user, and where they have come from, it would be great to be able to assign such users to basic user accounts so we can have granular access rights to IE's and Reps within Rosetta.