Since version 4, Rosetta includes a feature called AIP update. This feature is used mainly by libraries to alter data in the permanent repository, which might be necessary for a number of reasons. Metadata might need updating due to a typo or due to new research results, new files that weren't included in the original ingest might be added, or existing files might be altered with higher resolution versions or to repair defective files. Libraries tend to use this feature, because they regularly ingest SIPs that include a minimal set of metadata. Later, when full metadata have been allotted or OCR processing has been done, that data is added and Rosetta creates a new AIP version while preserving the original version.

In the status that AIP update has in current Rosetta versions, we see various design flaws that endanger stability, robustness and safety of our data. We'd like to outline the issues that we see and suggest a number of changes to improve on those issues and make AIP update more secure and usable for the Rosetta user community. Please note that we use AIP update exclusively via API, which is necessary for running automated ingest/update/access workflows. SLUB has met all of these issues in real world production.

One key requirement for successfully processing AIP updates in an automated workflow is the ability to distinguish between three alternatives. Any package that is ingested into Rosetta can be: 1. a new ingest, 2. an AIP update of an existing unlocked and finished IE in the permanent repository, 3. undecidable (internal processing in Rosetta hasn't finished yet, so the ingest has to wait for the moment). The basis for this decision is a combination of the data returned from querying the system's APIs, namely the number of SIPs that contain the external identifier (as given in the SIP MD), the number of SIPs that contain the external identifier (as returned by the SRU search through the indexed IE-DC MD) and the processing status of each of these SIPs. To do so, preservation repository software needs to provide gapless tracking of the processing status for each IE right from the ingest. IE metadata contain these identifies, but they are currently only indexed at the very end of the processing after moving the IE to the permanent repository. Until that happens, we cannot use the SRU search (which relies in the Solr index) to find out if an IE with a given external identifier is already in the system, regardless of its current processing status. Due to this limitation, we only ingest exactly one IE with each SIP, so there's always a distinct mapping from SIP ID to IE PID and vice versa. That way, we could use the SIP metadata that is stored in the database to find out if there's an IE with a given external identifier in the system even before it reaches the permanent repository. Until recently, we used a custom built workaround that is no longer supported in Rosetta 5.2, including a BIRT report and adventurous authentication methods to cover most of the gap between LOADING stage and indexing. However, there was a remaining gap between the moment when the SIP status was set to "module PERMANENT, stage FINISHED, status FINISHED" and the moment when the index processing has been completed. This gap was supposed to be closed in Rosetta 5.1, when a new API was created to replace the BIRT report and create a stable and supported solution. However, the new API does not cover the functionality of the BIRT report, making it unusable for controlling automated AIP updates. The legacy BIRT report solution returned all SIPs in the system for a given external ID, including legitimate SIPs as well as rejected and declined SIPs. The new API, however, only returns a SIP's status if exactly one SIP ID for a given external ID was found. If there's more than one SIP for a given external ID, Rosetta just returns an error instead of returning a status list for all SIPs that were found.
The solution for this situation would be as follows:
- Move the indexing of all metadata as far to the beginning to the SIP processing as possible. Performance considerations, which were used to justify keeping the indexing of the IE metadata at the very end of the processing and even running it after the SIP is in "module PERMANENT, stage FINISHED, status FINISHED" (so, practically lying to the user and external software), are not valid on recent hardware and do not justify neither the need for workarounds nor the danger imposed on the preservation data.
- Index SIP MD as well. As far as I saw in the Rosetta 5.3 announcements, this is planned for the next version and will solve some of hte problems that we're having.
- Enhance the getSIPStatusInfoByExternalId() API (documented here https://developers.exlibrisgroup.com/rosetta/apis/Sipweb services) to provide the functionality needed and described above.

Any information about exactly which files are supposed to be updated with which replacements are currently kept in a CSV file stored below /operational_shared/. In in our environment, this is an NFSv3 share that is mounted with nolock option set due to ExLibris requirements (which, in itself, would be dangerous enough). However, as writing to files on network shares is a non-atomic operation, the CSV files can get corrupted if there's a network glitch. This will result in an AIP update going wrong with no indication whatsoever, as we had to experience first hand. From our point of view, the correct way to do this would be by keeping all AIP update information in the database, where operations are transactional and can be recovered a lot easier even if there's an issue with the database (i.e. via Oracle FlashBack Recovery or something similar). This approach would offer multiple advantages:
1. The AIP update information is kept safe in the database.
2. The information is kept in just one place instead of splitting it between the database and the file system.
3. Network load on the NFS shares is reduced.
4. The database handles concurrent access a lot better than NFS shares do.
Taking the importance of the data in Rosetta installations around the world and the perspective preservation durations into consideration, its bordering irresponsible to prefer file operations over database operations for control mechanisms.

After calling the updateRepresentation() web service, Rosetta returns a RIP ID that can be used to see the status of the process. While this is a good idea in general, there are some issues with the actual implementation. As of Rosetta 5.1.0.2, the RIP ID is deleted from Rosetta immediately after the AIP update process completes. Small AIP updates tend to finish really fast, so if the submission application queries Rosetta about that RIP ID for the first time, it might already be gone and Rosetta returns an error indicating that the RIP ID is unknown. We have contemplated using this behaviour as an indicator that the AIP update has finished successfully, but according to ExLibris, this wouldn't be acceptable even for a workaround (their words). Anyway, the correct behaviour for a well-designed API would be to:
- issue a RIP ID on updateRepresentation() call
- return a "running" status for that RIP ID as long as the process hasn't finished
- return an "error" status for that RIP ID if anything unexpected occurs
- return a "finished" status for that RIP ID as soon as the AIP update process has finished successfully
- return any other status messages that might be needed
- EITHER delete only after at least 24h after successfully finishing the AIP update OR (preferred) never delete RIP IDs and use suitably large data type (i.e. long unsigned integer) for these IDs to make sure the system never runs out of IDs
- return an "unknown RIP ID" message only if the RIP ID actually has never existed or doesn't exist any more
SupportCase 00366236 "Error in RIP Loading, IE (FileNotFoundException: Unable to read file: Import Descriptor.csv)" describes some more details about how we found out about this issue.
In any case, software using the API must be 100% confident about the process status of AIP updates the manipulate the actual repository master data.
From what we see in the Rosetta 5.2 changelog, ExLibris has implemented some changes in this area. We have yet to test the extent and reliability of the implementation.

From what we understand about the inner workings of Rosetta, some of Rosetta's internal tasks are handled via web services as well. However, these web services are undocumented. We learned about one of these web services when we encountered a failed AIP update (SupportCase 00376523 "failed AIP update on IE3761638 [Rosetta 5.0.1.1]", error message: 'SocketException from an internal web service "PermanentManagerWS": java.net.SocketException: SocketException invoking http://langzeitarchiv.slub-dresden.de/dpsws/permanent/PermanentManagerWS: Connection reset'). The search for the root cause is ongoing, but our assumption is that an internal timeout for this web service hit us before the process had finished. We only noticed this error because we regularly grep the server.log for error messages; Rosetta itself didn't give any indication about something being wrong with our AIP update whatsoever. This means that we nearly missed a failed process that is supposed to alter master production data with the AIP still being locked in mid-process. We suggest to document all web services and improve API error notification to make sure cases like this one are caught even in fully automated production scenarios. For us, this is especially critical because of our specific production workflow where multiple AIP updates are needed to represent one change that came for image production. If one of those updates fails, the AIP ends up in a semantically inconsistent state where medatada might point to files that don't exist yet and were supposed to be added in the subsequent AIP update.

[continued in comments]

The migration plugin is part of preservation and is the way in Rosetta to migrate bitstreams from an obsolete format to an uptodate format. It is a script plugin receiving a file and delivering the newly migrated file.
For purpose of transparency it would be necessary to add a report about what has changed. For example when migrating pdf to pdf/a it might happen, that a font was included or even replaced. Having a report would implement a well documented preservation workflow. Depending on the migration tool the report might be quite complex.

So it would be nice to add this report as sourceMD on file level as part of the migration plugin.

There is a way to have Primo index the full text of PDFs that are in Rosetta which can help a user find which Rosetta IE has a document containing their term/phrase but then once the user clicks through to the Rosetta viewer there is no way for the user to know where their search term appears. Having the search term come through to the viewer would be great, but that would only work when there are a few PDFs. For IEs that contain dozens of PDFs the user would have to open each one and search for their term. It would be great if there were a way to search the full-text of all files in an IE at once in the viewer.

Rosetta offers an IE level export task that can be used with IE level sets and exports the whole IE/Representation. What we need in addition, is a file level export task. It should be possible to export sets of files (itemized sets based on a list of FL IDs, or logical sets based on search criteria, e.g. IE PID and file extension) to a given export path. Subdirectory name could be the set ID.

To date, the only options are file download – one at time – or IE export, which means unpacking the Rep.tar containers and selecting the required files. Both options are rather tedious.

Use Cases: A user requests a set of high-resolution images for publication, e.g. several pages from a digitized book or a collection of images from different sources.

Currently, when you export a file, REP or IE from Rosetta you will get an error if there is a problem with exporting the file. However, we have had at least one instance where the file seemed to export without issue but the file couldn't be opened (it was corrupt) and we were able to see that the checksum of the file did not match the checksum stored in Rosetta for that file. This helped us diagnose an issue with the way our storage layer was delivering the file. If Rosetta included an optional checksum calculation on export (the type of checksum could be specified in the export job parameters) this would make the exported files more trustworthy.

Please consider adding this feature.

Researchers often use zip or tar files in order to upload large amounts of research data in various formats. Bytestream extraction is a useful tool for analyzing the content of these container files. The resulting bitstream information is written in the METS and displayed in the BitStreams Tab in the Web Editor. The information includes BS ID, file name, file original path, extension, mime type and format ID (if successfully identified).

We would like to suggest the following improvements for the BitStreams Tab and the bitstream information in general:

1) Include Bitstream size. The relevant values can be found in the table “HDBITSTREAMREF” in the field “FILESIZEBYTES”.

2) Display bitstreams in order of path/filename: the bitstream list should reflect the order of the files in the zip or tar file. Currently they are displayed in random order.

3) Make bitstream information searchable

I would like an easier was to see the size of an IE/REP both in number of files and storage size. Right now, the only way I can see to get the size of the IE in terms of storage is to search at the file level for all files in that IE and then export the results to CSV and add up the size column (in bytes). Same goes for number of files except you can also see them when you’re in the Meditor in the metadata for the Rep. It doesn't seem like adding at least the number of files per REP as an option for a column in the results screen would be that difficult. Thanks!

Rosetta knows two CMS integration options: unidirectional and bi-directional. The bidirectional integration requires a Rosetta flag to be set in the CMS system. Via the flag, all Rosetta-relevant records from the CMS can be gathered and the metadata pool within Rosetta updated in bulk on a regular basis. However, institutions may be using a CMS for enrichment which belongs to a different / external institution, making the addition of a Rosetta-flag within the CMS system impossible.
This is where the unidirectional enrichment comes to play. The CMS is queried via SRU integration and records are pulled into Rosetta. The pre-requisite is that a CMS section is present within the ie.xml, containing a system name and identifier to automatically query the CMS. This works well exactly once - duirng the first enrichment of the IE from within Rosetta. While all required information to re-query the CMS is present within Rosetta, any re-enrichment attempts via Task Chains such as "Re-assign CMS" or via manual "remove" / "link" will not requery the CMS system but always pull the information from the rosetta-side shared metadata directoy containing the initial harvest. Not only is this behavior counter-intuitive, it also prohibits the update of CMS data for those institutions who do not have control over the source CMS they are enriching from.
Ex Libris has provided a script via which the re-enrichment of these records is possible. The script takes a list of CMS identifiers as an input parameter and returns newly harvested CMS records into the shared metadata folder. However, running the script requires shell access to the application servers and can therefore only be done by system administrators.
It seems that all the basic functionatily for re-enrichment is there, it's just not available in the UI. The users who typically conduct re-enrichment of CMS data, however, do not work via the shell, they work in the UI and expect the functionality to be included there.

Our idea proposed here is to make the re-enrichment mechamism avaiable via the script actionable via the UI. We urgently require this functionalilty as the experience of the past years has shown that CMS records DO change and that these changes are monitored by Rosetta staff users such as participating institutions in a consortia, who do not have sufficient rights to run shell scripts for updates.

For error-detection and workflow documentation it is sometimes necessary to be able to view all configurations in Rosetta - even all in the Administration area.
But not all users should be able to edit configuration elements in Rosetta, especially not in the Administration Area.
In our (ZBW/Germany) opinion, a view-only-user would be very handy. This person would be able to see all the configuration details and the view-option would be available to her/him, but not the edit-button.
She/he would not be able to break anything, but would be able to see all the details needed for documentation or other purposes.
This is escpecially important as sometimes configuration elements seem "to change themselves" when more than one tab in the browser is opened, even if only the view-option is selected.
Thank you

ExLibris' current mode of releasing new Rosetta versions is to announce a rough time slot ("April this year") for that new version/minor-release/servicepack/hotfix, release it and deliver a list of changes along with the new release. Up until the actual release, ExLibris will only give a very general roadmap of the planned changes. While this is an easy approach for ExLibris, it's hard for the customers to prepare for the release. Therefore, I propose for ExLibris to provide pre-release versions of Rosetta in order to enable developers to adapt their software (i.e. Submission Applications) to new features, changed API behaviour etc. These beta releases could also be used to gather customer feedback on the changes and improve features before the final release, just like it has been a best practice throughout the whole industry for many years now.

If, for any reason, the suggested developer preview releases are not feasible, I suggest that ExLibris at least releases their interface documentation for new APIs in order to enable developers to begin with their work prior to the actual release. API interfaces have to be specified very early in the design process, so they should be available early on. Developers could mock answers and write their routines before the release, and would only have to do some final tests when the actual working API becomes available.

From my point of view, both ExLibris and the community of customers would benefit from this approach.

In my opinion, the following prerequisites need to be fulfilled to implement this idea and reap the full benefits:
- smaller & more frequent patches, massively simplified/automated install process to enable fast on-site implementation
- rollback over multiple versions including DB schema (In test/dev/staging environments, the Rosetta install script could even take care of backing up the Oracle database using SSH and rman.)
- detailed change information: What new changes does the beta version contain? Which changes are in alpha/beta/release-candidate status? This includes online documentation of in-development features.
- detailed compatibility information (Which version can be installed on top of which other version? Which 3rd party tool versions are needed? Etc.)
- means of communicating development feedback that is separate from the "usual" support cases, so operational content doesn't get mixed up with development issue handling

Rosetta 5.2 has brought a new Storage Migration feature that enables institutions to restructure their permanent storage to fit changing requirements. However, the documentation currently warns that "Disconnecting legacy storage is not recommended as it may prevent the possibility of reverting to a previous IE version." (https://knowledge.exlibrisgroup.com/Rosetta/Product_Documentation/Version_5.2, Preservation Guide, page 185). As far as I can see, this limitation is due to the use of absolute paths in the IE XML files. The paths in older versions of these files cannot be updated without forging provenance information, which, of course, isn't an alternative. However, updating these paths would be the prerequisite for reverting to a previous IE version.

The documentation isn't clear about Rosetta's behaviour concerning older IE versions. These are the scenarios that I see:
- Rosetta might COPY older IE versions to a new storage to have the complete history stored on the new storage. The original copy on the legacy storage is kept to preserve Rosetta functionality concering version recovery.
- Rosetta might MOVE older IE versions to a new storage to have the complete history stored on the new storage. Reverting to older versions would not be possible anymore, but the legacy storage would be cleared of all files and could be removed. The complete provenance information would be kept in older versions of the IE XML, but with some invalid file paths.
- Rosetta might KEEP older IE versions on the legacy storage and write only new information to the new storage. Reverting to older IE versions would be possible, but the legacy storage could never ever be removed. You couldn't even clean up the mount points without losing access to older versions.

From my point of view, a possible way to go would be to change Rosetta's behaviour concerning file paths in the IE XML. Currently, absolute paths are used to point from the IE XML to the payload files/master images. However, using relative paths instead would make more sense here, because (at least on our storage. Comments?) IE XML files and payload files are always kept closely together anyway, so path complexity could be removed. Also, storage migrations would not make a path rewrite necessary, because the files can be addressed by the same relative path. For older IEs, that would mean that only versions created before the first storage migration cannot be reverted to. All subsequent versions would contain relative paths and could potentially be addressed even after a storage migration. For newer IEs (ingested after change to relative paths), that would mean that all versions can be addressed, even after storage migrations.

As this idea is somewhat storage specific (depending on storage layout, storage plugins etc.), I'd appreciate comments from other customers that see possible caveats. Context about SLUB's storage layout and our plans to redesign the storage can be found in the public SupportCase 00345262 "migrating permanent storage to new path [Rosetta 5.0.1.1]".