I would like an easier was to see the size of an IE/REP both in number of files and storage size. Right now, the only way I can see to get the size of the IE in terms of storage is to search at the file level for all files in that IE and then export the results to CSV and add up the size column (in bytes). Same goes for number of files except you can also see them when you’re in the Meditor in the metadata for the Rep. It doesn't seem like adding at least the number of files per REP as an option for a column in the results screen would be that difficult. Thanks!

Rosetta offers an IE level export task that can be used with IE level sets and exports the whole IE/Representation. What we need in addition, is a file level export task. It should be possible to export sets of files (itemized sets based on a list of FL IDs, or logical sets based on search criteria, e.g. IE PID and file extension) to a given export path. Subdirectory name could be the set ID.

To date, the only options are file download – one at time – or IE export, which means unpacking the Rep.tar containers and selecting the required files. Both options are rather tedious.

Use Cases: A user requests a set of high-resolution images for publication, e.g. several pages from a digitized book or a collection of images from different sources.

hen running a Publishing Sync Job and looking at the assciated 'View' we get the following information

Mon Jan 07 09:06:40 AEST 2019 INFO sync all configurations for IE publishing
Mon Jan 07 09:06:40 AEST 2019 INFO sync the configurationId: 18981614
Mon Jan 07 09:06:41 AEST 2019 INFO sync the profileId: 18981613
Mon Jan 07 09:06:41 AEST 2019 INFO Handling new IEs
Mon Jan 07 09:06:41 AEST 2019 INFO Handling Removed IEs
Mon Jan 07 09:06:41 AEST 2019 INFO Handling updated IEs
Mon Jan 07 09:06:41 AEST 2019 INFO Finished sync for the profileId: 18981613
Mon Jan 07 09:06:41 AEST 2019 INFO Finished sync for configurationId: 18981614
Mon Jan 07 09:06:41 AEST 2019 INFO sync the configurationId: 20975335
Mon Jan 07 09:06:41 AEST 2019 INFO sync the profileId: 20975337
Mon Jan 07 09:06:41 AEST 2019 INFO Skipping Profile: 20975337 because it is inactive.
Mon Jan 07 09:06:41 AEST 2019 INFO Finished sync for the profileId: 20975337
Mon Jan 07 09:06:41 AEST 2019 INFO sync the profileId: 20975338
Mon Jan 07 09:06:41 AEST 2019 INFO Skipping Profile: 20975338 because it is inactive.
Mon Jan 07 09:06:41 AEST 2019 INFO Finished sync for the profileId: 20975338
Mon Jan 07 09:06:41 AEST 2019 INFO sync the profileId: 20975389
Mon Jan 07 09:06:41 AEST 2019 INFO Handling new IEs

This is Ok but the additional information 'name/label' would be helpful

For example in the above -

Mon Jan 07 09:06:40 AEST 2019 INFO sync the configurationId: 18981614

the informatin would be better dispayed as follows

Mon Jan 07 09:06:40 AEST 2019 INFO sync the configurationId: 18981614 (Handle Publishing)

where 'Handle Publishing' relates to the 'Data Management: Publishing Configuration | Details | General Details | Name'

Another example

Mon Jan 07 09:06:41 AEST 2019 INFO sync the profileId: 20975337

the infomation would be better displayed as

Mon Jan 07 09:06:41 AEST 2019 INFO sync the profileId: 20975337 (Rosetta_2_Alma)

where Rosetta_2_Alma is the filed located in 'Data Management: Publishing Configuration | Details | Profile Details | Name'

Since version 4, Rosetta includes a feature called AIP update. This feature is used mainly by libraries to alter data in the permanent repository, which might be necessary for a number of reasons. Metadata might need updating due to a typo or due to new research results, new files that weren't included in the original ingest might be added, or existing files might be altered with higher resolution versions or to repair defective files. Libraries tend to use this feature, because they regularly ingest SIPs that include a minimal set of metadata. Later, when full metadata have been allotted or OCR processing has been done, that data is added and Rosetta creates a new AIP version while preserving the original version.

In the status that AIP update has in current Rosetta versions, we see various design flaws that endanger stability, robustness and safety of our data. We'd like to outline the issues that we see and suggest a number of changes to improve on those issues and make AIP update more secure and usable for the Rosetta user community. Please note that we use AIP update exclusively via API, which is necessary for running automated ingest/update/access workflows. SLUB has met all of these issues in real world production.

One key requirement for successfully processing AIP updates in an automated workflow is the ability to distinguish between three alternatives. Any package that is ingested into Rosetta can be: 1. a new ingest, 2. an AIP update of an existing unlocked and finished IE in the permanent repository, 3. undecidable (internal processing in Rosetta hasn't finished yet, so the ingest has to wait for the moment). The basis for this decision is a combination of the data returned from querying the system's APIs, namely the number of SIPs that contain the external identifier (as given in the SIP MD), the number of SIPs that contain the external identifier (as returned by the SRU search through the indexed IE-DC MD) and the processing status of each of these SIPs. To do so, preservation repository software needs to provide gapless tracking of the processing status for each IE right from the ingest. IE metadata contain these identifies, but they are currently only indexed at the very end of the processing after moving the IE to the permanent repository. Until that happens, we cannot use the SRU search (which relies in the Solr index) to find out if an IE with a given external identifier is already in the system, regardless of its current processing status. Due to this limitation, we only ingest exactly one IE with each SIP, so there's always a distinct mapping from SIP ID to IE PID and vice versa. That way, we could use the SIP metadata that is stored in the database to find out if there's an IE with a given external identifier in the system even before it reaches the permanent repository. Until recently, we used a custom built workaround that is no longer supported in Rosetta 5.2, including a BIRT report and adventurous authentication methods to cover most of the gap between LOADING stage and indexing. However, there was a remaining gap between the moment when the SIP status was set to "module PERMANENT, stage FINISHED, status FINISHED" and the moment when the index processing has been completed. This gap was supposed to be closed in Rosetta 5.1, when a new API was created to replace the BIRT report and create a stable and supported solution. However, the new API does not cover the functionality of the BIRT report, making it unusable for controlling automated AIP updates. The legacy BIRT report solution returned all SIPs in the system for a given external ID, including legitimate SIPs as well as rejected and declined SIPs. The new API, however, only returns a SIP's status if exactly one SIP ID for a given external ID was found. If there's more than one SIP for a given external ID, Rosetta just returns an error instead of returning a status list for all SIPs that were found.
The solution for this situation would be as follows:
- Move the indexing of all metadata as far to the beginning to the SIP processing as possible. Performance considerations, which were used to justify keeping the indexing of the IE metadata at the very end of the processing and even running it after the SIP is in "module PERMANENT, stage FINISHED, status FINISHED" (so, practically lying to the user and external software), are not valid on recent hardware and do not justify neither the need for workarounds nor the danger imposed on the preservation data.
- Index SIP MD as well. As far as I saw in the Rosetta 5.3 announcements, this is planned for the next version and will solve some of hte problems that we're having.
- Enhance the getSIPStatusInfoByExternalId() API (documented here https://developers.exlibrisgroup.com/rosetta/apis/Sipweb services) to provide the functionality needed and described above.

Any information about exactly which files are supposed to be updated with which replacements are currently kept in a CSV file stored below /operational_shared/. In in our environment, this is an NFSv3 share that is mounted with nolock option set due to ExLibris requirements (which, in itself, would be dangerous enough). However, as writing to files on network shares is a non-atomic operation, the CSV files can get corrupted if there's a network glitch. This will result in an AIP update going wrong with no indication whatsoever, as we had to experience first hand. From our point of view, the correct way to do this would be by keeping all AIP update information in the database, where operations are transactional and can be recovered a lot easier even if there's an issue with the database (i.e. via Oracle FlashBack Recovery or something similar). This approach would offer multiple advantages:
1. The AIP update information is kept safe in the database.
2. The information is kept in just one place instead of splitting it between the database and the file system.
3. Network load on the NFS shares is reduced.
4. The database handles concurrent access a lot better than NFS shares do.
Taking the importance of the data in Rosetta installations around the world and the perspective preservation durations into consideration, its bordering irresponsible to prefer file operations over database operations for control mechanisms.

After calling the updateRepresentation() web service, Rosetta returns a RIP ID that can be used to see the status of the process. While this is a good idea in general, there are some issues with the actual implementation. As of Rosetta 5.1.0.2, the RIP ID is deleted from Rosetta immediately after the AIP update process completes. Small AIP updates tend to finish really fast, so if the submission application queries Rosetta about that RIP ID for the first time, it might already be gone and Rosetta returns an error indicating that the RIP ID is unknown. We have contemplated using this behaviour as an indicator that the AIP update has finished successfully, but according to ExLibris, this wouldn't be acceptable even for a workaround (their words). Anyway, the correct behaviour for a well-designed API would be to:
- issue a RIP ID on updateRepresentation() call
- return a "running" status for that RIP ID as long as the process hasn't finished
- return an "error" status for that RIP ID if anything unexpected occurs
- return a "finished" status for that RIP ID as soon as the AIP update process has finished successfully
- return any other status messages that might be needed
- EITHER delete only after at least 24h after successfully finishing the AIP update OR (preferred) never delete RIP IDs and use suitably large data type (i.e. long unsigned integer) for these IDs to make sure the system never runs out of IDs
- return an "unknown RIP ID" message only if the RIP ID actually has never existed or doesn't exist any more
SupportCase 00366236 "Error in RIP Loading, IE (FileNotFoundException: Unable to read file: Import Descriptor.csv)" describes some more details about how we found out about this issue.
In any case, software using the API must be 100% confident about the process status of AIP updates the manipulate the actual repository master data.
From what we see in the Rosetta 5.2 changelog, ExLibris has implemented some changes in this area. We have yet to test the extent and reliability of the implementation.

From what we understand about the inner workings of Rosetta, some of Rosetta's internal tasks are handled via web services as well. However, these web services are undocumented. We learned about one of these web services when we encountered a failed AIP update (SupportCase 00376523 "failed AIP update on IE3761638 [Rosetta 5.0.1.1]", error message: 'SocketException from an internal web service "PermanentManagerWS": java.net.SocketException: SocketException invoking http://langzeitarchiv.slub-dresden.de/dpsws/permanent/PermanentManagerWS: Connection reset'). The search for the root cause is ongoing, but our assumption is that an internal timeout for this web service hit us before the process had finished. We only noticed this error because we regularly grep the server.log for error messages; Rosetta itself didn't give any indication about something being wrong with our AIP update whatsoever. This means that we nearly missed a failed process that is supposed to alter master production data with the AIP still being locked in mid-process. We suggest to document all web services and improve API error notification to make sure cases like this one are caught even in fully automated production scenarios. For us, this is especially critical because of our specific production workflow where multiple AIP updates are needed to represent one change that came for image production. If one of those updates fails, the AIP ends up in a semantically inconsistent state where medatada might point to files that don't exist yet and were supposed to be added in the subsequent AIP update.

[continued in comments]

we are using the Rosetta-Aleph-Integration in both directions:
1. creating the CMS-record in Rosetta and
2. setting the Rosetta-flag in Aleph as well as updating the Aleph record with the URN created in Rosetta.
In the past there was only one target field for the URN and the assignCMS task with updateURNTask worked without problems.

Due to changes in the cataloging rules - we are using now the international RDA-rules - we have to deal now with different target fields in Aleph for the URN depending on the type of material: e-only e-books, digital reproductions of printed books, online versions of printed books, etc.

The target field is configurable in the Rosetta-Aleph-Sync, but this is not sufficient with the new situation. We want to be able to configure profiles for the updateURNTask depending on the material type: to specify the target MARC-field, to populate subfields with fixed information depending on the material type and with the URN.
(It is not possible at the moment to update the URN in a subfield of an already existing field, a new field is created (case 00510676)).

Astrid Schoger on behalf of the Bavarian State Library