454 results found

1. Update Logical and Physical Structure Maps via Rosetta Back Office

   Hello,

   For IEs that have both a physical and a logical structure map, we would like to be able to globally change the default logical and physical structure map labels for Preservation Master in the Rosetta Back Office.

   Thanks,

   Ryan

   Ryan Edwards
   Digital Access and Systems Librarian
   Information Systems
   Getty Research Institute
   1200 Getty Center Drive, Suite 1100
   Los Angeles, CA 90049-1688
   reedwards@getty.edu
   310.440.7398

   2 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   1 comment  ·  Publishing & Delivery  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

2. Configure General Rep Viewer to display logical structmap

   The General Representation Viewer only displays the physical structmap if there is one. It should be possible to to configure the Delivery Rule / Representation Profile for the General Rep Viewer to show the logical structmap instead.

   1 vote

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   0 comments  ·  Publishing & Delivery  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

3. Containerize the Rosetta Server Infrastructure

   We would like to be able to install and run the Rosetta server in a docker container.

   Docker containers allow infrastructure to be managed more efficiently and at scale. Also, running Rosetta in a docker container would allow us to have the option to deploy it into cloud infrastructure.

   23 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   1 comment  ·  Environment & Infrastructure  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

4. Solr Startup Checker Plugin

   After having had some issues with the new default Solr port in Rosetta 5.4.0.0, I suggest creating a startup checker plugin for Rosetta to test for Solr availability. Possible error scenarios involve:
   1.) network down
   2.) Name resolution down/misconfigured
   3.) Solr default port closed in the local firewall / Solr not listening

   1. & 2. could be checked by pinging the other servers' DNS names (NOT the IPs), which would test network availability and DNS resolution in one step. As Rosetta servers usually are located in the same subnet, we don't have to worry about ICMP firewall restrictions.
   2. could be tested by using "$> telnet $HOSTNAME $PORT", like "$> telnet rosetta-host-1 8983" and SSH-ing into the remote systems to list processes listening on specific ports, like "netstat -tulpn | grep 8983" or "ss -tulpn | grep 8983".

   After having had some issues with the new default Solr port in Rosetta 5.4.0.0, I suggest creating a startup checker plugin for Rosetta to test for Solr availability. Possible error scenarios involve:
   1.) network down
   2.) Name resolution down/misconfigured
   3.) Solr default port closed in the local firewall / Solr not listening

   1. & 2. could be checked by pinging the other servers' DNS names (NOT the IPs), which would test network availability and DNS resolution in one step. As Rosetta servers usually are located in the same subnet, we don't have to worry about ICMP firewall restrictions.
   2. could be tested… more
   5 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   0 comments  ·  APIs & Plug-ins  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

5. Best fit mode for the JWPlayer in IE viewer

   With the new IE viewer in version 5.4 the JWPlayer (for video and audio) does not have a best fit parameter meaning that the right-hand side of the player is hidden under the metadata pane. The only way to see the whole player is to close the metadata pane manually or zoom out in your browser. We would like to see a “best fit” parameter added to the JWPlayer in Rosetta so that the behavior would match the jpeg viewer (which fits itself to the frame by default).

   Thanks,

   Ryan Edwards
   Digital Access and Systems Librarian
   Information Systems
   Getty Research Institute
   1200 Getty Center Drive, Suite 1100
   Los Angeles, CA 90049-1688
   reedwards@getty.edu
   310.440.7398

   With the new IE viewer in version 5.4 the JWPlayer (for video and audio) does not have a best fit parameter meaning that the right-hand side of the player is hidden under the metadata pane. The only way to see the whole player is to close the metadata pane manually or zoom out in your browser. We would like to see a “best fit” parameter added to the JWPlayer in Rosetta so that the behavior would match the jpeg viewer (which fits itself to the frame by default).

   Thanks,

   Ryan Edwards
   Digital Access and Systems Librarian
   Information Systems
   Getty Research… more

   1 vote

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   3 comments  ·  Publishing & Delivery  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

6. Scheduled Reports with Institution in Email-Header

   When Scheduling Reports for different Institutions, the Email received does not contain any information about where the Data comes from. If you want to know IEs in permanent, you would not know wheter it is about Institution A or B. It would be great if the Institution or a custom information-text-content would be sent either with the email or be included in the report itself. The Report in my special case is "Repository Stats" in "Data Management".

   •  

   •  

   2 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   0 comments  ·  Data Management  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

7. change to Rosetta METS XSD re: duplicate file IDs

   During a recent batch of deposits (see case 00477041 from the Getty for more info) we figured out that, due to an error in our one of our input files that creates the METS, our original METS file did not have unique file IDs in the structMaps for the preservation master and modified master representations. It was simply repeating the same file ID from the access file representation.

   For some reason, this did not cause a METS validation error when we validated against the Rosetta XSD before deposit, nor when it was deposited. However, it resulted in the METS in Rosetta not having any files listed in the structMaps that contained the duplicate files IDs. Since the fileIDs were correct in the fileSec it imported all the files properly, but they were not referenced from any structMap. Fixing this using the back office would have taken too much effort so we deleted the ca. 150 IEs and redid them from scratch.

   Might you be able to enhance the METS XSD so that it requires file IDs to be unique in each structMap? I know it is against Rosetta's data model to refer to the same file ID from different representations with the same structMap type so it would make sense if the XSD enforced this requirement.

   During a recent batch of deposits (see case 00477041 from the Getty for more info) we figured out that, due to an error in our one of our input files that creates the METS, our original METS file did not have unique file IDs in the structMaps for the preservation master and modified master representations. It was simply repeating the same file ID from the access file representation.

   For some reason, this did not cause a METS validation error when we validated against the Rosetta XSD before deposit, nor when it was deposited. However, it resulted in the METS in… more

   3 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   0 comments  ·  Deposit  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

8. Producer Mandatory Fields shall be controlled by a code table

   As of now (version 5.3) there is no code table to control which fields are optional or mandatory when adding a producer.

   We provide Rosetta as a service and the details (name, email, telephone etc.) of producers can be seen across the installation. This means that staff user 1 from institution A can see the details of the producers of all the other institutions. The workaround is to enter 123 as telephone.

   Basically we would like Rosetta to be fully compliant with the concept of multi-tenancy.
   Until this can be reached we would like to see code tables in use for any place where user-information is entered, stored and shown in Rosetta.

   We opened a ticket with Ex Libris but since this is not a bug we were asked to request a feature enhancement.

   As of now (version 5.3) there is no code table to control which fields are optional or mandatory when adding a producer.

   We provide Rosetta as a service and the details (name, email, telephone etc.) of producers can be seen across the installation. This means that staff user 1 from institution A can see the details of the producers of all the other institutions. The workaround is to enter 123 as telephone.

   Basically we would like Rosetta to be fully compliant with the concept of multi-tenancy.
   Until this can be reached we would like to see code tables in use… more

   5 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   0 comments  ·  User Management  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

9. Add report in migration plugin

   The migration plugin is part of preservation and is the way in Rosetta to migrate bitstreams from an obsolete format to an uptodate format. It is a script plugin receiving a file and delivering the newly migrated file.
   For purpose of transparency it would be necessary to add a report about what has changed. For example when migrating pdf to pdf/a it might happen, that a font was included or even replaced. Having a report would implement a well documented preservation workflow. Depending on the migration tool the report might be quite complex.

   So it would be nice to add this report as sourceMD on file level as part of the migration plugin.

   The migration plugin is part of preservation and is the way in Rosetta to migrate bitstreams from an obsolete format to an uptodate format. It is a script plugin receiving a file and delivering the newly migrated file.
   For purpose of transparency it would be necessary to add a report about what has changed. For example when migrating pdf to pdf/a it might happen, that a font was included or even replaced. Having a report would implement a well documented preservation workflow. Depending on the migration tool the report might be quite complex.

   So it would be nice to add… more

   42 votes

   Vote 0 votes Vote Vote

   Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   1 vote 2 votes 3 votes Remove votes

   You have left! (?) (thinking…)
   0 comments  ·  Preservation & Format Library  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Submit Rating

10. Configure the delivery rules to display only logical structure maps, and not physical ones

    We would like the ability to configure the delivery rules to display only logical structure maps, and not physical ones as for one of our institutions, we’d like NOT to deliver the physical structMap, and only the logical one.

    Thanks,

    Ryan

    Ryan Edwards
    Digital Access and Systems Librarian
    Information Systems
    Getty Research Institute
    1200 Getty Center Drive, Suite 1100
    Los Angeles, CA 90049-1688
    reedwards@getty.edu
    310.440.7398

    1 vote

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    2 comments  ·  Publishing & Delivery  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

11. Additional rule input parameter: material flow

    Since the organization levels in Rosetta are limited to consortium and institution, we often have to ingest objects of different origin an type into a single institution. This poses problems as for certain sources you want to activate more or less extraction and/or file extension rules that for other sources. Currently we have to leave the more permissive rules off, make the SIPs fail and end up in the TA, enable the permissive rules, rerun the SIPs and finally disable the permissive rules.

    This is a pain and requires many manual interventions that are error-prone and can influence other ingests that happen at the same time.

    Rules can be selected based on Producer name, but it is far more work to set up and maintain a number of Producers that it is to create multiple material flows. We therefore would like to suggest to add an extra selection parameter for selecting a number of material flows.

    Since the organization levels in Rosetta are limited to consortium and institution, we often have to ingest objects of different origin an type into a single institution. This poses problems as for certain sources you want to activate more or less extraction and/or file extension rules that for other sources. Currently we have to leave the more permissive rules off, make the SIPs fail and end up in the TA, enable the permissive rules, rerun the SIPs and finally disable the permissive rules.

    This is a pain and requires many manual interventions that are error-prone and can influence other ingests… more

    1 vote

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    0 comments  ·  Deposit  ·  Edit…  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

12. Ability to integrate our Ingest tool with Rosetta

    We are looking at supporting the ability to add a new format to our Ingest tool, in conjunction with this we would like that to trigger some type of API call to Rosetta and add the file format as an allowed format as well.

    In Rosetta you can add a new file extension in the Submission Format area of the UI. We have provided the option to add a new format in our Ingest tool, So once they do this, we would like this to trigger an some type of API call to Rosetta and add the new file extension as an allowed file extension without having to go into the UI and add the file extension manually in the Submission Format location.

    We are looking at supporting the ability to add a new format to our Ingest tool, in conjunction with this we would like that to trigger some type of API call to Rosetta and add the file format as an allowed format as well.

    In Rosetta you can add a new file extension in the Submission Format area of the UI. We have provided the option to add a new format in our Ingest tool, So once they do this, we would like this to trigger an some type of API call to Rosetta and add the new file extension… more

    1 vote

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    0 comments  ·  Environment & Infrastructure  ·  Edit…  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

13. Support the ability for Rosetta to request IEs in batches

    We would like Rosetta to work more efficiently when the storage layer is tape based, particularly for exports. Could Rosetta request IEs in batches from the storage layer. when exporting large collections from tape a file is retrieved from tape one at a time.

    10 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    0 comments  ·  Data Management  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

14. Support Oracle Internet Directory

    Support OID as an alternative to tnsnames.ora in Oracle Installation. OID is the DNS like service for Oracle that we use heavily in our institution.

    1 vote

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    1 comment  ·  Environment & Infrastructure  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

15. Recalculate fixity SHA2 and insert into Rosetta

    We have around 22,000,000 files that we would like to recalculate a new fixity hash, but this time it would be SHA256. We need a good way of getting this fixity back into Rosetta. We would like to store it along with the SHA1 value we previously generated. We are aware that the SHA1 fixity is located in xml clob data in the hdemetadata table. We are wondering if there is another location for this SHA1 value in another table.

    2 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    3 comments  ·  Data Management  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

16. Make IE Viewer share button configurable

    The share button in the IE Viewer currently always shows the browser URL. Often the IE has an OID or some persistent URL that is preferred over the transient viewer URL. Rosetta should allow the URL that is displayed to be configurable. Preferably with an XSLT per institution similar to the IE metadata XSLT.

    It is also desirable to allow for more than 1 URL in the share window. That would allow to add URLs for standalone vs embedded viewers, thumbnail view, CMS metadata view, etc.

    4 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    3 comments  ·  Publishing & Delivery  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

17. Bitstream information in Bitstreams Tab

    Researchers often use zip or tar files in order to upload large amounts of research data in various formats. Bytestream extraction is a useful tool for analyzing the content of these container files. The resulting bitstream information is written in the METS and displayed in the BitStreams Tab in the Web Editor. The information includes BS ID, file name, file original path, extension, mime type and format ID (if successfully identified).

    We would like to suggest the following improvements for the BitStreams Tab and the bitstream information in general:

    1) Include Bitstream size. The relevant values can be found in the table “HDBITSTREAMREF” in the field “FILESIZEBYTES”.

    2) Display bitstreams in order of path/filename: the bitstream list should reflect the order of the files in the zip or tar file. Currently they are displayed in random order.

    3) Make bitstream information searchable

    Researchers often use zip or tar files in order to upload large amounts of research data in various formats. Bytestream extraction is a useful tool for analyzing the content of these container files. The resulting bitstream information is written in the METS and displayed in the BitStreams Tab in the Web Editor. The information includes BS ID, file name, file original path, extension, mime type and format ID (if successfully identified).

    We would like to suggest the following improvements for the BitStreams Tab and the bitstream information in general:

    1) Include Bitstream size. The relevant values can be found in… more

    •  

    15 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    1 comment  ·  Data Management  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

18. untangle format identification/validation/MDextraction

    In the course of the discussion on the "File Extension Mismatch - Change Notice" posted on BaseCamp by Opher Kutner (ExL), I noticed that some aspects of file identification/validation/MDextraction in Rosetta need improvement. What got me thinking was the statement that format identification was done by the extractor, which isn't the case. Instead, the actual behaviour is that the MdExtractionPlugin also does the format validation, while the format identification is run separately in DROID. However, the job of extractors is neither format identification nor format validation. Instead, it's metadata extraction only. Usually, a preservation repository can't even know which extractor to apply without having successfully completed identification and validation before. The fact that format validation and metadata extraction are run in the same step introduces problems that could be avoided by dividing them up in separate steps.

    From my point of view, the correct workflow would be as follows:
    1. Run format identification against FL/PRONOM/YourFormatRegistryHere, get list of recognized formats (sorted descending by certainty of identification)
    2. Run format validation routine for elements from the list from step 1, only treat format IDs that were identified with a identification certainty above a certain percentage.
    2.a. If two elements from the list have the same identification probability, then both validators need to be executed. Human intervention is necessary to decide which format should be used. However, this should never occur, because PRONOM maintains a distinct prioritisation of format signatures.
    2.b. If validation successful, continue with 3.
    2.c. If validation fails, go back to 2. and try next element in list.
    2.d. If validation fails for all elements of the list, move file to TA.
    3. Choose the correct MD extractor, extract MD.

    This leaves us with the following classes of errors:
    1. Format identification fails, because:
    1.a. No recognized formats at all (empty list). This might be caused due to a missing signature in the format registry.
    1.b. No formats with sufficient certainty identified.
    1.c. One format identified with sufficient certainty, but file extension doesn't match. This might be either caused by:
    1.c.1. An extension that is actually wrong for that file type, and needs to be corrected.
    1.c.2. A wrong signature that misidentifies a file. (We had that problem in an older FL version, where TIFFs were misidentified as .NEFs due to an overly general signature for the NEF format.)
    1.d. Multiple formats identified with sufficient certainty, but no file extension matches.
    1.e. Internal error / Format identification plugin crashes.
    2. Format validation fails, because:
    2.a. If exactly one format was identified with sufficient certainty:
    2.a.1. One validator called, file validation doesn't return successfully.
    2.b. If multiple formats were identified with sufficient certainty:
    2.b.1. Validators for all detected formats were called, none returned successfully. This should never occur.
    2.c. Internal error / Validator crashes.
    3. MD extraction fails, because:
    3.a. No MD extractor plugin for that format available.
    3.b. No significant properties / DNX mapping configured for that plugin/format combination at all.
    3.c. File doesn't contain any of the significant properties that are configured for that plugin/format combination.
    3.d. Internal error / Format extractor plugin crashes.
    ELSE One format identified with sufficient certainty, no file extension mismatch, NO ERROR.

    The most important part here is that a format is only identified with certainty if a) the signature, MIMEtype and file extension given in the format registry match what is found in the actual file and b) if the validation has finished successfully. Only if both conditions are matched, we can be sure about the result of our format identification. I want to illustrate this with the example of the TIF format. In cases where one or both of the aforementioned conditions are not met, we DON'T have an invalid TIFF. What we DO have is something that looks suspiciously like a TIFF, but doesn't comply with the format specification, and therefore is NOT a TIFF. In other words: a format is ONLY successfully identified if the identification result is confirmed by the validation result.

    So, if we assume that the changes suggested above are implemented, the whole File Extension Mismatch Handling issue would become a lot easier. Every file that fails one of the stages ends up in TA and, ideally, will be returned to the producer to be repaired (or at least the producer will be notified about 1. the file being rejected 2. the reason for that including error message 3. the necessity to repair the file and re-ingest it). Alternatively, the TA user could run a repair operation on their own, but in the majority of the cases, the file that caused the error will have to be modified until the mismatch doesn't occur any more. This usually requires human intervention and analysis; just treating the error with a rule and ignoring it doesn't solve the underlying problem. As these underlying problems are usually a risk for the data in the repository, ignoring them is not an option.

    The consequences are that:
    - unknown formats can't be moved to permanent (and, as outlined, they shouldn't, because knowledge of the format is strict requirement for risk analysis).
    - format IDs can't be manually assigned to files at all, because that equals ignoring the issue instead and postponing the solution to the future, hoping that "someone will probably find time to do it then" (hint: they won't), thus effectively ignoring it.
    - the structure of plugin types in Rosetta needs to be refined, now strictly separating plugins for identification, validation and MD extraction, and also separating plugins by their intended format use. This probably means that existing plugins will need to be modified.
    - for cases where the files can't be returned to the producer and the repair operations have to be done by Rosetta staff:
    - a new plugin type "RepairPlugin" could be introduced to run very specific repair operations from within the TA workbench. The current manual approach is extremely cumbersome and doesn't scale at all.
    - repair operations in the TA workbench will need to support a higher level of automation to a) classify/group files not only by their IE but also by TypeOfError/numberOfErrors/ingestDate/anythingElseThatMightBeUseful b) run repair operations on all files from the same group or c) initiate repairs via an API (get errorMessages, get broken files from same group, repair outside of Rosetta, replace broken files, NOT using insecure ImportDescriptor.csv files). Also see these public SupportCases in SalesForce:
    - 00128163 bulk file replace in TA Rosetta 4.0.1.1
    - 00340712 enhance TA file replace assistant Rosetta 5.0.1.1
    - MD extractors are no longer assigned to Classification Groups (at least not exclusively). Instead, MD extractors should also be assignable to single format IDs. For practical reasons, formats could inherit their MD extractor setting from their Classification Group to set sensible defaults for all formats as long as no format-specific MD extractor is configured.

    In the course of the discussion on the "File Extension Mismatch - Change Notice" posted on BaseCamp by Opher Kutner (ExL), I noticed that some aspects of file identification/validation/MDextraction in Rosetta need improvement. What got me thinking was the statement that format identification was done by the extractor, which isn't the case. Instead, the actual behaviour is that the MdExtractionPlugin also does the format validation, while the format identification is run separately in DROID. However, the job of extractors is neither format identification nor format validation. Instead, it's metadata extraction only. Usually, a preservation repository can't even know which extractor… more

    7 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    5 comments  ·  Preservation & Format Library  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

19. Allow institutional parameters for (custom) plugins

    In a consortial landscape a plugin may be used by several institutions. However, each institution may need to use different parameters for a shared plugin.

    To give examples, consider these two business cases:
    (1) A plugin utililzes proprietary 3rd party software that is licenced per institution. In such case we must ensure that only the institutions that actually pay the licence fee have access to the functionality. The license key would have to be configured per institution.
    (2) A PDF migration plugin can take different input parameters which determine the output format to be created (e.g. to PDF/A-1b vs. PDF/A-2b). An institution will want to use these input criteria when defining the mirgration process as part of preservation action from within the institution itself.

    While in the case of processes parameters can currently be defined within an institution when scheduling the process (e.g., for fixity task which checksum algorithm should be used), parameters of plugins can currently not be passed to the institutional level. We would like to see this change in order to use plugins in a more flexible and institutional-friendly way. Currently there is no information available if and how this can be achieved for custom plugins. It might also make sense in the case of some bundled plugins, e.g., if an institution would like to use a different DROID max byte scan config than another one.

    In a consortial landscape a plugin may be used by several institutions. However, each institution may need to use different parameters for a shared plugin.

    To give examples, consider these two business cases:
    (1) A plugin utililzes proprietary 3rd party software that is licenced per institution. In such case we must ensure that only the institutions that actually pay the licence fee have access to the functionality. The license key would have to be configured per institution.
    (2) A PDF migration plugin can take different input parameters which determine the output format to be created (e.g. to PDF/A-1b vs. PDF/A-2b).… more

    5 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    0 comments  ·  APIs & Plug-ins  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating

20. Take account of the METS mimetype attribute during file ingest

    The METS standard has a mimetype attribute on files "<file MIMETYPE="application/pdf" ... /> but this information is currently ignored by Rosetta during the file-type identification process on ingest.

    Mimetye should be at least as reliable as file extensions (because mimetype is a standard, is actively maintained and widely use by a good portion of software for identifying file types).

    In particular, recognising mimetypes is likely to greatly increase the ease of ingest of content from internet-facing content management systems where it is very widely used

    See also https://tools.ietf.org/html/rfc6838

    3 votes

    Vote 0 votes Vote Vote

    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    1 vote 2 votes 3 votes Remove votes

    You have left! (?) (thinking…)
    0 comments  ·  Ingest  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Submit Rating