Manage Load File Import Templates

Project > Settings drop-down > Organization or Project Settings > eDiscovery Templates > Load File Import

Requires Organization - Load File Import - Add/Edit Permissions

A Load File Import template makes it possible to import Load Files into Digital Reef using the Imports > New Data Set from Load File option by providing the needed mappings between the fields in the Load File and the appropriate DR metadata fields. These mappings allow the information contained in the Load File to be identified and processed by Digital Reef. To create these mappings, you must first discover the fields in a specific Load File, which can then be individually mapped as desired. For example, when creating a Load File Import template for EDRM XML load files containing Google Vault data, you can map the Google field #Title to the Filename DR metadata field.

Digital Reef can import Load Files of type DAT, CSV, and EDRM XML. Each Load File Import template is permanently associated with the type of the Load File in which fields were discovered when it was created and cannot be used for other types. For example, if you discover fields in a CSV file to create a template, it can be used to import any CSV Load File, but not DAT or XML Load Files. In addition, some DAT and CSV Load Files use other than the default encoding and/or delimiters, and the Load File to be imported must match the encoding you specified for the Load File in which fields were discovered. (Because Digital Reef imports only EDRM XML Load Files, such as those containing Google Vault documents, it is not necessary to specify the encoding and delimiters for these.) In practice, however, a given Load Files Import template is primarily used to import similar Load Files from a specific source.

Load File Import templates can be created and managed at the Organization and Project levels using the Load File Import option on the respective Settings menus (access to which is determined by your permissions). This option provides a pane in which existing templates are listed, a pane in which Load File fields are discovered and listed, a pane in which DR metadata fields are displayed, and (in the center) the Field Mappings pane, in which you assemble the needed mappings from the Load File fields and the metadata fields in the other panes. The mappings you create can optionally include Transformers, which automatically make the changes you specify to each value in the discovered field before the value is assigned to the mapped metadata field in Digital Reef.

Saving and Loading Load File Import Templates

To create a Load File Import template, you can start with any existing template in the Templates pane, modify it, and save it as a new template, as well as starting a new one from scratch, using the following options. You can also modify existing templates, and export and import templates in XML files (which makes sharing easier). All of these options, save the Save button and the create icon ( so if i ), can be selected either by right-clicking an existing template in the Templates pane or by clicking a template's context menu icon () on the right.

Save to a Template

Save any changes you have made in the Field Mappings list to the currently loaded template by clicking the Save button.
Save your current Field Mappings list to another template by selecting Save to Template ... for any template in the Templates pane, which displays a list of the templates available to you (based on your permissions), and selecting one to save to. This automatically overwrites the selected template's current contents, without warning.

Note: When saving to a template automatically overwrites its current contents, some fields, such as those representing patterns, tags, domains, aliases and excluded content, are appended to the template you save to, leaving the fields currently in the template in place.

Load from a Template

Load a different template by clicking the one you want in the Templates pane. If you have made changes to the currently loaded template but not saved them, you must save or discard the changes before loading a template in this way.
Load a different template by selecting Load from Template ... for any template in the Templates pane, which displays a list of the templates available to you (based on your permissions), and selecting one. If you have made changes to the current template but not saved them, the changes are automatically discarded, without warning.

Change the Default Template

Change the default template that provides the initial contents of the Field Mappings list each time a user selects Load File Import from the Organization or Project settings (depending on which you chose) by selecting Set as Default for the template you want to make the default in the Templates pane. You do not need to save unsaved changes first, even if you are making the current template the default; the changes are preserved during the operation and can be saved afterward.

Create a New Template

Create a new template by clicking the create icon,, in the upper right corner of the Templates pane, and provide a name and optional description. The new template is created with the Field Mappings list as it was when you clicked . If you have made changes to the currently loaded template but not saved them, you must save or discard the changes before creating the new one.

Download or Upload Templates as XML

Save your current Field Mappingslist as a template in a loadable XML file and download it by selecting Download as XML ... for any template in the Templates pane. Downloaded template files are named Load_File_Importtemplate_name-organization_name.xml.
Upload an XML file and load the template it contains by selecting Load from XML ... for any template in the Templates pane and selecting a local XML template file. The current contents of the Field Mappings list are automatically overwritten by the contents of the loaded template, which you can then save as an existing or new template, optionally after having made further changes.

Save all of the existing Organization or Project Load File Import templates in a ZIP file containing one loadable XML file for each and download the ZIP file by using the context menu icon () in the upper right corner of the Templates pane (to the right of the title and the icon) to selectDownload all as XMLs .... Downloaded ZIP files are named Load_File_Import-organization_name.zip. If there are unsaved changes in the currently loaded template, the most recent saved version of that template is downloaded.
Upload a ZIP file containing multiple loadable XML template files and add all of the templates it contains to the Organization's templates by selecting Load from XMLs ... from the Templates pane's title context menu (as described above) and selecting a local ZIP template file. You do not need to save unsaved changes first; the changes are preserved during the operation and can be saved afterward. If the ZIP file contains any templates with the same names as existing Organization templates, they are saved as template-name_1.

If you have the needed permissions, additional context menu options for individual templates include Edit, which lets you modify the name and/or description of the selected template, and Delete, which lets you permanently delete a template.

Creating a Load File Import Template

Because Load Files can vary widely, it is important to thoroughly familiarize yourself with the contents and characteristics of the Load File in which you will discover fields before creating a Load File Import template.

Once you have created a Load File Import template, you can select it when you import a Load File into Digital Reef using the Imports > New Data Set from Load File option, as long as the Load File to be imported is of the same type as the one in which you discovered fields.

To add mappings and create a template, follow these steps:

Click Discover Fields on the toolbar to discover the fields in a Load File and add them to the Load File Fields list on the left. The DR Metadata Fields list on the right is automatically populated. Both lists have filter boxes to help you find the field you need more quickly.
Populate the Field Mappings section by dragging a field from the Load File Fields list into the Load File Field column, dragging a field from the DR Metadata Fields list into the DR Metadata Field column, and if necessary selecting a transformer in the Transformer column between them. By default no transformer is selected, which means that values of the field in the Load File are placed in the mapped DR metadata field in the new Data Set without alteration. If required, you can select among transformers for date/time, numeric, or text field values; for example, for a text field you can replace a specific character in values with a different character.
Repeat the previous step until you have created all the needed mappings. (Be sure to see Mapping Requirements and Guidelines for information about fields that must be mapped.) To speed the process, you can select multiple fields in the Load File Fields list and drag them together into the Field Mappings section; you can also delete fields in the Load File Fields list or Field Mappings section by selecting them and clicking Delete Selected.
If desired, select Preserve Families (see Mapping Needed to Preserve Families) and/or Process Load File Images (see Mapping Needed to Process Load File Images).
Save your changes. You will be reminded if your changes do not include one or more required field mappings.

Mapping Requirements and Guidelines

When the load file includes natives or text versions, you typically need at least one mapping entry to identify the location of the files. Depending on the discovered fields, entries may exist for one or both of the following:

The location of native files, using a Load File field such as NativeLink/NativeFile (hereafter referred to as NativeLink).
The location of text files from OCR processing, using a Load File field such as TextLink/OCRPath (hereafter referred to as TextLink).

In general, you must also have a mapping entry forextbegdoc to identify the external starting document with a document number. This mapping entry is required for any load file import setup.

If the load file is set up for image processing only, you might not have a mapping entry to darelativepath or ocrpath at all, but you will need to provide the required mapping entries, as described in Mapping to Process Load File Images.

Use the following mapping guidelines:

If you are providing only one mapping entry to identify the location of the files (whether it is NativeLink for native files or Textlink for text files from OCR processing), you must map to the Digital Reef darelativepath field (categorized under Digital Reef Properties). If your load file uses the DR\ prefix for the mapping entry, you can use a text transformer to trim the first three characters of the DR Base Path (DR\). If your load file does not use the DR\ prefix (and does not require any other adjustment), leave the mapping using a Pass-through Transformer.

If you are supplying more than one mapping entry for the location of the files, you can use a Load File field such as NativeLink for native files and map it to darelativepath, and then use a Load File field such as TextLink (for the OCR-processed text files) and map it to the Digital Reef ocrpath field (categorized under Digital Reef Properties). Supplying more than one mapping entry for the location of the files provides additional information.
As long as you provide a field mapping to identify the location of a native file and the native is available at that location, the filetype field for the document will report the appropriate filetype; if not, the filetype field will report an Unknown format because there is no native.
You must have a mapping entry to the Digital Reef extbegdoc field to ensure that the external starting document has a document number.
You may want to supply a mapping entry to the Digital Reef filename field to ensure that you see the expected filename.
You cannot provide mapping entries to the following fields (listed under the appropriate properties): contentmd5, docext, filemd5, and filetype. For restrictions concerning the mapping of family-related fields, see Mapping Needed to Preserve Families .

Note: Digital Reef can accommodate the situation in which the Load File provides natives for some files, but not all. In this case, a document without a populated NativeLink (or equivalent) field might have a populated TextLink (or equivalent) field that points to extracted text. To ensure that the software can identify the darelativepath information if the NativeLink (or equivalent) field is not populated, you can include an entry that maps the TextLink (or equivalent) field to the Digital Reef ocrpath field.

Viewing Load File Source Information

After import, you can verify Load File source information for a given document in the metadata field loadfiledocsource.This field either provides a semicolon-delimited list of the sources that were used to identify the document (for example, NATIVE;TEXT;IMAGE) or the value NONE if the Load File import did not provide any of those items for a document. Note the following:

NATIVE indicates that the native file for a given document was available based on Load File field mapping such as NativeFile/NativeLink to the darelativepath field.
TEXT indicates that a text file (for example, an OCR-processed text file) for a given document was available based on Load File field mapping such as TextLink or OCRPath to the ocrpath field.
IMAGE indicates that Load File images for a given document were available and processed separately based on an OPT or LFP Image Load File (available at the same location as the DAT Load File upon import).

The values in the loadfiledocsource field determine how representations for a document are made available:

When a native file is available (that is, the loadfiledocsource field includes the value NATIVE), that native is used for the native representation. Otherwise, an empty file is generated to represent the native.
When a text file is available (that is, the loadfiledocsource field includes the value TEXT), that text file is used for the text representation. Otherwise, text will be derived from the native.
When images are available (that is, the loadfiledocsource field includes the value IMAGE), those images are used to provide an image representation in PDF format.

Mapping Needed to Preserve Families

The Preserve Families option lets you require mapping of family information in an imported DAT file; when it is enabled, you must provide mappings to the extbegdoc and extbegattach DR metadata fields, as follows:

extbegdoc (generally required) — The external starting document with a document number (under Digital_Reef_Properties).

extbegattach (required) — The external starting attachment document with a document number (for example, for an email). This field is typically populated for the beginning attachment for an entire family, but a standalone document may also have this field populated. This field is categorized under Digital_Reef_Properties.

The following fields are also recommended to provide optimal family handling:

extenddoc (recommended) — The external ending document with a document number (under Digital_Reef_Properties). If this field is populated, the software will also populate the extdocrange and extdocattachrange fields.

extendattach (recommended) — The external ending attachment document with a document number (under Digital_Reef_Properties). If this field is populated, the software will attempt to preserve families when a family would otherwise be considered broken and generate an error during the Load File import.

If you clear this option, you are not required to map these fields, but you still have the option of doing so and preserving the family information. Without the required fields, all documents are treated as individual documents (without any MAG or DAG relationships).

Mapping Needed to Process Load File Images

To process the images for an OPT or LFP Image Load File (available at the same location as the DAT Load File upon import), you must enable and apply the Process Load File Images option on the Load File Settings tab, and you must supply a mapping entry for the following:

extbegdoc (generally required) — The external starting document with a document number (under Digital_Reef_Properties).

Optionally, you can provide a mapping entry for the following Digital Reef Properties:

extenddoc (optional) — The external ending document with a document number (under Digital_Reef_Properties).
extnumpages (optional) — The number of pages detected for a multi-page document (the number of pages for an attachment in a Load File Image PDF). A Load File field such as PgCount can be mapped to this field.

Upon successful processing of the images for an OPT or LFP Image Load File, the software creates a PDF that represents all of the images associated with a given Load File document.

After a successful import of a Load File with processed images, you can check the stored_image metadata field, which identifies External to indicate that the images were generated externally. You can also view the PDF created to contain the images for a document in the Image tab/view mode of the Document Viewer. The Image tab only appears for a document that has stored images as a result of a Load File Import with processed images, External Image Import, or an Export that requests PDFs.

Custom Metadata Field Mappings

At the Project level only, the Field Name list in the DR Metadata Fields section includes any Custom Fields that have been configured for the Project, as well as those created using the Custom Email Header Field Mapping section of the Project Index Settings. (Custom Fields at the Project level have the prefix drcustom- to avoid potential name conflicts with the predefined metadata fields.) If you find there is no appropriate DR metadata field for a mapping you want to include, you can create a Custom Field for it using Project Settings > Custom Fields.

Sample Load File Field Mappings

The following lists sample Load File Field Mappings (with no Transformers included).

Load File Field ---> DR Metadata Field

NativeFile ---> darelativepath

BegDoc# ---> extbegdoc

EndDoc# ---> extenddoc

BegAttach ---> extbegattach

Filename ---> filename

OCRPath ----> ocrpath