View and Manage System Storage

Home > Settings drop-down >System Settings > Provisioning > Storage
Project > Settings drop-down > System Settings > Provisioning > Storage

The System Settings > Storage page allows System Users in the IT Administrator role to review and manage the data storage used by Organizations. By default, System Users in other System roles can access this page to review and validate data storage. (For information about System-level permissions, see View and Manage System Role-Based Permissions.)

Data Storage for Organizations and Projects

Organizations in Digital Reef store four types of data. The first three—indexes, search results, and miscellaneous data—are collectively known as index storage and are stored together in a single index storage location that is shared by all Organizations system-wide. The index storage location must be configured by the IT Administrator before any Organizations can be created, and is typically placed on the fastest storage device available to optimize system performance. When a new Organization is created, a folder is created for it on the index storage, with subfolders for indexes, search results, and miscellaneous data.

The last type of data, known as document storage, consists of documents extracted from sources files when a new Data Set is added to a Project. These documents are stored on one or more document storage locations configured by the IT Administrator. At least one document storage location must exist before any Data Sets can be created.

When a new Data Set is created within a Project, the documents extracted are stored on the currently available document storage location, within a folder for the Project's Organization. This folder is created the first time an Organization stores documents on a location and is subsequently used for all document storage by that Organization on that location. A subfolder is created within the Organization's folder for each new Data Set. Thus, the entirety of each Data Set is always stored on the same document storage location, but the Data Sets of both Organizations and Projects can be distributed across multiple locations, as shown in the following illustration:

Note: In addition to index and document storage, Digital Reef maintains a system storage depot, primarily for archiving Project audit logs and storing virus definitions.

Document Storage Management

When there are multiple document storage locations, the IT Administrator manages them to determine which is used at any given time for newly-extracted documents; over the long run, this also determines how document storage is distributed across the locations. The IT Administrator has three tools for managing existing document storage, as follows:

  • Ranking the locations in order of priority. (This may be based on factors such as device type, capacity, and network latency.)

  • Setting usage thresholds on each location to trigger automatic deactivation/reactivation. When a location's usage exceeds its deactivation threshold (for example 80%) it becomes read-only for new Data Sets; when it then falls below its reactivation threshold (for example 70%), because usage has been reduced or capacity added, it becomes read-write for new Data Sets again.

  • Disabling locations for new Data Set storage as required, for example for maintenance, and then re-enabling them.

New Data Sets are always stored on the available document storage location, which is the highest-ranked location that is both enabled and active.

Bear in mind these additional points:

  • When a document storage location is automatically deactivated or manually disabled by the IT Administrator, existing Data Sets stored on it remain accessible and writable.

  • When a Data Set's source files are copied from their import location to document storage they are always copied to the Data Set folder containing the documents extracted from them, ensuring that the source files and extracted documents are stored together.

  • The IT Administrator can always add document storage locations as long as there are storage resources available.

  • The IT Administrator can delete a document storage location only if there are no Data Sets stored on it. (In practice, this requires that all Projects storing Data Sets on the location be disabled and then approved for deletion by a System Administrator responding to an Organization Administrator request.)

The System Storage Table

The table of index and document storage locations on the System Settings > Storage page includes the following columns:

  • Type – Either Index (one only) or Document (one or more).

  • Name – The name assigned to the storage by the IT Admin when configuring it and cannot be changed.
  • Enabled – If the checkbox is not selected, the location was disabled by the IT Administrator and cannot become available for storage of new data sets until it is enabled.
  • Mode – If Read/Write, the location is active and can become available to store new Data Sets; if Read-only it was automatically deactivated for exceeding its configured usage threshold and cannot.
  • Deactivate above , Reactivate below – Usage thresholds determining whether the location is active and can become available to store new Data Sets.
  • Used, Available, Total – Usage statistics for the location. Statistics for all locations are recalculated at frequent intervals.
  • Path, File System – The path and file system type entered when the IT Administrator configured the location; cannot be changed.

You can click and drag column headers to the left or right to reorder the columns.

System Storage Actions

The IT Administrator can take the following actions on the System Settings > Storage page. When you take one or more of the first three actions to manage existing document storage locations, you must click Save to save your changes (or click Discard to dismiss them).

  • Change the document storage priority ranking. Use the drag area at the left-hand end of each document storage location to change their priority ranking by moving one or more up or down.

  • Disable or enable a location. Clear or select the check-box under Enabled to disable or enable it for new Data Set storage.

  • Change a location's deactivation or reactivation thresholds. Enter usage percentages in the input boxes to change the usage level at which a location is deactivated for new Data Set storage and reactivated when usage is reduced. (The location must be enabled for you to change these.)

  • Create a new storage location. This can be the index storage location (if it does not yet exist) or a document storage location. Clicking the New Storage button in the upper left displays the New Storage dialog, which lets you enter a name for the location, designate it as index or document storage, enter the IP address or hostname of the storage host, select your intended storage path from those available on the host, and specify the file system as NFS or PanFS®. When you have provided all the needed input, click the Validate button to validate that the storage you've configured exists and is usable. If it is, you can click OK to add the storage location; if not, correct the entries as needed and validate again.

  • Validate a location's storage. Click the context menu control at the right-hand end of a location, select View/Edit to display the View / Edit Storage dialog, and click the Validate button to validate that the storage specified in the location exists and is usable.

  • Edit a location If there is no data currently stored on a location you can change the storage it represents by clicking the context menu control at the right-hand end of a location, selecting View/Edit to display the View / Edit Storage dialog, and updating the storage host, storage path, and file system as NFS or PanFS®. You must click the Validate button to validate that the updated storage, as above, before clicking OK to save the updated location.

  • Delete a storage location. If a location is not storing any Digital Reef data, you can disable it if it is enabled, click the context menu control at the right-hand end of each location, select Delete (which is not available if there is data stored on the location) to display the Delete Storage dialog, and click OK to confirm deletion of the storage.

Note: By default, all System Users can select System Settings > Storage to display the current locations, including their usage statistics, but only the IT Administrator can take the above actions. The one exception is validating an existing storage location as described above, which all System Users can do.

Storage Usage by Organizations

The bottom of the system storage page lists all Organizations, the date and time at which they were created, the amount of storage they are currently using, and the last date and time at which their storage usage was calculated.

You can see more detailed information for a specific Organization and recalculate its storage usage by selecting Organization Settings > Storage. You can see the total storage used by all Organizations, in a list similar to the one on the system storage page but with more detail, and recalculate storage usage for all of them by selecting System Settings > Organizations. You can also download