Sub-navigation

Drill down further into more specific configuration and information options by clicking through the sub-navigation items. In the large view, these appear as tabs:

In the small view, these become "accordion" panels. Clicking one of the collapsed accordions will expand it:

Item Tools

Certain views contain tables of data. These tables typically represent lists of editable items, such as lists of users present on the system. At the end of each row is an icon which opens a pop-up menu enabling actions such as editing or deleting the item.

Toolbar

Actions are often available in a toolbar above the table. Some of these are actions which are global to the table, such as creating a "new" item (eg. a new HotFolder). Others are redundant versions of the same actions available in the "Item Tools", but applicable to either single or multiple selections (Item Tools are for a single item). Tables that offer search functionality will also provide a search box in this toolbar.

Sign Out

Allows the administrator to sign out of the application and clear their credentials from the browser.

Note: It is possible to set FileCatalyst HotFolder to not request credentials for remote administration. In this case, the Sign Out button is replaced by a Refresh Session button, which clears the current session and refreshes the page.

Pause UI

Allows the administrator to visually pause updates to the UI. The application will be functioning as usual in the background, but the current view has its state paused. Activities which refresh data (such as sorting, page navigation, and editing individual items) will cause the lastest version of the data to appear. While paused, an indicator appears in the main application, and this menu option becomes "Resume UI"

Help

Opens this help document.

About

This invokes a pop-up dialog with information about installation environment and version.

In addition to a Close button, the dialog also contains the Diagnostic Report button.

Diagnostic Report should only be pressed if requested by the support team, at which time it will create a package of files to be used during troubleshooting. No additional visual feedback is presented to the user after pressing the button.

Task Schedule

The task schedule displays all configured tasks in FileCatalyst HotFolder. It shows the HotFolder, Site Name, and Direction (upload or download), as well as the Status (Idle, Transferring, Queued) and the date/time of the next execution.

Creating a New Task

Once you have defined at least one “Site” and one “HotFolder”, you may create a task. To do so, click the “+ New” button found in the toolbar above the schedule table. The Create Task dialog will appear. The form is pre-populated with default settings, with “Task Name” being the only required field. Once the task is configured, click “Create Task” to save it.

Managing Existing Tasks

To edit, delete, duplicate, disable or enable an existing task, or to view its activity, first select it on the schedule table. Next, click on the corresponding action in the toolbar. You can also click on the row's tool icon and select from the options on the sub-menu that appears. If multiple tasks are selected, you may use the menu from the toolbar to delete, execute, cancel, enable, or disable all of the tasks that are currently selected.

Execute a Task

To execute a task, highlight a task and click the “Execute” button.

Cancel a Task

To cancel a task, highlight a task and click the “Cancel” button.

Pause / Resume Scheduler

Clicking “Pause Scheduler” will prevent any further scheduled task execution. However, any tasks in progress will complete their execution. You may resume the scheduler by clicking the same button, which is now labeled, “Scheduler PAUSED - press to resume”.

Concurrent Tasks

The HotFolder can be configured to support execution of multiple tasks at a time. To do this, check the “Allow concurrent tasks” checkbox. To disable execution of concurrent tasks, uncheck the checkbox.

Title

Title of the activity view. It will show the task name, HotFolder used on the task, direction, and the FC Server Site ID involved in the transfer.

General Task Information

This table contains current information about the task. If the transfer is not in progress the table will be in an idle state. During transfer, it shows:

• Progress (in terms of number of files sent)
• ETA (estimated time of arrival)
• Elapsed Time
• Status (either a progress in terms of percent, or a description of current activity such as "Calculating MD5")
• Bytes Transferred
• Average Transfer Rate (for single-client) or Total Transfer Rate (for multi-client) for the current excecution
• Current RTT (latency)
• Average Packet Loss (for single-client) or Total Packet Loss (for multi-client) for the current execution

This header of the table also contains buttons to allow editing, starting or stopping the task in the same manner found in the section “Managing Existing Taks”.

Current File / Clients

This table contains information about current file transfers: the “current file” when single-client, or a list of active “clients” when multi-client. The available columns change depending on whether your task is set for single or multi-client usage.

Current File (Single Client):

• Filename
• ETA (estimated time of arrival)
• Elapased Time since the file transfer or client started
• Status (current activity such as "Calculating MD5")
• Progress %
• Transfer Rate (current)
• Packet Loss (current)

Clients (Multi-Client):

• Worker ID
• Processor ^*
• Filename
• Status
• Progress %
• Transferred (amount of data)
• File ETA
• Packet Loss
• Transfer Rate

* A Processor is a section of task execution that is independent from the transfer itself. For example, checking for file growth in the "Wait for file growth to cease" Advanced Progressive transfer option.

Note: If you are seeing a a multiple clients transfer the Show Overall Graphs and Logs will apear. When it is clicked the overall information for the task is displayed.

Graph, Visualization and Logs

Select an appropriate tab in order to change the information displayed below. “Visualization” is only available for single transfer threads. In other words, it is considered always-on for single-client tasks; for multi-client, an individual client must be selected from the “Clients” table above.

“Graph” (seen above) contains a line graph of effective and current transfer rate (in kbps):

“Visualization” is bundle of information representing resource usage for Source and Destination:

“Logs” shows the current logs for the task. Logs can be filtered by using the search form at the top of this container:

• Search: Clicking Search will open a new tab that displays the current log filtered for the provided keyword. You can match case should you wish.
• Scroll Lock: Enabling Scroll Lock will prevent the screen from scrolling to the latest text added to the text area.
• Retrieve: will retrieve a user provided number of lines from the logs available on the server.
• Copy to Clipboard: copies the text from the current log window to the clipboard.
• Clear: clears all of the text in the current text area.

Task Name

The name of the task.

Select HotFolder

Specify the HotFolder used by this task.

Select Site

Specify which Site is used by this task

Select Direction

Specifies whether to upload from this HotFolder to the server, or download files from the server to this HotFolder.

Frequency Options

A set of five radio buttons allows you to select the frequency of this task:

User Triggered

The default “user triggered” option requires the user to open the Scheduler, select a task, and click "execute". This default prevents accidental execution of tasks while the user is still configuring their HotFolder, but is typically changed to one of the scheduling options below.

Once

Selecting this option will enable corresponding fields allowing you to specify a date and time for a single execution. The task remains in the scheduler, allowing you to specify a new date or to manually execute.

Every minute

This is the most common choice. The scheduled task will check every minute to see if a new file transfer is required.

Always On + File System Events

As soon as the task completes, it will repeat. Essentially, the transfer filter is being monitored continuously to see if a new file transfer must be initiated. Selecting this option will also automatically check the “Monitor File System Events” box at the bottom of the page.

Note 1: The "Always On + File System Events" option is made available by enabling "Advanced View" on the HotFolder

Note 2: Using "File System Events" on network drives (Samba, CIFS, NFS, ATP) is not recommended. Any task transferring to or from mounted network drives should instead use a normal transfer schedule (ie: once a minute).

Scheduled Execution

Selecting this option enables a range of fields to create a repeated task. As the scheduled task is created, a plain-language representation of the task is updated at the top of the section.

After selecting the “Day(s)” for execution, you must select an interval. This may be:

• Once, at a particular time of day.
• Every X number of hours, at the specified minute past the hour.
• Every X number of minutes.
• Every X number of seconds.

Monitor File System Events

When this option is enabled, the task will first complete transfer of all files, and will then enter a “monitoring” state. When in the monitoring state, HotFolder will watch the file system at the source (local folder on uploads, and server folder on downloads) for events such as file creation, modification, deletion, and rename. When these events occur, the task will immediately transfer, delete, or rename the affected file(s).

Once the task enters monitoring mode, it will stay in this mode indefinitely or until it is canceled or an error occurs. The task is still considered to be running while in this state.

This option is useful when dealing with very large file sets that must be kept in sync. For a scheduled task, recursive listings, comparisons, and filtering determine which files have changed. On large file sets this process may takes several minutes and consume resources. However, with file system event monitoring enabled, recursive listings are not needed and resources will only be used when a file is changed.

Note on Monitoring Network storage: In many cases FileCatalyst is used to monitor directories that do not reside on the same physical machine but rather on a network attached storage. Depending on the OS and protocols used, the FileCatalyst software may not be notified of events that are triggered at a network location. For example, if a Linux machine is running FileCatalyst Server and has a user home directory pointing at a mounted directory (i.e. /mnt/network_dir), and changes are made to that directory from yet another machine pointing at that same directory, the events may not be registered.

Note 1: The "Monitor File System Events" checkbox is made available by enabling "Advanced View" on the HotFolder

Note 2: Because of how the multi-client manages itself, a multi-client transfer will identify as “Idle” when it is monitoring.

Protocol

This option allows you to specify the transfer mode you wish to use to connect and transfer files to/from the server. Options are UDP, TCP, HTTP (when HTTP Servlet is enabled), and Auto Detect. By selecting Auto Detect, the HotFolder will cascade from UDP to multi-stream TCP to single-stream TCP and finally to HTTP in order to find the optimal mode.

Auto-Resumes

With this option enabled, FileCatalyst HotFolder will automatically recover from failed network connections. This setting takes precedence over the “Transfer Files With Unique Name” options found within the incremental settings. If a file can be resumed and auto-resumes are enabled, the destination file will be resumed and a unique file will not be created at the destination location.

Congestion Control

When Congestion Control is enabled, FileCatalyst will attempt to dynamically modify its transmission rate in accordance with changing network conditions. This is useful if the end to end link speed is not known, or if the network is also being used by other applications with which FileCatalyst would otherwise interfere.

Start Rate

The rate at which FileCatalyst will begin transmitting data. It will then attempt to speed up the transmission rate until it either hits congestion, or it reaches the target rate specified in the bandwidth scheduler.

Three options are available:

• Use Target: Attempt to transfer at maximum speed allocated by the task. Will slow down if congestion is detected, but initial rate may adversely effect network link if it cannot sustain the transfer speed.
• Specify: Manually specify a start rate for transfer speed. We recommend a maximum of 1/10th of the known line speed if this is not a dedicated line.
• Auto: Have the transfer automatically launch an auto-detect when the task starts. This is the best solution for single file transfers. It is not recommended for multi-file session transfers, as each client will attempt to auto-detect transfer speed.

Congestion Control Strategy

RTT-based Strategy

RTT-based strategy works by establishing a baseline average RTT before the data starts to flow. Once the transmission begins, RTT is monitored continuously. While the RTT stays within a certain range of the baseline RTT, the speed of the transfer will be increased. Once the RTT begins to go above a certain range, the speed is decreased. How much the RTT is allowed to spike above the baseline average is controlled by the Congestion Control aggression setting FileCatalyst provides.

This type of congestion control works well on wireless or satellite links where there is packet loss from other sources besides congestion. TCP will slow down, for example, when a packet is lost due to interference or being too far from a cellular tower. When FileCatalyst is using the RTT based congestion control it ignores individual packet losses and focuses only on RTT. For these scenarios, FileCatalyst continues to maintain high speeds through the kinds of packet loss ”hiccups” that would trigger decreases speeds under TCP.

Loss-based Strategy

There are circumstances in which the RTT-based strategy may not work properly. For example, when a router's queue is very small, the RTT may never spike when there is congestion. The RTT will remain low, and therefore FileCatalyst would continue to increase its transmission rate even when there is congestion present. For these scenarios, the only way to detect the congestion is using a packet loss approach. As outlined previously, packet loss may come from other sources besides congestion, so this mode is best used on terrestrial (land-based) networks where all packet loss is due to real congestion.

Loss-based congestion control mode reacts to packet loss by slowing down (just like TCP); however, FileCatalyst is not nearly as aggressive as TCP. The primary reason for using a UDP-based file transfer solution like FileCatalyst is because TCP is so aggressive in its congestion avoidance that it often ends up under-utilizing your link. The FileCatalyst loss-based congestion control algorithm was designed such that it is able to maximize link utilization while still avoiding congestion. Like the RTT-based congestion control, the loss-based mode can be tuned to be more or less aggressive. More aggressive settings allow for a higher percentage of packet loss before slowing down, while for more passive settings the opposite is true.

Rate-based Strategy (Labs)

There are circumstances in which both Loss and RTT will not work properly. If UDP packets are buffered in stacks instead of queues (last in first out vs. first in first out), this interferes with the RTT strategy as the RTT can skip along the top of the stack buffers since it is last in first out. Stack buffering interferes with the Loss-based strategy as loss detection is based on packet ordering and stacks reverse packet order.

Rate-based congestion control mode compares the rate we are sending packets to the rate we are receiving them. If we are receiving slower than we are sending, then we know that the network, at that time, cannot handle the current send rate, and we slow down.

Aggressiveness

This setting allows the threshold (or range) that is used when comparing the current RTT to the initial RTT to be increased or decreased. If aggression is increased, the current RTT will be allowed to increase to a higher level before it is considered to be congestion. Conversely, if the aggression is lowered, smaller increases in RTT will be considered as congestion and transmission rates will slow down. In this way, the congestion control may be tuned to the sensitivity of the network on which the transfers are being performed.

Verify File Integrity After Transfer

During all UDP transfers, as blocks of data are transferred, they are verified using MD5 against the same block in the source file. This is done on the fly, and incurs no delay after a transfer completes. This provides a very high level of protection against corruption. Note that FTP/TCP mode transfers are verified on the fly as well, but it is built into the TCP protocol. FileCatalyst provides the additional option to perform a full MD5 checksum after each transfer instead of on the fly.

By checking the "Verify File Integrity After Transfer" option, integrity of the transferred file will be verified byte by byte on storage using MD5 after the transfer is complete. This option can be used in one of three modes:

Full

This will perform a byte by byte comparison of the entire file when the transfer is complete. Although this consumes less CPU on an ongoing basis, there is a delay between files being transferred as the checksum is calculated. If the files are large, it may cause delays of several seconds or even minutes depending on the speed of the I/O on the sending and receiving machines.

Concurrent

This will open up a separate session to perform checksums without interrupting the file transfer. For very fast transfers, disk I/O bottlenecks (competing between ongoing transfers and MD5 checksums) may result in lowered throughput.

Partial

Partial MD5 verification occurs after the file is completed, however it only compares certain portions of the file, and only at logical intervals that coincide with the transfer characteristics. This vastly decreases the time to perform the MD5 check. Note, however, that although the file is almost certainly intact, the Partial check cannot absolutely guarantee with 100% certainty.

Note: Partial MD5s are not available against servers older than 3.7. If partial MD5s are still selected at the time of transfer, then full MD5s will be used instead.

Keep File Modification Timestamp

If this item is selected, the modification date and time will be preserved when files are transferred to the server.

When multi-client is enabled, folders are also maintained. This behaviour is different from our legacy single client transfers. If you desire the legacy behaviour, it can be reverted via setting this system property: unlimited.fc.legacy_timestamp

Force File Ownership

By selecting this option on a download task, all files transferred will have their operating system user ownership set to the value specified.

Note: On Linux or Mac OSX, an additional field allows configuring group ownership.

Note: If the Maintain UID/GID system property is enabled, the user and group data will be updated to accept user and group identifiers. Please note that this functionality is only on Linux and Mac OSX operating systems.

Note: If Keep File Permissions is enabled when Force File Ownership is enabled, the "Keep File Permission" option is automatically deselected.

Transfer with Temporary Name

If this option is enabled, FileCatalyst will always upload or download files using a temporary name, and rename it back to the original name when the transfer is complete and verified. FileCatalyst Server will not report these temporary files when other HotFolder or FTP clients request file lists. This prevents them from downloading partial files. You may choose to either prepend the ".fctmp." prefix, or append the ".tmp" suffix to the files until they are finished transferring.

Task Priority

This allows you to specify the priority of a task. By default, bandwidth is split evenly between all running tasks. Setting the priority of a task will increase/decrease the weight of a task when bandwidth allocation is performed.

Enable Multi-Client (multiple concurrent file transfers)

HotFolder allows a single transfer task to utilize more than one transfer connection in order to push files across concurrently.

Multi-Client is particularly useful when transferring dynamic file sets (where data is being added in real-time to multiple files in a directory). When combined with progressive transfers, multi-client allows multiple growth files (log files, video transcoding, etc) to be sent from one directory.

NOTE: as you are utilizing more than one transfer connection to a server, this may potentially impact server-side license restrictions. As one might expect, Multi-Client utilize more resources (memory, ports) as more clients are added.

Multi-Client does not currently support Filecatalyst Workflow. Selecting specifying a Filecatalyst Workflow server will automatically execute the task as a single client transfer.

Automatically rescan and add new files to the running task

By default, the HotFolder scheduler will not re-execute a task if it is currently running (only one instance of the same task is allowed to run at a time).

Because multi-file sessions are ideally suited for transfers of growth files (ie: live streaming video) which may take an extended period of time to complete (2+ hour event), rescheduling allows you to rescan a directory even if the original transfer has not yet completed (live video event still on-going). New static files (or growth files) will get picked up by the task at scheduled intervals.

NOTE: Transfer cache is enabled and locked when this feature is used. Files will be sent only once when using this feature. Transfer cache can be manually cleared in the Fileset tab.

NOTE: When transferring a progressive file that grows much slower than our transfer rate, a situation occurs where we are always waiting for a file to grow before sending the next chunk. We call this chasing the tail. When this is happening a progressive file might not grow for a long enough time that we think it has finished growing and consider the transfer complete. With reschedule enabled, the next scan will pick up the file if it has indeed grown. Since we cannot know if the file has just grown, been replaced, or been edited, we need to treat it as a new transfer. This only affects our statistics displayed as we will increment the amount of data and number of files transferred. However this can be corrected increasing the time a progressive transfer will wait for the file to grow (see progressive transfers)

Enable Auto Archiving of small files.

This option will group together small files (10 MB or less in size) archiving them and sending them over as a single transfer. Use this feature if your transfers will contain many small files. The maximum size for an individual archive is 100MB, after which a new one will be created. Previous archives are transferred and extracted while new ones are being created.

Keep File Permissions (OSX/Linux/Solaris Only)

When this item is checked, file permissions will be saved when files are transferred.

Note: Source and Destination systems should already have the user account created and available in order to keep the file permissions.

Multi-Client changes from single client and limitations

Due to the nature of how parallel transferring works, it is highly recommended to use the AutoArchive of Small Files option in Multi-Client. This will keep the overhead of HTTP POSTs to a minimum during a transfer with a larger data set.

Changes from single client

Multi-Client has an added layer of redundancy: if an individual client fails a transfer, that transfer is pushed back onto an internal transfer queue to be attempted by a different client a set number of times before actually failing. This means that in cases where Single Client failed consistently, multi-client may succeed. Multi-Client can use auto archiving, incremental and compression simultaneously. Small files will still be auto archived and use the compression settings, ignoring the incremental while the larger, non archived files, will use incremental and compression. Normally archiving and incrementals are mutually exclusive for single client transfers. Note that while multi-client is enabled, some compression options in the Data Minimization pane will be disabled as they are handled by the auto archiving algorithms.

Limitations

Multi-Client transfers are not compatible with Filecatalyst Workflow so multi-client settings are ignored for transfers to a Filecatalyst Workflow site.

Enable Incremental

Incremental tasks limit transfer to files that are new or have been modified. This works in conjunction with “Do nothing with source files” option on the Post Task tab to allow you to treat the HotFolder as a working directory with only your modified files being transferred. With incremental enabled, the user also has the option to remove MD5 checks and only check the size of the file. Incremental options are as follows:

Transfer Entire File

Transfer the entire file if the remote file and the source file do not match.

Transfer File Deltas

Transfer the portions of the file that have changed (the "deltas"). These changes are then merged with the file on the target to recreate the source file.

Transfer Entire File with Unique Name

Transfer the entire file, with the target filename using a unique identifier. The unique name will use the following pattern:

<filename>_X.<extension>

Where X is a unique integer value.

NOTE:: If auto-resumes are enabled within the current task, a new file with unique name is created only if the destination file cannot be successfully resumed.

Transfer File Deltas with Unique Name

This option will only transfer the portions of the file that have changed. These changes are then merged with the file on the target to recreate the source file. The final file will be renamed such that is has a unique name using the following pattern:

<filename>_X.<extension>

Where X is a unique integer value.

NOTE: Incremental delta transfers are bypassed under circumstances where it is more beneficial to send the file normally. Deltas are bypassed if:

• either file (source or destination) is smaller than 200KB in size
• the destination file size is less than the square root of the source file size
• the minimum delta file will be larger than transferring the entire file

There are possible side effects using deltas in conjunction with Progressive transfers noted in this knowledge base article here.

Enable Compression

Compressing files as they are being sent to the server may reduce transfer time. However, if all files are already in compressed formats, it will only serve to add additional overhead to the transfer. Once you understand the compression options, you will know which method (if any) is right for your situation.

Compression Method

Two compression methods are available: Zip Deflater and LMZA.

• Zip Deflator: Zip Deflator is the standard compression library utilized by FileCatalyst, and is used for on-the-fly compression and for single zip archive (bundles multiple files together before transferring).
• LMZA: LMZA is an additional compression method available for on-the-fly compression (single archive is not supported), and allows a very high level of compression. The algorithm is very CPU and memory intensive for the sender. Profiling the transfer via HotFolder or Server visualization window and configuration of threads (block sender threads for transmitter, writer threads for receiver) may be required for achieving high speed transfers.

Single Archive

When sending several small files, there is an overhead to initiate the transfer of each file. With a high RTT, this means that far more time is spent “setting up” the transfer compared to actual data transmission. Furthermore, due to small file sizes, the transfer task never reaches full speed.

With the “Single Archive” option enabled, HotFolder will put all the files into a single package, transfer to the server, then extract it. This consolidates set-up and teardown, and allows the transfer task to reach its maximum speed.

NOTE: Single archive transfers are not compatible with Multi-Client transfers and will be disabled if Multi-Client transfers are enabled.

NOTE: If progressive transfers are disabled while enabling archiving, the archive file (tar/zip) must first be created locally before being transmitted. This may cause a noticeable delay before the transfer begins, but the total time for the transfer should still be significantly less.

NOTE: The maximum size of an individual zip file is 4GB, as is the maximum size of any single file contained in the zip file. For file sets exceeding 4GB in total size, please use the “Archive size limit” feature (described in the next section) to limit the size of each zip file created to under 4GB. If the zip file size is over 4GB due to no limit or a single file over 4GB in size, the zip file will not be able to be unzipped and will cause an error.

Archive size limit

This feature will limit the size of the zip archives to the specified value (in MB) when possible. If a file set is too large to fit into a single zip of the specified size, multiple zip archives will be created. These zip archives are automatically extracted and cleaned up as they are transferred. At most 5 archives will be created at one time. Additional archives are generated only after the oldest file is deleted. This helps reduce disk space required.

Compression Level Settings

This slider will allow you to select the level of compression to be applied when compression is enabled. Depending on the types of files and bandwidth constraints, it may be desirable to use more or less compression. For example, on a high bandwidth connection on which volume of data transfer is not measured or is otherwise not a concern, it may be faster to send the data either uncompressed or with low compression. However, on a low bandwidth satellite network it may be more optimal to use maximum compression.

Exclude Files

Any files in the HotFolder directory or server matching the given expression will be excluded from compression. You may specify multiple file filters delimited by a comma. This option also supports wildcards (*). For example, if you wish to ignore all Zip files, you would set this value to "*.zip". If you wish to ignore all Zip files, and MP3 files, you would set this field to "*.zip, *.mp3".

NOTE: The compression option is only available for either UDP or TCP based transfers. HTTP based transfers currently do not support this option.

Disabled

This option turns off progressive transfers. Files will not continue to transfer as they grow.

Basic Progressive

Files will transfer as they grow. This is the most basic of progressive features. This is our former "Enable Progressives" feature.

Advanced Progressive

If Advanced Progressive is enabled, one of the four sub-options (described below) must be selected. If one of these sub-options is not selected, the transfer will default to Basic Progressive. Note that in single client transfers only the "Transfer these files as they grow" sub-option will be permitted, as the other options are multi-client specific.

For each selected sub-option, an inclusive filter must be specified (e.g. "*.mxf" for mxf files) to indicate which file types the functionality should be applied to. Files that do not match this filter will be transferred as per Basic Progressive. A wait time (in seconds) may also be specified for each sub-option to modify how long the client waits to check if a file has grown.

Transfer these files as they grow:

These transfers will check for files matching this particular filter within the specified number of seconds. If no changes to the file have occurred within the set time, the file growth will be considered complete. This option is useful for files that grow in intervals with pauses, or if a hard drive is heavily taxed and the actual growth of the file is slower than normal.

Wait for file growth to cease:

These transfers will check for files matching this particular filter within the specified number of seconds. If no changes to the file have occurred within the set time, the file growth will be considered complete. The transfer will only happen after the file growth is considered complete. This option ensures only complete files are being transferred and not ones that are still growing.

Re-Transfer Headers/Footers when complete:

These transfers will wait for a specified amount of time, then the header and footer bytes (as configured) will then be sent and patched into the destination file. This option is useful when the file behaviour is specifically known to change the headers and footers during file growth. It is a low impact, focused delta transfer.

Re-Transfer Deltas when complete:

These transfers will wait for a specified amount of time, then the entire file will be checked for deltas. This option is useful in files where the growth behaviour is unknown but atypical. Checking the entire file has considerably more overhead than the "re-transfer headers/footers"; however, this option works well with a broader range of media files. The default byte limit of header and footer is 1MB and it can be configured (using FC.hotfolder.config.max.headerfooter.bytes) in fchf.conf.

Limitations

It is important to note that if the process growing the file locks said file (as is the case with Windows File Copy), progressive transfers will not work and the file will not transfer until the lock is released.

This feature is designed to handle different types of file growth and/or change during the course of a transfer. Some files grow in a predictable pattern, such as tail-appending log files, while other files may grow and/or change in ways that require more advanced options, such as slow growth files or media files with multiple parts (e.g. multiple audio channels or audio/video files).

Due to the fact that the way a file grows is completely out of our control, if you have growth files, you must configure these options to properly represent that growth or the results can be undefined. For example, if you have a progressively growing file that changes once every 10 seconds but you set up a timeout delay for only 3 seconds, this can fail in a number of different ways depending on when the new data is appended, how it is appended and where we are in the transfer process.

Since some of the dynamic file options will re-transfer all or parts of a file after it is completed, if After Transfer is selected under Verify File Integrity, the verification is moved to after the above options execute.

Remote Folder

By specifying a Remote Folder name, a folder by this name will be created or found on the FileCatalyst Server and all files will be transferred to and from that folder.

Enable Dynamic Folders

Selecting this will force the HotFolder to dynamically replace tags in the Remote Folder path when uploading. It supports the following tags:

• <%HOSTNAME%> is replaced with the hostname of the machine running the HotFolder.
• <%TIME%> is replaced with the time in the format: HH.mm.ss
• <%DATE%> is replaced with the date in the format: yyyy.MM.DD

File Filter

FileCatalyst HotFolder tasks allow you to define a filter which can be used to include or exclude particular files from the transfer. This filter is applied at the beginning of the transfer and can be configured through a wide range of available options that allow you to tune your fileset as needed.

Filter Phrase:

This option allows you to define a text phase that will filter any files in the HotFolder directory or on the Server that match the expression. To disable file filters entirely, leave this option blank.

Filter Mode:

This option allows you to define the type of filter that you wish to use. The available filter modes are as follows:

• Exclude Filter: Will exclude any files that match the given wildcard expression. The format for the expression is a comma-delimited list of files names, using a standard wildcard (“*”) character.

• Include Filter: Will only include files that match the given wildcard expression. The format for the expression is a comma-delimited list of files names, using a standard wildcard (“*”) character.

• Regular Expression Filter: This option allows you to build a complex regular expression to select which file(s) you want to include in a transfer.

  For additional documentation to how Java implements regular expressions, the following sources may be used:

  • https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
  • http://www.sitepoint.com/article/java-regex-api-explained/

  Construct	Matches
  x	The character x
  \\	The backslash character
  \0n	The character with octal value 0n (0 <= n <= 7)
  \0nn	The character with octal value 0nn (0 <= n <= 7)
  \0mnn	The character with octal value 0mnn (0 <= m <= 3, 0 <= n <= 7)
  \xhh	The character with hexadecimal value 0xhh
  \uhhhh	The character with hexadecimal value 0xhhhh
  \t	The tab character ('\u0009')
  \n	The newline (line feed) character ('\u000A')
  \r	The carriage-return character ('\u000D')
  \f	The form-feed character ('\u000C')
  \a	The alert (bell) character ('\u0007')
  \e	The escape character ('\u001B')
  \cx	The control character corresponding to x
  Character classes
  [abc]	a, b, or c (simple class)
  [^abc]	Any character except a, b, or c (negation)
  [a-zA-Z]	a through z or A through Z, inclusive (range)
  [a-d[m-p]]	a through d, or m through p: [a-dm-p] (union)
  [a-z&&[def]]	d, e, or f (intersection)
  [a-z&&[^bc]]	a through z, except for b and c: [ad-z] (subtraction)
  [a-z&&[^m-p]]	a through z, and not m through p: [a-lq-z](subtraction)
  Predefined character classes
  .	Any character (may or may not match line terminators)
  \d	A digit: [0-9]
  \D	A non-digit: [^0-9]
  \s	A white space character: [ \t\n\x0B\f\r]
  \S	A non-whitespace character: [^\s]
  \w	A word character: [a-zA-Z_0-9]
  \W	A non-word character: [^\w]
  POSIX character classes (US-ASCII only)
  \p{Lower}	A lower-case alphabetic character: [a-z]
  \p{Upper}	An upper-case alphabetic character:[A-Z]
  \p{ASCII}	All ASCII:[\x00-\x7F]
  \p{Alpha}	An alphabetic character:[\p{Lower}\p{Upper}]
  \p{Digit}	A decimal digit: [0-9]
  \p{Alnum}	An alphanumeric character:[\p{Alpha}\p{Digit}]
  \p{Punct}	Punctuation: One of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
  \p{Graph}	A visible character: [\p{Alnum}\p{Punct}]
  \p{Print}	A printable character: [\p{Graph}]
  \p{Blank}	A space or a tab: [ \t]
  \p{Cntrl}	A control character: [\x00-\x1F\x7F]
  \p{XDigit}	A hexadecimal digit: [0-9a-fA-F]
  \p{Space}	A whitespace character: [ \t\n\x0B\f\r]
  Classes for Unicode blocks and categories
  \p{InGreek}	A character in the Greek block (simple block)
  \p{Lu}	An uppercase letter (simple category)
  \p{Sc}	A currency symbol
  \P{InGreek}	Any character except one in the Greek block (negation)
  [\p{L}&&[^\p{Lu}]]	Any letter except an uppercase letter (subtraction)
  Boundary matchers
  ^	The beginning of a line
  $	The end of a line
  \b	A word boundary
  \B	A non-word boundary
  \A	The beginning of the input
  \G	The end of the previous match
  \Z	The end of the input but for the final terminator, if any
  \z	The end of the input
  Greedy quantifiers
  X?	X, once or not at all
  X*	X, zero or more times
  X+	X, one or more times
  X{n}	X, exactly n times
  X{n,}	X, at least n times
  X{n,m}	X, at least n but not more than m times
  Reluctant quantifiers
  X??	X, once or not at all
  X*?	X, zero or more times
  X+?	X, one or more times
  X{n}?	X, exactly n times
  X{n,}?	X, at least n times
  X{n,m}?	X, at least n but not more than m times
  Possessive quantifiers
  X?+	X, once or not at all
  X*+	X, zero or more times
  X++	X, one or more times
  X{n}+	X, exactly n times
  X{n,}+	X, at least n times
  X{n,m}+	X, at least n but not more than m times
  Logical operators
  XY	X followed by Y
  X|Y	Either X or Y
  (X)	X, as a capturing group

Filter Examples

Here are some example settings that can be used to help filter out necessary files in a transfer:

1. Mode: Exclude
   Target: File name
   Phrase: *.pdf

   Result of settings:
   Single phrase filter that transfers all files that do not end with the '.pdf' extension

2. Mode: Exclude
   Target: File path
   Phrase: folder1/**/*.docx,folder3/**/*.png

   Result of settings:
   Two phrase filter that transfers all files expect '.docx' files found in folder1, and '.png' files found in folder3

3. Mode: Include
   Target: File name
   Phrase: *.mov,*.mp4

   Result of settings:
   Two phrase filter that transfers all files that end with the '.mov' or '.mp4' extensions

4. Mode: Include
   Target: File path
   Phrase: **/videoProduction/**/4K Footage/**/*.mkv

   Result of settings:
   Single phrase filter that transfers all '.mkv' files that exist within the '4K Footage' sub-folders contained inside the 'videoProduction' parent folder.

5. Mode: Reg Expr
   Target: File name
   Phrase: ".*?\.dat$"
   Regular Expression Definitions:
   
   

   Expression Section	Description
   ".*?"	Any character, 0, 1, or multiple times with reluctant matching
   "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
   "dat"	The exact characters 'dat' all lowercase
   "$"	Must be at end of the line

   Result of settings:
   Filter that transfer only files that end with ".dat". Windows wildcard equivalent of "*.dat"

6. Mode: Reg Expr
   Target: File name
   Phrase: ".*?[0-9]+?.*?\.(mp3|wav)$"
   Regular Expression Definitions:
   
   

   Expression Section	Description
   ".*?"	Any character, 0, 1, or multiple times with reluctant matching
   "[0-9]+?"	Any numerical digit one or more times with reluctant matching
   ".*?"	Any character, 0, 1, or multiple times with reluctant matching
   "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
   "(mp3|wav)"	Characters may match exactly "mp3" OR "wav"
   "$"	Must be at end of the line

   Result of settings:
   Filter that transfer sound files (.mp3 or .wav) with numbers in the title

7. Mode: Reg Expr
   Target: File name
   Phrase: "^.*\.(?!tmp$)[^.]+$"
   Regular Expression Definitions:
   
   

   Expression Section	Description
   "^.*"	Matches starting characters, 0, 1, or multiple times.
   "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
   "(?!tmp$)"	Negative lookahead that excludes files with the extension "tmp"
   "[^.]+"	Match everything not eliminated by the previous negative lookahead.
   "$"	Must be at end of the line

   Result of settings:
   Filter that transfer everything except temporary files ending in ".tmp"

8. Mode: Reg Expr
   Target: File path
   Phrase: "^.*\.(?!(html|js|png)$)[^.]+$"
   Regular Expression Definitions:
   
   

   Expression Section	Description
   "^.*"	Matches starting characters, 0, 1, or multiple times.
   "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
   "(?!(html|js|png)$)"	Negative lookahead that excludes extensions that match "html", "js", or "png"
   "[^.]+"	Match everything not eliminated by the previous negative lookahead.
   "$"	Must be at end of the line

   Result of settings:
   Filter that transfers everything except files ending in ".html", ".js", or ".png"

Filter Target

This option allows you to specify the type of item that the file filter will apply to. The currently available options are as follows:

• Directory and file name: Applies the file filter to both file names and directory names.
  • Note 1: If a directory does not match the current filter settings, its contents will not be searched for other files that would have otherwise matched.
• File name: Applies the file filter to only file names. All directories that are found by the filter are automatically explored for other files that may match the current filter settings
• File path: Applies the file filter to the file path for a given file.
  
  
  • Note 1: Path-based filtering is relative to the source location currently defined by the transfer's current direction. If the task has an upload direction, then all path-based filter checks will be made relative to the current HotFolder location. If the task is a download, then all path-based filter calls will be made within the context of the currently defined remote folder. If no remote folder is currently defined, then the path based filter calls will be relative to the Server's home directory.
  • Note 2: If the task is a download task and the filter phrase starts with a leading slash, then all path based filter checks will be made within the context of the site's established home directory.

Transfer Files Based on Size

This option allows you to specify whether the HotFolder should transfer all files or only files within the specified size limits.

Transfer Files Newer than X days

This option allows you to specify that only files newer than the given value (in days) will be transferred. This is based on the current modification time of the file, not when it was placed into the folder.

Filter Test

The filter tester will allow you to view the files that will be transferred with the given filters before applying them to the task, this is especially useful when experimenting with complicated regular expressions. Using the "Test" button will populate the text field with all of the files that will be allowed under the given filter, "Apply" will save your settings to the "Fileset" panel but will still need to be applied via the "Fileset" "Apply" button.

Transfer Cache

The file transfer cache is a history of all the files you have uploaded/downloaded from the server for this task. This is used to indicate whether a file has changed on the server and has been flagged for upload or download again. File changes are tracked using the size and the last modified timestamp. These values are updated when a change is detected and the file is redownloaded. Thus, even if you remove the files from the local directory, they will not be downloaded again unless the cache is cleared. When this cache is deleted, it will re-download all files from the server. In the case of upload tasks, only files that have been changed locally will be uploaded again.

Note: Transfer cache is enabled and locked if multi-file transfers is selected and requeueing of running task is allowed.

The cache database directory is saved in the install directory as .fcdb by default. To save it somewhere else, specify the path in fchf.conf file with the following parameter key:

FC.config.derby.root=CACHEPATH

where CACHEPATH is replaced with the path of your target directory.

Clear Cache

Clears the current transfer cache entries. Files will be re-transferred even if they have previously been transferred.

Generate Cache

Generates the transfer cache without performing a transfer. This option is useful if both source and destination are already in sync. Cache generation occurs upon next task execution.

Remove File

When clicked, the user will have the ability to select a file for removal from the transfer cache. A browse dialog is presented showing the files in the source folder for this task. When a file is selected, if it is present in the transfer cache, it will be removed. If there was an error removing the file from the cache (i.e. it was not present), an error will be shown.

Note: When connecting to HotFolder service via remote admin, and attempting to remove a file from the transfer cache in an upload task, the full path of the file to be removed must be entered manually.

Always transfer directory structure (even if empty)

Empty directories are not transferred unless this is selected. If a filter hides all files in a directory, it is considered empty.

Transfer in priority sequence

This will apply a order the sequence of file transfers.

Oldest

Transfers files by last modified from oldest to newest.

Newest

Transfers files by last modified from newest to oldest.

Largest

Transfers files by size from largest to smallest.

Smallest

Transfers files by size from smallest to largest.

Synchronization Options

Delete files on destination not existing on source.

This option forces the source (HotFolder on uploads, Server on downloads) to make sure that both directories contain exactly the same fileset. This includes deletion of files which do not appear on the source.

File details (send details for Upload tasks, get details for Download tasks).

These flags are for a special configuration where two HotFolders want to synchronize files with each other with a server in between acting as a transit point (small temporary cache, not permanent file storage).

By enabling this feature, the source HotFolder will forward file listing information from the source in addition to the files themselves, allowing the destination HotFolder to download both the files and the listing and take appropriate actions.

The following options should be enabled to get this configuration setup:

On the Source HotFolder:

• "Transfer Cache" should be enabled (ensuring the same file is not sent up twice on the server)
• "Send details for Remote HotFolder synchronization" should be enabled.

On the Destination HotFolder:

• "Get remote HotFolder synchronization details" should be enabled.
• In Post Task tab, "Delete source file(s)" should be enabled.

This ensures that the destination HotFolder remain in-synch (Source HotFolder acting as the master repository), and that even through a Server acting as an intermediary, file creating and deletion events will be synchronized to the destination HotFolder.

Delete files on Destination that do not exist at source

When this item is checked, any files that exists on the destination that does not exist at the source will be deleted. In this way, the destination folder will always be kept in sync with the source folder because any new files added will be set to the destination, while anything removed will be deleted.

File Priority Settings

If this option is enabled, FileCatalyst allows you to specify which files should be transferred first. Depending on the number of files to be transferred, this setting may cause a delay before the actual transfer begins. This is a result of the file being sorted into the proper sequence.

After a Transfer is Complete

Allows the user to select what to do with the files after they have been transferred. You can choose either to do nothing, move to a folder called "Sent" under the HotFolder directory (Uploads only), or delete them.

Post URL

To allow integration with web applications, HotFolder can perform an HTTP POST to a URL or launch a local script following the completion of a task.

If this field is set to a URL, the contents of the HTTP POST will include the following parameters:

• "f"- contains a | (pipe) delimited list of the full remote paths that were transferred up until a cancelation. Files never transferred will not appear in this list.
• "lf" - contains a | (pipe) delimited list of the full local paths that were transferred or schedule to be transferred
• "s" - contains a | (pipe) delimited list of the sizes of each file
• "status" - Status of each file transferred. "0" = File transfer was not attempted, "1" = File transfer was successful, "2" = File transfer was cancelled, "3" = File transfer failed with an error. For example, if the transfer included 5 files, and all files were transferred successfully, you would see status=11111. If the transfer was cancelled during the 3rd file, you would see status=11200.
• "allfiles"- contains a | (pipe) delimited list of the full paths that were transferred or schedule to be transferred, this list has a 1 to 1 relationship with the status value where each status value corresponds to each value in allfiles

If this field is set to a script, the script will be called upon completion of the task. There are 4 arguments that can be passed to the script by FileCatalyst.

• <remotefiles> - contains a | (pipe) delimited list of the full remote paths that were transferred or schedule to be transferred
• <localfiles> - contains a | (pipe) delimited list of the full local paths that were transferred or schedule to be transferred
• <sizes> - contains a | (pipe) delimited list of the sizes of each file
• <statuscodes> - Status of each file transferred. "0" = File transfer was not attempted, "1" = File transfer was successful, "2" = File transfer was cancelled, "3" = File transfer failed with an error. For example, if the transfer included 5 files, and all files were transferred successfully, you would see status=11111. If the transfer was cancelled during the 3rd file, you would see status=11200.
• <allfiles>- contains a | (pipe) delimited list of the full paths that were transferred or schedule to be transferred, this list has a 1 to 1 relationship with the status value where each status value corresponds to each value in allfiles

As an example, you may set the Script to:

/home/user/script.sh <remotefiles> <localfiles> <sizes> <statuscodes> <allfiles>

and FileCatalyst will automatically insert the proper values into the command line before execution.On Windows you may execute a batch file like this:

cmd /c start c:\\file.bat <remotefiles> <localfiles> <sizes> <statuscodes> <allfiles>

Note 1: If the Post URL field is set to a script, and the script is called upon completion of a task, the task will block until the script fully completes. This completion includes the originating script and any sub-processes that the script creates when it executes.

Enable E-Mail Alerts

If this is enabled and the client specifies to send an e-mail on completion of a transfer, then an e-mail will be sent to the address provided whenever a transfer is finished.

Note: E-mail messages will only be sent if the Server has properly configured SMTP settings, and has enabled the e-mail alerts setting. If you wish to have e-mail alerts for your task, please contact your Server administrator to verify that the Server's SMTP settings are correct, and that the e-mail alerts option is enabled.

E-Mail Address

Email destination: If Server Administrator is selected, the email will be sent to the administator address configured on the Server. Otherwise you can sellect Custom Address and enter an email address. If you wish to send email to multiple addresses, they must be separated with a semicolon(;).

Send E-Mail on Successful Transfer

Indicates whether or not to send an e-mail on successful transfers.

Custom Message Text on Successful Transfer

This message is sent as the e-mail notification on a successful transfer.

Send E-Mail on Transfer Error

Indicates whether or not to send an e-mail when a transfer error occurs.

Custom Message Text on Transfer Error

This message is sent as the e-mail notification when a transfer error occurs.

Note: With the introduction of Alarms to HotFolder, it is recommended that FileCatalyst Central be instead used for capturing transfer faults. Network outages or user authentication errors are not handled by Task e-mail notifications.

Why Manage Your Bandwidth?

Your network is used for email, applications, web site access and any other form of corporate communication. All of these tools utilize and require a percentage of your overall bandwidth. If these tasks do not have sufficient bandwidth, they will begin to feel slow and unresponsive.

FileCatalyst is capable of maximizing usage of your allocated bandwidth. For example, if your organization has a T3 line, FileCatalyst is able to transfer at 45Mbps. However, this may not be practical during peak periods of the day when FileCatalyst may need to share available bandwidth with other applications.

Users themselves may manage and schedule their bandwidth usage by using the Bandwidth scheduler (see below). The user may even specify periods during which there should be no transfer activity by choosing "No Transfers" for the selected time.

Selecting an Appropriate Rate

The bandwidth scheduler allows users to limit or cap the level of bandwidth usage for a period of time. Example: if you have a T1 connection, you may wish to allocate 500 kbps of data transfer during the day and leave the remaining 1044 kbps for office use. However, at 6:00 PM you may wish to fully allocate 95% or 1467 kbps of bandwidth for data transfer. It is up to the users and the FileCatalyst administrator(s) to create an appropriate schedule.

Note:Congestion Control also automates the FileCatalyst transfer rate as networks become congested. As additional network traffic comes on-line and creates congestion, FileCatalyst will slow down or throttle back the transfer rate. It will remain in this state until the network is no longer congested. At which point the transfer will speed up until it resumes its set rate. This is a selectable feature which is enabled by editing individual tasks.

When configuring the FileCatalyst HotFolder to achieve high speed levels (500Mbps - 10Gbps for a single connection), several optimizations must be done in order to make sure the system is able to utilize multiple cores of a machine. These include configuring the number of reader threads per connection, packet processors (used for decoding AES packets on the receiver side), and the number of block writers.

# Reader threads

Number of threads PER CONNECTION spawned to read blocks off the disk and place them into the sender queue ready for transport. On the HotFolder, setting this value will only affect uploads. Most systems perform optimally when you configure only 1 thread to perform reading. There are two scenarios where this value should be moved up: when IO tests have shown that adding threads increases throughput for your storage device, or when utilizing high CPU load (more sender threads than CPU cores, or utilizing high CPU options such as AES encryption or LMZA compression), a single thread doing the block reads may be pushed down in priority and be CPU starved, causing a slowdown in transfer speeds. Multiple threads may help both of these scenarios.

Reader block size

By default, the block size used for each read operation is the same size specified by the client in the network block size. This can be adjusted for improved performance. Please run the advanced IO testing tool to see what works best for your system.

Packet processors

Packet processors are responsible for taking packets off the network socket, decrypting them, and preparing them for the block writers. When using AES at speeds greater than 200Mbps, it may be required to increase the number of packet processors on the receiving end to ensure multiple CPU's are utilized to decrypt the incoming data.

On the HotFolder, setting this value will only affect downloads. If uploading data to the server, similar settings should be set on the server applications.

# of Writer threads

Number of threads PER CONNECTION spawned to write blocks off the disk. On the HotFolder, setting this value will only affect downloads. Writer threads in FileCatalyst perform the following tasks:

• Uncompress data block (if using on-the-fly compression)
• Perform MD5 checksum on block (if using on-the-fly Verification)
• Write block data to disk

For systems using LMZA compression or possibly high speed transfers (> 1Gbps), setting multiple block writers may allow higher performance for single high speed connections.

Writer block size

By default, the block size used for each write operation is the same size specified by the client in the network block size. This can be adjusted for improved performance. Please run the advanced IO testing tool to see what works best for your system.

These values should be modified only after running through some performance IO testing.