Difference between revisions of "Large File Transfer"

From Dryad wiki
Jump to: navigation, search
(SCP)
m (Updated Splitting large files section)
 
(13 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Dryad needs to accept uploads of large files. These files must be transferred outside the normal submission system unless/until a better transfer mechanism is built into the submission system.
+
This page contains instructions  for Dryad users who want to deposit very large files. Files larger than 1GB may not work with the normal Dryad submission system. Uploading these files may fail due to unstable internet connections or web browser timeouts. In these situations, Dryad can accept content uploaded through alternate mechanisms.
  
== HTTP ==
+
== What to do if your files are large ==
  
Some institutions use HTTP for large transfers. Perhaps we can reconfigure our servers to work better over HTTP?
+
If you have files larger than 1GB and receive an error when using the the Dryad submission system, please contact the Dryad staff at [mailto:help@datadryad.org help@datadryad.org]. The staff will analyze your situation and make recommendations for how to proceed.
  
Pro:
+
== Uploading files through FTP ==
* can be built into the submission system easily
 
  
Con:
+
If you have files that are too large to transfer via the normal upload process, the curator may ask you to upload your files through FTP. You will be given instructions for accessing the Dryad FTP server. However, you will still need to complete the regular Dryad submission process using "dummy" files as placeholders for your actual files.
* times out easily, especially on low-quality Internet connections
 
  
== FTP ==
+
== Splitting large files ==
  
We are currently testing file transfers via ftp2.nescent.org. When a transfer is needed, NESCent IT must turn on the service, and they should be notified to shut down the service after a transfer is complete.
+
To ensure accessible downloads, Dryad recommends that you limit the size of each individual file (or directory of files) to 10GB. If you have a file larger than 10GB, it can be broken into smaller pieces. For example, if you have a large zip or tar archive with multiple subdirectories, please consider compressing and uploading each subdirectory separately. If you are unable to do so, you may split a file as follows:
  
Pro:
+
On Unix, Linux, or Mac OSX, you may use the "split" command, like this:
* simple and well understood
+
split -b 5000m myVeryLargeFile.txt
  
Con:
+
On Windows machines, you may split files with [http://www.hjsplit.org/ HJsplit].
* security issues
 
  
== SCP ==
+
== Recombining files that have been split ==
  
Pro:
+
On Unix, Linux, or Mac OSX, you may recombine files with the "cat" command, like this:
* Quick, easy, secure.
+
cat partaa partab partac > dataFile.txt
  
Con:  
+
On Windows, you may recombine files with the "copy" command, like this:
* Still not a standard install on Windows machines.
+
copy /B partaa+partab+partac dataFile.txt
  
== Mailed USB key ==
+
== Technical Details ==
  
Pro:
+
See [[Large File Technology]].
* can transfer very large files without worrying about size and bandwidth concerns
 
  
Con:
+
[[Category:Help]]
* doesn't seem "professional"
+
[[Category:Features]]
* may take a long time for international authors
 
 
 
== BioTorrents ==
 
 
 
Pro:  
 
* can transfer large files efficiently
 
 
 
Con:
 
* The process of creating a torrent file is too complex for a typical Dryad user.
 

Latest revision as of 13:41, 3 January 2018

This page contains instructions for Dryad users who want to deposit very large files. Files larger than 1GB may not work with the normal Dryad submission system. Uploading these files may fail due to unstable internet connections or web browser timeouts. In these situations, Dryad can accept content uploaded through alternate mechanisms.

What to do if your files are large

If you have files larger than 1GB and receive an error when using the the Dryad submission system, please contact the Dryad staff at help@datadryad.org. The staff will analyze your situation and make recommendations for how to proceed.

Uploading files through FTP

If you have files that are too large to transfer via the normal upload process, the curator may ask you to upload your files through FTP. You will be given instructions for accessing the Dryad FTP server. However, you will still need to complete the regular Dryad submission process using "dummy" files as placeholders for your actual files.

Splitting large files

To ensure accessible downloads, Dryad recommends that you limit the size of each individual file (or directory of files) to 10GB. If you have a file larger than 10GB, it can be broken into smaller pieces. For example, if you have a large zip or tar archive with multiple subdirectories, please consider compressing and uploading each subdirectory separately. If you are unable to do so, you may split a file as follows:

On Unix, Linux, or Mac OSX, you may use the "split" command, like this: split -b 5000m myVeryLargeFile.txt

On Windows machines, you may split files with HJsplit.

Recombining files that have been split

On Unix, Linux, or Mac OSX, you may recombine files with the "cat" command, like this: cat partaa partab partac > dataFile.txt

On Windows, you may recombine files with the "copy" command, like this: copy /B partaa+partab+partac dataFile.txt

Technical Details

See Large File Technology.