The dmscp data transmission tool

 

Manfred Stolle

Abstract:	Security aspects gain an increasing importance in the transfer of data via the Internet. Security conscious sites often refuse to provide access via ftp or rcp daemons. So the only remaining mean of data transfer is scp which provides a secure session control and a secure (encrypted) data transfer. Due to its poor performance scp is not very useful to transfer big amounts of data. In the paper we propose dmscp , a tool that provides a secure session control and an unencrypted data transfer between UNIX machines.
Author:	Manfred Stolle , Tel.: (030) 841 85-310, e-mail: stolle@zib.de
Version:	February 20, 2001
URL:	http://mss.zib.de/mss

1. Introduction

2. The Solution

2.1 Remote Objects

2.2 Why Fragmentation?

2.3 `disk full' on a Filesystem with Data Migration?

2.4 Checking the Quality of Data Transfer

3. The Data Transfer Tool dmscp

3.1 dmscp Parameters

3.2 How does dmscp work?

3.3 Optimizing the Network Transfer

3.4 User Front End

4. Performance

5. Future Plans

1. Introduction

In the Internet security aspects gain increasing importance. Faster computers and more sophisticated software give more and better opportunities to hackers to get illegal access to computers in the Net. So the classical access methods like telnet , rlogin , ftp are no longer adequate. An increasing number of (high performance computing) sites reduce the access to their machines to the Secure Shell (ssh) . By using the Secure Shell and its data transfer tool scp (Secure Copy) all data transfer goes via encryption. The encryption of the user data slows the data transfer significantly down. So the transfer of 10 MB from a Sun to a CRAY-J90 takes 3.3 sec via ftp and 140(!) sec via scp .

By using ssh as session control mean at least the metadata to open and close the session goes via an encrypted (secure) line. The performance of the data transfer can be significantly increased if the user data stream is detached from the control stream and goes via an unencrypted line. This way the user data can be read by software sniffing the Internet, but no information is compromised that increases the chance to break into the hosts transmitting the data.

In this paper we propose dmscp , a UNIX tool that (based on the ideas mentioned above) reaches a performance near to transfer rates of ftp/rcp additionally providing a better success check and a secure session control. dmscp is portable and easy to install by any user without requiring any special system features. A user front end yscp hat shows a behavior similar to scp/rcp is provided.

2. The Solution

The solution is a ssh based tool that

• uses the encrypted ssh data transfer for
• session control and
• exchange of metadata,
• transfers the data in plain (unencrypted) mode and
• checks the quantity and the quality of the data transferred.

The classical data transfer tools and scp are not very useful to pipe data from/to a process at the local host. This is one of the reasons for not using the design of scp/rcp to define the new tool.

The intention was to provide a more common mean to transfer and handle big amounts of data on hosts with data migration ( DMF - SGI/Cray ) that helps to solve several task classes.

Copying data to/from a remote file from/to a local file or a local pipe is one of these task classes.

2.1 Remote Objects

The basic idea is the remote object . The tool to be defined copies data to (or from) a remote object .

A remote object can be:

• an ordinary file on the remote host,
• a command on the remote host reading the data from stdin or writing it to stdout or
• a set of ordinary files ( fragments ) on the remote host.

With a remote command processing the data, tasks like copying a file tree become possible. A model like

	(cd sourcedir/. && tar -cf - sourcedirname) |\
	process_on_remote_host(cd destdir && tar -xf - sourcedirname)

would work.

2.2 Why Fragmentation?

By piping data into a data transfer tool it becomes possible to get data streams bigger than the biggest file at the source host.

If the remote (server) host does not have any mean of data migration the amount of transmitted data is limited by the size of the large file system at the server. Not so with data migration, provided the data is moved to an external media during the copy process. But most data migration systems allow to migrate whole files only!

If the data stream is cut into a set of files significantly smaller than the disk cache (system disk base of a file system with data migration) and each file is migrated immediately after closing really large amounts of data can be stored.

2.3 `disk full' on a Filesystem with Data Migration?

On a native UNIX file system a ` disk full ' error is fatal and leads to an immediate abort of the writing process. On a file system with data migration the same situation is not an error at all - it only means that the tape system is slower than the data transfer via the network. It can be expected that disk space will be available soon. So it makes sense for the writing process to wait for disk space to become available and resume the writing.

2.4 Checking the Quality of Data Transfer

Classical data transfer tools don't do any checks, they assume the network to work properly. We had to learn (the hard way) that one should not set too much trust into the network. Due to an error in lower network layers data bytes had been exchanged by zeros. The destination files had the same size as the source files but contained somewhere inside a lot of zero bytes.

The most safe way would be not only to compare the sizes of source and destination but also compute and compare their checksums. This is not only an expensive approach, it works only if the data source and destination are files (or fragments). If source or destination are processes (or pipes) you have to trust the operating system (you have to do that anyway :-) ) and check the data at the front to the network - in the transmission buffers.

So the tool to be provided has to

• compare the actual size of source and destination if they are files or fragments,
• compare the amount of data sent and received in the network buffer otherwise and
• compute and compare the checksums in the network transfer buffers.

3. The Data Transfer Tool dmscp

Our dmscp is a tool providing all the above mentioned features. It is designed as back end to build front ends for specific tasks. It runs on most UNIX platforms and provides DMF (SGI/Cray Data Migration Facility) specific functions. It is I/O optimized for UNICOS and IRIX.

dmscp is symmetric, meaning the same process runs at the local and the remote host. The local process invokes the remote process in a way that the remote process knows it is the server.

dmscp is easy to install. Except the Secure Shell daemon (sshd) on the remote host does not need any special system functions or daemons. It can be installed in a system directory or in the data area of the user, any user can install it by himself using a quickinstall script. dmscp only has to be in the search path of the user, a parameter allows to extend the search path before starting the remote dmscp .

dmscp is public domain. You can obtain the source via http://www.zib.de/zibdoc/zib/mass-storage/dmscp/ .

3.1 dmscp Parameters

dmscp runs in read or in write mode (from or to the remote host resp resp.). The (documented) parameters are shown in the following table (for more details see man dmscp past installation):

Every parameter can be at any place in the commandline. If a parameter occurs more than once the last occurrence is significant.

Examples :

• Copy the directory bzbstoll@avampire:/tmp/bzbstoll to /tmp/bzbstoll:

	dmscp -checksum -r -s avampire -u bzbstoll \
	-prefix 'setenv PATH "$HOME/bin:$PATH"' \
	-R 'cd /tmp/ && tar -cf  - ./bzbstoll' \
	| ( cd /tmp/ && tar -xf  - )

• Copy the directory /tmp/bzbstoll to bzbstoll@avampire:/tmp/bzbstoll2
  (The command in the -R option can be pretty complex!):

	( cd /tmp/ && tar -cf  - ./bzbstoll ) | \
	dmscp -checksum -w -s avampire -u bzbstoll \
	-prefix 'setenv PATH "$HOME/bin:$PATH"' \
	-R 'cd /tmp/ && mkdir ._tmpdir.123 && cd ._tmpdir.123 && tar -xf  - 1>&2 && cd .. &&  mv ._tmpdir.123/bzbstoll bzbstoll2 && rmdir ._tmpdir.123'

• Pack the current directory recursively and put the data into a set of fragments of 2GB each (The first fragment has the same size as the others).
  The ssh port is 1024, the destination directory is /archivedir, the name of the remote object (fragment set) is myarchive, each fragment is to be migrated after closing.
  The timeout is 10 h:

	find . -print|cpio -ocv |\
	dmscp -w -s server -u user -p 1024 -d /archivedir \
	-F 2g -S 0 -T 36000 -f myarchive -FMigrate

• Unpack this archive into the current directory, release the disk cache of each fragment on the server immediately after closing the fragment file ( -F 1 indicates that the remote object is a set of fragments):

	dmscp -r -s server -p 1024 -u user -d /archivedir \
	-F 1 -f myarchive -frelease -T 360000 |\
	cpio -icdv

3.2 How does dmscp work?

Several processes are needed to do the job. The initial process has the task to analyze the command line. If it learns that it is the server it does the job itself (see below). If it is the client it forks and the child does the real work. The parent process gathers the reporting messages and checks the success.

• Before forking the client parent process creates a named pipe and an unnamed pipe. It receives the reports from its child via the unnamed pipe. The named pipe is used to transmit messages from the remote dmscp to the local child.
• The (client) child process builds a commandline like:

	ssh user@server "cd destdir && dmscp\
	-s -server -f destfile' >/tmp/<named_pipe>

and invokes it. Any reply from the remote dmscp (stdout/stderr) is reported to the parent process.

• After having finished the command line analysis the server process looks for a free port, grabs it and reports (the secure way) its number to the client together with a passphrase (a codeword to get access to that port). Then it watches this port until a client comes that tells the right passphrase (or a timeout occurs). So unauthorized access to the port is prevented .
• If the remote object is a command the next step is to transmit the commandline via the data socket. This way any commandline can be transmitted to the server.
• Now everything is prepared to transmit the data. Both client and server open their data objects (files, pipes) and transmit the data. (If the operating system is UNICOS* or IRIX the local I/O and the network I/O are parallel, this requires under IRIX an additional process.) On the server the transmission can be interrupted to open the next fragment respectively to do DMF related jobs.
• When everything is done the server reports (secure line) the amount and (optional) the checksum of the data to the server. The client child reports the same to its parent and the parent calls it a success if everything is fine by exiting with success state.

If the server ( remote dmscp ) runs into a fatal error it calls a panic routine . panic cleans up (as good as possible), reports to the client child, closes the data socket and dies. The (client) child process reports to its parent, cleans up and dies. The client parent process reports the error to the user. The panic function in the client child process works similarly.

3.3 Optimizing the Network Transfer

The network bandwidth of a file transfer program is highly influenced by the tcp windowsize.
(see http://www.psc.edu/networking/perf_tune.html and http://www.dast.nlanr.net/Features/Autobuf/ ) Unfortunately the proposed method to optimize the tcp windowsize does not work on all Unix platforms. So dmscp doesn't contain an automatic network tuning but an option for hand tuning. During installation time the default windowsize can be modified. It is predefined with 64Kbyte. With the option -wsz [k|m]# (example -wsz 8m ) dmscp sets the tcp windowsize.

A thumb rule for a good value is: tcp_windowsize=bandwidth*delay of the network connection.

3.4 User Front End

At Konrad-Zuse-Zentrum dmscp is used for various tasks. First it was used for backup and archival problems not solvable by NetBackup ( Veritas ). Here we need the fragmentation ! The user front end is a set of Korn Shell ( ksh ) scripts using dump/restore, cpio etc. to pack/unpack the data. A secure tool has to be provided because the only way to address the server is the Secure Shell .

Lately we have been forced to decide to cut the telnet etc. lines to our high performance computers (some nice hacker twisted our arm) and found out that scp is not an adequate tool to transfer that kind of data our costumers are dealing with. So we got another dmscp front end on the track.

yscp ( Yet another Secure Copy Program ) does basically the same as scp with the exception that the user data is transmitted without encryption. This way we cannot avoid the overhead of connecting by ssh (in fact yscp connects twice to each remote site) but we gain the performance of ftp/rcp in the data transfer itself, only slightly diminished by the (optional!) checksum computing of dmscp .

yscp is much faster than scp , much more secure than ftp/rcp and safer than all of them thanks to the properties of dmscp .

yscp is a perl program of 500 lines, dmscp is 2700 lines of C code.

Examples (for more details see man yscp):

• Copy a file to a remote host:

	yscp sourcefile user@server:destdir/destfile

	yscp sourcefile user@server:destdir

• Copy a directory to a remote host (recursively):

	yscp -r sourcedir user@server:destdir

4. Performance

The following table shows some (wall clock) times in seconds spent on copying 100 MB from a SGI/O2 to some hosts via a fast local network (without any network optimization).

The following table shows some (wall clock) the transmission bandwidth between a Sun,Ultra-60 in the ZIB (hlr.zib.de) and several remote hosts with optimized tcp windowsize

5. Future Plans

The data migration related features of dmscp are designed for DMF (CRAY/SGI - UNICOS, UNICOS/mk, IRIX) only. A more common approach is to provide a set of parameters to define a command to be executed

• before opening the first fragment (or the remote process),
• before opening any fragment,
• after closing the last fragment (or the remote process) and
• after closing any fragment.

Depending on demands from the user community asynchronous I/O would be possible for most platforms by using shared memory, doing the network I/O and the disk (pipe) I/O by different processes synchronized by signals.