How to transfer directories into remote server using tar and ssh combined.
$ tar cf - mydir/ | ssh gate 'cd /tmp && tar xpvf -'
The example in the previous section can be expanded to build a basic data transfermodel. Any information input into one end of the connection will be output to the other end,and input and output can be easily captured in order to emulate file transfer. Start by using nc to listen on a specific port, with output captured into a file: $ nc -l 1234 > filename.out Using a second machine, connect to the listening nc process, feeding it the file which is to be transferred: $ nc host.example.com 1234 < filename.in After the file has been transferred, the connection will close automati- cally.
destination-server# nc -l 9999 | tar xvzf -
source-server# tar cvzf - | nc destination-server-ip 9999
It is quite simple to build a very basic client/server model using nc.
On one console, start nc listening on a specific port for a connection.
For example:
$ nc -l 1234
nc is now listening on port 1234 for a connection. On a second console
(or a second machine), connect to the machine and port being listened on:
$ nc 127.0.0.1 1234
There should now be a connection between the ports. Anything typed at
the second console will be concatenated to the first, and vice-versa.
After the connection has been set up, nc does not really care which side
is being used as a `server' and which side is being used as a `client'.
The connection may be terminated using an EOF (`^D').server:/mnt /mnt nfs rw,hard 0 0 #mount from server
# showmount -a
All mount points on local host:
edcert20.ucs.indiana.edu:/home
edcert21.ucs.indiana.edu:/usr/local
# showmount -d
Directories on local host:
/home
/usr/local
# showmount -e
Export list on local host
/home edcert21.ucs.indiana.edu edcert20.ucs.indiana.edu
/usr/local edcert21.ucs.indiana.edu# df -F nfs Filesystem Type blocks use avail %use Mounted on edcert21.ucs.indiana.edu:/home nfs 68510 55804 12706 81% /usr/share/helpUse the command nfsstat -s to display NFS activity on the server side. For example:# nfsstat -s Server RPC: calls badcalls nullrecv badlen xdrcall duphits dupage 50852 0 0 0 0 0 0.00 Server NFS: calls badcalls 50852 0 null getattr setattr root lookup readlink 1 0% 233 0% 0 0% 0 0% 1041 2% 0 0% read wrcache write create remove rename 49498 97% 0 0% 0 0% 0 0% 0 0% 0 0% link symlink mkdir rmdir readdir fsstat 0 0% 0 0% 0 0% 0 0% 75 0% 4 0%The output may be interpreted using the following guidelines.
# nfsstat -c
Client RPC:
calls badcalls retrans badxid timeout wait newcred
369003 62 1998 43 2053 0 0
Client NFS:
calls badcalls nclget nclsleep
368948 0 368948 0
null getattr setattr root lookup readlink
0 0% 51732 14% 680 0% 0 0% 95069 25% 542 0%
read wrcache write create remove rename
210187 56% 0 0% 2259 0% 1117 0% 805 0% 337 0%
link symlink mkdir rmdir readdir fsstat
120 0% 0 0% 7 0% 0 0% 5510 1% 583 0%
This output may be interpreted using the guidelines given below.# umount /mntdir
# vxvol -g <diskgroup> stopall# vxdg deport <diskgroup># vxdg import <diskgroup># vxrecover -g <diskgroup> -sb# /usr/lib/fs/vxfs/fsadm /mount/pointnolargefiles# /usr/lib/fs/vxfs/fsadm -o largefiles /mount/point# vxplex [-g diskgroup] -o rm dis plex# ioscan -fnC disk (identify mirror disk) # pvcreate -Bf /dev/rdsk/c2t5d0 (make a bootable physical volume) # mkboot -l /dev/rdsk/c2t5d0 (create LVM disk layout) # mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c2t5d0 (-lq = switch off quorum) # vgextend /dev/vg00 /dev/dsk/c2t5d0 # for P in 1 2 3 4 5 6 7 8 9 10 > do > lvextend -m 1 /dev/vg00/lvol$P /dev/dsk/c2t5d0 > sleep 1 > done
# bdf -l # lvlnboot –v
# lvlnboot -r /dev/vg00/lvol3 (prepare a root LVM logical volume) # lvlnboot -s /dev/vg00/lvol2 (prepare a swap LVM logical volume) # lvlnboot -b /dev/vg00/lvol1 (prepare a boot LVM logical volume) # vgcfgbackup vg00 # lifls -C /dev/rdsk/c2t5d0 (confirms as a boot device)
# pvcreate -Bf /dev/rdsk/c0td50 # vgcfgrestore -n /dev/vg00 /dev/rdsk/c0td50 # vgchange -a y /dev/vg00 # pvcreate -Bf /dev/rdsk/c2t5d0 (make a bootable physical volume) # mkboot -l /dev/rdsk/c2t5d0 (create LVM disk layout) # mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c2t5d0 (-lq = switch off quorum) # vgsync /dev/vg00[NB. It will only be necessary to run the mkboot commands above if it is a system disk replacement.]
# lvlnboot -r /dev/vg00/lvol3 (prepare a root LVM logical volume) # lvlnboot -s /dev/vg00/lvol2 (prepare a swap LVM logical volume) # lvlnboot -b /dev/vg00/lvol1 (prepare a boot LVM logical volume) # vgcfgbackup vg00
# init 4
# PATH=$PATH:/sbin:/usr/sbin # mount -a # pvcreate -Bf /dev/rdsk/c2t6d0 # mkboot -l /dev/rdsk/c2t6d0 # mkboot -a "hpux (;0) /stand/vmunix" /dev/rdsk/c2t6d0 # vgcfgrestore -n /dev/vg00 /dev/rdsk/c2td60 # vgchange -a y /dev/vg00 # lvlnboot -r /dev/vg00/lvol3 (prepare a root LVM logical volume) # lvlnboot -s /dev/vg00/lvol2 (prepare a swap LVM logical volume) # lvlnboot -b /dev/vg00/lvol1 (prepare a boot LVM logical volume) # init 4
# lvdisplay -v /dev/vg00/lvol1
# lvdisplay -v /dev/vg00/lvol8
# for P in 1 2 3 4 5 6 7 8 9 10 > do > lvreduce -m 0 /dev/vg00/lvol$P /dev/dsk/c2t5d0 > sleep 1 > done # vgreduce /dev/vg00 /dev/dsk/c2t5d0
for lv in $(vgdisplay -v vg00 | grep -i 'lv name' | awk '{print $NF}')
do
echo "#################################"
echo ${lv}
lvextend -m 1 ${lv} ${dsk}
donehm()
{ lvdisplay -v $1 | sed -n -e '/Logical extents/,$p' | grep stale | wc -l
}
# setboot Primary bootpath : 0/1/1/0.1.0 HA Alternate bootpath : 0/1/1/0.0.0 Alternate bootpath : 0/1/1/0.1.0 Autoboot is ON (enabled)
# lvlnboot -v vg00
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c2t1d0s2 (0/1/1/0.1.0) -- Boot Disk
/dev/dsk/c2t0d0s2 (0/1/1/0.0.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Root: lvol3 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Swap: lvol2 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Dump: lvol2 on: /dev/dsk/c2t1d0s2, 0
cat > /tmp/part << eof > 3 > EFI 500MB > HPUX 100% > HPSP 400MB > eof
for lv in $(vgdisplay -v vg00 | grep -i 'lv name' | awk '{print $NF}')
do
echo "#################################"
echo ${lv}
lvextend -m 1 ${lv} ${dsk}s2 # note the s2 partition again
donehm()
{ lvdisplay -v $1 | sed -n -e '/Logical extents/,$p' | grep stale | wc -l
}
# setboot Primary bootpath : 0/1/1/0.1.0 HA Alternate bootpath : 0/1/1/0.0.0 Alternate bootpath : 0/1/1/0.1.0 Autoboot is ON (enabled)
# lvlnboot -v vg00
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c2t1d0s2 (0/1/1/0.1.0) -- Boot Disk
/dev/dsk/c2t0d0s2 (0/1/1/0.0.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Root: lvol3 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Swap: lvol2 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Dump: lvol2 on: /dev/dsk/c2t1d0s2, 0
| URL: | http://olearycomputers.com/ll/hpux_mirror.html |
# zfs send tank/cindy@today | ssh newsys zfs recv sandbox/restfs@today
prodigy# zfs snapshot poolName/fileSystem@labelName# zfs list
NAME USED AVAIL REFER MOUNTPOINT
poolName 5.63G 141G 27.5K /poolName
poolName/fileSystem 2.97G 141G 1.92G /poolName/fileSystem
poolName/fileSystem@label 0 - 1.92G -
prodigy# # zfs destroy poolName/fileSystem@label
#
- The following syntax creates recursive snapshots of all home directories in the tank/home file system. Then, you can use the zfs send -R command to create a recursive stream of the recursive home directory snapshot, which also includes the individual file system property settings.
# zfs snapshot -r tank/home@monday
# zfs send -R tank/home/@monday | ssh remote-system zfs receive -dvu pool
ZFS Clone
Clones can be only created from snapshot and they are actually new FS with initial content as original FS, so they are writable.
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool 1.49G 527M 528M /mypool
mypool/home 993M 527M 20K /mypool/home
mypool/home/user1 993M 527M 993M /mypool/home/user1
mypool/home/user1@Monday 0 - 993M -
# zfs clone mypool/home/user1@Monday mypool/home/user2
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool 1.49G 527M 528M /mypool
mypool/home 993M 527M 20K /mypool/home
mypool/home/user1 993M 527M 993M /mypool/home/user1
mypool/home/user1@Monday 0 - 993M -
mypool/home/user2 0 527M 993M /mypool/home/user2
Once clone is created from snapshot, the snapshot can not be deleted until clone exists.
# zfs get origin mypool/home/user2
NAME PROPERTY VALUE SOURCE
mypool/home/user2 origin mypool/home/user1@Monday -
# zfs destroy mypool/home/user1@Monday
cannot destroy 'mypool/home/user1@Monday': snapshot has dependent clones
use '-R' to destroy the following datasets:
mypool/home/user2
ZFS Snapshot
ZFS snapshot is read-only copy of the file system.
When taken, it consumes no additional disk space, but when data changes, snapshot is growing since references to old data (unique to snapshot), so space cannot be freed.
Let's take snapshot of FS mypool/home/user1. Snapshot name is Monday. So you can see what's command's syntax.
# zfs snapshot mypool/home/user1@Monday
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool 1.49G 527M 528M /mypool
mypool/home 993M 527M 20K /mypool/home
mypool/home/user1 993M 527M 993M /mypool/home/user1
mypool/home/user1@Monday 0 - 993M - (consumes no additional data within zpool)
You even cannot destroy FS if it has any snapshot.
# zfs destroy mypool/home/user1
cannot destroy 'mypool/home/user1': filesystem has children
use '-r' to destroy the following datasets:
mypool/home/user1@Monday
Let’s change some data and see now that used data of snapshot is growing.
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool 1.49G 527M 528M /mypool
mypool/home 993M 527M 20K /mypool/home
mypool/home/user1 993M 527M 993M /mypool/home/user1
mypool/home/user1@Monday 28K - 993M -
Snapshot(s) is stored in hidden directory named .zfs/snapshot/, located in the root of FS.
So you can go there and restore files if needed, remember it is read-only copy of FS.
# /mypool/home/user1/.zfs/snapshot> ls
total 3
dr-xr-xr-x 2 root root 2 Nov 16 13:23 .
dr-xr-xr-x 3 root root 3 Nov 16 13:23 ..
drwxr-xr-x 3 root root 6 Nov 16 17:09 Monday
Beside restoring individual files from snapshot, you can roll back whole FS to previously taken snapshot.
# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
mypool/home/user1@Monday 993M - 993M -
# zfs rollback mypool/home/user1@Monday
And all changes since snapshot Monday has been taken up to now are deleted and FS is same as it was on that Monday. OPTIONS SUMMARY
Here is a short summary of the options available in rsync.
-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don’t send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place
--append append data onto shorter files
--append-verify --append w/old data in file checksum
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the tree
-k, --copy-dirlinks transform symlink to dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
-E, --executability preserve executability
--chmod=CHMOD affect file and/or directory permissions
-A, --acls preserve ACLs (implies -p)
-X, --xattrs preserve extended attributes
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve modification times
-O, --omit-dir-times omit directories from --times
--super receiver attempts super-user activities
--fake-super store/recover privileged attrs using xattrs
-S, --sparse handle sparse files efficiently
-n, --dry-run perform a trial run with no changes made
-W, --whole-file copy files whole (w/o delta-xfer algorithm)
-x, --one-file-system don’t cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that exist on receiver
--remove-source-files sender removes synchronized files (non-dir)
--del an alias for --delete-during
--delete delete extraneous files from dest dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during xfer, not before
--delete-delay find deletions during, delete after
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from dest dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of dirs even if not empty
--max-delete=NUM don’t delete more than NUM files
--max-size=SIZE don’t transfer any file larger than SIZE
--min-size=SIZE don’t transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at end
-m, --prune-empty-dirs prune empty directory chains from file-list
--numeric-ids don’t map uid/gid values by user/group name
--timeout=SECONDS set I/O timeout in seconds
--contimeout=SECONDS set daemon connection timeout in seconds
-I, --ignore-times don’t skip files that match size and time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare received files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
--skip-compress=LIST skip compressing files with suffix in LIST
-C, --cvs-exclude auto-ignore files in the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter=’dir-merge /.rsync-filter’
repeated: --filter=’- .rsync-filter’
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don’t exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *from/filter files are delimited by 0s
-s, --protect-args no space-splitting; wildcard chars only
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we’re doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read daemon-access password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating dest
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
--iconv=CONVERT_SPEC request charset conversion of filenames
--checksum-seed=NUM set block/file checksum seed (advanced)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (see below for -h comment)
Rsync can also be run as a daemon, in which case the following options are accepted:
Rsync can also be run as a daemon, in which case the following options are accepted:
--daemon run as an rsync daemon
--address=ADDRESS bind to the specified address
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--config=FILE specify alternate rsyncd.conf file
--no-detach do not detach from the parent
--port=PORT listen on alternate port number
--log-file=FILE override the "log file" setting
--log-file-format=FMT override the "log format" setting
--sockopts=OPTIONS specify custom TCP options
-v, --verbose increase verbosity
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
-h, --help show this help (if used after --daemon)