I fall in love with rsync lately. It is particularly useful when I sync my hadoop stuff (scripts and input, which add up to a few GBs) between local and my hadoop cluster. After running the sync script for a few times, I cannot ssh to the machine anymore. This post is about how I debug it and the lessons learned.
This is how the problematic script
sync.sh looks like: (the IP is masked for obvious reasons) (warning: this script is faulty, do not use)
#!/usr/bin/env bash rsync -rave "ssh -i '/path/to/aws.pem'" . email@example.com:
1. I use ssh key to
ssh to the machine
-e option enables
ssh with pem, which is useful for
ssh to AWS
-a archive mode equals -rlptgoD (no -H,-A,-X), meaning that the
-r is actually redundant
This is how I execute the script:
The output of the script even when no file is changed:
$ ./sync.sh sending incremental file list ./
There should be no output when no files are changed. The next time I execute the script, it does not work because ssh failed!
Part of the output of
ssh in verbose mode:
debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/carson/.ssh/id_rsa debug1: Authentications that can continue: publickey
After Googling for a while, I realize the problem is that the
rsync synced directory permissions of the local directory with remote home directory. The immediate solution is to fix the remote home directory permissions. Luckily I can
ssh to another user in the same machine and switch user to fix this problem.
How to fix the permissions:
chmod o-w ~/; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys
The output of
ssh (in verbose mode) looks different now.
OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016 debug1: Reading configuration data /home/carson/.ssh/config debug1: /home/carson/.ssh/config line 1: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com [xxx.xxx.xxx.xxx] port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file /path/to/aws.pem type -1 debug1: key_load_public: No such file or directory debug1: identity file /path/to/aws.pem-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000 debug1: Authenticating to ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com:22 as 'hduser' debug1: SSH2_MSG_KEXINIT sentoperation debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: firstname.lastname@example.org debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: email@example.com MAC: <implicit> compression: none debug1: kex: client->server cipher: firstname.lastname@example.org MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:- debug1: Host 'ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com' is known and matches the ECDSA host key. debug1: Found key in /home/carson/.ssh/known_hosts:28 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: carson@carson-mint debug1: Server accepts key: pkalg ssh-rsa blen 535 sign_and_send_pubkey: signing failed: agent refused operation debug1: Trying private key: /path/to/aws.pem debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. Permission denied (publickey).
Notice the line:
sign_and_send_pubkey: signing failed: agent refused operation
Thank Google, the solution is
ssh-add. It appears that
ssh-agent cannot find any keys attached.
- Be careful when using
rsyncpreserves permissions under archive mode (since
- Sidetrack: What is /.
*to sync files under a directory
*does not sync hidden files (a.k.a. dotfiles) during expansion
.has to be used if hidden files are to be synced