Contents

Debugging rsync and ssh

¹/Nov 2016

Background

I fall in love with rsync lately. It is particularly useful when I sync my hadoop stuff (scripts and input, which add up to a few GBs) between local and my hadoop cluster. After running the sync script for a few times, I cannot ssh to the machine anymore. This post is about how I debug it and the lessons learned.

This is how the problematic script sync.sh looks like: (the IP is masked for obvious reasons) (warning: this script is faulty, do not use)

#!/usr/bin/env bash
rsync -rave "ssh -i '/path/to/aws.pem'" . hduser@ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com:

Background info: 1. I use ssh key to ssh to the machine

Sidetrack: 1. the -e option enables ssh with pem, which is useful for ssh to AWS 2. the -a archive mode equals -rlptgoD (no -H,-A,-X), meaning that the -r is actually redundant

This is how I execute the script:

$ ./sync.sh

Problem

The output of the script even when no file is changed:

$ ./sync.sh                      
sending incremental file list          
./

There should be no output when no files are changed. The next time I execute the script, it does not work because ssh failed!

Part of the output of ssh in verbose mode:

debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/carson/.ssh/id_rsa
debug1: Authentications that can continue: publickey

Solution

After Googling for a while, I realize the problem is that the rsync synced directory permissions of the local directory with remote home directory. The immediate solution is to fix the remote home directory permissions. Luckily I can ssh to another user in the same machine and switch user to fix this problem.

Sidetrack: Thread about ssh and home directory permission

How to fix the permissions:

chmod o-w ~/; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys

Problem 2

The output of ssh (in verbose mode) looks different now.

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/carson/.ssh/config
debug1: /home/carson/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com [xxx.xxx.xxx.xxx] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /path/to/aws.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file /path/to/aws.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Authenticating to ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com:22 as 'hduser'
debug1: SSH2_MSG_KEXINIT sentoperation
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:-
debug1: Host 'ec2-xxx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /home/carson/.ssh/known_hosts:28
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: carson@carson-mint
debug1: Server accepts key: pkalg ssh-rsa blen 535
sign_and_send_pubkey: signing failed: agent refused operation
debug1: Trying private key: /path/to/aws.pem
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

Notice the line:

sign_and_send_pubkey: signing failed: agent refused operation

Solution 2

Thank Google, the solution is ssh-add. It appears that ssh-agent cannot find any keys attached.

Sidetrack: Thread about signing failed: agent refused operation

Takeaway

Be careful when using .
- rsync preserves permissions under archive mode (since -a includes -p)
- Sidetrack: What is /.
- Use * to sync files under a directory
Actually * does not sync hidden files (a.k.a. dotfiles) during expansion
- . has to be used if hidden files are to be synced

Tags// development, linux,

Overthought

Carson Ip