rsync-ssh-detailed

This is a detailed version of the note rsync-ssh. Plan:

Normalize terminology and flow: prerequisites → one-time setup → daily workflow → troubleshooting.
Replace hardcoded hostnames/keys with placeholders, add security notes, and keepalive options.
Improve SSH config (ControlMaster, keepalives) and show a clear “gateway → lynx via local port 2222” model.
Fix typos, standardize script names, and add small robustness tweaks to the sync script (quotes, common excludes, help, dependency checks).
Provide a concise “quick start” and a reliable long-running-task path with tmux.
Keep everything as a single, easy-to-skim document.

Remote dev with rsync + SSH tunnels (AWS gateway → lynx)

This guide lets you:

Send code from your local machine to a remote server (lynx) through an AWS gateway.
Run R code remotely (with renv).
Optionally pull a saved R environment back to local.
Keep a VPN GUI running on AWS and access it from your browser.

Quick start (daily)

Start/ensure gateway tunnel + VPN UI are running (see Daily workflow).
Test: ssh lynx
From project dir: ./sync-and-run.sh ‘lr=0.05 epochs=200’
For long runs: use tmux on lynx (see Long-running jobs).

Prerequisites

Local: ssh, rsync, tmux, a web browser, R (optional for local testing).
AWS gateway (Ubuntu) with: ssh, tmux, xpra, Cisco AnyConnect (vpnui).
Remote target (lynx): ssh server, R, renv, rsync.
Your AWS key file has permission 600 and is not committed.

Install on Ubuntu (gateway/lynx as needed):

sudo apt-get update
sudo apt-get install -y rsync tmux xpra r-base

One-time setup

1) Create an SSH launcher script for the AWS gateway

Save this as ~/bin/ssh_to_gateway, then chmod +x ~/bin/ssh_to_gateway. Replace placeholders in ALL_CAPS.

#!/usr/bin/env bash
set -euo pipefail
# Required replacements:
#   AWS_KEY=~/.ssh/AWS_MyServer.pem
#   AWS_USER=ubuntu
#   AWS_HOST=ec2-xx-xx-xx-xx.region.compute.amazonaws.com
#   LYNX_HOST=lynx.dfci.harvard.edu
# Ports forwarded locally:
#   8787 → lynx:8787 (RStudio Server)
#   2222 → lynx:22   (SSH into lynx via local port 2222)
#   8000 → aws:8000  (xpra html5 for Cisco AnyConnect GUI)

exec ssh -o ExitOnForwardFailure=yes \
         -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
         -i "$AWS_KEY" \
         -L8787:"$LYNX_HOST":8787 \
         -L2222:"$LYNX_HOST":22 \
         -L8000:localhost:8000 \
         "$AWS_USER@$AWS_HOST"

Notes:

If you won’t use xpra html5, you can drop -L8000.
-X is not needed when using xpra’s web UI.

2) Keep the VPN GUI running on the AWS gateway

Option A (simple): in the AWS shell, use tmux and xpra html5.

# On your local: run the gateway script
~/bin/ssh_to_gateway
# On the AWS shell you just opened:
tmux
# inside tmux:
xpra start --start=/opt/cisco/anyconnect/bin/vpnui --bind-tcp=0.0.0.0:8000
# detach tmux: Ctrl+b then d

You can now open http://localhost:8000 in your browser (thanks to the 8000 forward). Tip: tmux a to reattach; tmux ls to list sessions; xpra stop to stop xpra.

Option B (advanced/optional): run the SSH tunnel in the background with -Nf and start xpra in a separate login. Use autossh or a systemd user service for maximum resilience.

3) Configure passwordless SSH to lynx through the tunnel

With the gateway connection active (so local port 2222 forwards to lynx), generate and copy a key:

# Local:
ssh-keygen -t ed25519 -C "lynx-key"   # press Enter through prompts
ssh-copy-id -p 2222 saha@localhost     # replace 'saha' with your lynx username

Test:

ssh -p 2222 saha@localhost

4) Create an SSH config entry for lynx (multiplexed + keepalive)

Add to ~/.ssh/config on your local machine:

Host lynx
  HostName localhost
  Port 2222
  User saha
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 10m
  ServerAliveInterval 30
  ServerAliveCountMax 3

Now test:

ssh lynx
# On exit you may see: "Shared connection to localhost closed."
# To fully close the master:
ssh -O exit lynx

5) Create your project and sync script

On your local machine:

mkdir -p ~/proj && cd ~/proj
R -q -e "if (!requireNamespace('renv', quietly=TRUE)) install.packages('renv'); renv::init()"

Create sync-and-run.sh in the project root, then chmod +x sync-and-run.sh.

#!/usr/bin/env bash
# Usage:
#   ./sync-and-run.sh [--no-sync] [--no-run] [--pull-env] [--help] 'args for run.R'
# Examples:
#   ./sync-and-run.sh 'lr=0.05 epochs=200'   # sync + run
#   ./sync-and-run.sh --no-sync 'lr=0.05'    # run only
#   ./sync-and-run.sh --no-run               # sync only
#   ./sync-and-run.sh --pull-env             # pull remote_env.RData to local
set -euo pipefail

# -------- Settings (edit as needed) --------
REMOTE=lynx
REMOTE_RUN_DIR='tmp/proj'   # disposable copy for runs on lynx
SSH_CMD=ssh                 # set to 'autossh' if installed
RSYNC_SSH=(-e "$SSH_CMD")
EXCLUDES=(
  --exclude '.git'
  --exclude 'renv'
  --exclude '.Rproj.user'
  --exclude '.RData'
  --exclude '.Rhistory'
  --exclude '.DS_Store'
)
# -------------------------------------------

DO_SYNC=1
DO_RUN=1
PULL_ENV=0

while [[ $#--gt-0-| -gt 0 ]]; do
  case "$1" in
    --help)
      sed -n '2,40p' "$0" | sed '/^set -euo pipefail/,$d'
      exit 0
      ;;
    --no-sync) DO_SYNC=0; shift ;;
    --no-run)  DO_RUN=0;  shift ;;
    --pull-env) PULL_ENV=1; shift ;;
    *) break ;;
  esac
done

RUN_ARGS="$*"

# Optional: quick preflight checks
command -v rsync >/dev/null || { echo "rsync not found"; exit 1; }
command -v "$SSH_CMD" >/dev/null || { echo "$SSH_CMD not found"; exit 1; }

if [[ $DO_SYNC -eq 1 ]]; then
  rsync -az --delete -P "${RSYNC_SSH[@]}" "${EXCLUDES[@]}" \
    ./ "$REMOTE:$REMOTE_RUN_DIR/"
fi

if [[ $DO_RUN -eq 1 ]]; then
  "$SSH_CMD" -t "$REMOTE" bash -lc "
    set -euo pipefail
    cd \"$REMOTE_RUN_DIR\"
    mkdir -p _outputs
    Rscript -e \"if (!requireNamespace('renv', quietly=TRUE)) install.packages('renv'); renv::restore()\"
    R --quiet -e \"source('run.R'); save.image('remote_env.RData')\" --args $RUN_ARGS
  "
fi

if [[ $PULL_ENV -eq 1 ]]; then
  rsync -az -P "${RSYNC_SSH[@]}" \
    \"$REMOTE:$REMOTE_RUN_DIR/remote_env.RData\" ./
fi

What it does:

Syncs local project to lynx:$REMOTE_RUN_DIR (excluding heavy/irrelevant files).
Restores R packages via renv on the remote and runs run.R with args.
Optionally pulls remote_env.RData back to your local machine.

6) Long-running jobs

SSH connections can drop. Use tmux on lynx:

ssh lynx
tmux new -s rsession
# inside tmux, run your R script manually or with the sync script invoked beforehand
# detach: Ctrl+b then d
# later:
tmux attach -t rsession
# list sessions: tmux ls

Daily workflow

Start a local tmux and connect to the AWS gateway:

tmux
~/bin/ssh_to_gateway
# authenticate VPN inside the xpra window (see step 2)

Detach local tmux if desired (Ctrl+b then d).

In your browser, open http://localhost:8000 and authenticate the VPN. If xpra seems down:

# Reattach to the AWS shell:
tmux a
xpra stop || true
xpra start --start=/opt/cisco/anyconnect/bin/vpnui --bind-tcp=0.0.0.0:8000
# Detach again from tmux when done.

For a new project, one-time on lynx (directories, renv init, etc.). Otherwise, skip.
Develop locally and test.
From your project root, run:

./sync-and-run.sh 'lr=0.05 epochs=200'
# flags: --no-run, --no-sync, --pull-env

If the lynx connection behaves oddly:

ssh -O exit lynx   # resets the multiplexed master connection