remote-gpu-trainer – skill DevOps & Hạ tầng cho agent AI

remote-gpu-trainer — Remote GPU Job Orchestration

Overview

Deploy and babysit long-running GPU jobs on rented boxes you don't own, across any platform, and

get the result off the box before the meter or a preemption kills it. The core insight: **you are a

short-term tenant on someone else's machine** — so the job is to *detach the work, make the result

outlive the instance, and stop the meter safely*, not to provision a cluster.

This skill is platform-agnostic at the core, platform-specific at the edges: a fixed set of

operating principles + a 6-phase lifecycle that hold everywhere, plus one profile per platform

(profiles/<platform>.md) that owns every concrete path, proxy, billing verb, and spot semantic. Its

defensible value is the union the big orchestrators skip: **Chinese cgroup-isolated rentals + bare-SSH

cheap boxes + the disk-budget / monitoring / teardown reality** that *is* the job on metered hardware.

When to Use This Skill

Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a **RENTED

or remote instance they do not own** — training, eval, ablation sweeps, batch inference, or large data

processing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云/矩池云/Featurize/

揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual):

"远程 GPU 训练", "GPU 租赁", "GPU rental", "租卡", "spot 抢占", "spot preemption", "断点续训",

"resumable training", "tmux 训练守护", "防 SSH 断线", "scp/rsync 上传", "多实例 ablation",

"远程 GPU 监控", "省钱关机/销毁实例", "stop vs terminate billing", "checkpoint 磁盘满",

"CUDA OOM/显存不足", "loss NaN/loss spike", "loss 不下降/不收敛", "overfit 单 batch",

"FSDP/DeepSpeed 配置", "多卡训练 hang", "dataloader worker/数据增广 bug". NOT for purely local

single-GPU training, in-instance multi-GPU DDP (use torchrun/accelerate), managed multi-cloud

price-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).

When NOT to use — and what to use instead

| Situation | Use instead |

|---|---|

| Local single-GPU, or multi-GPU DDP inside one box | torchrun / accelerate directly |

| Managed multi-cloud price-shopping + auto spot-recovery across Western clouds | SkyPilot (has its own Agent Skill) — then come back here to make your *code* resume-correct so its recovery actually works |

| Open BYOC dev environments | dstack |

| Zero-ops serverless inference | Modal |

| "Is this metric / ablation delta real?" | REQUIRED: verifying-dl-experiments (this skill owns *running* the job; that one owns *whether the number is true*) |

This skill is for the blind spot those tools leave: AutoDL + Chinese platforms, bare SSH/Slurm/K8s

rentals, and the operational gotchas (inode caps, mirror stalls, cgroup OOM, silent sync, spot grace

windows, irreversible teardown) that survive whichever provisioner you use.

Operating principles (the WHY — 10 invariants)

These hold on every metered, isolated, rented GPU; only the paths/CLI change. One line each; the deep

form with cross-platform nuance is in references/principles.md (read it before Phase 0).

1. Minimize paid wall-clock. The meter runs the whole time — smoke locally on CPU before renting, launch detached, release the instant verification passes.

2. Cheap checks before expensive compute. A 1–2 batch CPU smoke (logger off) kills import/config/shape/scale bugs for ~free. (Smoke *content* → verifying-dl-experiments.)

3. Trust artifacts you loaded, not log lines that claim success. "synced/saved/done" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process/artifact.

4. Know what survives stop vs destroy. Per platform, identify exactly which mount survives a *stop* and which survives a *terminate* — the data you need often lives on the volatile one. (The single biggest portability trap.)

5. Storage fails on the dimension you're not watching. Disk dies on inodes before bytes; the real hog hides in a symlinked cache; clean by value (keep tiny evidence, drop big scratch); monitor df -i, not just df -h.

6. Never mutate inputs under a live run. A running job holds its scripts in memory by byte-offset; overwriting one mid-run re-executes blocks. Version filenames.

7. Design for retry — failure is probabilistic, transfers are flaky, mirrors are route-specific. Make wrappers idempotent + resumable; retry the *identical* config; wrap bulk transfers in timeout+resume loops; a mirror/proxy speeds ONE route — validate on the same route the real transfer uses.

8. Checkpoint-to-durable + idempotent resume is the universal spine. File checkpoint to the platform's durable location + unconditional load-latest-on-startup is the *one* mechanism that survives an SSH drop, a Slurm walltime kill, a K8s reschedule, a spot preemption, and a Colab disconnect. The detach primitive (tmux/sbatch/Job/commit) is the swappable plug; this is the invariant.

9. Cost and destructive actions are the user's call. Never auto-release/terminate, never delete durable files without confirmation; if cleanup can't free space, ask to expand the disk rather than silently shrink the experiment.

10. Teach the user the platform, don't just drive it. Most users don't know a platform's non-obvious conveniences (one-click SSH-key registration, GPU-availability notifications, built-in panels) or its danger clocks (auto-release/auto-delete timers on a *stopped* box — AutoDL releases a 关机 instance after 15 days → data disk gone; a stop that keeps billing; low-balance purge). Surface them on first contact — #9 stops the agent *doing* the dangerous thing, #10 *warns the human* before the clock fires. Per-platform list → each profile's Surface to the user block.

> Monitoring constraints (substrate for #3): tool timeouts and exit notifications depend on the host. Check its documented limits; a never-exiting watcher may never notify. An unquoted | can create an unintended shell pipeline. The four-layer monitoring architecture is built on these facts → references/monitoring_patterns.md.

Code discipline (the wrapper & training scripts you write)

Two rules govern the launch/wrapper/training code this skill has you write — corollaries of #1 and #8, not new invariants:

1. Reuse before writing. Take the lowest rung that already works before adding code: the base image's pre-installed stack + platform features → a framework/library utility (torchrun / accelerate / HF) → your existing scripts/ templates → minimal new code. On a metered box a needless pip install also burns paid wall-clock and can break the image's ABI — Phase 1's rule (*the prebuilt image is the env; don't conda create on a rental*) is exactly this principle applied to dependencies.

2. Floor — minimum bounds scope, not correctness. Shrinking code must never drop what makes an expensive run survivable: checkpoint-to-durable + idempotent resume (#8), atomic writes, the error handling that prevents losing a long run, or seed/determinism logging. Keep one minimal self-check for non-trivial logic.

Pick your platform profile FIRST

Read the matching profile before Phase 0 — it owns every path, proxy, credential location, billing

verb, and spot rule the phases below delegate to. Each follows the same 8-field schema

(profiles/_schema.md).

> New here? The path is: (1) find your platform in the table below → (2) read that profile's LAUNCH

> section (it walks rent → register SSH key → reach the box) → (3) come back and run the 6 phases from Phase 0.

> Already have a box you can ssh into? Skip straight to Phase 0.

| You're on… | Profile | Kind | Detach primitive | Meter-stop verb |

|---|---|---|---|---|

| AutoDL (deepest, battle-tested) | profiles/autodl.md | ssh-rental | tmux | 关机 (stops meter, keeps disk — the AutoDL exception) |

| RunPod | profiles/runpod.md | ssh-rental | tmux | terminate (stop still bills 2×; destroys volume disk) |

| vast.ai | profiles/vastai.md | ssh-rental (spot) | tmux | destroy (stop bills disk forever) |

| Lambda | profiles/lambda.md | cloud-api | tmux | terminate (no stop state) |

| Paperspace | profiles/paperspace.md | cloud-api | tmux | destroy + release IP + delete storage (shut-down stops compute only) |

| 恒源云 / 矩池云 / Featurize / 揽睿星舟 | profiles/china.md | ssh-rental | tmux | per-platform (data disk often bills while stopped) |

| Bare SSH box / Slurm / K8s / Colab-Kaggle | profiles/generic-ssh.md | ssh / slurm / k8s | tmux / sbatch / Job / commit | manual (a forgotten box bills 24/7) |

> Profile confidence: AutoDL is battle-tested from the author's daily use; the other six profiles are

> built from each platform's official docs + community reports (cited inline, verified <month>) and not

> yet independently live-tested — lean on the Phase-0 live measurements and **re-verify any teardown/

> billing fact against

… (nội dung đầy đủ có trong gói tải về)