Batch And Cli

---
name: batch-and-cli
description: "Use hailtop.batch DAGs and packaged hailctl commands for Hail
  Batch, cloud configuration, authentication, cluster helpers, and file
  description workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Batch and CLI

Use this sub-skill when the task mentions Hail Batch, `hailtop.batch`, `Batch`, `LocalBackend`, `ServiceBackend`, `new_job`, resource files, Docker/resource settings, `hailctl batch`, `hailctl config`, `hailctl auth`, `hailctl dataproc`, `hailctl hdinsight`, `hailctl describe`, billing projects, requester-pays, regions, or `remote_tmpdir`.

## Route First

- Read `references/batch-workflows.md` for Python `hailtop.batch` DAG construction, local/service backend choice, dependencies, input/output files, resource groups, Docker images, resource requests, and scatter/gather planning.
- Read `references/cloud-and-cli.md` for packaged `hailctl` command families, Batch service configuration, auth, Dataproc, HDInsight, `describe`, and cloud side-effect boundaries.
- Read `references/troubleshooting.md` for LocalBackend/Docker/tmp failures, ServiceBackend auth/billing/remote tmpdir errors, requester-pays access, command interpolation mistakes, and cloud skip conditions.
- Run `scripts/batch_local_smoke.py --help` first; use `scripts/batch_local_smoke.py --dry-run` for a no-execution template or `scripts/batch_local_smoke.py --run` for a local-only smoke when `hailtop.batch` is importable.

## Boundaries

- For Hail query backend selection, `hl.init`, `hl.init_batch`, Spark/local setup, package import failures, or generated runtime asset issues, route to `../setup-and-backends/SKILL.md` unless the task is specifically about `hailtop.batch` DAG execution.
- For Table or MatrixTable analysis code, route to `../tables-and-expressions/SKILL.md` or `../genomics-analysis/SKILL.md`; use this sub-skill only to package those scripts as Batch jobs.
- For sparse `VariantDataset` or GVCF combiner workflows, route data-processing logic to `../variant-datasets/SKILL.md`; keep this sub-skill focused on orchestration.
- Do not recommend private development, service-maintenance, or deployment commands as public user workflows. Prefer packaged `hailctl` commands visible in installed help.

## Verified Entry Points

- Python API: `hailtop.batch.Batch`, `LocalBackend`, `ServiceBackend`, `Batch.new_job`, `Batch.new_python_job`, `Batch.read_input`, `Batch.read_input_group`, `Batch.write_output`, `Job.depends_on`, `BashJob.command`, `BashJob.declare_resource_group`, `BashJob.image`, `Job.cpu`, `Job.memory`, `Job.storage`, and `Job.regions`.
- Installed signatures include `Batch(name=None, backend=None, attributes=None, requester_pays_project=None, default_image=None, default_memory=None, default_cpu=None, default_storage=None, default_regions=None, default_timeout=None, default_shell=None, default_python_image=None, default_spot=None, project=None, cancel_after_n_failures=None)`.
- Installed signatures include `LocalBackend(tmp_dir='/tmp/', gsa_key_file=None, extra_docker_run_flags=None)` and `ServiceBackend(*args, billing_project=None, bucket=None, remote_tmpdir=None, google_project=None, token=None, regions=None, gcs_requester_pays_configuration=None, gcs_bucket_allow_list=None)`.
- Verified CLI help covers `hailctl --help`, `hailctl auth --help`, `hailctl batch --help`, `hailctl config --help`, `hailctl dataproc --help`, `hailctl describe --help`, `hailctl hdinsight --help`, and `python -m hailtop.hailctl --help`.

## Safety Defaults

- Start with `LocalBackend` for examples, smoke tests, and DAG debugging; it avoids Batch Service credentials and cloud billing.
- Treat `ServiceBackend`, `hailctl batch submit`, `hailctl batch init`, `hailctl dataproc start`, `hailctl dataproc stop`, `hailctl dataproc modify`, `hailctl hdinsight start`, and `hailctl hdinsight stop` as cloud-mutating or billing-impacting.
- Use placeholders for billing projects, buckets, tokens, and remote tmpdirs; do not store credentials in scripts, skills, prompts, or logs.
- Close explicit backend objects after use with `backend.close()` or the async close pattern used by the surrounding application.