If you have a GitHub repository and GitHub Actions is enabled, you’re probably using a runner for CI hosted on GitHub. This has become the default for many projects because it’s as simple as adding a workflow, writing runs-on: ubuntu-latest, and GitHub provides the machine.
Although this default is useful, it also has limitations. GitHub Actions can be slow or down for maintenance, the hosted machine is generic, and GPU access isn’t something that can be easily turned on for most open source projects. For Trackio, these limitations started to become important. We needed reliable CPU CI for basic unit tests and front-end checks, as well as GPU CI for tests that needed to run on real CUDA hardware.
So we built an alternative where GitHub Actions takes care of the CI and jobs run with Hugging Face Jobs.
As a result, Trackio’s CI runs on Hugging Face jobs that stream back real-time logs, reducing CI time for CPU jobs by about 30% and enabling a completely new test suite to run on GPU machines.
This article provides step-by-step instructions on how to recreate the same configuration in your GitHub repository. If you’re using an agent, see this article. This is because CLI instructions are provided along with browser-based instructions for humans.
Let’s start with a quick introduction to Hug Face Job!
What is Hug Face Job?
Hugging Face jobs allow you to run commands or scripts on Hugging Face’s serverless infrastructure using almost any hardware flavor. A job is essentially:
Commands to run Docker images from Docker Hub or Hugging Face Space Environment variables and secrets for hardware options such as CPU, t4-small, and h200 GPUs
For example, you can run:
n/a work Python:3.12 Run Python -c. “print(‘Hello world’)”
or
n/a work UV run –flavor a10g-small “https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py”
That makes Jobs a natural fit for CI. CI jobs are already command-driven, run in a clean environment, and often benefit from choosing exactly the right hardware. For ML libraries, the GPU case is particularly attractive. You can run test suites on real GPU hardware without having to maintain your own always-on runners.
An important step is to connect your GitHub action to your HF job. This is explained below.
architecture
For this setup, I created huggingface/jobs-actions, a small bridge that turns GitHub Actions jobs into temporary self-hosted runners that run inside HF jobs.
The complete flow looks like this:
Pull requests trigger GitHub Actions workflows. GitHub queues jobs for which the runs-on label is not available, such as hf-jobs-cpu-upgrade or hf-jobs-t4-small, and sends a signed workflow_job.queued webhook to the dispatcher through the GitHub app. Dispatcher Space validates the webhook, checks the hf-jobs-* label, creates a short-lived GitHub runner registration token, and starts the HF job on matching hardware. The HF job launches a temporary GitHub Actions runner and registers it with your repository using its one-shot token. GitHub assigns pending workflow jobs to their runners. The runner runs the CI job, reports the status to GitHub, and exits.
From GitHub’s perspective, it’s just a self-hosted runner. From Hugging Face’s perspective, this is simply a job that launches a container that runs workflow steps from a GitHub action in your repository.
Step 1: Duplicate the dispatcher space
The first thing you need is a dispatcher. This is a small Docker space that receives GitHub workflow_job webhook events and launches HF jobs in response.
The GitHub app requires a webhook URL, and that URL is retrieved from the space, so create this first. This space must be under its own namespace or under the Hugging Face organization with write access.
Web settings
Go to hackingface/jobs-actions-dispatcher and click Clone this Space.
use:
Owner: HF User or Organization Name: jobs-actions-dispatcher Hardware: cpu-upgrade
Use cpu-upgrade for actual CI to ensure that the dispatcher remains available for GitHub webhooks. cpu-basic is fine for testing and will probably work, but it may sleep when inactive. If a webhook arrives during GitHub startup, the workflow can remain in the queue forever.
Once built, open the cloned space. You’ll see a section called “Required Space Secret”, which you can ignore for now. The landing page displays the GitHub app webhook URL that you’ll need in the next step. It will look like this:
https://YOUR-HF-NAMESPACE-jobs-actions-dispatcher.hf.space/webhook
CLI setup
If you want to set up dispatcher space using an agent or use the CLI workflow:
export HF_NAMESPACE=Your hf user or organization
export Space_ID=”$HF_NAMESPACE/Job Action Dispatcher”
Duplicate huggingface/jobs-actions-dispatcher in hf repository ”$SPACE_ID” \ —type space \ –flavor cpu-upgrade \ –exist-ok
Then set it like this:
export DISPATCHER_URL=“https://${HF_NAMESPACE}-jobs-actions-dispatcher.hf.space”
Step 2: Create and install a GitHub app
Next, create and install a GitHub app from the dispatcher space itself. This app requires permission to listen for queued workflow jobs and create a temporary self-hosted runner registration token.
Web settings
Open the cloned dispatcher space.
https://YOUR-HF-NAMESPACE-jobs-actions-dispatcher.hf.space
In the setup form, enter the GitHub repository where your HF job needs to run CI.
your Github organization/your repository
Next, click the button to create your GitHub app. GitHub will ask you to choose a name for your app. You can use any name that is available in your GitHub account or organization. The final screen after submission shows you exactly how to upload your app’s credentials to the dispatcher space using the hf CLI.
Important note: You must provide a Hugging Face token with permission to launch the job. This corresponds to a personal account or an organization that claims jobs. This token must be stored in the dispatcher space as the HF_TOKEN secret.
Finally, install your app in the same GitHub repository you entered in the space. When setting up Trackio, I installed it to gradio-app/trackio.
Agent-assisted setup
The GitHub app’s manifest flow is still browser-based, but agents can follow the same space-driven path.
export HF_NAMESPACE=Your hf user or organization
export GITHUB_REPO=YOUR-GITHUB-ORG/YOUR-REPO Open “https://${HF_NAMESPACE}-jobs-actions-dispatcher.hf.space”
Paste $GITHUB_REPO into the space, click the Create GitHub app button, select an available app name, and follow the generated GitHub instructions.
Once the app exists, install it to your repository from the app settings page. For GitHub organizations, the installation settings are:
https://github.com/organizations/YOUR-GITHUB-ORG/settings/installations
Step 3: Final configuration of the dispatcher
At this point you need to configure your dispatcher space. The GitHub app setup flow generated commands to upload your app’s credentials, webhook secret, and Hugging Face token to your space.
By default, HF jobs are launched in the same namespace as the dispatcher space. If you want to bill the job to another Hugging Face user or organization, optionally set HF_NAMESPACE as a space variable.
export SPACE_ID=YOUR-HF-NAMESPACE/jobs-actions-dispatcher Add hf space variable ”$SPACE_ID” -e HF_NAMESPACE=restart billing namespace hf space ”$SPACE_ID”
The token you configured in step 2 must correspond to this namespace.
Step 4: Change the run-on
The actual workflow changes are minor. instead of:
Running: ubuntu – latest
Use one of the labels handled by the dispatcher.
Running: hf-job-cpu-upgrade
For GPU tests, use GPU labels.
Running: hf-jobs-t4-small
For any GitHub actions you want to run in your HF job, this one line change is all you need to do.
Step 5: Test it out
To add a minimal smoke test workflow from the CLI, follow these steps:
mkdir -p .github/workflows
cat > .github/workflows/hf-jobs-test.yml ‘EOF’ Name: HF Jobs Tested on: pull_request: Push: Branch: (main) workflow_dispatch:
work:
test: Run: hf-jobs-cpu-upgrade Step: – Use: action/checkout@v4 – Run: echo “Hello from Hug Face Jobs”
EOF git add .github/workflows/hf-jobs-test.yml git commit -m “Run CI with hug face job”
git push
To check from the CLI:
gh run list –repo YOUR-GITHUB-ORG/YOUR-REPO —limit 5hf work ps –namespace ”$HF_NAMESPACE”
hf space log ”$SPACE_ID”
For example, this Trackio PR #565 should allow you to view logs just like a regular GitHub Action.
That’s it!
Considerations for choosing the right Docker image
I used ubuntu:22.04 for my initial CPU setup and installed missing system packages on every run. This worked, but was slower than necessary. The latest ubuntu images on GitHub include many developer tools by default. Not in the bare Ubuntu image.
For Trackio, UI testing requires the Playwright browser, Node, ffmpeg, sqlite, git, and regular Linux build dependencies. Hugging Face Jobs supports using any Docker image, so we switched to Microsoft Playwright images. This worked fine.
mcr.microsoft.com/playwright:v1.60.0-jammy
For GPU jobs, I used:
nvidia/cuda:12.4.0-runtime-ubuntu22.04
result
Here are the Trackio CI numbers:
Runner setup Runtime GitHub Comparison to average GitHub ubuntu-latest baseline 1 minute 40 seconds Baseline HF job CPU, Playwright image 1 minute 10 seconds -30 seconds, about 30% faster HF job GPU, t4-small label 45 seconds GitHub hosted GPU No baseline
The biggest accomplishment was GPU CI. The Trackio GPU check was run on the HF job and passed in 45 seconds, costing less than 1 cent at t4-small rates during that time.
The CPU results were also promising. With the right images, the Linux test job was faster than the baseline hosted on GitHub. This suggests that HF Jobs can be a practical CI backend, especially for ML projects that require custom images or accelerators.
The logs were also a nice surprise. GitHub Actions logs are useful, but large logs can slow down the web UI. HF job logs can be easily retrieved from the CLI.
n/a work Log > logs.txt
This makes it easy to inspect using local tools or coding agents. In our bridge, we mirrored the GitHub Actions job log to the HF job log, so we had enough information to debug the execution on either system.
Finally, although I didn’t need it for Trackio’s CI, HF jobs also support mounting volumes. This is very useful when you need to quickly load datasets and models from Hugging Face as part of CI.
Hopefully this gives you everything you need to try out HF jobs to run GitHub Actions.
