NVIDIA agent workflows with Humiris

Before you start

“NVIDIA GTX” describes a consumer GPU family, not a universal NIM target. Check the exact GPU architecture, VRAM, operating system and container profile on the selected model card. NVIDIA recommends data-center GPUs such as A100 80 GB, H100 or L40S for many production NIM workloads; a smaller model may run on a compatible workstation GPU.

01 · ARCHITECTURE

Separate agents from GPU replicas

An agent is workflow state plus a sequence of model/tool steps. It should not own a permanent model process. Humiris accepts every model request, selects a compatible route, and sends it to a shared NVIDIA inference pool.

CLIENTSApps · APIs · agent triggers

→

WORKFLOWSQueue · state · retries

→

HUMIRISRoute · govern · observe

→

NVIDIANIM · Dynamo · GPU pool

One public Humiris key and route per environment.
NVIDIA credentials and private endpoints stay server-side.
Every workflow step carries a run ID, tenant ID and policy context.
Tools execute in separate workers with least-privilege credentials.

02 · LOCAL NVIDIA WORKSTATION

Start with one compatible NIM

Use this path for development on Linux with an NVIDIA GPU, current drivers, NVIDIA Container Toolkit and Docker. Select the image and profile from the model card instead of assuming that every NIM fits your GPU.

bash

# 1. Check that Docker can see the NVIDIA GPU
nvidia-smi
docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi

# 2. Authenticate to NVIDIA NGC without printing the key
export NGC_API_KEY="<your-ngc-key>"
echo "$NGC_API_KEY" | docker login nvcr.io   --username '$oauthtoken' --password-stdin

# 3. Keep model artifacts between restarts
export LOCAL_NIM_CACHE="$PWD/.nim-cache"
mkdir -p "$LOCAL_NIM_CACHE"
chmod 777 "$LOCAL_NIM_CACHE"

# 4. Use the image and GPU profile shown on the model card at build.nvidia.com
export NIM_LLM_IMAGE="nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.9"
docker run --rm --gpus all   --name humiris-nim   -e NGC_API_KEY   -v "$LOCAL_NIM_CACHE:/opt/nim/.cache"   -p 8000:8000   "$NIM_LLM_IMAGE"

Verify NIM before adding Humiris

bash

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<model-id-returned-by-v1-models>",
    "messages": [{"role":"user","content":"Reply with NIM ready"}],
    "max_tokens": 32
  }'

The NIM LLM API exposes OpenAI-compatible Chat Completions, Completions and Responses endpoints, including streaming. A failed direct smoke test is an NVIDIA/runtime issue; fix it before testing routing.

03 · CONNECT HUMIRIS

Register the NVIDIA endpoint once

Connect the provider.Open Humiris → Providers → NVIDIA NIM or Custom. Use the reachable NIM base URL ending in /v1, its model ID, and a server-side bearer token when the endpoint is protected.
Test the connection.Humiris must receive a successful model list or chat response before the model becomes selectable.
Create a Mix route.Choose Text, Image or Voice, add the compatible NVIDIA models, then set cost, latency, privacy and quality constraints.
Set fallbacks deliberately.For private data, keep fallback inside an approved NVIDIA route. Never silently cross a privacy boundary.
Create a Humiris API key.Use separate keys for development, staging and production; attach budgets and rate limits.

typescript

import OpenAI from "openai";

const humiris = new OpenAI({
  apiKey: process.env.HUMIRIS_API_KEY,
  baseURL: "https://api.humiris.ai/api/v1",
  timeout: 45_000,
  maxRetries: 2,
});

const response = await humiris.chat.completions.create({
  // The route can prefer NVIDIA NIM and fall back only when policy allows it.
  model: process.env.HUMIRIS_ROUTE ?? "humiris/auto",
  messages: [
    { role: "system", content: "You are a governed support agent." },
    { role: "user", content: "Summarize case #1842 and propose next actions." },
  ],
});

console.log(response.choices[0].message.content);

04 · AGENT WORKFLOW

Make the workflow bounded and resumable

Each run needs an explicit step limit, deadline, budget, approval policy and idempotency key. Persist state after every step so a worker crash resumes the run rather than restarting it.

json

{
  "workflowId": "customer-support-v1",
  "route": "humiris/auto",
  "limits": { "maxSteps": 8, "deadlineMs": 120000, "maxCostUsd": 0.20 },
  "policy": {
    "sensitiveData": "nvidia-private-route",
    "allowExternalFallback": false,
    "humanApprovalFor": ["refund", "account_delete"]
  },
  "steps": [
    { "id": "classify", "type": "model", "output": "intent" },
    { "id": "retrieve", "type": "tool", "tool": "knowledge.search" },
    { "id": "draft", "type": "model", "dependsOn": ["classify", "retrieve"] },
    { "id": "review", "type": "model", "condition": "risk >= medium" },
    { "id": "respond", "type": "action", "approval": "policy" }
  ]
}

05 · 100–1,000+ AGENTS

Scale workers independently from inference

Put workflow steps in Redis Streams, BullMQ, Kafka, NATS JetStream or another durable broker. Stateless CPU workers consume tasks and call Humiris. NVIDIA NIM/Dynamo scales the GPU model pool separately.

typescript

import OpenAI from "openai";
import { Worker } from "bullmq";

const humiris = new OpenAI({
  apiKey: process.env.HUMIRIS_API_KEY,
  baseURL: "https://api.humiris.ai/api/v1",
  timeout: 45_000,
});

new Worker("humiris-agent-steps", async (job) => {
  const { runId, stepId, messages, route = "humiris/auto" } = job.data;

  // jobId must be runId:stepId so a retry cannot execute the same action twice.
  const result = await humiris.chat.completions.create({
    model: route,
    messages,
  });

  return {
    runId,
    stepId,
    output: result.choices[0].message.content,
  };
}, {
  connection: { url: process.env.REDIS_URL },
  concurrency: 32,
  limiter: { max: 200, duration: 1000 },
});

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: humiris-agent-worker
spec:
  replicas: 20
  selector:
    matchLabels: { app: humiris-agent-worker }
  template:
    metadata:
      labels: { app: humiris-agent-worker }
    spec:
      containers:
        - name: worker
          image: registry.example.com/humiris-agent-worker:1.0.0
          envFrom:
            - secretRef: { name: humiris-agent-secrets }
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits: { cpu: "2", memory: "2Gi" }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: humiris-agent-worker
spec:
  minReplicas: 20
  maxReplicas: 100
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: humiris-agent-worker
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 65 }

Layer	Scale signal	Protection
Agent workers	Queue depth and oldest task age	Concurrency cap, retries with jitter
Humiris gateway	Requests/sec and p95 latency	Tenant rate limits, circuit breakers
NVIDIA inference	Tokens/sec, TTFT, KV cache, GPU utilization	Dynamic batching, admission control
Tools	Per-tool queue and API quotas	Scopes, sandbox, human approval
State	Write latency and connection saturation	Idempotency, outbox, partitioning

Ramp with evidence

bash

# Do not start at 1,000 concurrent runs. Establish a baseline, then ramp.
k6 run -e HUMIRIS_API_KEY="$HUMIRIS_API_KEY"   -e HUMIRIS_ROUTE="humiris/auto"   tests/humiris-load.js

# Recommended stages:
# 1) 10 agents / 5 min     — correctness and traces
# 2) 100 agents / 15 min   — queue, timeout and fallback behavior
# 3) 500 agents / 30 min   — sustained throughput
# 4) 1,000+ agents         — only after SLO and cost gates pass

06 · PRODUCTION NVIDIA

Choose NIM Operator or NVIDIA Dynamo

NIM Operator

Use when you want Kubernetes-native lifecycle management for NIM microservices, model caches and NIMService resources. Install NVIDIA GPU Operator first.

NVIDIA Dynamo

Use for distributed inference graphs, KV-aware routing, disaggregated prefill/decode, multinode serving and independent scaling of inference components.

For a shared production cluster, install the NVIDIA GPU Operator, then the NIM Operator or Dynamo platform. Cache models before scale-out, expose only a private service to Humiris, and use Kubernetes NetworkPolicy plus workload identity or a secret manager.

07 · GO-LIVE CHECKLIST

Do not launch a swarm without these controls

Per-tenant quotas and a global emergency stop

Max steps, max cost and deadline on every run

Idempotency for every model and tool action

Human approval for payments, deletion and outbound messages

Prompt/tool version pinned in every trace

Redaction before logs and memory persistence

p50/p95/p99 latency, TTFT, tokens/sec and GPU metrics

Dead-letter queue with replay tooling

Provider and model circuit breakers

Load test plus failure, timeout and fallback drills

Official NVIDIA references

NVIDIA NIM for LLMs quickstart Install NVIDIA NIM Operator NIM Operator platform support NVIDIA Dynamo on Kubernetes

One NVIDIA model pool.Thousands of governed agent runs.

Separate agents from GPU replicas

Start with one compatible NIM

Verify NIM before adding Humiris

Register the NVIDIA endpoint once

Make the workflow bounded and resumable

Scale workers independently from inference

Ramp with evidence

Choose NIM Operator or NVIDIA Dynamo

Do not launch a swarm without these controls

Official NVIDIA references

One NVIDIA model pool.
Thousands of governed agent runs.