GPU & Node Health
GPU temperature, power, clock speed and why it is throttling, memory (ECC) and driver (XID) errors, and utilization, read from NVIDIA’s own GPU health tool (DCGM). Server CPU, memory, disk, and network-card state alongside it.
AI Cluster Monitoring
AI cluster monitoring.
Catch the bad link before it stalls the job.
One bad InfiniBand port (the high-speed link between GPU servers) or one GPU logging hardware errors can slow an entire training job while the cluster stays online, and the bill keeps running. Automated checks monitor GPU health and the fabric between the servers. We tune alerts to your cluster’s real baseline, and send critical alerts to a senior engineer who has run production InfiniBand fabrics at hyperscale. Monitoring is automated; human response is Mon–Fri, 8 AM–5 PM Pacific.
Experience
InfiniBand fabrics run at hyperscale before we offered this to you.
This service comes from hands-on production operation of AI research infrastructure: the InfiniBand fabric that links the GPUs, and the Ethernet and storage networks around it. The numbers below are what our engineer ran, so the person on the alert has already seen a fabric at this scale. Full background on the About page.
Compute fabric
~2,400
InfiniBand-attached nodes operated in production.
IB switches
24
8 spine, 16 leaf, run under both UFM and OpenSM (the two InfiniBand fabric managers).
Ethernet
~560
Leaf/spine and edge switches around the compute and storage tiers.
Footprint
4
Sites, ~5,600 inventoried devices.
Prior-employer engagement; the client is not named. Figures describe the environment operated, not an Ambio Edge customer.
What Gets Monitored
GPUs, nodes, and the fabric between them.
A training job runs at the speed of the slowest link between its GPUs. We cover the whole path, GPU to switch to storage, so a slowdown is traced to one port or one node instead of guessed at. If your own team or integrator runs the cluster, this works as a co-managed layer: they keep the keys, we watch the fabric.
InfiniBand Fabric Monitoring
Link errors, link-downs, port resets, and ports that came up slower or narrower than they should have. Health of the subnet manager (the service that keeps the fabric routed) on UFM or OpenSM. Alerts fire on the rate of new errors, not on lifetime totals, so you hear about a link that is going bad now, not one that had a bad day last year.
RoCE / Ethernet Fabric Monitoring
For clusters that run GPU traffic over Ethernet (RoCE, RDMA over Converged Ethernet) instead of InfiniBand: switch link errors and drops, the congestion signals (PFC pause, ECN marks) that mean the fabric is asking senders to slow down, and uplink and link-bundle (LACP) state. These are the signals that quietly stretch every training step.
Storage Network Path
Error and throughput telemetry on the links between the GPU servers and storage, so a slow data-loading step is traced to a port rather than blamed on the model or the dataset.
Topology Drift
What the fabric actually looks like, compared against the cabling plan you signed off. A moved cable or a missing link is flagged when it happens, not weeks later as a mystery slowdown.
Alerting & Routing
Alert rules tuned to your cluster’s baseline, with severity tied to job impact, delivered to your team’s alert channel. Critical alerts also go to a senior engineer during business hours (Mon–Fri, 8 AM–5 PM Pacific), with the affected ports and nodes already identified. Outside those hours alerts still fire to your channel.
Your Engineer
One point of contact. Fabric-level context.
Alerts identify the affected port or node and when the issue began.
JJ
Jeff Johnson
Principal Network Architect
The senior engineer who writes your alert rules stays your point of contact when an alert fires. No handoffs. No loss of context when a link degrades.
Background: Operated production InfiniBand and Ethernet fabrics for hyperscale AI research infrastructure: ~2,400 IB-attached nodes, 24 IB switches, and ~560 Ethernet switches across 4 sites.
Cisco Certified
Fortinet Certified
CompTIA Certified
Fortinet Engage Partner
20+ Yrs
Network-Only
Technology Partners
Fortinet partner. Vendor-neutral advice.
We are a Fortinet Engage Advocate partner and supply FortiGate, FortiSwitch and FortiAP. We also monitor and manage the Cisco, Aruba, Arista and other enterprise gear you already run. You don’t have to rip anything out to work with us.
Scope
Monitoring and alerting. Clearly bounded.
If you run a GPU cluster without a dedicated fabric engineer, or your integrator left after the build, this is the gap it fills. Your scheduler, your MLOps stack and your fabric design stay with you.
✓ Included
- Continuous GPU, node, and fabric telemetry
- Alert rules and dashboards tuned to your cluster
- Senior-engineer alert routing with a written diagnosis
- Works alongside your existing Prometheus / Grafana stack
× Not Included
- Fabric design, cabling, or hardware procurement
- Job scheduler, Kubernetes, or MLOps administration
- Model training or tuning work
- Hands-on fixes inside your cluster (quoted separately when you want them)
FAQ
Common AI cluster monitoring questions.
What does pricing look like?
AI cluster monitoring is quoted per cluster, not off the per-site plan card. Three things drive the price: how many nodes, which fabric (InfiniBand, RoCE, or both), and how much of the monitoring stack you already run. You get a written, fixed monthly quote after one scoping call, whether or not you go ahead. The dashboards, alert rules and monitoring plan are yours to keep.
Do you design or build the cluster fabric?
No. This service is monitoring and alerting only. You or your integrator own the architecture; we watch it, catch degradation early, and hand your team a specific diagnosis: which port, which node, since when.
How does telemetry leave our environment?
Through the Ambio edge connector: outbound-only encrypted tunnels from a small VM or appliance on your side. No inbound ports are opened at your site. Collectors can also read from your existing Prometheus, DCGM exporter, or UFM API instead of polling devices directly.
We already run Prometheus and Grafana. Do you replace it?
No. If your cluster already exports DCGM and node metrics, we build on that: add fabric telemetry, write alert rules tuned to your baseline, and route alerts to a senior engineer during business hours. The dashboards and rules stay yours.
Start Here
Tell us about the cluster.
We’ll tell you what to watch.
One 30-minute call to walk through your node count, fabric, and current tooling. You get a written monitoring plan and a fixed monthly quote whether or not you continue with us. It is yours to keep. Office and branch networks around the cluster are covered separately under managed network monitoring from $695 per site per month. See a sample report (PDF) to judge how we write things up.
What Happens Next
Response from a senior engineer within 1 business day.
A direct conversation — no sales team, no runaround.
A written monitoring plan for your cluster.
Clear next steps if we are a fit — no pressure if we are not.