MMOOZZGG.com

Hardware

AI server maintenance: what the articles about “your own AI” leave out

3 min read10 sources

AI server maintenance is the regular work that keeps a GPU server running. Hardware breaks, models age, updates break compatibility, and an engineer costs more than electricity. Below are reliability figures from the largest published report, their translation to a small server and the list of maintenance tasks that someone must do every month.

On this page · 4 sections

How often it breaks

While a model was trained on a cluster of 16,384 server cards, 419 unplanned interruptions happened in 54 days, about one every three hours. Of these, 148 (30.1%) came from failures of the cards and their connections, and another 72 (17.2%) from memory on the cards. The team kept more than 90% of useful time because automation handled 416 failures. Our translation to a small scale: 419 failures over 884,736 card-days give one failure per about 2,100 days of one card running at full load. That means one failure every year and a half for a server with 4 cards, and every 8–9 months for a cluster of 8 cards. This is an estimate from someone else’s data, not a promise. The maker does not list memory error correction in the specs of gaming cards, and some early RTX 5090 cards, less than 0.5% according to the maker, had a factory chip defect with no effect on AI workloads; such cards were replaced under warranty.

What to do every month

Update drivers and the serving engine, and check that the model answers the same way after the update. Watch temperature and memory errors. Back up settings, prompts and the document base. Update the model when a stronger one comes out, and run your own set of test requests again. Keep a plan for a card failure: a spare card, a GPU rental for a week, or switching agents to a cloud model for non-confidential data.

Who does it and at what cost

The median market salary of a DevOps engineer in Russia is $2,700–$3,000 a month; in Moscow it is up to $3,800 for a mid-level specialist and $4,400–$5,900 for a senior one, according to a market review of 3 September 2026. A server with one model does not need a full-time person: by our estimate, budget 10–20% of a position, $300–$700 a month, or a contractor’s support. Check the warranty period in the seller’s invoice; replacing a card outside warranty costs $6,000–$28,000 at prices of 29 September 2026.

Backup

The rule: a service on which a department’s work depends does not live on a single card without a plan. The cheapest backup is rental: H100 from $3 an hour and RTX 4090 from about $0.50 an hour with hourly billing in Russian clouds. Sign the rental contract in advance so that you can switch on in an hour, not in a week.

Questions and answers

Can you do without an engineer?

On a laptop or a Mac Studio for one team, yes, if one of the staff can update software. For a server that serves a department, no.

How can you tell that a card is starting to fail?

Memory errors in the logs, rising temperature at the same load, random reboots under load. Monitoring is the first thing to install after the model.

Sources and fact-checking

Checked by the editors on 29 September 2026.

  1. Tom’s Hardware, failures while training Llama 3 on 16,384 H100s, 2024 (tomshardware.com)
  2. 3DNews, RTX 5090 in Russia from $4,200, 12 March 2025 (3dnews.ru)
  3. Habr (Slurm), DevOps engineer salaries, 3 September 2026 (habr.com)
  4. Servermall, NVIDIA A100 40/80 GB, prices as of 29 September 2026 (servermall.ru)
  5. Servermall, NVIDIA H100 SXM5 80 GB, price as of 29 September 2026 (servermall.ru)
  6. Intelion Cloud, H100/H200 rental (checked 29 September 2026) (intelion.cloud)
  7. Tproger, cloud GPUs for ML in 2026, 21 May 2026 (tproger.ru)
  8. Tom’s Hardware, NVIDIA: defect in less than 0.5% of RTX 5090 cards, no effect on AI workloads, 22 February 2025 (tomshardware.com)
  9. GPU Index, RTX 4090 rental in Russia: 173 offers, from about $0.40 an hour (checked 30 September 2026) (gpuindex.ru)
  10. GPU Index, H100 rental in Russia: 41 offers, from $3 an hour (checked 30 September 2026) (gpuindex.ru)

Found a mistake? Tell us

Publications

Contact us