How often it breaks
While a model was trained on a cluster of 16,384 server cards, 419 unplanned interruptions happened in 54 days, about one every three hours. Of these, 148 (30.1%) came from failures of the cards and their connections, and another 72 (17.2%) from memory on the cards. The team kept more than 90% of useful time because automation handled 416 failures. Our translation to a small scale: 419 failures over 884,736 card-days give one failure per about 2,100 days of one card running at full load. That means one failure every year and a half for a server with 4 cards, and every 8–9 months for a cluster of 8 cards. This is an estimate from someone else’s data, not a promise. The maker does not list memory error correction in the specs of gaming cards, and some early RTX 5090 cards, less than 0.5% according to the maker, had a factory chip defect with no effect on AI workloads; such cards were replaced under warranty.
What to do every month
Update drivers and the serving engine, and check that the model answers the same way after the update. Watch temperature and memory errors. Back up settings, prompts and the document base. Update the model when a stronger one comes out, and run your own set of test requests again. Keep a plan for a card failure: a spare card, a GPU rental for a week, or switching agents to a cloud model for non-confidential data.
Who does it and at what cost
The median market salary of a DevOps engineer in Russia is $2,700–$3,000 a month; in Moscow it is up to $3,800 for a mid-level specialist and $4,400–$5,900 for a senior one, according to a market review of 3 September 2026. A server with one model does not need a full-time person: by our estimate, budget 10–20% of a position, $300–$700 a month, or a contractor’s support. Check the warranty period in the seller’s invoice; replacing a card outside warranty costs $6,000–$28,000 at prices of 29 September 2026.
Backup
The rule: a service on which a department’s work depends does not live on a single card without a plan. The cheapest backup is rental: H100 from $3 an hour and RTX 4090 from about $0.50 an hour with hourly billing in Russian clouds. Sign the rental contract in advance so that you can switch on in an hour, not in a week.