Never!
Today I had the pleasurable experience of doing a remote upgrade on my server. Usually, remote upgrades on boxes with no LOM is a nail-biting edge-of-the-seat experience, not so when they're running OpenBSD - for me, at least.
I started by fetching the source and building a new kernel, a step made necessary by my use of RAIDFrame with autoroot. Most people wouldn't need to build a new kernel. Then I installed the kernel and rebooted, untarred the 4.0 tarballs and followed the rest of the instructions from the OpenBSD FAQ. I even had time to upgrade Wordpress.
Time used: 20 minutes