• 0 Posts
  • 8 Comments
Joined 9 months ago
cake
Cake day: December 23rd, 2023

help-circle

  • Gonna try my best here:

    Crowdstrike is an anti-virus program that everyone in the corporate world uses for their windows machines. They released a update that made the program fail badly enough that windows crashes. When it crashes like this, it tries to restart in case it fixes the issue, but here it doesn’t, and computers get stuck in a loop of restarting.

    Because anti-virus programs are there to prevent bad things from happening, you can’t just automatically disable the program when it crashes. This means a lot of computers cannot start properly, which means you also cannot tell the computers to fix the problem remotely like you usually would.

    The end result is a bunch of low level techs are spending their weekends manually going to each computer individually, and swapping out the bad update file so the computer can boot. It’s a massive failure on crowdstrikes part, and a good reason you shouldn’t outsource all your IT like people have been doing.


  • Was relocating a $10k piece of networking hardware and dropped it into another 10k VM server. Server was fine, network hardware was not. It was during a project that was a real mess, thought it was a stupid mistake though.

    We were able to work around it, though we did lose the contract we were trying to make. Honestly though given how rushed and panicked that whole three months was though, we were lucky nobody suffered anything worse. Real shitshow.