I have two, one is actually complicated and one was so obtuse that I never would have figured it out in a million years:
Actually complicated: I still don’t know how it happened, but somehow an update on Arch filled the
bootpartition with junk files, which then caused the kernel update to fail because of no disk space, which then kind of tanked the whole system. It took ages, but with a boot disk and chroot-ing back into the boot partition I eventually managed to untangle it all. I was determined to see it through and not reinstall.Ridiculous: One day when using Ubuntu, the entire system went upside-down. As in, everything was working perfectly fine, but literally the screen was upside-down. After much Googling I had no luck figuring it out, then I accidentally found the solution - I’d plugged a PS4 controller into the USB on the laptop to charge it, and for some reason Ubuntu interpreted the gyroscope on the controller as “rotate the screen display” so when I moved it, the screen spun round. I only figured it out by accident when I plugged it back it and it spun back to normal lol.
Ridiculous: One day when using Ubuntu, the entire system went upside-down. As in, everything was working perfectly fine, but literally the screen was upside-down. After much Googling I had no luck figuring it out, then I accidentally found the solution - I’d plugged a PS4 controller into the USB on the laptop to charge it, and for some reason Ubuntu interpreted the gyroscope on the controller as “rotate the screen display” so when I moved it, the screen spun round. I only figured it out by accident when I plugged it back it and it spun back to normal lol.
LMAO what the fuck?
deleted by creator
This deserves some sort of funniest Linux problem award.
The controller thing is goddam hilarious
This is up there with the
redacted(just looked it up it’s called the 500-mile email)Ah I remember that one! Classic. I also remember a story about someone who lost an entire PC in their apartment. It was running and connected to the network, they could ping it, but couldn’t physically find it lol.
This is a phenomenal read. Thank you for sharing lol
Ridiculous
I had a similar one. I had a usb-powered fan cooling pad that my laptop was sitting on. My laptop would randomly go into boot loops when I turn it on. I thought it was a grub issue so I always had my usb stick ready to re-install grub. Did some dusting one day and forgot to plug in the cooling fan, then the boot loop never happened again. Turns out it was the fan plugged into the usb that was causing it.
A couple years ago on Reddit I saw a story where a dude working IT support had to drive to a remote office or replace a workstation that wouldn’t boot. When he got there the lady whose desk it was had some shitty USB fan or maybe an led Christmas tree plugged into one of the USB ports. He unplugged that and the pc booted fine.
I think this is likely related to USB cables as power cables and USB ports/voltages.
I have seen a lamp completely fry a MacBook. I wouldn’t be surprised to see something similar cause a boot loop.
I’ve found that the silliest desktop problems are usually the hardest to solve, and the “serious” linux system errors are the easiest.
System doesn’t boot? Look at error message, boot from a rescue disk, mount root filesystem and fix what you did wrong.
Wrong mouse cursor theme in some Plasma applications, ignoring your settings? Some weird font rendering issue? Bang your head against a wall exploring various dotfiles and rc files in your home directory for two weeks, and eventually give up and nuke your profile and reconfigure your whole desktop from scratch.
A couple of weeks ago I moved Firefox to one side. Window disappeared, but Firefox was still running “somewhere” on my desktop, but was not actually be rendered to the screen. Killing the process and relaunching just resulted in it be rendered to this weird black hole. Log out of gnome and log back in? Same! Reboot? Same!
Ended up deleting it’s config folder and re-attaching to Firefox sync in order to have it working again. No idea what went wrong, nor will I ever most likely.
There really should be a hotkey for “move window to primary display” or somesuch. The worst is when just the top “cleat” of the window is inaccessible, making it impossible to simply move the window yourself.
Alternately, a CLI tool to just trash a specific app’s window settings, or a system control panel that lets you browse these settings, would be incredible.
Hold down meta and you can drag the window from anywhere (on gnome at least thats a default)
In every GUI I’ve used, there are tiling or snapping hotkeys, something like Super + Arrow keys or something, that will usually put the window somewhere sane.
I laughed so hard reading your comment. I totally agree.
I manage a machine that runs both media transcodes and some video game servers.
The video game servers have to run in real-time, or very close to it. Otherwise players using them suffer noticeable lag.
Achieving this at the same time that an ffmpeg process was running was completely impossible. No matter what I did to limit ffmpegs use of CPU time. Even when running it at lowest priority it impacted the game server processes running at top priority. Even if I limited it to one thread, it was affecting things.
I couldn’t understand the problem. There was enough CPU time to go around to do both things, and the transcode wasn’t even time sensitive, while the game server was, so why couldn’t the Linux kernel just figure it out and schedule things in a way that made sense?
So, for the first time I read up on how computers actually handle processes, multi-tasking and CPU scheduling.
As FFMPEG is an application that uses ALL available CPU time until a task is done, I came to the conclusion that due to how context switching works (CPU cores can only do one thing, they just switch out what they do really fast, but this too takes time) it was causing the system to fall behind on the video game processes when the system was operating with zero processing headroom. The scheduler wasn’t smart enough to maintain a real-time process in the face of FFMPEG, which would occupy ALL available cycles.
I learned the solution was core pinning. Manually setting processes to run on certain cores of the CPU. I set FFMPEG to use only one core, since it doesn’t matter how fast it completes. And I set the game processes to use all but that one core, so they don’t accidentally end up queueing for CPU time on a core that doesn’t have the headroom to allow the task to run within a reasonable time range.
This has completely solved the problem, as the game processes and FFMPEG no longer wait for CPU cycles in the same queue.
Well that’s interesting… I’d have thought, possibly naively, that as long as a thread had work to do it would essentially behave like ffmpeg does?
Perhaps there’s something about the type of work though, that it’s very CPU-bound or something?
I think the difference is simply that most processes only have a certain amount that needs accomplishing in a given unit of time. As long as they can get enough CPU time, and do so soon enough after getting in line for it, they can maintain real-time execution.
Very few workloads have that much to do for that long. But I would expect other similar workloads to present the same problem.
There is a useful stat which Linux tracks in addition to a simple CPU usage percentage. The “load average” represents the average number of processes that have requested CPU time, but have to queue for it.
As long as the number is lower than the available number of cores, this essentially means that whenever one process is done running a task, the next in line can get right on with theirs.
If the load average is less than the number of cores available, that means the cores have idle time where they are essentially just waiting for a process to need them for something. Good for time-sensitive processes.
If the load average is above the number of cores, that means some processes are having to wait for several cycles of other processes having their turn, before they can execute their tasks. Interestingly, the load average can go beyond this threshold way before the CPU hits 100% usage.
I found that I can allow my system to get up to a load average of about 1.5 times the number of cores available, before you start noticing it when playing on one of the servers I run.
And whenever ffmpeg was running, the load average would spike to 10-20 times the number of cores. Not good.
That makes complete sense - if you’ve got something ‘needy’, as soon as it’s queuing up, I imagine it snowballs, too…
10-20 times the core count is crazy, but I guess it’s had a lot of development effort into parallelizing it’s execution, which of course goes against what your use case is :)
Theoretically a load average could be as high as it likes, it’s essentially just the length of the task queue, after all.
Processes having to queue to get executed is no problem at all for lots of workloads. If you’re not running anything latency-sensitive, a huge load average isn’t a problem.
Also it’s not really a matter of parallelization. Like I mentioned, ffmpeg impacted other processes even when restricted to running in a single thread.
That’s because most other processes will do work in small chunks that complete within nanoseconds. Send a network request, parse some data, decode an image, poll HID device, etc.
A transcode meanwhile can easily have a CPU running full tilt for well over a second, working on just that one thing. Most processes will show up and go “I need X amount of CPU time” while ffmpeg will show up and go “give me all available CPU time” which is something the scheduler can’t actually quantify.
It’s like if someone showed up at a buffet and asked for all the food that no-one else is going to eat. How do you determine exactly how much that is, and thereby how much it is safe to give this person without giving away food someone else might’ve needed?
You don’t. Without CPU headroom it becomes very difficult for the task scheduler to maintain low system latency. It’ll do a pretty good job, but inevitably some CPU time that should have gone to other stuff, will go the process asking for as much as it can get.
Grub.
Seriously. Tha was some fat as shit because I didn’t know what I was doing.
I broke my bootloader fucking with uefi settings. I was in a panic for a few hours because I hadn’t bothered to learn how that shit worked until then.
It sure was a relief when i got back into my system.
Not fixed but there is an Arch problem that is and will always be the bane of mi existence.
For some reason when I click with the trackpad buttons the touchpad gets frozen for like a second (it’s like they are recognised by the system as keyboard buttons, I have enabled that option to temporarily disable it when using keyboard).
I’ve checked for hours and days the libinput documentation and some synaptics libraries, even legacy ones. It is to this day the only problem that has lead me to reinstall my system but the problem remains.
It’s not even like I have some niche setup, I mean, surely there must be thousands of Arch users running with a ThinkPad X1 Carbon Gen 7, and surely not every single one of them must be running it like this, right?
It has come to a point where I just gave up and got used to my system as is, but I’m sure I would be running fanfare if some day I am able to fix it.
Around 2017 I spent three days on and off trying to diagnose why my laptop running elementary OS had no wifi support. I reinstalled the wifi drivers and everything countless times. It worked for many days initially then just didn’t one day when I got on the laptop. Turns out I had accidentally flipped the wifi toggle switch while it was in my bag. I forgot the laptop had one. Womp womp.
I had a friend come over to my place to fix her laptops wifi. After about an hour searching for any setting in windows that i could have missed, i coincidentally found a forum where one pointed out this could be due to a hardware wifi switch…
I once exited vim without having to look up the commands.
Full kernel corruption after a botched sudo full-upgrade.
I got the wonderful “bailing out you are on your own” shit as well.
Read a guide online about a hail mary ext file system journal recovery protocol, I ran it, like most things without reading too deeply.
Kernel was successfully repaired, Kubuntu kept on truckin’
Working for a VoIP company in the early 2010s I rm -rf’d the /bin/ directory. As root. On a production server. On site.
I ended up booting from my phone (android app for iso booting) then manually coppied over the files from another machine. Chrooted and some stuff was broken but rebuilding from the package manager reinstalled everything that was missing. Got the system back up in around 40 mins after that colossal screw up. Good fun and a great learning experience. Honestly, my manager should not have had me doing anything on a root shell with no training.
My first Linux machine crashing. This was way before Redhat, Ubuntu, Arch, or OpenSUSE. This was installed from 60+ floppy disks on a 386-40 with 8MB of RAM.
This machine ran happily, but it crashed under heavy load. I checked out causing the load by using different applications, but could not nail it to a certain software. So the next thing I checked was the RAM. Memtest86 ran for a day without any problems. But the crashes still came. So I got the infrared camera from the lab to see if some hardware overheats. Nope, this went nowhere, either.
Then I tested the harddisk. Read test of the whole HD went without problems. I copied the data on a backup medium and did a write and read test by dd’ing /dev/zero over the whole disk, and then dd’ing the disk to /dev/null. Nothing did show up.
I reinstalled the Linux, and it crashed again. But this time, I noticed that something was odd with the harddisk. I added a second swap partition, disabled the first, and the machine ran without problems. Strange…
So I wrote a small program that tested the part of the disk occupied by the old swap space: Write data, read data, and log everything with timestamps. And there was the culprit: There was an area on the HD where I could write any data, but when I read blocks from that area, a) It took a very long time for the read, b) the blocks I read were containing all zero, regardless of what I had written, and worst of all c) there was no error indication whatsoever from the controller or drive. Down at the kernel level, the zeroed blocks were happily served by the HD with an “OK”. And the faulty area was right in the middle of the original swap partition.
Nice read! Did you delete the old swap space or left it as-is?
I took no risks and binned the disk. I wanted to buy a bigger one, anyway.
Getting WiFi to work in 2003
NDISWrapper: we’re just gonna trick the Windows driver into thinking it’s running on Windows and intercept the system calls.
That was certainly an era.
Oh god I remember that. Luckily in 2003 my main computer was scraped together from discarded parts at my father’s day job, so it was ethernet only
In 2024 on a laptop I still have wifi problems though. Most recently, if I closed and opened the laptop lid (suspend + resume), the wifi hardware just disappeared off the face of the kernel.
Turns out that the
iwlwifikernel module just irreversibly crashes when the laptop suspends and can only be fixed with a reboot.So I had the fun task learning about systemd pre-suspend hooks to unload the driver before suspend and load it again on resume.
Turns out wifi drivers still suck in 2024
