Hey yall, just wanted to chime in with a quick update on why we were down and have gone down a few times these past few days.
So… for those who don’t know a bit on the backend way of how lemmy works right now, it’s a lil more than a mess. Lemmy itself uses very little computing resources, you could probably run a decent instance on a toaster (I wouldn’t suggest it though) but the problem that arises rapidly is the way photos are managed. The photo database downloads images both that are uploaded and federated. There is an option to disable caching for federated NSFW posts by unchecking enable NSFW within the instance admin panel. That’s the only option related disabling federated images as of right now.
You see where I’m going with this, with this instance being as large as it is? Our photo database has been filling up pretty rapidly. We are attempting to troubleshoot solutions to prevent caching all federated content. Unfortunately as our backend team is attempting to figure that out, our pictures database just keeps ballooning more and more.
We have deleted some stuff in hopes to give more time, but we unfortunately have no choice but to migrate to object storage ASAP. Tomorrow there will likely be downtime as we migrate things over, and unfortunately there is no telling how long that will take.
Thank you for being patient and understanding while we navigate this. If you want to see updates to our status when we go down, check out our mastodon page here or come to our matrix public operations channel for updates (link in the sidebar.)
Things should work today though. Emphasis on should.
We are working on open collective still, but hosting this instance does cost money. If you are able to, donations are greatly appreciated. Especially as we will have to factor in object storage into our costs now.
You must log in or register to comment.
Oooh neat! Which object storage are you guys using? Ceph? MinIO? Something else? Would love to discuss gory details!
Just remove direct uploaded pictures, there are a number of image hosting sites that already exist that are both setup for this kind of thing and that people already use and link to.
This is already the case for videos for the very same reason.
The problem with a 3rd party hosting site is that there’s no guarantee they suddenly won’t go out of business, or hide or delete all their nsfw content like imgur and Tumblr and many others in the past.
Just hosting images for another site like this one isn’t profitable because they can’t sell ads for the hotlinked images served to users of that other site.
It is a conundrum to find a working long term solution.
I can’t believe you’re only just moving to object storage! Offload that stuff as much as possible (this is probably a grandmother suck eggs situation, I know).
When you say “photo database”, do you mean Lemmy literally stores the binary image files in the db, or just the storage area for images (ofc linked to from the db)?
By default lemmy as a software, the whole photo database is stored directly into a file folder that links itself to the drive. The entire photos. And of course the drive just rapidly inflates like a balloon since it stores not just local images but federated images
