I really like the overall concept of Lemmy, so I decided to set up lemm.ee to support the Lemmy network with my skillset. I have previously had the privilege of being responsible for running large platforms online (end-to-end, everything from operations to software engineering), and so far, this experience seems to be extremely relevant for running Lemmy in its current state.
As for paying for hosting, my initial plan was to to just pay for everything myself as kind of a hobby, but the userbase at lemm.ee has been very gracious in first asking me several times to share costs, and then actually sending money once I set up donations. I’m not sure yet if this donations-based funding will be sustainable, or if it will fall off after the initial hype dies, but for now it’s really awesome to see that there are several other people who believe in lemm.ee and want to share financial responsibility for it.
That’s great! It’s kind of a crowdfunded instance, then. Makes me wonder if it would be feasible to implement some sort of collection box plugin or something…
Yes … if anyone is a developer looking for ways to provide value to the fediverse … I suspect the donation process is probably of high value.
I don’t know the best way for it to be done … but something so that it’s easy for users to setup a single or regular donation and easy for devs and admins to put the relevant button right into their platform … all so that whoever is willing to donate has every opportunity to do so.
Yes, and I know it’s counter to the core motivations of this movement, but probably need a centralized repository for donation that can be a universal door for funds that can then be distributed to vulnerable, but active, instances. Needs to be run by a collective of reps from instances meeting a minimum threshold of support for the community. Also needs to be nimble enough to revoke funding is an instance takes a hard evil turn.
Or maybe just an app/site that recommends a distribution of a set monthly amount (e.g. 30 bucks) to the instances you use the most as a user?
I don’t think we need that at all. Each instance can look after its own hosting and funding.
A centralised pot is asking for trouble - what about theft or corruption? What happens about disagreements like defederation? Seems a lot more trouble than it’s worth.
I’ve been hosting a gaming server plus other related stuff myself for some years now.
While the user base will definitely be different, relying solely on donations is unfortunately not sustainable long-term. Donations fluctuate massively based on time of year in my experience.
So it’s always good to periodically remind your community that lemm.ee needs donations to survive long-term.
When I do those reminders, users come out of the woods in droves to donate. It’s less that they’re unwilling to donate and more that they just forget to donate.
What does it actually cost? I have no basis for a ballpark guess even. I’ve seen this question asked to a number of admins and haven’t seen a direct answer.
It’s hard to judge how sustainable a donation based approach is without that info.
The costs will vary wildly depending on how the instance has been set up. If you set up all necessary services on a single VPS (as is the most common approach for smaller instances), then you can probably get by on $10-$20 a month. Splitting different services onto different servers, adding backups, load balancing, CDNs, redundancy, caches, etc will quickly increase the cost. Bigger instances need more powerful servers, that will increase the cost further.
On lemm.ee, we are currently not using very high-end servers, but we ARE using all the other things I mentioned above, and the monthly cost is currently hovering around $200 (that’s for 3 servers, a managed database, object storage, load balancing, a global CDN, and an e-mail provider). This is still on the very cheap side in the grand scheme of running online platforms, but definitely much more than I would want to pay for a single-user instance for example.
$200 is still on the low end, trust me - high end managed databases and compute resources are in the thousands, adding redundancy to that will double or triple it 😃
Ye if you’re adding like super redundancies etc sure. But I mean, it’s lemmy. We don’t need all that. My whole VM config is in ansible. I can literally scrap the whole thing, and redeploy it in 10 minutes. I just need to have a DB backup in case of some sort of catastrophic failure
IaC is awesome 👍 lemm.ee is deployed using Terraform.
My lemmy_server redundancy is mostly just so I can do infra changes without downtime (just take one node out of the load balancer, redeploy it and put the new one in), but it does also help a bit with general performance. I agree it’s not strictly necessary, the vast majority of other instances are just running on a single server, but I do think it’s very nice to have.
For back-ups, I have point-in-time recovery, so I can restore the database to any random timestamp - and it has actually come in useful once already when two weeks ago I was able to restore to a good state about a few minutes after a problem - I think nobody even noticed that anything happened in that case 😃
Problem with Terraform is it assumes cloud providers, which tend to be expensive (or at least, I don’t know a way to do terraform on hetzner dedis :D )
My solution is to get cheapskate VPS and dedis and loadbalance them as frontends. The VM request is manual, but I only have to do this once anyway. It’s what I’m currently doing with the AI Horde. Of course, that doesn’t help when there’s DB changes but still.
What object storage are you using btw? I’m thinking to move to R2 or smt since I’ve had good experience with them until now. Contabo’s is way cheaper but when I tried to use it for high-demand stuff it dropped dead on the spot. But it might be ok for Lemmy.
I can’t thank you enough for sharing your knowledge, I am very interested in learning about server management and being able to read your thoughts is something I find extremely invaluable, please keep up with the great work! 😁
Yeah pretty much my setup. Full DB and pictrs backups uploaded to Backblaze B2 every few hours. Config files all backed up. Also send a copy to my home lab. Encrypted on the server before it gets sent out.
Worst case - we lose a few hours of data.
I keep about a week’s worth of backups on B2 (and 2 days on my home server) which is just over 200GB now. But that costs next to nothing on B2.
Maybe eventually if my instance grows more I might consider doing a replica DB on another server.
My total costs are like, $7.50 a month but I only have 15 actually active users. I don’t need to grow, but I’m willing to. If the costs increase too much I’ll ask for donations and if that stops covering things I’ll just close registrations. That’s the great part of not being a business chasing infinite growth.
I really like the overall concept of Lemmy, so I decided to set up lemm.ee to support the Lemmy network with my skillset. I have previously had the privilege of being responsible for running large platforms online (end-to-end, everything from operations to software engineering), and so far, this experience seems to be extremely relevant for running Lemmy in its current state.
As for paying for hosting, my initial plan was to to just pay for everything myself as kind of a hobby, but the userbase at lemm.ee has been very gracious in first asking me several times to share costs, and then actually sending money once I set up donations. I’m not sure yet if this donations-based funding will be sustainable, or if it will fall off after the initial hype dies, but for now it’s really awesome to see that there are several other people who believe in lemm.ee and want to share financial responsibility for it.
That’s great! It’s kind of a crowdfunded instance, then. Makes me wonder if it would be feasible to implement some sort of collection box plugin or something…
Yes … if anyone is a developer looking for ways to provide value to the fediverse … I suspect the donation process is probably of high value.
I don’t know the best way for it to be done … but something so that it’s easy for users to setup a single or regular donation and easy for devs and admins to put the relevant button right into their platform … all so that whoever is willing to donate has every opportunity to do so.
Yes, and I know it’s counter to the core motivations of this movement, but probably need a centralized repository for donation that can be a universal door for funds that can then be distributed to vulnerable, but active, instances. Needs to be run by a collective of reps from instances meeting a minimum threshold of support for the community. Also needs to be nimble enough to revoke funding is an instance takes a hard evil turn.
Or maybe just an app/site that recommends a distribution of a set monthly amount (e.g. 30 bucks) to the instances you use the most as a user?
I don’t think we need that at all. Each instance can look after its own hosting and funding.
A centralised pot is asking for trouble - what about theft or corruption? What happens about disagreements like defederation? Seems a lot more trouble than it’s worth.
I’ve been hosting a gaming server plus other related stuff myself for some years now.
While the user base will definitely be different, relying solely on donations is unfortunately not sustainable long-term. Donations fluctuate massively based on time of year in my experience. So it’s always good to periodically remind your community that lemm.ee needs donations to survive long-term.
When I do those reminders, users come out of the woods in droves to donate. It’s less that they’re unwilling to donate and more that they just forget to donate.
What does it actually cost? I have no basis for a ballpark guess even. I’ve seen this question asked to a number of admins and haven’t seen a direct answer.
It’s hard to judge how sustainable a donation based approach is without that info.
The costs will vary wildly depending on how the instance has been set up. If you set up all necessary services on a single VPS (as is the most common approach for smaller instances), then you can probably get by on $10-$20 a month. Splitting different services onto different servers, adding backups, load balancing, CDNs, redundancy, caches, etc will quickly increase the cost. Bigger instances need more powerful servers, that will increase the cost further.
On lemm.ee, we are currently not using very high-end servers, but we ARE using all the other things I mentioned above, and the monthly cost is currently hovering around $200 (that’s for 3 servers, a managed database, object storage, load balancing, a global CDN, and an e-mail provider). This is still on the very cheap side in the grand scheme of running online platforms, but definitely much more than I would want to pay for a single-user instance for example.
And honestly, 200 is on the high-end even with this setup. lemmy.dbzer0.com is way less
$200 is still on the low end, trust me - high end managed databases and compute resources are in the thousands, adding redundancy to that will double or triple it 😃
Ye if you’re adding like super redundancies etc sure. But I mean, it’s lemmy. We don’t need all that. My whole VM config is in ansible. I can literally scrap the whole thing, and redeploy it in 10 minutes. I just need to have a DB backup in case of some sort of catastrophic failure
IaC is awesome 👍 lemm.ee is deployed using Terraform.
My
lemmy_serverredundancy is mostly just so I can do infra changes without downtime (just take one node out of the load balancer, redeploy it and put the new one in), but it does also help a bit with general performance. I agree it’s not strictly necessary, the vast majority of other instances are just running on a single server, but I do think it’s very nice to have.For back-ups, I have point-in-time recovery, so I can restore the database to any random timestamp - and it has actually come in useful once already when two weeks ago I was able to restore to a good state about a few minutes after a problem - I think nobody even noticed that anything happened in that case 😃
Problem with Terraform is it assumes cloud providers, which tend to be expensive (or at least, I don’t know a way to do terraform on hetzner dedis :D )
My solution is to get cheapskate VPS and dedis and loadbalance them as frontends. The VM request is manual, but I only have to do this once anyway. It’s what I’m currently doing with the AI Horde. Of course, that doesn’t help when there’s DB changes but still.
What object storage are you using btw? I’m thinking to move to R2 or smt since I’ve had good experience with them until now. Contabo’s is way cheaper but when I tried to use it for high-demand stuff it dropped dead on the spot. But it might be ok for Lemmy.
I can’t thank you enough for sharing your knowledge, I am very interested in learning about server management and being able to read your thoughts is something I find extremely invaluable, please keep up with the great work! 😁
Yeah pretty much my setup. Full DB and pictrs backups uploaded to Backblaze B2 every few hours. Config files all backed up. Also send a copy to my home lab. Encrypted on the server before it gets sent out.
Worst case - we lose a few hours of data.
I keep about a week’s worth of backups on B2 (and 2 days on my home server) which is just over 200GB now. But that costs next to nothing on B2.
Maybe eventually if my instance grows more I might consider doing a replica DB on another server.
My total costs are like, $7.50 a month but I only have 15 actually active users. I don’t need to grow, but I’m willing to. If the costs increase too much I’ll ask for donations and if that stops covering things I’ll just close registrations. That’s the great part of not being a business chasing infinite growth.