I already get rate-limited like crazy on lemmy and there are only like 60,000 users on my instance. Is each instance really just one server or are there multiple containers running across several hosts? I’m concerned that federation will mean an inconsistent user experience. Some instances many be beefy, others will be under resourced… so the average person might think Lemmy overall is slow or error-prone.
Reddit has millions of users. How the hell is this going to scale? Does anyone have any information about Lemmy’s DB and architecture?
I found this post about Reddit’s DB from 2012. Not sure if Lemmy has a similar approach to ensure speed and reliability as the user base and traffic grows.
https://kevin.burke.dev/kevin/reddits-database-has-two-tables/
Bigger instances will indeed run multiple copies of the various components, it’s pretty standard software in that regard.
Usually at first that will start by moving the PostgreSQL database to its own dedicated box, and then start adding additional backend boxes, possibly adding more caching in front so that the backend doesn’t have to do as much work. Once the database is pegged, the next step is usually a write primary and one or more read secondaries. When that gets too much, you get into sharding so that you can spread the database load across multiple servers. I don’t know much about PostgreSQL but I have to assume it’s better than MySQL in that regard and I’ve seen a 1 TB MySQL database in the wild running just fine.
I think lemmy.world in general is hitting some scalability issues that they’re working on. Keep in mind the software is fairly new and is just being truely tested at large scale, there’s probably a ton of room for optimization. Also lemmy.world is still on 0.17 and apparently 0.18 changed the protocol a lot in a way that makes it scale much better, so when they complete that upgrade it’ll probably run a lot better already.
The part that worries me about scalability in the long term is the push nature of ActivityPub. My server is already getting several POST requests to
/inboxper second already, which makes me wonder how that’s gonna work if big instances have to push content updates to thousands of lemmy instances where most of the data probably isn’t even seen. I was surprised it was a push system and not a pull system, as pull is much easier to scale and cache at the CDN level, and can be fetched on demand for people that only checks lemmy once in a while.I need to start digging into Lemmy’s code and get familiar with the internals, still only a couple days in with my private instance.
I don’t have a coding background but this was very informative. Thanks for sharing
Would it be feasible to change it to a pull system at this point? I don’t think the Lemmy part of that is a problem, but ActivityPub may need to make big changes and I’m not sure how practical that is.
The implementation as far as I understand it is plain stupid. It prevents small instances from participating at any significant scale and seems happy to just drop data over the wire without reconciling. Seriously amateurish.
I’m not sure about the data being dropped, my instance was misconfigured for a day or two, and as soon as I fixed it, the data came right in. Instances repeatedly trying to push data to my instance is what clued me in that something was missing from my NGINX config. It backfilled pretty fast.
Although I wouldn’t mind if there was a fallback pull mechanism to remediate failed pushes.
Interesting. Curious if you have a better understanding of ActivePub - do you happen to know if the protocol guarantees synchonicity and what mechanism guarantees it?
I don’t, really going off on ~1 week of running my essentially single user instance and watching it do its thing. I need to read the spec and experiment with it when I have some more free time.
Pure speculation but my guess would be that the servers are expected to retry for a certain amount of time. I know there’s been some tickets opened for some big instances going out of sync with eachother and fixes being worked on to address those. I don’t know if it only fixes it forward or if that also backfills.
Also nothing preventing Lemmy from implementing a fallback way of doing a resync if it detects drift. “Hey lemm.ee, I lost everything since an hour ago, backfill please”.
Yeah, or batching changes and confirming receipt with a hash. From what I’ve been reading, the design seems a little janky.
The part that worries me about scalability in the long term is the push nature of ActivityPub. My server is already getting several POST requests to /inbox per second already, which makes me wonder how that’s gonna work if big instances have to push content updates to thousands of lemmy instances where most of the data probably isn’t even seen. I was surprised it was a push system and not a pull system, as pull is much easier to scale and cache at the CDN level, and can be fetched on demand for people that only checks lemmy once in a while.
I think there’s a benefit to the push model, as the instances can prioritize who to push to first if there’s scaling issues, instead of having to throttle GETs, effectively the end result is anyway the same that nothing ends up to other instances in real time (which is fine). I don’t know how lemmy works exactly, but could the push model just be a detail of activitypub https://flak.tedunangst.com/post/what-happens-when-you-honk ?
It definitely is part of the ActivityPub protocol, but I only glanced at the spec so far. I should probably follow that link and implement a toy ActivityPub app to get more familiar with how it works.
I think there’s a benefit to the push model, as the instances can prioritize who to push to first if there’s scaling issues, instead of having to throttle GETs,
The downside to this is smaller instances are penalized in that scenario, which would in turn could cause users to flock to megainstances until it becomes centralized again.
As I said, GETs are cacheable, so if one slaps Cloudflare in front you can handle millions of GETs for relatively cheap.
Maybe it’s batched however? I really need to read the spec. Pushing to thousands of servers every 1/5/10 minutes certainly would give a fair amount of headroom to make it work I guess.
Poorly. Lemmy will scale poorly.
I won’t be surprised if the larger instances start locking down more as a way to sustain themselves, like restricting communities or only allowing text posts.
Sometimes you have just to accommodate to the situation and keep going until it settles down. The error I think here is thinking something can’t have flaws and issues, even more if it’s not behind a corporations. And no one wants corporations.
It isn’t about accommodating to the situation, but planning for long term growth.
Right now, instances of Lemmy don’t have any way to fund server costs other than asking for donations. Outside of Wikipedia, that isn’t a sustainable business model. How is Lemmy supposed to survive if, every time a sub gains critical mass, it shuts down?
planning for long-term growth
Which is part of any scaling effort, and you can’t really guess through predicting and resolving bottlenecks, it takes some serious expertise. And as far as I know, the Lemmy devs have never built a high-scale service before, and I think that is possibly the single biggest risk to the growth and success of the Lemmy project in general.
Source: that’s my job, I’ve been doing that for some of the most high-scale services in the world for about a decade. I absolutely could help, actually I’d love to, but I definitely won’t under current Lemmy leadership, for reasons: https://lemmy.world/comment/596235
How about helping Kbin?
I think Kbin is something good being built by good people, I get what they’re trying to do, but unfortunately I don’t have a lot of faith that it will turn out to be a successful project.
In terms of technical scaling, I’m puzzled that they went with an interpreted language if the goal is scale. I get that the basic usage of Kbin’s features may not require a ton of CPU-heavy operations, or a fine handling of the memory; but once it meets sufficient scale, there will have to be some scale edge-case bottlenecks where you’ll want to step out of the beaten path and get lower-level, so I’m a bit confused about why they chose a technology that will make those harder to get past rather than easier. PHP is great for rapid prototyping, but I’d argue that’s not what the vision should be here.
About community scale, I’m not expert, but they seem to really care to offer a karma system; and we’ve seen the karma-farming behavior that this has been incentivizing on Reddit. I don’t see why it would be any different here if enough people end up joining. Lemmy is intentionally not offering a karma system, and it really feels like the healthier move long-term.
I think all it would take would be for the Lemmy devs to admit that they’re in over their heads, and that their political affiliations have been a hindrance to the project, to the point that they transition the governance of it to other people. I really hope they do that. If they do soon enough, they’re so far ahead and built on so much more long-term thinking, that I think it would pretty much make Kbin kinda obsolete. I have no special information about this, so I could be wrong, and I hope for them that I am; but I can see that as a pretty likely outcome.
(That, and on the shorter-term, I wouldn’t contribute to a product I don’t use, and I can’t use it for now because my usage is 100% mobile, and the current lack of API means no native client. I wish the mobile web was better than it is as an application platform…)
Have you found that their political leanings have affected you in any way? Just curious if you have some sort of bias that’s making you think people on the left can’t produce efficient software.
It hasn’t. But letting terrible people have power affects the world in normalizing violence and hatred. It’s not about left or right, if they were American racists against Chinese people, I would have the exact same problem. I’m personally quite on the left, but without the hate.
I am living safe and not being targeted with hateful violence like the Uyghurs or North Koreans are, so this is far, far more important than what can affect me.
When the protocol favors monoliths, we’re right back to the Reddit problem
Scalability doesn’t mean “favoring monoliths”. It’s just scalability and honestly, 60k users shouldn’t bring a service down. 60k users is not even close to being a monolithic instance.
Wouldn’t that create a natural balance though? A large instance starts struggling so people are incentivised to move to smaller instances or start new instances and so spread the load more evenly. That’s how it would scale. I’m surprised how many of the larger instances haven’t closed signups yet but that wouldn’t be a bad thing if they did.
The issue isn’t on the user end, but the sub end since that is where all the data is stored.
So, according to your proposal, the best thing a sub should do when it is getting popular is to go private with its existing subscribers and any new people who want to participate should go create their own sub in a different instance.
I wasn’t talking about subs, I’m talking about when an instance gets too popular. Ideally you’d want lots of small instances, ideally communities should be spread evenly as well and if your users are spread out that should happen more or less naturally.
Where do you think the main costs of internet traffic and data are?
As far as I’m aware lemmy does not support load balancing or high availability as it currently stands. But development is still in its infancy and I’m sure that’s a top priority
I don’t think it officially supports it but it does work! Lemmy.world is currently running on multiple containers load balanced by nginx. look at u/ruud latest post about it
I mean it can….it’s just very DB heavy. It would be on an admin to scale up and scale out a single instance witb multiple dbs, replication etc.
It would be nice to be able to assign dbs to a task (ie: one for federation updates, one for local community posts, one to service web requests. There may be a way to do that already but I’m not aware, it may need to be in code.
Also syncing/federation across instances seems to be a mixed bag. And my instance will sometimes waste threads trying to sync with instances they have come and gone. As a result some communities id love to see updates on don’t come through.
Ideally they figure a way to continue to optimize federation and allow smaller instances to just pick up the load.
Mine is open, but I’m not getting any registration requests. I’m not upset about it but their main join page still seems to optimize for larger instances. It would make more sense to optimize for smaller ones to better distribute load. And focus dev work on better l/smoother syncing between federated instances.
Some locking down is a concern. I would love to see a lemmy of trust group if that came to pass. Where you can join the group and federate. My biggest concern with open federation is the legal risk of things like CSAM or CP getting synced onto your instances even if you have the nsfw box unchecked.
It’s not only about scaling a single instance, but scaling the fediverse.
Currently, each instance sends all events to all federated instances. That means, essentially each instance needs to store and process a significant part of the entire fediverse. That’s insane and has to be addressed.
My expectation, or at least hope, is that Lemmy will grow horizontally, i.e. more instances for more specialised content, instead of vertically, i.e. more communities in singular, larger instances. Since it’s all federated, you can get to stuff in other instances.
I just had an idea. Let’s compare reddit and lemmy as land use metaphors.
Reddit is like one monolithic megacity. It’s full of communites, some big, encompassing entire neighbourhoods, and others smaller, having one street, one block, maybe even just one building.
Lemmy is like a country, with every instance a city. Some cities are big and varied, others are smaller and specialised, like ones dedicated entirely to fishing or aviation or being German. And you can choose a city to settle in and move between cities for your content. Some cities will be more open to sharing content with residents of other cities, and others will put up bigger restrictions. There are jokes about parts of the userbase on 4chan or Tumblr forming their own subcommunities, and the fediverse allows this in a very material way.
My expectation is that more cities may emerge as people develop more specialised communities. And since there are many cities, there is some resilience in the system. If an instance goes down, you’ve lost one instance. Out of christ knows how many. Chances are some of its content is duplicated across other instances, so nothing of value is lost. Meanwhile, if (/when) Reddit goes down, all of Reddit is gone.
In short, I hope lemmy develops more, smaller, specialised instances over time. Reddit allowed very niche insterests to have a corner, and despite that, I think the fediverse is more suited to allow for that than a centralised service.
Education probably. Back in the day people didn’t have any problem understanding that different forums had different capabilities. When MMOs were in full swing, people didn’t have problems understanding what being on thr popular server during peak hours meant.
Everyone has just gotten too used to centralization with a lot of money behind it. Eventually people will adjust their expectations. Even if Meta’s fediverse attempt takes off, there are always going to be niche communities that exist outside of those spheres, so if people want that, they’ll have to move.
The point of the fediverse is having a choice. Some people are going to chose megacorp of the week’s offering and that’s okay as long a little pockets exist for when people get mad at the megacorp. Also federation leaves space for multiple dominant platforms in a way the current system doesn’t.
In short, eventually some instances are going to be bankrolled either through a robust crowdcourcing effort or through being a company. That’s okay. The purpose of the fediverse is to allow for smaller niche ideas to be able to breathe without having to adhere to one group’s ideals. “If you don’t like it, make your own” is a fair statement now
The way i see it the best way forward would be each community runs there own instance and what we now call communities should become subtopics of that community.
So for example. Asklemmy could be an instance and its members are all people who believe in the value of that instance and want to be involved in sustaining it.
Explainlikeiam5 could be a subcommunities of this instance because its philosophy is largely the same. If asklemmy has plenty of scientist members they could open a askscience subcommunity too.
The majority of user traffic would all come from other smaller homerun instances.
Big instances that try to be everything at once are a side effect of the massive growth we are experiencing, they work now but will slowly become more centralized and are therefore doomed to fail (in my opinion)
To recalculate. How can we help Lemmy grow? By being proactive users that maintain something small we chose to care about.
Well, I run an instance, too. It’s not big at all, but I was thinking about the issue of scaling, too. You can only scale up a single server so much…
But on the other hand, Lemmy is still young. We’ll find solutions to that problem.
Also, interesting article. I only took a glance at it, but having only two tables kind of suggests that Reddit is using a relational database. So, if they’re not “normalizing” everything, why not use a completely different paradigm, like what MogoDB etc. has?
The database isn’t really the problem in the current state of things. The server is because:
- Until 0.18 there was no caching (for the UI) and the poorly implemented websockets
- The developers have admited that they aren’t proficient in SQL, in which case, why not using an ORM instead? Sure, they aren’t perfect but they will do better than the average developer at scale.
- There is no queue system for activityPub requests
- Because there is no queue, user requests and federation have the same priority when it shouldn’t and one can bottleneck the other
- Live inserts are used meaning that regardless of the DB used, performance is going to be killed since inserting data 1 at a time several times a second is a major waste of resource
Tl;dr: It’s trying to do everything and not that well. So users suffer because they have to share resources with non-UI related tasks.
The database suffer because it has to do an insert of 1 object X 50 times in a second when it could do it once for all 50 items.
Federation suffers because you can’t offload it to a seperate machine farm whose job will be to receive and send ActivityPub requests and send/read data from the correct queues to do so.
Federation also does a lot of live HTTP connects to other peers. It looks up users for messages. The whole design is very resource intensive, one single vote, comment, post at a time. There is also a lot of boilerplate JSON overhead in sending something as simple as a single vote.
It’s a challenge, for sure. It is known that there are some inefficiencies in the codebase, which are actively being worked on. But besides that, it’s tricky to know where bottlenecks are until the user influx happens, particularly with the novel federation architecture. Maybe it’s impossible to scale, maybe not, but we only now are seeing a testable use case. I would expect optimization work to start bearing fruit, but these thing take time.
60k is a massive instance btw. Remake your account somewhere else to help out
I love Lemmy but your question is legit. I just signed up with lemmy.world because lemmy.ml is slow/not responding.
Before making a post in lemmy.world guess what? lemmy.world isn’t responding. I know they have scheduled maintenance at 9 CET but it was 20 minutes before that.
Lemmy uses PostgreSQL, with a number of tables. It’s pretty standard stuff, looks like, and it’s in Rust. It’s assumed there’s only going to be one server per instance right now, but I’d expect that one server could keep up with a reasonable volume.
I already get rate-limited like crazy on lemmy and there are only like 60,000 users on my instance.
Is that a typical number of users to have already? Wow, we really are growing.
Rate limiting is it’s own thing, I guess some work will need to be done to find a non-exploitable way to do it that still scales.
It’ll scale horizontally, like E-mail. Lots of different servers operated by various entities. There may be some big players, like Outlook or Gmail, but overall there could be thousands of instances.
Since Mastodon is using ActivityPub too. How it worked after Elon?
Mastodon is a couple years older than Lemmy, it was already a fairly mature product. People are thinking Lemmy is badly coded but we’re all using a beta product with no user limits.
60,000 users is also a LOT for a Lemmy instance. They’ve gone from ~8,000 to 60k+ in a matter of weeks. I’m sure there will be many performance and efficiency improvements coming soon. Hopefully people spread out a bit more across other instances with less users!
The ideal way that ActivityPub federation works IMO is a bunch of smaller nodes coming together to make a large network.
If you have a bunch of people all on one or two instances then you’ll have a “central hub” of the network that’s constantly overloaded.
That’s my advice to community builders on this platform… Spread out across smaller instances, don’t just all sign up to a big one.
Moving the scaling problem from a few instances to federation is likely to cause more harm than good.
I don’t see how syncing a post across a hundred instances is more efficient than having a hundred users see the post on one instance.
