Another user posted the blog where they discuss their speedup techniques: https://tailscale.com/blog/more-throughput/

It’s likely that the kernel version can use similar techniques to surpass the performance of the userspace version that tailscale uses, but no one has put in the work to to make the kernel implementation as sophisticated as the userspace one.

I had a look through the comments on this HN thread the other day and came away more intrigued by https://github.com/openobserve/openobserve than hyperdx. Hyperdx is built on top of clickhouse whereas open observe has it’s own storage engines based on parquet files that can be accessed from local disk, S3, or a few other protocols.

I haven’t tried either option yet… I’m, currently using netdata for metrics and don’t do anything special for logs or tracing, but at tiny self-hosting scale I often find software with it’s own storage engines (often sqlite) to be extra hassle-free. I’m curious to kick the tires on openobserve for that reason.

This is a very strong explanation of what’s going on. And as a follow-up, I believe that ZeroTier present a single Ethernet broadcast domain, and so WoL tricks are more likely to work naturally there than with Wireguard. I haven’t used ZeroTier, and I do use Wireguard via Tailscale/Headscale. I’ve never missed the Ethernet features of ZeroTier and they CAN result in a very chatty wan if you’re not careful. But I think ZT would make this straightforward.

Though as other people note… the simplest/least-disruptive change is probably to expose some scripty thing on the rpi that can be triggered via be triggered over a routed protocol and then have the rpi emit the Ethernet broadcast packets from the physical network.

I use k8s at work and have built a k8s cluster in my homelab… but I did not like it. I tore it down, and currently using podman, and don’t think I would go back to k8s (though I would definitely use docker as an alternative to podman and would probably even recommend it over podman for beginners even though I’ve settled on podman for myself).

1. K8s itself is quite resource-consuming, especially on ram. My homelab is built on old/junk hardware from retired workstations. I don’t want the kubelet itself sucking up half my ram. Things like k3s help with this considerably, but that’s not quite precisely k8s either. If I’m going to start trimming off the parts of k8s I don’t need, I end up going all the way to single-node podman/docker… not the halfway point that is k3s.
2. If you don’t use hostNetworking, the k8s model of traffic routes only with the cluster except for egress is all pure overhead. It’s totally necessary with you have a thousand engineers slinging services around your cluster, but there’s no benefit to this level fo rigor in service management in a homelab. Here again, the networking in podman/docker is more straightforward and maps better to the stuff I want to do in my homelab.
3. Podman accepts a subset of k8s resource-yaml as a docker-compose-like config interface. This lets me use my familiarity with k8s configs iny podman setup.

Overall, the simplicity and lightweight resource consumption of podman/docker are are what I value at home. The extra layers of abstraction and constraints k8s employs are valuable at work, where we have a lot of machines and alot of people that must coordinate effectively… but I don’t have those problems at home and the overhead (compute overhead, conceptual overhead, and config-overhesd) of k8s’ solutions to them is annoying there.

1. If a service supports sqlite, I often will use that option. It provides everything a self-hoster needs from a DB with basically no operational overhead.
2. If I do need a proper RDBMS (because the software I’m using doesn’t support sqlite), I’m going to use…
   1. A single Postgres container.
   2. Configured with multiple logical “databases” (the container for schemas and tables), one DB for each app connecting.

I do this because I’m always memory constrained and the rdbms is generally the most memory-hungry part of any software stack. By sharing one db-process across all the apps that need it I get the most out of my db cache memory, etc. And by using multiple logical db’s, I get good separation between my apps, and they’re straightforward to migrate to a truly isolated physical DB if needed… but that’s never been needed.

… advertisement and push they did on sites like reddit…

The lemmy world admins advertised on Reddit? Can you link an example?

… their listing on join-lemmy.org…

Until recently EVERY lemmy instance was listed on join-lemmy.

And with the name Lemmy.world they did nothing to dissuade anyone from thinking that.

They run a family of servers under the world tld, including at least mastodon, lemmy, and calckey. They’re all named similarly.

I also saw nothing from .world not claiming to be the bigger instance(super lemmy)

They ARE the biggest instance, but that happened organically. It’s not based on any marketing claims from the admin team about being a flagship/super/mega/whatever instance. People just joined, and the admins didn’t stop them (nor should they). It’s not a conspiracy to take over lemmy. It’s just an instance that… until recently… happened to work pretty well when some were struggling.

I think the issue is that .world has put itself forward as some sort of super lemmy.

Citation needed. All the admins of lemmy world ever purported to do was host a well-run general-purpose (aka not topic-oriented) lemmy instance. It was and remains that, and part of being a well-run general purpose instance is managing legal risk when a small subset of the community generates an outsized portion of it.

Being well run meant that they scaled up and remained operational during the first reddit migration wave. People appreciated that, but continuing to function does not amount to a declaration of being a super lemmy.

World also has kept signups open through good times, and more recently bad. Other instances at various times shut down signups or put irritating steps and purity tests along the way. Keeping signups open is a pretty bare-minimum bar for running a service though, it is again not a declaration of being a super-lemmy.

Essentially lemmy world just… kept working (until recently when it has done a pretty poor job of that). I dunno where you found a declaration that lemmy world is a super-lemmy, but it’s not coming from the lemmy world admins, it’s likely randos spouting off.

Why would you use LVM to configure the RAID-1 devices? Btrfs supports raid1 natively.

Two tips:

I have not tried running WINE yet but I plan on doing so soon.

Steam “just works” on Linux, you can install it via flatpak (which I use) or from their deb repo. It includes “Proton”, which is a fancy bundle of wine and some extra open source valve sauce to make it nice and easy to use. Any game that runs on the steam deck also runs on Linux via proton, and there’s no messing around at all. It looks and feels just like steam on Windows, and thousands of games just work with no setup or config beyond clicking the big blue and green buttons to install and run. Not EVERY games works, but tons do. I’d heavily recommend this over raw wine to a beginner.

The second tip is not to ask what you can do on Linux. The answer, to a first approximation, is that you can do everything on Linux that you can do on Windows or OSX. I daily drive all three, and mostly do the same stuff on them. Instead, ask YOURSELF what you WANT to do on Linux. Then Google and ask us HOW to do it… or what the nearest approximation is if the precise thing you want to do doesn’t work on Linux.

I use postgres for my install and had a similar thing happen to me. I tried moving an org credential to a folder, which moved the folder to the org, and kicked all other credentials to “no folder”.

Thanks for confirming with your DB. That saves me sweating whether I should rebuild on PG at least, and also makes me feel better that it’s a folder bug and not generalized database corruption.

Having finished the heavy organizing, my rate of big org transfers has slowed and I haven’t reproduced again yet. Hopefully this will be uncommon enough to be a non-issue. Thanks again for the info.

Thanks for the suggestion, but sync seems to be working ok… at least on the read side. I was able to verify the pre-existing good state and the bad state afterward from multiple clients. If sync played into it, it must have been on a write somehow.

A very common DDoS attack uses UDP services to amplify your request to a bigger response, but then spoof your src ip to the target.

Having followed many reports of denial of service activity of Lemmy, I don’t think this is the common mode. Attacks I’d heard of involve:

• Using regular lemmy APIs backed by heavy database queries. I haven’t heard discussion of query rates, but Lemmy instances are typically single-machine deployments on modest 4-core to 32-core hardware. Dozens to thousands of queries per second to the heaviest API endpoints are sufficient to saturate them. There’s no need for distributed attack networks to be involved.
• Uploading garbage images to fill storage.

Essentially the low-hanging fruit is low enough that distributed attacks, amplification, and attacks on bandwidth or the networking stack itself are just unnecessary. A WAF is still a good if indeed OPs instance is getting attacked, but I’d be surprised if wafs has built-in rules for lemmy yet. I somewhat suspect one would have to do the DB query analysis to identify slow queries and then write custom waf rules to rate limit the corresponding API calls. But it’s worth noting that OP has provided no evidence of an attack. It’s at least equally likely that they dos’ed themselves by running too many services on a crappy VPS and running out of ram. The place to start is probably basic capacity analysis.

Some recent sources:

• https://sh.itjust.works/post/2732891
• https://lemmy.world/post/2923697

Very true and good points, and when it comes to snap I mostly agree with you. I would guess the “war on Ubuntu” going on is more due to Ubuntu’s history of making controversial decisions that go against the grain of what most other distros are doing at the time (creating and dropping Mir, creating Unity instead of using GNOME and then switching back to GNOME when they finally got Unity working well, installing an Amazon app out of the box in one version), many of which angered a lot of Linux community members before who are still angry despite Ubuntu rolling back most of those decisions, and they’ve found snap a great current scapegoat issue to use to vent their long-standing frustrations with Ubuntu at.

I agree with just about every word here. I lived through all this stuff. Mir and Unity were hugely disruptive to the OSS desktop community beyond Ubuntu and I was as salty about them as anyone. If someone is aware of this history and just fucking done with Ubuntu’s bullshit they’ll get no flak from me. I rarely see this coherent an argument made though, it’s much more often “snap bad, use this other distro that’s downstream of Ubuntu and shares all the same foundations but has a different default desktop and disables snap by default”, which I think is pretty nonsense and is rampant in the comments of this post.

But I’ve done my share of distro hopping and if someone wants to use something else for any reason or no reason… more power to them. I will make the counterpoint that no one has to care about snap specifically and if you just pretend it doesn’t exist then your life will be no different. And if history is any indicator, snap has about 2y left before they abandon it anyway.

I have to wonder if NLNet has some process for amending commitments made in light of new lessons learned. By a wide variety of metrics, the impact of the project has been increased beyond all imagination and ambition that people could have had in January. And the technology and quality of the project has improved way way faster as its accrued new contributors. This is really a case where the the right milestones to measure by have changed.

One might also hope that a call for help from contributors on these specific milestones might just get them back on track.

But speculation aside… yeah your description of their funding challenges is accurate.

If you’re serious about this, there’s a post up calling for sysops: https://lemmy.world/post/2769245

It’s somewhat of a commitment, rather than drop-in drop-out… but that’s what it takes to make a difference here. There are already several sharp and experienced database engineers working on the Lemmy world team. The problem is that the site is under repeated denial of service attack, and there isn’t one bad query to fix… each time one query gets addressed, the attackers move on to a new one.

While it’s always possible that someone has missed a silver bullet, it’s much more likely that a a series of ongoing independent mitigations and optimizations are needed to achieve a tipping point where lemmy is more or less protectable with some hidden dos-able bits rather than more or less trivially dos-able everywhere.

Tell me more about why I care that snap is setting up loop devices and not that docker is setting up virtual ethernet devices and nftables chains. System tools do system things, news at 11.

I say again, this impacts my life not at all and there is nothing easier to ignore than snap.

… those “pending update, close the app to avoid disruptions” popups are kind of disrupting.

I don’t exactly disagree that it’s slightly irritating but:

1. No one declares war on an operating system the way snap haters have over a “restart to update” message. It’s an irritation, but it’s not an irritation proportional to the response snap gets out of people.
2. Restarting to enable an update or complete an update is not something unique to snap. Except for a tiny number of very advanced live-patching systems like the one some kernel updaters use, every updater either nags you to shutdown to do the update, nags you to restart to finish the update, or doesn’t nag you and the update just doesn’t take effect till you restart (apt falls in this category and it’s not unambiguously better than nagging because you’re silently vulnerable when security patches are shipped until you restart). So again, this is just an extremely unremarkable thing that tons of updaters deal with similarly.

I do nothing.

• I use the Firefox snap. It takes like 800 extra milliseconds to start up on my 10y old laptop and it moves my profile dir. It otherwise impacts my life not at all and is just fine. If it ever bothers me, there PPAs, flatpak, or a dozen other ways to install Firefox that are all perfectly simple.
• I install other stuff from flatpaks or PPAs or using docker.

The angst around snap is inscrutable to me. There are 30 million easy ways to install software and they all work on Ubuntu. There is nothing in my life that’s easier to ignore than snap.

Lemmy world is under persistent denial of service attack in recent weeks: https://lemmy.world/post/2923697

The admins are aware and responding daily, the technical specifics of the attack keep changing as they close off one avenue of attack, the attackers switch to a slightly different approach in a game of cat and mouse.

There’s nothing you can do but wait, it will come back online… or use alts on other instances. Lemmy world has a competent admin team who is working hard to weather these attacks, but lemmy the software is not prepared for this kind of adversarial resource consumption so it’s a very hard job to both layer protections on top of lemmy and also to fix underlying issues so it’s natively more resilient.

It seems like the subscriber count only shows for whatever particular instance you are on, but adding together the subscriber count of the major instances exceeds 10,000.

Fwiw, the community sidebar in the community’s home instance shows a rough global subscriber count. So for this community that’s https://lemmy.world/c/sciencefiction which when opened in a browser rather than an app shows 9.81k… pretty close to 10k as you estimated.

One can pretty quickly verify that the community’s home instance shows more than the local subscribers by looking at small instances hosting bigger communities. For example, the instance sidebar at pathfinder.social shows less than 350 accounts on the server, but their biggest community at https://pathfinder.social/c/pf2general shows over 1.3k.subs.