• 1 Post
  • 289 Comments
Joined 2 years ago
cake
Cake day: June 7th, 2023

help-circle




  • the FDA actually has less than 10 employees. Everyone else is an outside contractor.

    You can search for FDA employees here (the US Government has a lot of transparency around employees). Also, it wouldn’t surprise me to find a lot of contractors in scientific and technical roles in the FDA, the FedGov uses a lot of contractors. I actually spent time, at a site, first as a contractor and then as a direct employee. The only thing that materially changed was whether or not my badge said “contractor” on it. I literally sat in the same seat and did the same job. Pay was pretty close, but the benefits were way better. The line between contractor and govie was pretty blury and more of a running joke than a real wall. Maybe that was just the departments I worked for, but I suspect my experience was more typical.







  • It depends on the type of backup:

    For a filesystem backup, restore one or more files to a secondary location. E.g. pick a few files out of the backup and try to restore them to a temporary folder. Then hash the original and restored files to verify integrity.
    For a full machine backup (e.g. VM backup), restore a copy of the machine to a test location. Spin up the test machine to verify that it can boot.
    For a database backup, restore a copy of the database to a test location (e.g. change the database name as part of the restore process), compare a few tables against the real database to verify integrity.

    Pretty much, it’s going to be some version of “Restore X to a test location and verify integrity”. You want to both prove that the backup can be restored and that the restored copy is actually intact.




  • I’m sure there are several out there. But, when I was starting out, I didn’t see one and just rolled my own. The process was general enough that I’ve been able to mostly just replace the SteamID of the game in the Dockerfile and have it work well for other games. It doesn’t do anything fancy like automatic updating; but, it works and doesn’t need anything special.


  • I see containers as having a couple of advantages:

    1. Separation of dependencies - while not as big of issue as it used to be, just knowing that you won’t end up with the requirements for one application conflicting with another is one less issue to worry about. Additionally, you can do anything you want to one container, without having an effect on another container. You don’t get stuck wanting to reboot or revert the system, but not wanting to break a different running service.
    2. Portability - Eventually, you are going to replace the OS of that VM (at least, you should). Moving a container to a new OS is dead simple. Re-installing an application on a new OS, moving data and configs can be anywhere from easy to a pain in the arse, depending on the software.
    3. Easier fall back - Have you ever upgraded an application and had everything go to shit? In my years working as a sysadmin, I lost way too many evenings to this sort of bullshit. And while VM snapshots should make reverting easy, sometimes it just didn’t work out that way. Containers force enough separation of applications that you can do just about anything to one container and not effect others.
    4. Less dependency on a single install - Have you ever had a system just get FUBAR, and after a few hours of digging the answer seems to be, just format the drive and start over? Maybe you tried some weird application out and the uninstall wasn’t really clean. By having all that crap happen in containers, you can isolate the damage. Nuke the container, nuke the image, and the base OS is still clean.
    5. Easier version testing - Want to try out upgrading to version 2 of an application, but worried that it may not be fully baked yet or the new configs may take a while to get right? Do it off in a separate container on a copy of the data. You can do this with VMs and snapshots; but, I find containers to be less overhead.

    That all said, if an application does not have an official container image, the added complexity of creating and maintaining your own image can be a significant downside. One of my use cases for containers is running game servers (e.g. Valheim). There isn’t an official image; so, I had to roll my own. The effort to set this up isn’t zero and, when trying to sort out an image for a new game, it does take me a while before I can start playing. And those images need to be updated when a new version of the game releases. Technically, you can update a running container in a lot of cases; but, I usually end up rebuilding it at some point anyway.

    I’d also note that, careful use of VMs and snapshots can replicate or mitigate most of the advantages I listed. I’ve done both (decade and a half as a sysadmin). But, part of that “careful use” usually meant spinning up a new VM for each application. Putting multiple applications on the same OS install was usually asking for trouble. Eventually, one of the applications would get borked and having the flexibility to just nuke the whole install saved a lot of time and effort. Going with containers removed the need to nuke the OS along with the application to get a similar effect.

    At the end of the day, though. It’s your box, you do what you are most comfortable with and want to support. If that’s a monolithic install, then go for it. While I, or other might find containers a better answer for us, maybe it isn’t for you.


  • I’m not going to defend everything the TSA does. And they do have a lot of problems. But, the lines at the checkpoint are the result of trade-offs in security. For all things security related, it’s about managing risk. You will never eliminate risk, so you need to pick and choose where to apply controls to reduce the worst risks and accept some risk in other areas.

    Think about the possible outcomes from terrorist attacks on airports. There are several possible scenarios:

    1. The attacker kills a few people in the airport using a direct weapon (gun, knife, etc.)
    2. The attacker kills a lot of people in a small area with a area weapon (bomb, gas, chemical, etc.)
    3. The attacker destroys an airplane in flight, killing everyone onboard.
    4. The attacker hijacks the airplane and takes everyone onboard for ransom.
    5. The attacker hijacks the airplane and uses it as a weapon, killing everyone on board and more people on the ground.

    We could probably come up with other cases, but I think this covers the bulk of it. So, let’s dive into managing these risks. What are the effects of such attacks, if successful?
    Looking at case 1, how many people are likely to be killed? Well, that depends on the police response time and the effectiveness of the attacker’s weapon. But, based on other mass casualty events, this probably falls into the range of 10-30 people. It could move outside this range, but this is pretty typical of such situations. To pick a number in the middle, will say they the expected loss for such an attack is around 20.
    With Case 2, again there is variability. But, it’s also something we have analogs for and may be able to put a range of casualties on. The Boston Marathon bombing in 2013 killed 6. The attack on Kabul Airport in 2021 during the US evacuation killed 182, though that also included multiple gunmen attacking after the explosion. Let’s put the loss rate around 50 for as single bomb, assuming a very packed area and a very effective bomb.
    For Case 3, the numbers are a bit easier to get a handle on. Typical airliners carry anywhere from 100-200 passengers. The 737 MAX 8-200 is designed for 200, while the Airbus A200-100 carries around 100 passengers. We’ll pin the loss rate here at 150, as attackers are likely to target larger aircraft for this sort of attack.
    Case 4 is basically Case 3, but with an optional loss of only money. For that reason, I’m going to remove this case, but wanted to mention it to avoid the “well akshuly” crowd, since this is a historic problem.
    That leaves Case 5. And it’s Case 4’s situation, plus some number of people on the ground. Certainly, not every such use of an airplane as a weapon will be as successful as the attack on 9/11. And that also involved multiple successful attacks. But, let’s assume that such attacks will hit populated buildings and cause significant damage. We’ll pin the expected loss at 200, This is 150 for the airplane and 50 on the ground, somewhat equivalent to Case 2 with a bomb in a crowded area.

    Ok, so we have expected losses, now lets talk about how often we expect such attacks to happen? And yes, this is a rough guess. But, since terrorists are unlikely to publish their plans, it’s the best we can do. We also face a difficulty in that these are still (thankfully) pretty rare events. And trying to extrapolate from a small set of data points is always a fraught exercise. So, fell free to quibble over these numbers, but I don’t think any numbers which fall into a reasonable range will change things much.
    Case 1 - This attack as a pretty low barrier to entry. If a person can be found to perform the attack, arming them isn’t terribly hard. So, we let’s assume we get 2 of these attacks a year. I don’t think we’re actually getting that, but out goal is just to get into the right ballpark.
    Case 2 - This attack takes a touch more work, bomb making isn’t that hard, but making a really effective one isn’t easy either. This type of attack does have the advantage that it doesn’t always require the attacker to die in the process. So, it might be easier to find someone willing to engage in such an attack. Let’s call this 1 per year.
    Case 3 - This also requires a bomb, but it may not need to be quite as big to be effective. Granted, modern aircraft can be amazingly resilient (see Aloha Flight 243). This attack also results in the attacker dying, so that can be a bit harder to source. So, lets say this happens once every other year, or 1/2 per year.
    Case 5 - So, no bomb this time, but you have to have an attacker not only willing to die in the process, but also go through enough flight training to fly the aircraft to it’s target. And you need the training itself. Plus, the attacker needs to get a weapon onto the aircraft. And since they need to overpower 100-200 people who might just take exception to the hijacking, you probably need multiple attackers willing to die in the attack. This is a pretty high bar to clear; so, let’s say that these attacks happen at a rate of 1 every 5 years.

    Ok, so let’s consider our Annualized Loss Expectancy (ALE) with what we have:

    Case Loss Expectancy Frequency ALE
    1 20 2 40
    2 50 1 50
    3 150 0.5 75
    5 200 0.2 40
    Total - - 205

    Alright, so lets start talking about controls we can use to mitigate these attacks. By raw numbers, the thing we should care about most is Case 3, as that has the highest ALE. So, what can we do about bombs on airplanes? Making them more resilient seems like a good start, but if we could do that, the military would have done it long ago. So, really the goal is to keep bombs out of airplanes. And that’s going to mean some sort of screening. We could just say “no carry on, period” and move the problem to the cargo hold. This would reduce the frequency of Case 3 and Case 5, as it would be much harder to get a bomb or weapon onto an airplane, without a bag to hide them in. But, travelers are not likely to give up all carried on bags. So, that really leaves us with searching bags and controlled checkpoints to do it. Of course, as has been noted, this would likely mean that Cases 1 and 2 become deadlier. Let’s put some numbers to it. Let’s say that checkpoints reduce the frequency of Cases 3 and 5 by a factor of 4 and increase the Loss Expectancy of Cases 1 and 2 by 1.5.

    Case Loss Expectancy Frequency ALE
    1 30 2 60
    2 75 1 75
    3 150 0.125 18.75
    5 200 0.05 10
    Total - - 163.75

    And we could push the numbers around for the effect of the checkpoints. And we could look at other controls or controls in combination. But, this is the sort of risk analysis which would need to be done to make such decisions. And, ideally, the numbers chosen would be done with a bit more care than my rectal extraction method. Can I say that anyone at the TSA/DHS/etc did this sort of analysis? No, but I suspect there has been some work on it. And it probably does lead to the conclusion that the expected loss is lower for airports with checkpoints than airports without. Though, that doesn’t excuse the TSA’s abysmal track record for tests done by the FBI.




  • My list of items I look for:

    • A docker image is available. Not some sort of make or build script which make gods know what changes to my system, even if the end result is a docker image. Just have a docker image out on Dockerhub or a Dockerfile as part of the project. A docker-compose.yaml file is a nice bonus.
    • Two factor auth. I understand this is hard, but if you are actually building something you want people to seriously use, it needs to be seriously secured. Bonus points for working with my YubiKey.
    • Good authentication logging. I may be an outlier on this one, but I actually look at the audit logs for my services. Having a log of authentication activity (successes and failures) is important to me. I use both fail2ban to block off IPs which get up to any fuckery and I manually blackhole entire ASNs when it seems they are sourcing a lot of attacks. Give me timestamps (in ISO8601 format, all other formats are wrong), IP address, username, success or failure (as a independent field, not buried in a message or other string) and any client information you can (e.g. User-Agent strings).
    • Good error logging. Look, I kinda suck, I’m gonna break stuff. When I do, it’s nice to have solid logging giving me an idea of what I broke and to provide a standardized error code to search on. It also means that, when I give up and post it as an issue to your github page, I can provide you with some useful context.

    As for that hackernews response, I’d categorically disagree with most of it.

    An app, self-contained, (essentially) a single file with minimal dependencies.

    Ya…no. Complex stuff is complex. And a lot of good stuff is complex. My main, self-hosted app is NextCloud. Trying to run that as some monolithic app would be brain-dead stupid. Just for the sake of maintainability, it is going to need to be a fairly sprawling list of files and folders. And it’s going to be dependent on some sort of web server software. And that is a very good place to NOT roll your own. Good web server software is hard, secure web server software is damn near impossible. Let the large projects (Apache/Nginx) handle that bit for you.

    Not something so complex that it requires docker.

    “Requires docker” may be a bit much. But, there is a reason people like to containerize stuff, it avoids a lot of problems. And supporting whatever random setup people have just sucks. I can understand just putting a project out as a container and telling people to fuck off with their magical snowflake setup. There is a reason flatpak is gaining popularity.
    Honestly, I see docker as a way to reduce complexity in my setup. I don’t have to worry about dependencies or having the right version of some library on my OS. I don’t worry about different apps needing different versions of the same library. I don’t need to maintain different virtual python environments for different apps. The containers “just work”. Hell, I regularly dockerize dedicated game servers just for my wife and I to play on.

    Not something that requires you to install a separate database.

    Oh goodie, let’s all create our own database formats and re-learn the lessons of the '90s about how hard databases actually are! No really, fuck off with that noise. If your app needs a small database backend, maybe try SQLite. But, some things just need a real database. And as with web servers, rolling your own is usually a bad plan.

    Not something that depends on redis and other external services.

    Again, sometimes you just need to have certain functionality and there is no point re-inventing the wheel every time. Breaking those discrete things out into other microservices can make sense. Sure, this means you are now beholden to everything that other service does; but, your app will never be an island. You are always going to be using libraries that other people wrote. Just try to avoid too much sprawl. Every dependency you spin up means your users are now maintaining an extra application. And you should probably build a bit of checking into your app to ensure that those dependencies are in sync. It really sucks to upgrade a service and have it fail, only to discover that one of it’s dependencies needed to be upgraded manually first, and now the whole thing is corrupt and needs to be restored from backup. Yes, users should read the release notes, they never do.
    The corollary here is to be careful about setting your users up for a supply chain attack. Every dependency or external library you add is one more place for your application to be attacked. And just because the actual vulnerability is in SomeCoolLib.js, it’s still your app getting hacked. You chose that library, you’re now beholden to everything it gets wrong.

    At the end of it all, I’d say the best app to write is the one you are interested in writing. The internet is littered with lots of good intentions and interesting starts. There is a lot less software which is actually feature complete and useful. If you lose interest, because you are so busy trying to please a whole bunch of idiots on the other side of the internet, you will never actually release anything. You do you, and fuck all the haters. If what you put out is interesting and useful, us users will show up and figure out how to use it. We’ll also bitch and moan, no matter how great your app is. It’s what users do. Do listen, feedback is useful. But, also remember that opinions are like assholes: everyone has one, and most of them stink.