So, did you hear about the SolarWinds hack?

[thewayne]

SolarWinds provides network security appliances to corporations. And governments. And universities. These organizations depend on companies like SolarWinds because security is VERY hard and it helps to get outside help. You can take my word for it to a degree - I worked in IT for over three decades, and while I did not specifically work in computer/network security, I had enough exposure to it to understand what a huge, tiresome, endless, and thankless job it was. It was truly neverending. You never had enough money or resources, and you were always outnumbered by the baddies.

Now what happens when a company like SolarWinds gets hacked?

I guess the first thing to talk about would be the nature of the hack. Let's suppose an update that SolarWinds pushes out to all of their customers was compromised. Now, ALL of your customers are vulnerable to being infiltrated by the people who compromised SolarWinds.

Perhaps it was a nation-state who did it.

Perhaps Russia.

Let's take a look at who makes up SolarWinds customer list. Here's a listing from SolarWinds' web site:



Notice any familiar names there?

I thought not.

So basically, Russian hackers, some of which are largely synonymous with the Russian government, compromised pretty much the entirety of the United States Government. And the U.S. Military. And effectively the entirety of the S&P 500.

Yeah, and our President has poo-poohed Russian election interference and sucked up to Putin for how long?

https://krebsonsecurity.com/2020/12/u-s-treasury-commerce-depts-hacked-through-solarwinds-compromise/

https://www.schneier.com/blog/archives/2020/12/another-massive-russian-hack-of-us-government-networks.html

Comments

[graydon]

Really do have to start over with the net; the existing design makes inappropriate assumptions about the participants.

There's likely a DARPA project. Trick is to get the new version to be open enough people will consider trusting it, even more than the massive logistical change.

[thewayne]

And that is the core problem.  The internet was designed to be open, not secure.  And bolting security on to it is night unto a never-ending challenge.  Back when it was created, as I understand it, machine to machine connections did not have passwords.

[dewline]

World-wide retrofitting, ad nauseum, right?

[thewayne]

There's a new privacy model designed by Apple and someone else, I don't recall, that makes DNS fairly securely private.  I should post about that.  But that just gives users more privacy, it doesn't increase security.  That's a network perimeter and server OS issue.  For most users, security isn't much of an issue for their devices because they don't have valuable information.  Occasionally someone falls victim to ransomware, but that's more of a drive-by attack, not targeted.  Worst a user is looking at is throwing away a hard drive for a couple hundred bucks and starting fresh with a loss of data.  Stealing corporate and gov't data is the big ticket target these days. And, of course, securing a toehold in corporate, government, university, and utility networks for potential future exploitation. It's a little funny.  Just a week or so ago the uni that I work for started sending emails that they were implementing two-factor authentication (2FA) for email and I think for online Microsoft product use, I don't recall if it was just for employees or across the board, because it was the "gold standard" for security.  I almost did a spit take.  I guess they didn't read about all of the bitcoin wallets that were stolen because these 15 and 16 y/o's social engineered Verizon and AT&T to change cell phone credentials and intercepted 2FA verification and committed some pretty big thefts. Yeah, gold standard.  If they want gold standard, they'd issue those Verisign random number generator security dongles to employees or give them the app for their phones.  I had one for World of Warcraft for a while, had to enter a random number when logging on.

[dewline]

Facebook is crying foul about that privacy model in full-page ads, I'd read this morning...?

[makovette]

https://support.mozilla.org/en-US/kb/firefox-dns-over-https

Best
Thing
Ever :D

[kraig]

You're probably thinking of DNS over HTTPS, which afaik Apple didn't spearhead, but Cloudflare did. Cloudflare is in many ways a shitty and reprehensible company, but this (and its followon, Oblivious DNS over HTTPS) is probably A Generally Good Thing, even if it does make the lives of folks like me - network defenders - difficult.

The thing with the Solarwinds customer list is that Solarwinds has lots of products and as far as I know, it was one specific product that was compromised - one they use for network and service management/alerting. And the indications are that it wasn't some worm that caused passive data exfil, compromises etc - it required some active intervention from the baddies. Still awful that it happened, but not every customer will have been using that particular product and not every one of those would necessarily be of interest to the black hats here.

[thewayne]

Yeah, it was Oblivious DNS that I was reading about.  What I had read - and I could be mistaken, it was not a deep dive, was Apple had co-developed it with Cloudflare. From reading more on SolarWinds, it was definitely one specific product they offer that was compromised - Orion.  And SolarWinds should really be taken to task: a default password of SolarWinds123?!  That is such ridiculous idiocy that it isn't worth mention.  The only good news from that hack is that a bunch of security companies plus Microsoft managed to identify the C&C server and get control of the domain and give it to Microsoft, so theoretically the hack is neutralized, but that assumes the baddies only have one C&C server and the deployed package won't look elsewhere if PhoneHome#1 is no longer available. I should do another post now that more information has come out.

[kraig]

Ah, that could be it. It seems like an Apple thing to do, for sure.

Default passwords are what they are; it doesn't matter how good or bad they are, they'll still be defaults. The crime is not forcing an admin to change that the very instant it is used. My new cablemodem did exactly that.

If you haven't read https://www.zdnet.com/article/partial-lists-of-organizations-infected-with-sunburst-malware-released-online/ then you'll likely find it interesting.

One of the more interesting pieces of news that's come out since I last commented/you replied is that, probably unsurprisingly, there was probably more than one group inside of Solarwinds.

[thewayne]

I did see that!  Quite interesting.  Quite a haul for the Russians, and zero repercussions from Trump.  Of course, in a month Biden is going to be in office.... The scary bit is the strongly- and widely-held belief of multiple backdoors installed, so they may have access to these networks for a few years!

[graydon]

When I was an undergrad, a typo in an init script could start your session on a different machine someone else was already using.

Designing in security really is a complete "start again", because it can't just be the network layer. (it has to be all the hardware implementing the network layer and pretty soon that's all the hardware.)

Worth doing, but no one's even close to putting up the money.

[thewayne]

Yup.  I was digging through some old hardware in the library that I work at and came across some old networking gear.  Thought they might be switches.  Nope.  They were hubs.  I laughed out loud at that.  Might as well toss 'em in the trash.  Then I thought if I could put them in promiscuous mode, I might be able to use 'em at home to snoop on traffic if I ever started working with pfSense or WRT or something. A friend and I made the mistake of going to a "business college" (trade school), complete waste of time and money.  Fortunately it wasn't a huge cost sink at that time.  We pulled an interesting hack which would probably get us arrested these days.  HP3000 computer.  People would leave their sessions signed on and walk away.  So we wrote a password stealer.  It would clear the screen, look like the login screen, asking for the user name and password, then simulate a glitch or login failure or something with a bogus error message, clear the screen and do an actual logoff, returning to the real login screen.  In the background, it wrote the user name/password to a public text file. Interestingly, that school had the first conviction in the State of Arizona for a computer crime.  So they buy this (at the time) new and lovely HP3000.  And very quickly the students start complaining about the lack of disk space (not because of our little prank).  The only other big user with serious expertise of this system is Kaibab Forestry Industries up in Flagstaff.  So one day when the IT director is off, they arrange for Kaibab's head tech to come down and hack their system.  And he finds this huge program consuming vast gobs of space in the school's system.  And it's a forestry management system.  And it looks strangely familiar to the tech. It's Kaibab's internally developed system, with the identifying information being scraped off. Somehow the school's IT director had gotten ahold of the source code and was in the process of modifying it to make it look like his product so that he could resell it. He was charged with some counts of computer theft and fraud and who knows what else, to which he pled guilty.

[kathmandu]

Yow, that's chutzpah.

[kraig]

If you want to snoop your own traffic, there's cleaner ways to do it than hubs, which aren't really compatible with the environment you'd be likely to have at home in 2020. I don't know what your home internet is like, but here 150 meg down is pretty standard for cable connections now (I just upgraded to 500M). Your hub can likely do 100M max, and if you've got multiple devices connected to it, your speed is going to be severely affected as network traffic floods to each port.

Of course, something you already have can be better than a better solution you need to acquire, but for a bit under $100 Canadian I just ordered a Netgear 8 port switch with port mirroring and VLAN capabilities - model GS108PE-300NAS - for exactly that purpose, snooping my own traffic. That model has 4 POE ports because I wanted to play with a VOIP phone, but you could likely get the 5 port equivalent sans POE for about half that price.

[thewayne]

Ah, no more taking hubs and putting them in promiscuous mode, eh?  Yeah, it's been a lot of years since I had my CCNA.  You're absolutely right, those old hubs are probably capped at 10 mbs and our connection at home is a fiber at 50/50.  Considering we live literally on top of a mountain in the middle of a national forest, that's pretty darn spiffy!

[kraig]

I think you're thinking of putting the network card on your monitoring server into promisc, hubs by design already flood traffic to every port, then your NIC will ignore any unicast traffic not directed to its address without PROMISC.

[thewayne]

You're right.  With the hub automatically flooding everything across all ports, it would be the card in the monitoring device that would be in promiscuous.  Probably just end up with a RasbPi for monitoring if I ever get around to sniffing my own traffic.

[kraig]

Zeek (zeek.org) is fun to put on mirror ports. We made extensive use of this BC, but even with many people WFH and a lot not VPN'ing, it's still useful.

[thewayne]

I will look in to that, thanks!

[devilc]

ARRRGH!

[thewayne]

Now, don't hold back, K!  Tell us what you're really feeling! Yeah.  You'd think someone would have compared checksums or something to verify the patch before they pushed it out to make sure it was authentic.  Wanna bet they're going to do that from now on, as will a number of other companies?

[silveradept]

I'm not sure how you would prevent that kind of hack being that destructive, apart from there being more companies and such doing this work, but it sounds like that's the sort of thing where if you spread that effort and money across multiple companies, you get less ability in defending from all of the attacks and all of the actors.