The call is coming from inside the house, but from which device?
October 12, 2022 7:18 AM Subscribe
I got two emails last month and two more this month from abuse@verizon.com warning me about "unauthorized traffic originating from my IP". I need to track down any compromised device/devices and fix this. However, nobody at Verizon tech support or Verizon security has any record of the emails being sent to me so they're saying the emails are fake. But they look and sound very real! More details inside, including one of the full emails.
So first of all, I believe the issue is real. I strongly suspect we have genuine unauthorized activity occurring. I have a 14 yr old kid who is a wannabe programmer and I am 99% certain he has Done Some Shit which has created some kind of security hole.
Backstory: This kid turned my old laptop into a server that he can access from his friends' homes and from school last year. He turned an old Kindle into an Android device two years ago by not only wiping the drive but also mucking about with the wiring or chip or something? And I don't even know what all he got up to with my old android phone which he was obsessed with until 6 months ago. It's shenanigans all the way. Crucially, though, he is not fully competent at these shenanigans yet. Last week he got into trouble at school for replacing the Windows wallpaper for the entire school district with a rickroll... unintentionally. He was trying to replace the wallpaper on just one friend's computer but he did the whole fucking school district by accident.
So I have no trouble imagining that it was an unintentional security hole caused by any one of his million projects. He is honest and earnest and sweet so I believe him when he says he's not trying to access other people's devices at all, so this unauthorized activity is coming from elsewhere. But he is also 14, he doesn't know shit but he thinks he knows it all, he says "trust me I'll figure it out this weekend, I've got this" but LOL no I am finding competent help for this, and he has lost access to all his devices for the week.
Current questions:
1. I've copied and pasted the latest email below. It looks real, right?? This is the fourth one so far. I am personally only mildly conversant in tech things, but even I can tell there is something wrong here. Verizon tech support has been of no use. The first two hours of my call I was shunted from one clueless person to another - so clueless that even I knew more about this than they did. Finally in the last 45 minutes I reached the security team who seemed to know what they were talking about, but in the end they too concluded after internal conferences with their internal teams that they have no idea where these emails are coming from, they have no record of having sent this to me, so it must be fake. They agree that the originating address is real but their theory is that it's spoofed. How credible is this???
2. Assuming for the moment that there is really unauthorized activity happening, how can I track down the compromised device from which the activity is originating?? Our household contains the following internet connected devices: an ipad, two android phones, two macbooks, one gaming laptop that's half windows and half linux, an old windows laptop that's now on Linux Mint and rigged to act like a server, a TV, an alexa device, and a wifi enabled AC/heating controller. I have no idea how to find out which port number correlates with which device! How can I find out?
3. IS there a "Verizon Global IP Abuse" team and if so how can I reach them? If not how can I find some competent help from Verizon? Which department should I call and what magic words should I use?
Thank you! Copy-pasted email follows:
------------------------------------------
Dear Verizon Online Customer,
We have received 17 prior notifications regarding the issue below.
On 10-11-22, your account was reported to have been used in an attempt to gain unauthorized access to another system, or to transmit malicious traffic to another Internet user.
It is possible your system may have been infected by a virus or a botnet that is causing this action.
Report and/or Logs:
Oct 11 22:37:50 servername sshd[12725]: Failed password for invalid user test8 from [my ip] port [NumberA] ssh2
Oct 11 22:38:42 servername sshd[12773]: Invalid user obc from [my ip] port [NumberB]
Oct 11 22:38:42 servername sshd[12773]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my ip]
Oct 11 22:38:45 servername sshd[12773]: Failed password for invalid user obc from [my ip] port [NumberB] ssh2
Oct 11 22:39:36 servername sshd[12921]: Invalid user direction from [my ip] port [NumberC]
Oct 11 22:39:36 servername sshd[12921]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my ip]
Oct 11 22:39:38 servername sshd[12921]: Failed password for invalid user direction from [my ip] port [NumberC] ssh2
Oct 11 22:40:26 servername sshd[12975]: Invalid user radio from [my ip] port [NumberD]
Oct 11 22:40:26 servername sshd[12975]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my ip]
Oct 11 22:40:28 servername sshd[12975]: Failed password for invalid user radio from [my ip] port [NumberD] ssh2
Please immediately ensure your anti-virus software is properly updated, and then run a full-system virus scan on your computer(s). Follow the removal instructions for any viruses found, as indicated by your anti-virus software.
Additional information and removal instructions about viruses and your anti-virus software may be found on the website of your anti-virus software manufacturer.
It is difficult to verify the presence of an exact virus which has infected a computer without a full system scan with up-to-date anti-virus software. Therefore, you may wish to contact the Technical Support Department of your anti-virus software manufacturer, should you need assistance with this process, or have any questions or concerns about a possible virus infection.
Please be aware of the importance in taking immediate actions to stop further virus related activity. If you are unable to take immediate action, it would be advisable to remove the computer(s) which may be infected from any networks and the Internet connection until it has been properly cleaned. For DSL customers, this may be easily done by unplugging the network cable that connects the computer to the DSL modem, or the telephone line that connects the DSL modem to the telephone jack.
Verizon Policy:
If you do not take appropriate action to resolve this issue, we will be forced to take further action, which could include the suspension of your service until the issue is resolved, in order to ensure the safety of our network, and the safety of other Internet users.
Please carefully review these agreements, which can be viewed at:
http://www.verizon.com/about/terms/
Any future violation will result in further action being taken, up to,
and including, the termination of your service.
So first of all, I believe the issue is real. I strongly suspect we have genuine unauthorized activity occurring. I have a 14 yr old kid who is a wannabe programmer and I am 99% certain he has Done Some Shit which has created some kind of security hole.
Backstory: This kid turned my old laptop into a server that he can access from his friends' homes and from school last year. He turned an old Kindle into an Android device two years ago by not only wiping the drive but also mucking about with the wiring or chip or something? And I don't even know what all he got up to with my old android phone which he was obsessed with until 6 months ago. It's shenanigans all the way. Crucially, though, he is not fully competent at these shenanigans yet. Last week he got into trouble at school for replacing the Windows wallpaper for the entire school district with a rickroll... unintentionally. He was trying to replace the wallpaper on just one friend's computer but he did the whole fucking school district by accident.
So I have no trouble imagining that it was an unintentional security hole caused by any one of his million projects. He is honest and earnest and sweet so I believe him when he says he's not trying to access other people's devices at all, so this unauthorized activity is coming from elsewhere. But he is also 14, he doesn't know shit but he thinks he knows it all, he says "trust me I'll figure it out this weekend, I've got this" but LOL no I am finding competent help for this, and he has lost access to all his devices for the week.
Current questions:
1. I've copied and pasted the latest email below. It looks real, right?? This is the fourth one so far. I am personally only mildly conversant in tech things, but even I can tell there is something wrong here. Verizon tech support has been of no use. The first two hours of my call I was shunted from one clueless person to another - so clueless that even I knew more about this than they did. Finally in the last 45 minutes I reached the security team who seemed to know what they were talking about, but in the end they too concluded after internal conferences with their internal teams that they have no idea where these emails are coming from, they have no record of having sent this to me, so it must be fake. They agree that the originating address is real but their theory is that it's spoofed. How credible is this???
2. Assuming for the moment that there is really unauthorized activity happening, how can I track down the compromised device from which the activity is originating?? Our household contains the following internet connected devices: an ipad, two android phones, two macbooks, one gaming laptop that's half windows and half linux, an old windows laptop that's now on Linux Mint and rigged to act like a server, a TV, an alexa device, and a wifi enabled AC/heating controller. I have no idea how to find out which port number correlates with which device! How can I find out?
3. IS there a "Verizon Global IP Abuse" team and if so how can I reach them? If not how can I find some competent help from Verizon? Which department should I call and what magic words should I use?
Thank you! Copy-pasted email follows:
------------------------------------------
Dear Verizon Online Customer,
We have received 17 prior notifications regarding the issue below.
On 10-11-22, your account was reported to have been used in an attempt to gain unauthorized access to another system, or to transmit malicious traffic to another Internet user.
It is possible your system may have been infected by a virus or a botnet that is causing this action.
Report and/or Logs:
Oct 11 22:37:50 servername sshd[12725]: Failed password for invalid user test8 from [my ip] port [NumberA] ssh2
Oct 11 22:38:42 servername sshd[12773]: Invalid user obc from [my ip] port [NumberB]
Oct 11 22:38:42 servername sshd[12773]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my ip]
Oct 11 22:38:45 servername sshd[12773]: Failed password for invalid user obc from [my ip] port [NumberB] ssh2
Oct 11 22:39:36 servername sshd[12921]: Invalid user direction from [my ip] port [NumberC]
Oct 11 22:39:36 servername sshd[12921]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my ip]
Oct 11 22:39:38 servername sshd[12921]: Failed password for invalid user direction from [my ip] port [NumberC] ssh2
Oct 11 22:40:26 servername sshd[12975]: Invalid user radio from [my ip] port [NumberD]
Oct 11 22:40:26 servername sshd[12975]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my ip]
Oct 11 22:40:28 servername sshd[12975]: Failed password for invalid user radio from [my ip] port [NumberD] ssh2
Please immediately ensure your anti-virus software is properly updated, and then run a full-system virus scan on your computer(s). Follow the removal instructions for any viruses found, as indicated by your anti-virus software.
Additional information and removal instructions about viruses and your anti-virus software may be found on the website of your anti-virus software manufacturer.
It is difficult to verify the presence of an exact virus which has infected a computer without a full system scan with up-to-date anti-virus software. Therefore, you may wish to contact the Technical Support Department of your anti-virus software manufacturer, should you need assistance with this process, or have any questions or concerns about a possible virus infection.
Please be aware of the importance in taking immediate actions to stop further virus related activity. If you are unable to take immediate action, it would be advisable to remove the computer(s) which may be infected from any networks and the Internet connection until it has been properly cleaned. For DSL customers, this may be easily done by unplugging the network cable that connects the computer to the DSL modem, or the telephone line that connects the DSL modem to the telephone jack.
Verizon Policy:
If you do not take appropriate action to resolve this issue, we will be forced to take further action, which could include the suspension of your service until the issue is resolved, in order to ensure the safety of our network, and the safety of other Internet users.
Please carefully review these agreements, which can be viewed at:
http://www.verizon.com/about/terms/
Any future violation will result in further action being taken, up to,
and including, the termination of your service.
Two thoughts:
1. Who the fuck hyphenates antivirus? Literally no one.
2. Since literally no one hyphenates antivirus, I bet I can quote google search a sentence from this and find other people who have received the exact same email.
And yes, the link Azrael has above that I see in preview, and a 6 year old reddit post.
posted by phunniemee at 7:26 AM on October 12, 2022 [1 favorite]
1. Who the fuck hyphenates antivirus? Literally no one.
2. Since literally no one hyphenates antivirus, I bet I can quote google search a sentence from this and find other people who have received the exact same email.
And yes, the link Azrael has above that I see in preview, and a 6 year old reddit post.
posted by phunniemee at 7:26 AM on October 12, 2022 [1 favorite]
I'm probably not much more of an expert than you are, but this also jumped out at me: "a wifi enabled AC/heating controller." These IoT devices have notoriously shitty security, and it seems possible that an attacker has found that device and is using it for nefarious purposes.
I respect the kid's hacker ethos, but online security is incredibly hard. And I know you'll probably have a hard time convincing him he doesn't know everything (I remember being a teenager). Maybe tell him to start reading Bruce Schneier's blog to get a better appreciation of how hard it is.
posted by adamrice at 7:29 AM on October 12, 2022
I respect the kid's hacker ethos, but online security is incredibly hard. And I know you'll probably have a hard time convincing him he doesn't know everything (I remember being a teenager). Maybe tell him to start reading Bruce Schneier's blog to get a better appreciation of how hard it is.
posted by adamrice at 7:29 AM on October 12, 2022
Those are weird errors. I was fully expecting the message about unauthorized traffic to be something about the laptop server, because sometimes internet providers crack down on that so that you can't run a business on the non-business plan, but all those lines are about ssh which is a very particular type of connection. Verizon wouldn't have logs for that on a random devices unless it was the modem they gave you, but in that case they just wouldn't have a ssh server running on it.
I think it's a fake email just based on that.
One of numbers A/B/C/D is 22, right? (That's the standard ssh port) Are any of them 80 or 443? Those are typical web server ports and generally wouldn't be used for ssh, buuut because they are common I wouldn't be surprised to see those numbers pop up in something fake.
posted by Nonsteroidal Anti-Inflammatory Drug at 7:42 AM on October 12, 2022 [1 favorite]
I think it's a fake email just based on that.
One of numbers A/B/C/D is 22, right? (That's the standard ssh port) Are any of them 80 or 443? Those are typical web server ports and generally wouldn't be used for ssh, buuut because they are common I wouldn't be surprised to see those numbers pop up in something fake.
posted by Nonsteroidal Anti-Inflammatory Drug at 7:42 AM on October 12, 2022 [1 favorite]
Response by poster: AzraelBrown that is SO HELPFUL OMG it's exactly the same shit! And yes it's related to the server, I KNEW IT. I've unplugged the server immediately. However I don't know what to do next. Planning to show my son that blog post and hope he can parse it and figure out what to do. But how do *I* know he's done it right? What are some steps that I can just check with him verbally that he did X and Y and Z? If anyone can link me to a guide or something that would be very helpful. (PS: I would also like to pull up that list of all attempted accesses like this guy does on his blog.)
Nonsteroidal Anti-Inflammatory Drug , no, all the ssh port numbers are five digit numbers. None of them are "22" or "80" or "443". I pulled up a list of ports in use via my router and none of these numbers even match! I have no fucking idea what is going on and it's likely I am doing all of it wrong. So... now what? PS: I use my own modem, Verizon did not give me it.
Thank you so much, I knew I could count on you all!
posted by MiraK at 7:50 AM on October 12, 2022
Nonsteroidal Anti-Inflammatory Drug , no, all the ssh port numbers are five digit numbers. None of them are "22" or "80" or "443". I pulled up a list of ports in use via my router and none of these numbers even match! I have no fucking idea what is going on and it's likely I am doing all of it wrong. So... now what? PS: I use my own modem, Verizon did not give me it.
Thank you so much, I knew I could count on you all!
posted by MiraK at 7:50 AM on October 12, 2022
Best answer: This kid turned my old laptop into a server that he can access from his friends' homes and from school last year.
A server made accessible to the wider Internet by an inexperienced kid is not super likely to be well secured, and is quite plausibly vulnerable to being exploited by malicious randoms for their own malicious ends.
That said,
1. I've copied and pasted the latest email below. It looks real, right??
Looks fake to me. Generally if an outfit with which you have a business relationship is going to send you a nastygram, they'll address you by name. Also, that snippet of server log looks like the kind of thing that's designed to cause fear and trembling in people who don't read that kind of log on the regular; if I were trying to set up a scam and I was looking for a way to make my opener look all technical and shit, that's exactly the kind of thing I'd stick in the middle of it.
If Verizon's security team has no record of sending you any such thing, get them to say so in writing. That way, in the exceedingly unlikely event that Verizon does end up suspending your service, you've got an email record that can demonstrate that (a) you raised this with them and (b) they told you the alerts were most likely fake.
If you'd care to forward me one of those emails as an attachment (procedures for doing this will vary by client but are easy to look up) I'd be happy to cast an eye over its headers and tell you whether or not it actually originated inside Verizon. Chances are it won't have. Email address is in profile.
You can also reach me as flabdablet on Keybase if you and/or your kid would like me to do a bit of penetration testing against his server and follow up with you over something less horribly insecure than email. Yes I'm an Internet random you don't know from a bar of soap, but I'm also a MeFi member of long standing who values his handle's reputation and I'd be happy to offer whatever hands-on help I could at no charge.
posted by flabdablet at 7:58 AM on October 12, 2022 [25 favorites]
A server made accessible to the wider Internet by an inexperienced kid is not super likely to be well secured, and is quite plausibly vulnerable to being exploited by malicious randoms for their own malicious ends.
That said,
1. I've copied and pasted the latest email below. It looks real, right??
Looks fake to me. Generally if an outfit with which you have a business relationship is going to send you a nastygram, they'll address you by name. Also, that snippet of server log looks like the kind of thing that's designed to cause fear and trembling in people who don't read that kind of log on the regular; if I were trying to set up a scam and I was looking for a way to make my opener look all technical and shit, that's exactly the kind of thing I'd stick in the middle of it.
If Verizon's security team has no record of sending you any such thing, get them to say so in writing. That way, in the exceedingly unlikely event that Verizon does end up suspending your service, you've got an email record that can demonstrate that (a) you raised this with them and (b) they told you the alerts were most likely fake.
If you'd care to forward me one of those emails as an attachment (procedures for doing this will vary by client but are easy to look up) I'd be happy to cast an eye over its headers and tell you whether or not it actually originated inside Verizon. Chances are it won't have. Email address is in profile.
You can also reach me as flabdablet on Keybase if you and/or your kid would like me to do a bit of penetration testing against his server and follow up with you over something less horribly insecure than email. Yes I'm an Internet random you don't know from a bar of soap, but I'm also a MeFi member of long standing who values his handle's reputation and I'd be happy to offer whatever hands-on help I could at no charge.
posted by flabdablet at 7:58 AM on October 12, 2022 [25 favorites]
Response by poster: I'm copy-pasting the full log from another email a couple of days ago, just for more info, in case it helps:
The following intrusion attempts were detected:
Oct 10 20:07:53 dopracenabicykli sshd[25888]: Invalid user trouble from [my ip] port [5DigitNumberE]
Oct 10 20:07:53 dopracenabicykli sshd[25888]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=pool-[my ip].[mycityname].fios.verizon.net
Oct 10 20:07:55 dopracenabicykli sshd[25888]: Failed password for invalid user trouble from [my ip] port [5DigitNumberE] ssh2
Oct 10 20:07:55 dopracenabicykli sshd[25888]: Received disconnect from [my ip] port [5DigitNumberE]:11: Bye Bye [preauth]
Oct 10 20:07:55 dopracenabicykli sshd[25888]: Disconnected from [my ip] port [5DigitNumberE] [preauth]
Oct 10 20:11:44 dopracenabicykli sshd[25977]: Invalid user edt from [my ip] port [5DigitNumberF]
Oct 10 20:11:44 dopracenabicykli sshd[25977]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=pool-[my ip].[mycityname].fios.verizon.net
Thank you so much, flabdablet, I'll get in touch!
posted by MiraK at 8:02 AM on October 12, 2022
The following intrusion attempts were detected:
Oct 10 20:07:53 dopracenabicykli sshd[25888]: Invalid user trouble from [my ip] port [5DigitNumberE]
Oct 10 20:07:53 dopracenabicykli sshd[25888]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=pool-[my ip].[mycityname].fios.verizon.net
Oct 10 20:07:55 dopracenabicykli sshd[25888]: Failed password for invalid user trouble from [my ip] port [5DigitNumberE] ssh2
Oct 10 20:07:55 dopracenabicykli sshd[25888]: Received disconnect from [my ip] port [5DigitNumberE]:11: Bye Bye [preauth]
Oct 10 20:07:55 dopracenabicykli sshd[25888]: Disconnected from [my ip] port [5DigitNumberE] [preauth]
Oct 10 20:11:44 dopracenabicykli sshd[25977]: Invalid user edt from [my ip] port [5DigitNumberF]
Oct 10 20:11:44 dopracenabicykli sshd[25977]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=pool-[my ip].[mycityname].fios.verizon.net
Thank you so much, flabdablet, I'll get in touch!
posted by MiraK at 8:02 AM on October 12, 2022
OMG it's exactly the same shit! And yes it's related to the server, I KNEW IT.
If that were a genuine Verizon nastygram, I would expect to find many more than two results from a Google search on a text snippet from it (DuckDuckGo returns no results at all).
Also, the fact that both the results that Google does indeed return have got random chunks of sshd log entries embedded in them, as opposed to any of the thousands of other kinds of malicious traffic reports that could plausibly trigger an ISP nastygram, says "low-volume scammer" to me.
Those sshd log entries, by the way, are absolutely run of the mill reading for anybody who has ever exposed a ssh server to the wider Internet. The school server I used to admin would generate literally thousands of those per day in response to unsuccessful automated probes coming from all over the Net. They're basically background noise and certainly not something I could see a server admin getting het up about enough to start raising a stink with Verizon over.
posted by flabdablet at 8:07 AM on October 12, 2022 [4 favorites]
If that were a genuine Verizon nastygram, I would expect to find many more than two results from a Google search on a text snippet from it (DuckDuckGo returns no results at all).
Also, the fact that both the results that Google does indeed return have got random chunks of sshd log entries embedded in them, as opposed to any of the thousands of other kinds of malicious traffic reports that could plausibly trigger an ISP nastygram, says "low-volume scammer" to me.
Those sshd log entries, by the way, are absolutely run of the mill reading for anybody who has ever exposed a ssh server to the wider Internet. The school server I used to admin would generate literally thousands of those per day in response to unsuccessful automated probes coming from all over the Net. They're basically background noise and certainly not something I could see a server admin getting het up about enough to start raising a stink with Verizon over.
posted by flabdablet at 8:07 AM on October 12, 2022 [4 favorites]
Best answer: Verizon wouldn't have logs for that on a random devices unless it was the modem they gave you, but in that case they just wouldn't have a ssh server running on it.
The logs are presumably from remote targeted systems, submitted with the abuse reports.
I agree that some of the wording of the email is odd—but I'm not sure what the point would be without some kind of phishing link included that the recipient would be likely to click (i.e., not one to the ISP terms of service), and given the old posts I'm going with genuine (although fladbablet et al's skepticism could also be correct).
The jakob.space analysis and the links in it are informative. I might turn this into a breach analysis project for your son: let him start with the links above and see how far he can get; including through networking in various infosec hobby/career communities if necessary. @notshenetworks Discord might be a good place to meet younger people starting out in the field who would react to this like helping out with a particularly cool science fair project.
On the more somber side, the oldest cautionary tale for your son (and some of the most famous internet lore) is the original Internet Worm — also an exploratory project gone wrong.
If he doesn't know what happened to Aaron Swartz, that might be a wake up call in terms of how even the most elite and tech-oriented educational institutions react to well-intentioned subversive dabbling.
Some of the other well-known hacking chronicles also teach valuable lessons about hazards of exploration and what incident responders have to deal with—but tend to glamorize for-profit underground hacking a but more, so maybe not those right now.
posted by snuffleupagus at 8:11 AM on October 12, 2022 [2 favorites]
The logs are presumably from remote targeted systems, submitted with the abuse reports.
I agree that some of the wording of the email is odd—but I'm not sure what the point would be without some kind of phishing link included that the recipient would be likely to click (i.e., not one to the ISP terms of service), and given the old posts I'm going with genuine (although fladbablet et al's skepticism could also be correct).
The jakob.space analysis and the links in it are informative. I might turn this into a breach analysis project for your son: let him start with the links above and see how far he can get; including through networking in various infosec hobby/career communities if necessary. @notshenetworks Discord might be a good place to meet younger people starting out in the field who would react to this like helping out with a particularly cool science fair project.
On the more somber side, the oldest cautionary tale for your son (and some of the most famous internet lore) is the original Internet Worm — also an exploratory project gone wrong.
If he doesn't know what happened to Aaron Swartz, that might be a wake up call in terms of how even the most elite and tech-oriented educational institutions react to well-intentioned subversive dabbling.
Some of the other well-known hacking chronicles also teach valuable lessons about hazards of exploration and what incident responders have to deal with—but tend to glamorize for-profit underground hacking a but more, so maybe not those right now.
posted by snuffleupagus at 8:11 AM on October 12, 2022 [2 favorites]
Not only would Verizon not have those logs, so said email is basically guaranteed to be fake, they aren't going to cut off your service for running a server.
Point your kid to resources regarding setting up SSH with key-based authentication and disabling password authentication and it won't be a terrible security risk, either. If they want to access other services remotely they can use SSH tunnels very safely.
posted by wierdo at 8:13 AM on October 12, 2022 [2 favorites]
Point your kid to resources regarding setting up SSH with key-based authentication and disabling password authentication and it won't be a terrible security risk, either. If they want to access other services remotely they can use SSH tunnels very safely.
posted by wierdo at 8:13 AM on October 12, 2022 [2 favorites]
It's possible you have something in your house that has been compromised and is running a scanner. I used to run honeypots and saw this sort of stuff constantly. The scanner is looking for devices using common passwords. If it gains access to one, it'll drop a copy of the malware and then scan from there, ad infinitum. All of the compromised devices are running a bit of script, which phones home (usually to an IRC server), awaiting instructions from whoever is assembling the herd of bots.
The bigger the herd, the better because the operator can use them to DDoS for fun and/or profit. The groups competing for control of devices - badly configured routers and cable modems, in the main - will sometimes even check for competitive control of the device and kick out/delete the incumbent before dropping their code.
posted by jquinby at 8:15 AM on October 12, 2022 [5 favorites]
The bigger the herd, the better because the operator can use them to DDoS for fun and/or profit. The groups competing for control of devices - badly configured routers and cable modems, in the main - will sometimes even check for competitive control of the device and kick out/delete the incumbent before dropping their code.
posted by jquinby at 8:15 AM on October 12, 2022 [5 favorites]
Also, googling for
posted by snuffleupagus at 8:18 AM on October 12, 2022 [2 favorites]
dopracenabicykli finds "@dopracenabicykli · Environmental Conservation Organization" [https://www.facebook.com/dopracenabicykli/about/?ref=page_internal] — if that is a unique name (any Slovakian speakers lurking?) might not be impossible to contact an admin there and ask if they've submitted abuse reports.posted by snuffleupagus at 8:18 AM on October 12, 2022 [2 favorites]
Residential FiOS service doesn't allow servers in the T&C, if I recall, so that's not automatically a red flag to me (a current FiOS customer). But the part about "we tried to creach you 17 times" smells like BS: they're your provider, and they know where to find you.
That said, those log entries do look like the traces of outbound automated scanning from some old software that a kid installed and someone else's automated scanner found and hijacked. Whether it's legit or not, I would assume that the old laptop is already rooted.
If you know how, get the laptop off the Internet and patch it up. If you don't, just get it off the Internet now. And close up whatever hole he opened in your home firewall ASAP!
Your son seems to have a keen interest in cybersecurity, but that's very much a "playing with matches" thing here in terms of data loss & actual legal liability until he learns some stuff. (Source: I am a nerd, have edited SANS course books, and don't open any incoming ports on my FiOS router.)
posted by wenestvedt at 8:26 AM on October 12, 2022 [4 favorites]
That said, those log entries do look like the traces of outbound automated scanning from some old software that a kid installed and someone else's automated scanner found and hijacked. Whether it's legit or not, I would assume that the old laptop is already rooted.
If you know how, get the laptop off the Internet and patch it up. If you don't, just get it off the Internet now. And close up whatever hole he opened in your home firewall ASAP!
Your son seems to have a keen interest in cybersecurity, but that's very much a "playing with matches" thing here in terms of data loss & actual legal liability until he learns some stuff. (Source: I am a nerd, have edited SANS course books, and don't open any incoming ports on my FiOS router.)
posted by wenestvedt at 8:26 AM on October 12, 2022 [4 favorites]
Oh. The other thing I would probably do in a situation like this is insert a hardware firewall router under my control. A PC with two network cards running Untangle or PFsense would do. Untangle might even be able to detect the activity (although I wouldn't rely on it).
posted by snuffleupagus at 8:28 AM on October 12, 2022
posted by snuffleupagus at 8:28 AM on October 12, 2022
(Excellent tags, BTW.)
posted by wenestvedt at 8:33 AM on October 12, 2022 [1 favorite]
posted by wenestvedt at 8:33 AM on October 12, 2022 [1 favorite]
Whether it's fake or not, your son needs guidance. A poorly secured server can and will be used by ill-intentioned hackers to spread viruses and do crime. Call a local college and see if they have a student or staff member who will take a security job to secure your home network, see what's accessible on your router, and who's using it, and maybe mentor your kid. Your son might enjoy computing classes at a community college, and can probably start some coding classes. If there's a computer user's meetup group or similar, that would be a resource.
I'd make the next ask.me a question about best sites for computing ethics. When I worked at a University IT Dept., students got expensive fines for stuff. And doing stuff that might allow viruses to spread causes actual harm.
posted by theora55 at 8:34 AM on October 12, 2022 [3 favorites]
I'd make the next ask.me a question about best sites for computing ethics. When I worked at a University IT Dept., students got expensive fines for stuff. And doing stuff that might allow viruses to spread causes actual harm.
posted by theora55 at 8:34 AM on October 12, 2022 [3 favorites]
Best answer: Maybe try the second item on this page, titled "Cybersecurity": https://cyber.org/find-curricula?f%5B0%5D=type%3Acourse
posted by wenestvedt at 8:36 AM on October 12, 2022 [1 favorite]
posted by wenestvedt at 8:36 AM on October 12, 2022 [1 favorite]
SSH password auth gets a bad rap and obviously key-based auth is both more convenient and more secure (assuming half-decent key management practices are in place), but ssh does rate-limiting on authentication attempts and as long as its password is randomly generated and has at least 45 bits of entropy (nine characters, each of which could be a lowercase letter or a digit, is enough for that) then it's not going to let anybody in before fail2ban can shut them off at the firewall.
A bit of security by obscurity actually works pretty well for ssh too. The aforementioned school server used to get probed thousands of times per day because the school district's IT department, which was in charge of the district VPN's Internet firewall, insisted that if I was going to run a ssh server in the school then I had to expose it in the standard way on port 22. My own home servers, which have exposed a couple of ssh servers on completely non-standard high-numbered ports for the last ten years, have yet to log any ssh probe traffic.
But again, none of this is directly relevant to MiraK or KiddoK, because none of the failed ssh logins shown in any of the Terrible Emails are coming from their own server's logs; they're allegedly coming from somebody else's logs and allegedly identify MiraK's public IP address as the source of the probes. I'm currently not convinced that any of MiraK's boxes are doing any such thing.
That said, there seems to be an assumption at play here that the server KiddoK set up is indeed a ssh server. If it's not - if it's something like a Windows file share exposed to the Internet - then hells yeah that shit needs shutting down like yesterday. Not because some scammer is handing out simulated Verizon nastygrams, but because if that's what it is then it's pretty much a dead certainty that it has been compromised and is doing god knows what.
SSH is really good, and if you're going to expose any kind of server at all to the Internet then it's easily the least likely to cause grief; plus, it's built in a way that lets you use an ssh connection as a secure tunnel for pretty much anything else, making it an excellent wrapper for other kinds of server. My own home servers expose only ssh for exactly that reason.
posted by flabdablet at 8:45 AM on October 12, 2022 [5 favorites]
A bit of security by obscurity actually works pretty well for ssh too. The aforementioned school server used to get probed thousands of times per day because the school district's IT department, which was in charge of the district VPN's Internet firewall, insisted that if I was going to run a ssh server in the school then I had to expose it in the standard way on port 22. My own home servers, which have exposed a couple of ssh servers on completely non-standard high-numbered ports for the last ten years, have yet to log any ssh probe traffic.
But again, none of this is directly relevant to MiraK or KiddoK, because none of the failed ssh logins shown in any of the Terrible Emails are coming from their own server's logs; they're allegedly coming from somebody else's logs and allegedly identify MiraK's public IP address as the source of the probes. I'm currently not convinced that any of MiraK's boxes are doing any such thing.
That said, there seems to be an assumption at play here that the server KiddoK set up is indeed a ssh server. If it's not - if it's something like a Windows file share exposed to the Internet - then hells yeah that shit needs shutting down like yesterday. Not because some scammer is handing out simulated Verizon nastygrams, but because if that's what it is then it's pretty much a dead certainty that it has been compromised and is doing god knows what.
SSH is really good, and if you're going to expose any kind of server at all to the Internet then it's easily the least likely to cause grief; plus, it's built in a way that lets you use an ssh connection as a secure tunnel for pretty much anything else, making it an excellent wrapper for other kinds of server. My own home servers expose only ssh for exactly that reason.
posted by flabdablet at 8:45 AM on October 12, 2022 [5 favorites]
Oh, one last thought and I'll shut up: if this is a scam, the nastygrams will be the softening-up phase and the next step will be phone calls from "Verizon's technical support department".
So if you get a call claiming to be from Verizon, and talking about unauthorized traffic from your IP address and/or threatening to cut off your service, then I recommend just hanging up on them without saying another word. If you want to talk to your ISP's tech support folks, you call them.
posted by flabdablet at 8:51 AM on October 12, 2022 [4 favorites]
So if you get a call claiming to be from Verizon, and talking about unauthorized traffic from your IP address and/or threatening to cut off your service, then I recommend just hanging up on them without saying another word. If you want to talk to your ISP's tech support folks, you call them.
posted by flabdablet at 8:51 AM on October 12, 2022 [4 favorites]
Nonsteroidal Anti-Inflammatory Drug , no, all the ssh port numbers are five digit numbers. None of them are "22" or "80" or "443". I pulled up a list of ports in use via my router and none of these numbers even match!
Those port numbers in that position in a sshd log entry are the source port number for the TCP connection over which the ssh login attempt is being made. As such, they're assigned essentially at random by the source machine's operating system. Unlike the standard destination port numbers on which services are exposed like 22 for ssh, 80 for http and 443 for https, source port numbers do not have pre-agreed meanings and unlike source IP addresses they carry no personally identifying information.
The only way a source port number in a sshd connection failure log entry is ever of any use at all is for debugging those connection failures using something like the Wireshark traffic analyzer running on the originating side at the time the logins are being attempted. After the fact they're of no use whatsoever.
But they do look nice and scary and quoting them in a log entry is good for promoting a general feeling of being under attack by technical forces beyond the ken of mere mortals.
posted by flabdablet at 9:40 AM on October 12, 2022 [4 favorites]
Those port numbers in that position in a sshd log entry are the source port number for the TCP connection over which the ssh login attempt is being made. As such, they're assigned essentially at random by the source machine's operating system. Unlike the standard destination port numbers on which services are exposed like 22 for ssh, 80 for http and 443 for https, source port numbers do not have pre-agreed meanings and unlike source IP addresses they carry no personally identifying information.
The only way a source port number in a sshd connection failure log entry is ever of any use at all is for debugging those connection failures using something like the Wireshark traffic analyzer running on the originating side at the time the logins are being attempted. After the fact they're of no use whatsoever.
But they do look nice and scary and quoting them in a log entry is good for promoting a general feeling of being under attack by technical forces beyond the ken of mere mortals.
posted by flabdablet at 9:40 AM on October 12, 2022 [4 favorites]
I really do intend to shut up and let somebody else have a go. Honestly. Real soon now...
I just looked up "FIOS fixed IP address" and it looks like that's something Verizon charges for. So unless you've paid for a fixed IP address, which you wouldn't do unless you were intending to run some kind of business server, chances are your public IP address is dynamic, which means you get a new one handed out every time your modem powers up, which means there's a really good chance that even if those log entries are even remotely genuine then it was somebody else in the same FIOS address pool as you, and not necessarily your house, that was responsible for them.
Which, again, makes it super unlikely that any sysadmin would ever have bothered to complain to Verizon about probes coming from a pool-[anything].[yourcityname].fios.verizon.net host even once, let alone multiple times as alleged. And even if they had, I would expect that it would take active involvement from Verizon's security team to cross-check the times of those log entries against the actual customer who had been handed that IP address at those times and so they would have some kind of record of these incidents.
posted by flabdablet at 9:51 AM on October 12, 2022
I just looked up "FIOS fixed IP address" and it looks like that's something Verizon charges for. So unless you've paid for a fixed IP address, which you wouldn't do unless you were intending to run some kind of business server, chances are your public IP address is dynamic, which means you get a new one handed out every time your modem powers up, which means there's a really good chance that even if those log entries are even remotely genuine then it was somebody else in the same FIOS address pool as you, and not necessarily your house, that was responsible for them.
Which, again, makes it super unlikely that any sysadmin would ever have bothered to complain to Verizon about probes coming from a pool-[anything].[yourcityname].fios.verizon.net host even once, let alone multiple times as alleged. And even if they had, I would expect that it would take active involvement from Verizon's security team to cross-check the times of those log entries against the actual customer who had been handed that IP address at those times and so they would have some kind of record of these incidents.
posted by flabdablet at 9:51 AM on October 12, 2022
October is cybersecurity awareness month! I am currently on a webinar from Proofpoint, and they are describing an attack technique called TOAD (Telephone-Oriented Attack Delivery). Here an email instructs you to call a phone number, where the person who answers will walk you through downloading malware to "solve" the "problem" they emailed you about . I can TOTALLY see how this model fits your experience so far!!!
--
BTW: ...super unlikely that any sysadmin would ever have bothered to complain to Verizon about probes coming from a pool-[anything].[yourcityname].fios.verizon.net host even once....
Yeah, we see stuff like this in our logs all the damn time. And no, we never follow it up because the IP addresses get recycled too fast to be tracked, if the affected sysadmin ever cared to do so.
posted by wenestvedt at 10:18 AM on October 12, 2022 [3 favorites]
--
BTW: ...super unlikely that any sysadmin would ever have bothered to complain to Verizon about probes coming from a pool-[anything].[yourcityname].fios.verizon.net host even once....
Yeah, we see stuff like this in our logs all the damn time. And no, we never follow it up because the IP addresses get recycled too fast to be tracked, if the affected sysadmin ever cared to do so.
posted by wenestvedt at 10:18 AM on October 12, 2022 [3 favorites]
Any email that promises immediate negative consequences if you don't take immediate urgent actions should be viewed with high suspicion, as scammers are notorious for using that sense of urgency to bypass your instincts towards caution. IMHO, if you've called Verizon about it, and they say that it's fake and there's nothing you need to do about it, then I'd believe them. A customer service agent should be able to see any notes on your account. The recommendations to not trust cold calls or cold emails are spot on; always hang up and dial a known-good support number for Verizon instead.
On the other hand, a teenager running services exposed to the internet at large is a red flag to me. These days there are entire departments at companies tasked with preventing, detecting and mitigating intrusions from very sophisticated threat actors, and with the amount of people working from home these days, no target is too small. I would personally not feel comfortable with someone that inexperienced managing something like that without competent and close supervision.
I would log into your router, disable all of the port-forwarding rules, and change the router admin password to something difficult to guess. Any ports open to the internet must go through you now. Note that if he still has physical access to the router he may be able to factory reset it to get around the password requirement.
If you do allow him to run services that require ports open to the internet at large (or he runs them secretly, also a possibility to be aware of now) then I strongly recommend that you make frequent offline backups to mitigate any ransomware attacks on your systems. As the adage goes, you have to be lucky every single time, but attackers only have to be lucky once.
posted by Aleyn at 2:57 PM on October 12, 2022 [1 favorite]
On the other hand, a teenager running services exposed to the internet at large is a red flag to me. These days there are entire departments at companies tasked with preventing, detecting and mitigating intrusions from very sophisticated threat actors, and with the amount of people working from home these days, no target is too small. I would personally not feel comfortable with someone that inexperienced managing something like that without competent and close supervision.
I would log into your router, disable all of the port-forwarding rules, and change the router admin password to something difficult to guess. Any ports open to the internet must go through you now. Note that if he still has physical access to the router he may be able to factory reset it to get around the password requirement.
If you do allow him to run services that require ports open to the internet at large (or he runs them secretly, also a possibility to be aware of now) then I strongly recommend that you make frequent offline backups to mitigate any ransomware attacks on your systems. As the adage goes, you have to be lucky every single time, but attackers only have to be lucky once.
posted by Aleyn at 2:57 PM on October 12, 2022 [1 favorite]
Best answer: There are some mistaken ideas floating around in this thread. So I'll try to add my own and hope they're not also mistaken. :)
posted by whatnotever at 7:09 PM on October 12, 2022 [9 favorites]
- The notice says Verizon has received 17 reports, not that they have tried to contact MiraK 17 times.
- It doesn't direct MiraK to call any phone number, email anyone, or go to any suspicious website. I don't see any scamming potential in the text of the email.
- It would not be surprising if Verizon has many different units that have no idea what the others are doing.
- Whoever emailed this to MiraK knows both their email address and their IP address. It's unlikely a random attacker on the internet with no specific information about MiraK would be able to connect their IP address with their email address. Verizon can. That's a point in favor of this really being from Verizon.
- Dynamic IP addresses are often assigned to ISP customers for a very long time, persisting through modem reboots. I don't know about Verizon, personally, but on Xfinity, for example, one can have the same dynamic IP for years.
- If MiraK's son is relatively inexperienced and has set up an internet-accessible server, it is very likely that server has been compromised and is part of a botnet.
- The logs are, as many have noted, exactly the type of logs one would see on a targeted server when a computer somewhere is in a botnet and scanning for weak passwords across the internet.
- fail2ban, a common security tool mentioned above, has a setting that will automatically email an ISP's abuse email address when a scanning attack like this is detected. If a botnet-infected machine is constantly hitting random machines all over the internet, it will hit several that have that setting enabled. This will trigger reports from multiple disconnected servers back to the ISP's abuse address, all pointing to the same IP.
- It would not be surprising if Verizon created an automated system that emails a customer when they receive enough of these automated reports of what is almost certainly botnet activity coming from that customer. Providing some information and a threatening nudge will be enough to get some customers to correct the malicious activity once they are aware of it, and so it could be seen as worth doing overall.
- All of the above fits the previous example linked in the first answer, in which there definitely was a compromised server on the customer's network.
posted by whatnotever at 7:09 PM on October 12, 2022 [9 favorites]
Best answer: Whatnotever makes good points, and if I had any degree of access to your LAN then that is indeed a scenario l I'd be investigating as plausible.
But putting my parent hat on for a moment: MiraK, I know nothing about your kiddo or the kind of relationship you have with him, but I've seen enough compassionate, well considered, well grounded, thoughtful and thoroughly sound human relations advice from you over the years to be pretty confident it's a good one, so I hope you don't see what follows as overstepping the mark.
I would strongly advise you not to adopt an adversarial position against a 14 year old tech-oriented kid over this issue. Maintaining a relationship built fundamentally on trust and open communication and collaborative problem solving with the small human beings that it's our job to keep alive matters much more, to my way of thinking, than the security of our home networking gear.
Kiddo says he's got this? Give him a chance to show you how he's got this. Have him walk you through his threat modelling and the steps he intends to take to mitigate those threats. If there is tech he needs to teach you before you can make heads or tails of what he's thinking, get him to try teaching it and make a good faith effort to wrap your head around it.
He might well have created a security hole and all the attendant hassle and headaches through inexperience and overconfidence, but that doesn't make running a small scale Internet server on a residential Internet service bad or wrong or irresponsible in and of itself. There's an opportunity here to strengthen both his tech skills and yours and your trust in each other, and it would be a pity to waste that in a reflexive flurry of You're Grounded, Mister.
posted by flabdablet at 9:11 PM on October 12, 2022 [6 favorites]
But putting my parent hat on for a moment: MiraK, I know nothing about your kiddo or the kind of relationship you have with him, but I've seen enough compassionate, well considered, well grounded, thoughtful and thoroughly sound human relations advice from you over the years to be pretty confident it's a good one, so I hope you don't see what follows as overstepping the mark.
I would strongly advise you not to adopt an adversarial position against a 14 year old tech-oriented kid over this issue. Maintaining a relationship built fundamentally on trust and open communication and collaborative problem solving with the small human beings that it's our job to keep alive matters much more, to my way of thinking, than the security of our home networking gear.
Kiddo says he's got this? Give him a chance to show you how he's got this. Have him walk you through his threat modelling and the steps he intends to take to mitigate those threats. If there is tech he needs to teach you before you can make heads or tails of what he's thinking, get him to try teaching it and make a good faith effort to wrap your head around it.
He might well have created a security hole and all the attendant hassle and headaches through inexperience and overconfidence, but that doesn't make running a small scale Internet server on a residential Internet service bad or wrong or irresponsible in and of itself. There's an opportunity here to strengthen both his tech skills and yours and your trust in each other, and it would be a pity to waste that in a reflexive flurry of You're Grounded, Mister.
posted by flabdablet at 9:11 PM on October 12, 2022 [6 favorites]
Best answer: Unless this is a two part scam and a future e-mail has the malicious attachment/link/phone number/whatever, I don't see anything in the e-mail that is particularly scammy, just surprising that Verizon actually cares enough to contact a customer over suspicious traffic.
You can try going to Greynoise and putting in your IP address and see what they say about it. They operate honeypot sensors all over the internet that detects bad traffic. If your IP is part of a botnet that's been doing widespread mischief they might have a record of it.
posted by Candleman at 9:33 PM on October 12, 2022 [4 favorites]
You can try going to Greynoise and putting in your IP address and see what they say about it. They operate honeypot sensors all over the internet that detects bad traffic. If your IP is part of a botnet that's been doing widespread mischief they might have a record of it.
posted by Candleman at 9:33 PM on October 12, 2022 [4 favorites]
Best answer: This is an example of what you'd be looking for - see the two tags related to SSH in the bottom right? Those are both things that the reported activity would likely be labeled as.
posted by Candleman at 11:21 PM on October 12, 2022 [1 favorite]
posted by Candleman at 11:21 PM on October 12, 2022 [1 favorite]
Response by poster: Okay, some updates!
The kid is running an SSH server but it is password protected. He says he will change this method.
Thank you for the resources you've provided for him to learn about security! It's a fabulous idea for me to direct him to these.
I am checking Greynoise ASAP.
And as for being grounded, I very much appreciate the commenters here urging me to turn this into a Teaching And Trust moment and I was all set to do that by the time he got home from school. But then he promptly spent 3.5 hours doing about 15 minutes' worth of homework because he got distracted by his phone & laptop, and then he lied to me about it, and it's a recurring issue so I'm afraid he IS Grounded, Mister. So I'm going to give this grounding a couple of days before putting him back on Project Server.
This is just THE BEST community, you all know that? This thread is the definition of "treat the person not the symptom." I'm very grateful to everyone here.
posted by MiraK at 3:22 AM on October 13, 2022 [7 favorites]
The kid is running an SSH server but it is password protected. He says he will change this method.
Thank you for the resources you've provided for him to learn about security! It's a fabulous idea for me to direct him to these.
I am checking Greynoise ASAP.
And as for being grounded, I very much appreciate the commenters here urging me to turn this into a Teaching And Trust moment and I was all set to do that by the time he got home from school. But then he promptly spent 3.5 hours doing about 15 minutes' worth of homework because he got distracted by his phone & laptop, and then he lied to me about it, and it's a recurring issue so I'm afraid he IS Grounded, Mister. So I'm going to give this grounding a couple of days before putting him back on Project Server.
This is just THE BEST community, you all know that? This thread is the definition of "treat the person not the symptom." I'm very grateful to everyone here.
posted by MiraK at 3:22 AM on October 13, 2022 [7 favorites]
He says he will change this method.
Fastest best first step would be just to change all existing Linux user account passwords to new ones generated at random, then have people write their new password on a sticky note and stick it to the screen until it's sunk into finger muscle memory.
A fairly natural mistake for the larval sysadmin is to set a sooper seekrit sooper strong root password and declare Job Done. But root isn't the only user account usable from outside with an ssh server, merely the most potentially damaging.
If he's exposed an ssh service to the world, the default way that ends up working on Mint will be that any of the box's existing Linux user accounts can log in via that service and start issuing shell commands. So if any existing user account has a shitty password (and to a very good first approximation, shitty = created by a human being rather than a random number generator) then there's a fair risk that a probe bot has managed to do that.
On a default Mint installation with ssh enabled, just being able to connect to sshd and knowing any user account's username and password is enough to enable the installation and use of software that could scan the Internet looking for more ssh servers and trying to guess usernames and passwords for those, in the process causing the machines it scans to create log entries like the ones you've pasted. You need root access to hide such software, but not merely to install and run it.
But if the account passwords are all decent (i.e. machine generated, as above) then probe bots can hammer away at the ssh service for centuries and get nowhere.
Best case if this is a Mint server breach, is that the user account being used wasn't one with sudo enabled and there are no rootkits installed, just a bit of digital graffiti from some script kiddie with a botnet, and it will be pretty easy to work out which user account(s) are affected and clean them up.
posted by flabdablet at 4:01 AM on October 13, 2022 [1 favorite]
Fastest best first step would be just to change all existing Linux user account passwords to new ones generated at random, then have people write their new password on a sticky note and stick it to the screen until it's sunk into finger muscle memory.
A fairly natural mistake for the larval sysadmin is to set a sooper seekrit sooper strong root password and declare Job Done. But root isn't the only user account usable from outside with an ssh server, merely the most potentially damaging.
If he's exposed an ssh service to the world, the default way that ends up working on Mint will be that any of the box's existing Linux user accounts can log in via that service and start issuing shell commands. So if any existing user account has a shitty password (and to a very good first approximation, shitty = created by a human being rather than a random number generator) then there's a fair risk that a probe bot has managed to do that.
On a default Mint installation with ssh enabled, just being able to connect to sshd and knowing any user account's username and password is enough to enable the installation and use of software that could scan the Internet looking for more ssh servers and trying to guess usernames and passwords for those, in the process causing the machines it scans to create log entries like the ones you've pasted. You need root access to hide such software, but not merely to install and run it.
But if the account passwords are all decent (i.e. machine generated, as above) then probe bots can hammer away at the ssh service for centuries and get nowhere.
Best case if this is a Mint server breach, is that the user account being used wasn't one with sudo enabled and there are no rootkits installed, just a bit of digital graffiti from some script kiddie with a botnet, and it will be pretty easy to work out which user account(s) are affected and clean them up.
posted by flabdablet at 4:01 AM on October 13, 2022 [1 favorite]
Oh, and disable the router's port mapping that's exposing ssh to the world before doing anything to the server. Don't enable it again until after the server's been cleaned up and secured.
posted by flabdablet at 4:06 AM on October 13, 2022 [2 favorites]
posted by flabdablet at 4:06 AM on October 13, 2022 [2 favorites]
Maybe I missed it somewhere above, but unless there's some reason not to I'd probably just wipe the thing and start over at the point I'd confirmed it had been compromised (once doing as much investigation as I felt like doing).
posted by snuffleupagus at 4:29 AM on October 13, 2022 [2 favorites]
posted by snuffleupagus at 4:29 AM on October 13, 2022 [2 favorites]
Candle man: Unless this is a two part scam and a future e-mail has the malicious attachment/link/phone number/whatever...
Yep, that is exactly how the scam works. :7)
The email sets you up and the human caller convinces you via social engineering.
I am not convinced this isn't both things at once: The Kid got the machine rooted and also a spray-and-pray script happened to land in MiraK's email box.
Good luck, MiraK: the smart ones are the hardest to raise! :7)
posted by wenestvedt at 4:32 AM on October 13, 2022 [2 favorites]
Yep, that is exactly how the scam works. :7)
The email sets you up and the human caller convinces you via social engineering.
I am not convinced this isn't both things at once: The Kid got the machine rooted and also a spray-and-pray script happened to land in MiraK's email box.
Good luck, MiraK: the smart ones are the hardest to raise! :7)
posted by wenestvedt at 4:32 AM on October 13, 2022 [2 favorites]
Response by poster: We looked at Greynoise this morning on my phone before he went to school, turns out this IP is classified as malicious. Eeep. SSH Bruteforcer and SSH Worm.
posted by MiraK at 4:40 AM on October 13, 2022 [1 favorite]
posted by MiraK at 4:40 AM on October 13, 2022 [1 favorite]
I'd probably just wipe the thing and start over at the point I'd confirmed it had been compromised
Depending what it's used for, it might have stuff sitting in user home folders that needs to get preserved across an OS nuke and pave. If that's the case then it's definitely worth going through those with a fine tooth comb for stuff that looks like ssh probe scripts.
posted by flabdablet at 5:19 AM on October 13, 2022 [1 favorite]
Depending what it's used for, it might have stuff sitting in user home folders that needs to get preserved across an OS nuke and pave. If that's the case then it's definitely worth going through those with a fine tooth comb for stuff that looks like ssh probe scripts.
posted by flabdablet at 5:19 AM on October 13, 2022 [1 favorite]
So MiraK just sent me the emails concerned as attachments, and they did come from abuse@verizon.com and they do have Verizon DKIM signatures. The scam hypothesis is disproved, and Verizon tech support needs to pull its head out of its arse.
posted by flabdablet at 5:37 AM on October 13, 2022 [8 favorites]
posted by flabdablet at 5:37 AM on October 13, 2022 [8 favorites]
As for what to do next: the suspected box has been taken offline for the time being, so it seems to me that the best thing to do next would be to do a daily Greynoise search for the next few days and make sure the "last seen" date on the results page stops advancing.
If it does stop, that says that the bot responsible for the malicious activity probably was most likely hosted on the Mint installation that's now offline. If "last seen" keeps on clocking up, the bot is on something that's still online.
posted by flabdablet at 5:47 AM on October 13, 2022 [4 favorites]
If it does stop, that says that the bot responsible for the malicious activity probably was most likely hosted on the Mint installation that's now offline. If "last seen" keeps on clocking up, the bot is on something that's still online.
posted by flabdablet at 5:47 AM on October 13, 2022 [4 favorites]
In your shoes I'd also be exercising whatever web-based process Verizon offers its customers to open formal tech support tickets.
If they have an online tech support system that's fit for purpose, then somewhere along the way it will give you an opportunity to upload files and attach them to the ticket, and you could and should upload the same .eml files you attached to the mail you sent to me.
Then ask Verizon how they reconcile the fact of those mails having valid Verizon DKIM signatures with the fact that phone tech support told you they were probably fake.
Let them know that you've treated their warning mails as genuine, and that you've taken a suspected bot host offline for closer examination, and ask them for formal confirmation that you've done what you've been asked to do and are no longer facing any kind of threat of service suspension or cancellation following from this particular issue.
posted by flabdablet at 6:07 AM on October 13, 2022 [1 favorite]
If they have an online tech support system that's fit for purpose, then somewhere along the way it will give you an opportunity to upload files and attach them to the ticket, and you could and should upload the same .eml files you attached to the mail you sent to me.
Then ask Verizon how they reconcile the fact of those mails having valid Verizon DKIM signatures with the fact that phone tech support told you they were probably fake.
Let them know that you've treated their warning mails as genuine, and that you've taken a suspected bot host offline for closer examination, and ask them for formal confirmation that you've done what you've been asked to do and are no longer facing any kind of threat of service suspension or cancellation following from this particular issue.
posted by flabdablet at 6:07 AM on October 13, 2022 [1 favorite]
That advice will help to clear your name -- and also get them to do a decent scan of your network.
Maybe sometime soon also force a restart of your Verizon router, to try to get a new IP address and get out from any blocks that might have been dropped on you. (Not 100%, but if you really got flagged, let the next poor sucker inherit your bad reputation.)
And when you check his files for the exploit scripts, make sure to look for hidden files before you copy stuff off the affected host to prep for a wipe-and-resintall.
(Dang, this kid is going to learn so much from this! It's actually pretty cool, if the two of you can stay focused on the task and not let blame creep in: it's an amazing opportunity for hands-on incident response.)
posted by wenestvedt at 6:22 AM on October 13, 2022 [1 favorite]
Maybe sometime soon also force a restart of your Verizon router, to try to get a new IP address and get out from any blocks that might have been dropped on you. (Not 100%, but if you really got flagged, let the next poor sucker inherit your bad reputation.)
And when you check his files for the exploit scripts, make sure to look for hidden files before you copy stuff off the affected host to prep for a wipe-and-resintall.
(Dang, this kid is going to learn so much from this! It's actually pretty cool, if the two of you can stay focused on the task and not let blame creep in: it's an amazing opportunity for hands-on incident response.)
posted by wenestvedt at 6:22 AM on October 13, 2022 [1 favorite]
The Greynoise ratings for things like SSH attacks are highly accurate - there's no way to forge your IP address with them short of being an extremely advanced attacker who is not going to do clumsy password spraying attacks - so it's safe to assume that some device on your network has been compromised.
I agree that it's likely to be a teenager's experimental box but there are other possibilities, including your router itself having gotten hacked.
As far as cleanup, wipe and reinstall is what I'd recommend. I do this stuff professionally at a high level and while I have in some cases tracked down and cleansed every bit of tricky backdoors that had been added by an attacker without having to reinstall, it's not something most people can easily do.
As far as the initial access point? Mint's default SSH installation does not allow root (AKA the account with the highest level of permissions) access so unless kiddo went out of his way to enable remote root access or has a guessable username and password (unlikely), it probably wasn't SSH that was how the attacker got it. It could be something else on the system like an out of date Minecraft server but that's part of why I'd encourage you to keep an eye out on other things like the router.
posted by Candleman at 7:52 AM on October 13, 2022 [3 favorites]
I agree that it's likely to be a teenager's experimental box but there are other possibilities, including your router itself having gotten hacked.
As far as cleanup, wipe and reinstall is what I'd recommend. I do this stuff professionally at a high level and while I have in some cases tracked down and cleansed every bit of tricky backdoors that had been added by an attacker without having to reinstall, it's not something most people can easily do.
As far as the initial access point? Mint's default SSH installation does not allow root (AKA the account with the highest level of permissions) access so unless kiddo went out of his way to enable remote root access or has a guessable username and password (unlikely), it probably wasn't SSH that was how the attacker got it. It could be something else on the system like an out of date Minecraft server but that's part of why I'd encourage you to keep an eye out on other things like the router.
posted by Candleman at 7:52 AM on October 13, 2022 [3 favorites]
The laptop is a likely culprit, so I'll just add that the honeypots generally caught scanners and malware intended for badly-configured hardware like cable modems, routers, IoT gadgets (cameras, etc), and other things that ended up shipping with little-to-no security - unset or easily-guessed passwords and so on (looking at you, Mikrotik). As part of your clean-sweep, take a peek at your cable modem setup and ensure it's up-to-date with whatever your ISP suggests/requires.
If you do end up wiping the laptop and receive any more of these emails going forward, it means that there's something else on your network that's been compromised.
And welcome to cybersecurity, where the threats are annoying and the attack surfaces, fractals. :)
posted by jquinby at 8:22 AM on October 13, 2022 [3 favorites]
If you do end up wiping the laptop and receive any more of these emails going forward, it means that there's something else on your network that's been compromised.
And welcome to cybersecurity, where the threats are annoying and the attack surfaces, fractals. :)
posted by jquinby at 8:22 AM on October 13, 2022 [3 favorites]
I ran an nmap scan against MiraK's IP address and it came up with all ports below 1000 filtered. I also checked port 2222 because that showed up on her Greynoise results page as having been targeted, hinting that it might also be something the bot knows how to get in through, and that was filtered too.
We know that the laptop (currently disconnected) was running a ssh server that was world-reachable, and to my mind that makes it the most likely candidate for compromise. Yes, Mint's default ssh configuration disables root by default and a probing bot would have needed to guess a username but that's not impossibly hard; lots of those, on home boxes, are just somebody's first name. Mint also gives the first user account created on the box wide-open sudo by default if I recall correctly, making the privilege escalation required for installing rootkits pretty trivial.
Next most likely compromise candidate would be any Windows box that's been used outside the house LAN at some point.
Anyway, the laptop is shut down for the time being and if Greynoise sees no further activity from MiraK's IP address for a week or so then that's probably where the bot was running and close examination of its entrails will probably be educational.
posted by flabdablet at 9:09 AM on October 13, 2022 [3 favorites]
We know that the laptop (currently disconnected) was running a ssh server that was world-reachable, and to my mind that makes it the most likely candidate for compromise. Yes, Mint's default ssh configuration disables root by default and a probing bot would have needed to guess a username but that's not impossibly hard; lots of those, on home boxes, are just somebody's first name. Mint also gives the first user account created on the box wide-open sudo by default if I recall correctly, making the privilege escalation required for installing rootkits pretty trivial.
Next most likely compromise candidate would be any Windows box that's been used outside the house LAN at some point.
Anyway, the laptop is shut down for the time being and if Greynoise sees no further activity from MiraK's IP address for a week or so then that's probably where the bot was running and close examination of its entrails will probably be educational.
posted by flabdablet at 9:09 AM on October 13, 2022 [3 favorites]
Might be an opportunity for the kid to learn how to do static analysis. (You examine an image rather than the running system.)
posted by snuffleupagus at 7:21 AM on October 14, 2022 [1 favorite]
posted by snuffleupagus at 7:21 AM on October 14, 2022 [1 favorite]
Greynoise first saw bot activity from that IP address on 2022-09-13, so one quick and easy static check that might yield useful results would be a search for files modified on that date. The bot would presumably have started running as soon as it was installed, so there's a fair chance that the bot scripts would turn up amongst those.
find /path/to/image/mountpoint -newermt 2022-09-12 -not -newermt 2022-09-15 -exec ls -l {} + | less
find should be run from a root shell, or via sudo, so that it gets to look inside all the mounted image's directories.
Note that the date search range covers 2022-09-12 to 2022-09-14, just to allow for the possibility that the system running find and Greynoise's reporting format might be using different timezones.
posted by flabdablet at 10:47 PM on October 14, 2022 [2 favorites]
find /path/to/image/mountpoint -newermt 2022-09-12 -not -newermt 2022-09-15 -exec ls -l {} + | less
find should be run from a root shell, or via sudo, so that it gets to look inside all the mounted image's directories.
Note that the date search range covers 2022-09-12 to 2022-09-14, just to allow for the possibility that the system running find and Greynoise's reporting format might be using different timezones.
posted by flabdablet at 10:47 PM on October 14, 2022 [2 favorites]
« Older Tell me about this ungoogleable onion/meat/pepper... | Scam resistant sandbox please! Newer »
This thread is closed to new comments.
However, in google a couple others have gotten this email and this may have some useful info.
posted by AzraelBrown at 7:24 AM on October 12, 2022 [7 favorites]