Improving odds of getting past the Gmail spam filter (legitimately)?
January 7, 2020 1:41 PM Subscribe
I run a legit web app/service that puts legit computational work onto a legit local cluster. At the conclusion of that job, success or failure, the service sends an email to the end user. However, that email can get put into the end user's spam folder. So I'm looking for general advice on what I can do — without the obvious cheat of telling end users to check their spam folder — to change how I am doing things to get past Gmail's spam filter.
I don't want this legit email to go to the spam folder, because end users (like my boss, who wants to use this service) say the service doesn't work. And it legitimately doesn't! At least not as well as it could and should.
Telling people, like my boss, to check their spam folders is a non-starter. So I'm hoping there are ways to do this that sidestep that response. (Thanks for understanding before typing that in as an answer.)
For more technical specifics, I am using a Slurm job scheduler. A second, dependent job runs a pretty simple bash script on success or failure of the parent job, piping an echo-ed statement to mail.
After that, I don't really have a good understanding of what mail is doing, other than that emails kind of get to some people (like me) and not others (like my boss).
I honestly don't know much about email/SMTP servers. I haven't configured one on the host from which mail is sent. But I do have root access on this box, and I have UNIX chops, and I can do that, if need be and I know what to do and roughly how. If this can be configured in a way that makes email look more like the legitimate message that it is, for instance, pointers on how to do this would be useful information.
Additionally, information on how to structure the subject, sender, and email body itself is useful. Stepping back, the email content itself is pretty simple and kinda looks spammy, I suppose — inside the envelope is just a message saying: here is a link to your results web page, or your job failed and so please email so-and-so with your job ID. The subject is the job name, which is some boilerplate with a timestamp tacked on, unless the end user changed it back when submitting a job to the service.
Do people register their email servers with Google or something? I get loads of spam from political campaigns I never signed up for, so there must be some way to do this. Do they pay Google to get around the company's spam filter? I'm in a non-profit research lab, so hopefully I don't have to do something like that. But out of general curiosity I would like to know what options exist in that direction, if any.
I'm happy to provide more information, or if there's a good Stack Exchange site where I can go to ask this sort of question (without getting nit-picking rules-lawyers jumping in and saying the question isn't appropriate, etc.) I'd appreciate any tips.
I don't want this legit email to go to the spam folder, because end users (like my boss, who wants to use this service) say the service doesn't work. And it legitimately doesn't! At least not as well as it could and should.
Telling people, like my boss, to check their spam folders is a non-starter. So I'm hoping there are ways to do this that sidestep that response. (Thanks for understanding before typing that in as an answer.)
For more technical specifics, I am using a Slurm job scheduler. A second, dependent job runs a pretty simple bash script on success or failure of the parent job, piping an echo-ed statement to mail.
After that, I don't really have a good understanding of what mail is doing, other than that emails kind of get to some people (like me) and not others (like my boss).
I honestly don't know much about email/SMTP servers. I haven't configured one on the host from which mail is sent. But I do have root access on this box, and I have UNIX chops, and I can do that, if need be and I know what to do and roughly how. If this can be configured in a way that makes email look more like the legitimate message that it is, for instance, pointers on how to do this would be useful information.
Additionally, information on how to structure the subject, sender, and email body itself is useful. Stepping back, the email content itself is pretty simple and kinda looks spammy, I suppose — inside the envelope is just a message saying: here is a link to your results web page, or your job failed and so please email so-and-so with your job ID. The subject is the job name, which is some boilerplate with a timestamp tacked on, unless the end user changed it back when submitting a job to the service.
Do people register their email servers with Google or something? I get loads of spam from political campaigns I never signed up for, so there must be some way to do this. Do they pay Google to get around the company's spam filter? I'm in a non-profit research lab, so hopefully I don't have to do something like that. But out of general curiosity I would like to know what options exist in that direction, if any.
I'm happy to provide more information, or if there's a good Stack Exchange site where I can go to ask this sort of question (without getting nit-picking rules-lawyers jumping in and saying the question isn't appropriate, etc.) I'd appreciate any tips.
A very large part of what determines if mail gets to its destination without ending up in the Spam folder is the reputation of the IP address doing the sending. The Mail Transfer Agent (MTA) on your server that your service is running on is likely sending email directly to the destination server. The destination server doesn't know the reputation of your server, or it's possibly in a block of IP addresses where spam has originated from in the past, so, out of an abundance of caution, the recipient's system sends the mail to spam.
You could build up the reputation of this server by diligently telling folks to mark these spam emails, following best practices, etc, but the far easier path is to pay someone else. You should instead have your MTA relay your mail through what's known as a transactional email service. These are services that act as intermediaries for your email and have processes in place to make sure that the reputation of their IP addresses stays good. There are quite a few out there: SendGrid, MailGun, Postmark, and Amazon SES. If you're not sending millions of emails, they're pretty cheap.
You ask about registering email servers with Google. You can kind of do this with their postmaster tools. But what would probably get your further is using SPF, DKIM, and DMARC to tell receiving servers that your mail is legitimately being sent from your domain name. All of the above transactional email providers will help you set you up with this acronym soup.
posted by zsazsa at 2:04 PM on January 7, 2020 [6 favorites]
You could build up the reputation of this server by diligently telling folks to mark these spam emails, following best practices, etc, but the far easier path is to pay someone else. You should instead have your MTA relay your mail through what's known as a transactional email service. These are services that act as intermediaries for your email and have processes in place to make sure that the reputation of their IP addresses stays good. There are quite a few out there: SendGrid, MailGun, Postmark, and Amazon SES. If you're not sending millions of emails, they're pretty cheap.
You ask about registering email servers with Google. You can kind of do this with their postmaster tools. But what would probably get your further is using SPF, DKIM, and DMARC to tell receiving servers that your mail is legitimately being sent from your domain name. All of the above transactional email providers will help you set you up with this acronym soup.
posted by zsazsa at 2:04 PM on January 7, 2020 [6 favorites]
pipeski's idea about sending through Gmail is a good one if your rates are low and especially if it's an internal-only tool. Gmail limits to about 100-150 emails per day sent via SMTP.
posted by zsazsa at 2:35 PM on January 7, 2020 [1 favorite]
posted by zsazsa at 2:35 PM on January 7, 2020 [1 favorite]
Response by poster: The scope of that kind of traffic sounds like it would work well for my needs.
I've only sent email through Gmail through its web UI. For scripting purposes, would I do something like this:
posted by They sucked his brains out! at 2:44 PM on January 7, 2020
I've only sent email through Gmail through its web UI. For scripting purposes, would I do something like this:
$ env MAILRC=/dev/null \
from=scriptreply@domain \
smtp=host \
smtp-auth-user=login smtp-auth-password=secret \
smtp-auth=login \
mailx -n -s "subject" ...other email body stuff...
Replacing host with smtp.gmail.com (and the rest with credentials)?posted by They sucked his brains out! at 2:44 PM on January 7, 2020
zsazsa has the right idea.
Where is the actual SMTP server? Are are you just sending the emails from a rando AWS EC2 instance or your desktop or something? If so, Google et. al. punishes the shit out of emails that come this way, since firing up an instance and sending a bunch of spam emails is incredibly easy.
In fact, if it is EC2 or Digital Ocean or Rack Space or some other VPS, it's likely blocked by default because of SpamHaus's PBL. Go to their Blacklist Removal Center to see if your IP is on any of their lists. If it's the PBL, you can pretty easily self remove yourself and Google should be happy in a few hours.
If you aren't on the PBL, but one of their more serious lists, you might want to check out one of the other MTAs like SendGrid.
posted by sideshow at 3:09 PM on January 7, 2020 [1 favorite]
Where is the actual SMTP server? Are are you just sending the emails from a rando AWS EC2 instance or your desktop or something? If so, Google et. al. punishes the shit out of emails that come this way, since firing up an instance and sending a bunch of spam emails is incredibly easy.
In fact, if it is EC2 or Digital Ocean or Rack Space or some other VPS, it's likely blocked by default because of SpamHaus's PBL. Go to their Blacklist Removal Center to see if your IP is on any of their lists. If it's the PBL, you can pretty easily self remove yourself and Google should be happy in a few hours.
If you aren't on the PBL, but one of their more serious lists, you might want to check out one of the other MTAs like SendGrid.
posted by sideshow at 3:09 PM on January 7, 2020 [1 favorite]
You should instead have your MTA relay your mail through what's known as a transactional email service. These are services that act as intermediaries for your email and have processes in place to make sure that the reputation of their IP addresses stays good. There are quite a few out there: SendGrid, MailGun, Postmark, and Amazon SES. If you're not sending millions of emails, they're pretty cheap.
Seconding that this is going to be the easiest and most reliable solution to your deliverability problems. Getting a self-hosted mail-server set up to reliably deliver transactional email is really hard to do right unless you only have to deliver inside your own organization where you control all of the email infrastructure already.
posted by Aleyn at 3:27 PM on January 7, 2020 [1 favorite]
Seconding that this is going to be the easiest and most reliable solution to your deliverability problems. Getting a self-hosted mail-server set up to reliably deliver transactional email is really hard to do right unless you only have to deliver inside your own organization where you control all of the email infrastructure already.
posted by Aleyn at 3:27 PM on January 7, 2020 [1 favorite]
Also came in to say what zsazsa and Aelyn did: use a commercial service and set an SPF record. I have responsibility for a number of apps that generate emails and MailGun + an SPF record tied to our mail domain (which has a good reputation) has been the best solution for us in a setting where most customers receiving these emails are running O365/Exchange and aggressive spam/malware settings.
posted by kovacs at 5:43 PM on January 7, 2020 [2 favorites]
posted by kovacs at 5:43 PM on January 7, 2020 [2 favorites]
« Older How do I build a small, usable replica of AOL? | Title I schools + college admissions? Newer »
This thread is closed to new comments.
Alternatively, if the volume of emails being sent out is pretty small, getting your service to log into a GMail account to send the emails could be a quick fix.
posted by pipeski at 2:02 PM on January 7, 2020 [5 favorites]