Advice Needed: email monitoring system

Tags: #<Tag:0x00007f224eebc5b8>

My company email server is hosted by an ISP, but it is unreliable, and sometimes messages get stuck for several hours. The problem is that some of these messages are sent to an email-to-sms gateway, so we get time-critical SMS messages that take hours to arrive. Not good. We get anomalies like this, where a mail sent at 16:57 finally arrives 4 hours late, but another mail sent later arrives almost immediately:

I plan to write an app that sends a single short message to a Gmail mailbox every, say, 5 minutes. Then I plan to check the Gmail mailbox after 5 minutes to see if it has arrived. If so, send another test message. If not, send a notification to the ISP helpdesk informing them of the problem. I will use the Gmail server to do this, since the normal mail server would be stalled at this point.

Before I reinvent the wheel, can anyone offer me any advice on the best way to do this? I plan to use the NetTalk email examples to assist with this process.

I haven’t used the TIMER property on a MAIN form before. If you have any advice on the correct way to do this, please let me know. I don’t want to make too many newbie errors if I can help it.

Thanks in advance
Donn
Clarion 11.0.13630 PE | NetTalk 12 App

People have been having all sorts of troubles with gmail because of recent authorisation changes.
Your slow email sounds like it might be a result of Grey Listing. This is a process where email is delayed until a reputation is built up. Unfortunately it can be affected by other domains sharing the same IP.

Have you talked to the hosting provider about it?

Hi Sean
Thanks. The flaky email server doesn’t normally send anything to Gmail. I plan on sending messages to Gmail only because it is more reliable than my current mail server. I could use another local email service instead of Gmail, so I’ll definitely bear that in mind.

My hosting provider explained that if we send too many emails then they halt the queue until someone can manually inspect it for spam. This can take any random amount of time from 10 minutes on a good day, to 4 hours or longer on a bad day. My plan is to alert them of their self-inflicted problem within 5 minutes of it showing up.

Donn

oh ok.
That sort of thing is usually triggered by number of emails sent in a time period.
If you get the provider to let you know the limits you can alter your side to rate limit.

Donn,

Have you looked at Twilio or similar provider?

Twilio is affordable and the SMS messages are darn near instant.

Don

I would look for a different provider and if thats not an option, I’d host what you can so you have data on where things are failing. Now a days, I’m finding there are too many points of failure in services which means even more passing of the buck, as they say.

I’ve always understood (for the UK dont know what its like outside the UK), text messages have to be sent within 24hrs, so with that in mind I wouldnt rely on txt messages if they are time critical. What I would do is use something like Freeswitch or Asterisk to ring the number and play a prerecorded message as you will know instantly if someone has got the message. Txt is unreliable imo because of the 24hr delivery window over the network.

Now if you need to do text to speech, Freeswitch has a good engine where you can write a report, convert it to mp3 and play it to a number that answers. TTS - FreeSWITCH - Confluence

You need to look at ATA’s gateways Gateways - FreeSWITCH - Confluence these are the devices that port the call from the computer program to the phone line, like a fancy modem, but make sure you lock this all down as this is an attack vector. Personally, I’d have a dedicated machine running Freeswitch based on past experience, to increase the isolation from the rest of the network, controlled by a firewall between the rest of your network and the freeswitch server. You can run freeswitch on a cheap raspberry pi and you can get rack mounts for multiple raspberrypi’s Raspberry Pi Rack Mount: 5 Best Racks for Pi Clusters | SHB (smarthomebeginner.com) to maintain that physical isolation from hackers which cloud servers cant always give you especially when thinking about Hertzbleed Attack and a bit of remote memory corruption like this Notes on OpenSSL remote memory corruption – Guido Vranken

and if you want to automatically handle the call at the other end, you can sent up freeswitch to handle the other end, record the call to mp3 and then speech to text it. But if you are going this route, you might just as well use DTMF tones to pass data! :grinning:

Interactive Voice Response is very easy on freeswitch IVR - FreeSWITCH - Confluence.

The main point is text is not reliable, only voice calls, so maybe look at a monitoring system that uses the voice part of the telecom’s network.

The other thing is, dont underestimate the extremism within some parts of society to monitor everything that goes on, its got a fervour that’s on a par with religion in my experience, so hacking is to be expected from even the most unlikely, trusted parts of life.

I recommend using a different email provider - one that is reliable and speedy. Especially for outgoing mails. There are many services (Postmark, SendGrid etc) that are dedicated to just sending mail, and this seems to be a better approach than trying to monitor an unreliable service.

1 Like

You say that, but they would be classed as spammers so would get blocked in some situations.

It’s the exact opposite Richard. Because they are legit services, and because you pay for them to send mail (if you have high volume), they have a higher reputation and thus they’re more reliable than spammers.

Costs are negligible - either free or $10 or so for most use cases. but having reliable email is certainly worth a few $.

I think Symantec blocked them for MS Exchange but the user’s had to put the anti spam settings on a high setting and I know I used to block.

Thanks everyone. I have learnt a lot more than I expected, and there are definitely some options I can explore.

I have always followed the notion that “you can’t solve a problem that you can’t measure” and so I set about sending an email every 5 minutes to a Gmail address that I could monitor. Initially the ISP guessed (wrongly) that the problem was with Gmail. When I pointed out that the delay was occurring in the SMTP server itself, before it passed the message on to another mail server within the ISP, they claimed that we were sending too much mail.

So we upped the limit of the number of messages per hour to 3600 (the same as Gmail) and increased the number of simultaneous connections allowed.

By this time my monitoring program (thanks to NetTalk) was able to produce a report of the delivery delay for each message. Here is part of the report on Saturday

9 JUL 2022 10:25:01 to 10:25:47 delay  0:00:45
9 JUL 2022 10:30:01 to 10:30:49 delay  0:00:47
9 JUL 2022 10:35:01 to 11:38:18 delay  1:03:16 <============
9 JUL 2022 10:40:00 to 11:37:11 delay  0:57:10 <===========
9 JUL 2022 10:45:00 to 11:39:44 delay  0:54:43 <==========
9 JUL 2022 10:50:00 to 11:41:48 delay  0:51:47 <==========
9 JUL 2022 10:55:00 to 11:38:40 delay  0:43:39 <========
9 JUL 2022 11:00:01 to 11:40:13 delay  0:40:11 <========
9 JUL 2022 11:05:00 to 11:05:48 delay  0:00:47
9 JUL 2022 11:10:00 to 11:10:57 delay  0:00:56
9 JUL 2022 11:15:01 to 11:15:47 delay  0:00:45
9 JUL 2022 11:20:01 to 11:39:23 delay  0:19:21 <===
9 JUL 2022 11:25:00 to 11:38:55 delay  0:13:54 <==
9 JUL 2022 11:30:00 to 12:09:10 delay  0:39:09 <=======
9 JUL 2022 11:35:00 to 12:05:05 delay  0:30:04 <======
9 JUL 2022 11:40:00 to 12:08:49 delay  0:28:48 <=====
9 JUL 2022 11:45:01 to 11:45:56 delay  0:00:54
9 JUL 2022 11:50:01 to 12:05:22 delay  0:15:20 <===
9 JUL 2022 11:55:00 to 12:09:03 delay  0:14:02 <==
9 JUL 2022 12:00:00 to 12:08:24 delay  0:08:23 <=
9 JUL 2022 12:05:01 to 12:05:46 delay  0:00:44
9 JUL 2022 12:10:01 to 12:10:55 delay  0:00:53

Then the ISP realised that two remote PCs, one in Namibia and one in Botswana, had been trying to send messages using an old password. Naturally, that caused the SMTP server to stop sending ANY mail from that account for the next hour. Seriously??!

Once I updated the two PCs we haven’t had any delays longer than 2 minutes, which is well within the tolerances I can deal with. I have improved the program to send me a report every night at 10pm, and to CC it directly to the ISP support desk if there are delays that need to be investigated. Also, I get an SMS notification if too many monitoring emails do not arrive in time. So far I haven’t needed either.

So hopefully the main issues have been resolved, for now. If the SMS system becomes unreliable I now have plenty of options, thanks to your suggestions. Many thanks.