Improved Schedule Monitoring

By Beate Thomsen, Co-founder & Product Design - October 13, 2020

In our latest release, which came out this past Sunday, we have improved the monitoring of your Schedules in your MyRapidi Configuration interface.

Before you would receive one email containing a list of all the Schedules that were due to be notified as failing at that given time. The next notification email would not be sent before twelve hours later (in order not to spam you with too many emails). This was sometimes not very good as it could create some situations, where you were not notified in a timely manner of a failing schedule - for example, another schedule that failed some hours after the first email was sent out and where you wanted to get notified about this.

Some of you have also mentioned that you would like to be able to specify who gets notified, in a more fine-grained way - for example for each Schedule or for a specific Service (if you have multiple Services). 

We found that the best solution to the above was to monitor and notify based on each Schedule (and for each Service). So that is what we have implemented in this release.

The monitoring process now automatically creates a record for each active Schedule (with Status different from 'Stopped') and keeps track of when you have been notified of a given error on this Schedule. The process will now wait 7 days before it alerts you again about the same issue. If however, the Schedule runs through successfully after some hours and then again gets an error, then we regard this as a new error and will notify you again.

The notification emails concern now only one specific Schedule, so it should be clearer what it is about and you can archive the email when you have fixed that specific issue.

We have also changed the layout of these emails in order to make them look better and make them easier to read.

It is clear that sending out notification emails about each Schedule instead of just one notification email for all failing Schedules could lead to a lot more emails.  So this brings us to the next topic of setting up the Schedules to only notify you when needed.

Setting up the Notify Interval on a schedule

The notify interval on a Schedule is what tells the monitoring process when to notify you about that specific Schedule. It is important to understand that the notification will happen not only for a Schedule that fails to run through without error but also for a Schedule that fails to run through in time (or at all). The latter can easily happen if the scheduling process is busy running other Schedules that possibly take a long time to run.

Each time a Schedule runs through without errors, it will calculate a new "Next Date/Time" for when the Schedule should run the next time. After that, it will calculate the "Notify Date/Time" by adding the notify interval to the "Next Date/Time". If the schedule does not run through successfully before the "Notify Date/Time" (whereby a new Notify Date/Time would be calculated), the monitoring process will notify you.

With this in mind, it is important to set a notify interval that leaves the scheduler process enough time to run all due-to-run Schedules and also finish this specific Schedule before the time elapses. For example, if you set your notify interval to e.g. 10 minutes you will most likely get notified very often (and very often without the Schedule actually having an error). If instead, you set your notify interval to e.g. 3 or 4 hours for a Schedule that runs each hour, that will normally be fine. Or if you have set the Schedule to run each night at 1 am, set the notify interval to 7 or 8 hours so that you will alerted in the morning which should normally be sufficient.

If you have Schedules that need to run often, you can use our "High Priority" feature - this will run the Schedule before other normal-priority Schedules and help to get it run often. You might also consider adding another schedule process (service process) to be sure that these high-priority Schedules are run in time. The "High Priority" feature is available in our new interface and for later Rapidi releases only. 

Read more on the wiki on how to set up your Schedules

Manage your notification Email addresses

Before you would manage who receives the notification emails on a per-service basis - directly in the Service List.

We have now moved this into the Account Admin area (Account Settings > Schedule Monitor). Here you can manage your notification email addresses in a more fine-grained way. This is available in the new interface.

For example, you can set up an email address to receive all the notifications on all schedules for a specific service, and/or you can set up an email address to only receive notifications on a specific schedule (on a specific service).

If no email address is specified all the notification emails with be sent to all the Account Administrators.

Schedule Monitor page on myrapidi
We hope you like the changes we have done this time. We do apologize if this resulted in a lot of emails being sent to you initially but hope that you will like the new system after getting it tuned to your specific needs.

Please feel free to send us any feedback or contact our support if you need any assistance.

About the author

Beate Thomsen, Co-founder & Product Design

Picture of
As co-founder of the Rapidi Data Integration platform, Beate has spent over 15 years on its development, building it around her motto: 'keep it simple, functional yet beautiful.'