BUILDING YOUR OWN WEBSITE MONITOR

Overview

Like most of us, I am sure you love to do a lot of tinkering under the hood and trying out DIY solutions. This is one such example of building a website monitor. There are a multitude of free and paid website monitoring systems and they do exceptionally well in informing you when a site is goes down and comes back up again. This article is not meant to be a complete replacement for these, but as a breeding ground for you to expand on building a quick & working website monitor.

The Basics

We are trying to create a basic, yet fully functional website monitoring system. It is designed to run with a very small footprint and was tested with a 128MB NAT VPS. It relies on external applications – Slack & a Git Server (GitLab in this example).

The Git server, stores the URLs for monitoring and you can add/remove entries via version control. Slack is the destination for notifying you when a website goes down. You could expand this example to use any webhook such as Amazon SNS or use Twilio’s SMS API to send you a message.

Let me outline the process here:

Store a list of URLs to check in a Git repository as a text file
Create an incoming webhook for your Slack workspace
Run the shell script to retrieve the URL list from the repo, check the status code from a curl request to the URL
Check the status code with the previous status code (from the URL list file), post an appropriate message to Slack (Website has recovered/Website is down)
If a status code has changed, update the text file with the new status code
At the end of processing, if any URLs status code has changed, commit the change and push the file to the Git repository

Let’s get to implementing this. Since all projects get a name, I am going to call this

200orBust

Creating a Git Repo and Building The URL List

In your git server, create a text file with URLs in this format

The format is the full URL (http/https) and a separator \ followed by 200. All sites are initially added as “Up”, i.e. HTTP Status of 200.

Commit the change and push to the git server. Note the raw URL of the text, for e.g.,

https://gitlab.com/userid/reponame/raw/master/urllists.txt

Setting Up a Slack Incoming Webhook

If you already have a slack workspace, use this link (https://api.slack.com/incoming-webhooks) to create a new incoming webhook integration for your workspace. Note the webhook URL. It is of the format – https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXX XXXXXXXXX I recommend that when you create a new webhook, you also create a new channel for monitoring site availability so as to not clutter up your existing channels. I have named mine #site-up-down

The Code

Before we begin, we need to do an initial clone of the git repo. Use git clone

Now, we are all set to begin our test. Here is the complete code (I have removed some comments for brevity)

We start out with pulling the URLlist from our git repo. We have used a function to perform the actual “post-to-slack” functionality so that we can parameterize the icons and emojis.

The next for loop reads through the file urllists.txt, parses it (converting the input line into the URL to be checked and the previous HTTP status code).

We do a quick curl to get the HTTP status code. I will quickly explain what this does

We pass on the user agent via the option -A. This is required, as some sites such as amazon.com return a 503 status code when no user agent is provided (this prevents bots from screen scraping)

The -L option is to follow redirects. This helps get the right status code for querying websites that use load balancers to redirect your request.

Now, the output of the curl is stored in $status_code which is checked against the previous status code and if changed, send the appropriate message to slack. We are also setting a flag UPDATED to true so that it triggers a git commit and push to the repo at the end.

Here is how the slack messages appear in your channel

Setting Up the Script to Run Automatically

You can have this script run automatically through cron. To set it up, pull up crontab – e and add the path to your job like so

Since the cron does not run from the directory of the shell script, you will see that the very first command in our script is cd /home/user/200orbust/200orbust.sh

The above cron job runs every minute and posts to slack of failures and when sites come back up again.

Quick word of caution, keep track of the size of your syslog that tracks the execution of the cron job. You may have to periodically reset it to avoid the syslog becoming a massive file after a month of execution.

Hope this script has helped you in designing a quick and useful server monitor. Let us know how you expanded this example. Happy Monitoring!