Send Gmail Messages to Graylog
In this post, we will look into using a script to send new Gmail messages to Graylog
Graylog is an excellent platform for collecting and storing log data from servers, but that's not the only tasks that it can be suitable for. Objectively, any form of textual data can be added to Graylog for retention, alerting or reposting.
In this example, I'll show you how you can use Graylog to monitor your Gmail inbox for new messages. You can receive alerts for new messages, even receive alerts for particular words contained within the body of the message. The Python script is flexible, and you can always rely on Graylog's superb filters in case you need extra work done.
This script can be useful for monitoring a support email inbox, an inbox which receives alert messages, or anything else which requires a little intelligence.
Setting Up the Script
I'm going to use Python for this script. You'll be needing a couple of extra libraries to make it work. I used Python 2.7.6 on OS X and it runs without any issue.
- Gelfclient - Simple GELF library for Python
- Gmail for Python - It's a little older, but works fine as of March, 2015
Installing the Libraries
You'll need to run the following on the device you'll be running the Python script on. This requires Root / Administrator access to install the libraries correctly
pip install gelfclient pip install gmail
Install the Script
I've uploaded the script as a gist file at Github. It should be relatively self-explanatory, but I'll go through the highlights, just in case.
myGraylogServer- Welp, that's going to be your Graylog server. IP address or FQDN is fine
username- Your username at gmail.com
password- Password for the account. If you use 2-Factor Auth for Gmail, then you can create your own application password for it.
Running the Script
When the script runs, it will look for any new messages in your Gmail inbox. If there are no new messages, then it quits.
If there are new messages, it will split the message up into a number of fields:
message_id- Gmail's internal message ID number
sent_at- When the email was sent, according to Google
sent_from- Email address the message came from
subject- You should have a clue
body- Body of the email message
Script will mark any new messages as read once it finishes forwarding them to Graylog. I didn't explicitly test HTML email, so unsure how HTML messages will look in Graylog. The handful of Google Team default emails look OK to me.
If HTML mail is unacceptable, then you might try parsing the body variable through something like Beautiful Soup, to remove the tags.
With the email messages safely in Graylog, you can slice and dice them how you like them. Make a search for
And you should see the new records coming in. Messages should look like this
Add that search into a Stream, and you can perform alerting via streams, or reporting via dashboards. Intelligently forward messages to other users depending on content of the email message itself. Sky's the limit.
So, this is just a basic overview of how you can use a Graylog to capture actionable information, and even alert off of it's contents. Anything that exposes an API (or is easy to scrape) can be a source for your logging needs.