Send Gmail Messages to Graylog

In this post, we will look into using a script to send new Gmail messages to Graylog

Graylog is an excellent platform for collecting and storing log data from servers, but that's not the only tasks that it can be suitable for. Objectively, any form of textual data can be added to Graylog for retention, alerting or reposting.

In this example, I'll show you how you can use Graylog to monitor your Gmail inbox for new messages. You can receive alerts for new messages, even receive alerts for particular words contained within the body of the message. The Python script is flexible, and you can always rely on Graylog's superb filters in case you need extra work done.

This script can be useful for monitoring a support email inbox, an inbox which receives alert messages, or anything else which requires a little intelligence.

Setting Up the Script

I'm going to use Python for this script. You'll be needing a couple of extra libraries to make it work. I used Python 2.7.6 on OS X and it runs without any issue.

• Gelfclient - Simple GELF library for Python
• Gmail for Python - It's a little older, but works fine as of March, 2015

Installing the Libraries

You'll need to run the following on the device you'll be running the Python script on. This requires Root / Administrator access to install the libraries correctly

pip install gelfclient
pip install gmail

Install the Script

I've uploaded the script as a gist file at Github. It should be relatively self-explanatory, but I'll go through the highlights, just in case.

• myGraylogServer - Welp, that's going to be your Graylog server. IP address or FQDN is fine
• username - Your username at gmail.com
• password - Password for the account. If you use 2-Factor Auth for Gmail, then you can create your own application password for it.

Running the Script

When the script runs, it will look for any new messages in your Gmail inbox. If there are no new messages, then it quits.

If there are new messages, it will split the message up into a number of fields:

• message_id - Gmail's internal message ID number
• sent_at - When the email was sent, according to Google
• sent_from - Email address the message came from
• subject - You should have a clue
• body - Body of the email message

Script will mark any new messages as read once it finishes forwarding them to Graylog. I didn't explicitly test HTML email, so unsure how HTML messages will look in Graylog. The handful of Google Team default emails look OK to me.

If HTML mail is unacceptable, then you might try parsing the body variable through something like Beautiful Soup, to remove the tags.

Graylog Alerting

With the email messages safely in Graylog, you can slice and dice them how you like them. Make a search for

source:gmail

And you should see the new records coming in. Messages should look like this

Add that search into a Stream, and you can perform alerting via streams, or reporting via dashboards. Intelligently forward messages to other users depending on content of the email message itself. Sky's the limit.

In Closing

So, this is just a basic overview of how you can use a Graylog to capture actionable information, and even alert off of it's contents. Anything that exposes an API (or is easy to scrape) can be a source for your logging needs.