How To poll out lotus notes(or any other mail server) emails into a Ruby App

A Utility to poll out emails from Mail server and bringing into a Ruby Application. Few steps to be used:

1. Get created your mailbox on lotus notes server with POP3 enabled.
2. Mailman gem
3. Script/task to poll the emails(to receive formatted emails and with attachments)
4. Some checks so that any email should not lost if any issue on server-side for the time being.
5. God gem(to monitor the script/task, to start/stop/restart).

STEP 1: Get created your mailbox server.
A mailbox server is what required for storing your email on notes database and then poll them in Ruby app.
It Should be POP3 enabled & ‘UTF-8’ enabled otherwise it could trouble with attachments & formats.

STEP 2: Add gem Mailman to your Gemfile
                Mailman is an incoming mail processing micro framework (with POP3 and Maildir support), that works with  Rails

                 gem ‘mailman’, :require =>  false
& run bundle install

STEP 3: Mailman Script
The script to read & fetch emails from notes database with attachments & proper format.
Lets create a rake task/script for the same.
Create a task execute_mailman.rake in lib/tasks
===============================================================================================

require “mailman”
puts ‘Mailman Started’
namespace :execute do
desc “it will pull the mails from mail server”
task :mailman => :environment do
Mailman.config.logger = Logger.new(“log/mailman.log”)
Mailman.config.poll_interval = 3 # to check for new emails in every 3 seconds
Mailman.config.pop3 = {
server: “pop.gmail.com” #”Your_mail_server_name.COM”,
port: 995,
ssl: false,
username: ENV[‘MAIL_SERVER_USERNAME’],
password: ENV[‘MAIL_SERVER_PASSWORD’]
}

Mailman::Application.run do
default do
begin
p “Found a new message from #{message.from}”
to_list = ”
to_list << message.to.join(‘,’) if message.to.present?
from_list = ”
from_list << message.from.join(‘,’) if message.from.present?
cc_list = ”
cc_list << message.cc.join(‘,’) if message.cc.present?
bcc_list = ”
bcc_list << message.bcc.join(‘,’) if message.bcc.present?
fl = “0”
if message.multipart?
if message.html_part
if message.html_part.charset.present?
if message.html_part.body
fl = “1”
# to check if email has 7-bit Korean Charachter
if message.html_part.charset == “ISO-2022-KR”
the_message_html = message.html_part.body.decoded.encode(‘UTF-8’, ‘us-ascii’, :invalid => :replace, :undef => :replace)
else
the_message_html = message.html_part.body.decoded.encode(‘UTF-8’, message.html_part.charset, :invalid => :replace, :undef => :replace)
end
end
end
end
# if message has text_part present
if message.text_part
if message.text_part.body
if message.text_part.charset.present?
fl = “1”
if message.text_part.charset == “ISO-2022-KR”
the_message_text = message.text_part.body.decoded.encode(‘UTF-8’, ‘us-ascii’, :invalid => :replace, :undef => :replace)
else
the_message_text = message.text_part.body.decoded.encode(‘UTF-8’, message.text_part.charset, :invalid => :replace, :undef => :replace)
end
end
end
end
em = Email.create( :from => from_list, :to => to_list, :subject => message.subject, :copyto => cc_list, :blindcopyto => bcc_list, :received_at => message.date, :body => the_message_html)

# to recieve the email attachments
the_message_attachments = []
message.attachments.each do |attachment|
file = StringIO.new(attachment.decoded)
file.class.class_eval { attr_accessor :original_filename, :content_type }
file.original_filename = attachment.filename
file.content_type = attachment.mime_type
attachment = Attachment.new
attachment.attached_file = file
attachment.email_id = em.id
em.attachments << attachment
attachment.save(:validate => false)
the_message_attachments << attachment

end
else
if message.text_part.present?
if message.text_part.charset.present?
fl = “1”
if message.text_part.charset == “ISO-2022-KR”
the_message_text = message.text_part.body.decoded.encode(‘UTF-8’, ‘us-ascii’, :invalid => :replace, :undef => :replace)
else
the_message_text = message.text_part.body.decoded.encode(‘UTF-8’, message.text_part.charset, :invalid => :replace, :undef => :replace)
end
end
end
if message.html_part.present?
if message.html_part.charset.present?
fl = “1”
if message.html_part.charset == “ISO-2022-KR”
the_message_html = message.html_part.body.decoded.encode(‘UTF-8′,’us-ascii’, :invalid => :replace, :undef => :replace)
else
the_message_html = message.html_part.body.decoded.encode(‘UTF-8’, message.html_part.charset, :invalid => :replace, :undef => :replace)
end
end
end

if fl != “1”
if message.body.present?
if message.charset.present?
if message.charset == “ISO-2022-KR”
the_message_html = message.body.decoded.encode(‘UTF-8’, ‘us-ascii’, :invalid => :replace, :undef => :replace)
else
the_message_html = message.body.decoded.encode(‘UTF-8’, message.charset, :invalid => :replace, :undef => :replace)
end
end
end
end
the_message_attachments = []
em = Email.create(:from => from_list, :to => to_list, :subject => message.subject, :copyto => cc_list, :blindcopyto => bcc_list, :received_at => message.date, :body => the_message_html)
end
rescue Exception => e
Mailman.logger.error “Exception occurred while receiving message:\n#{message.subject} , #{message.from}”
p “Error ”
Mailman.logger.error [e, *e.backtrace].join(“\n”)
Kernel.exit
end
end
end
end
end

================================================================================================

In your config/environments/development.rb add following settings:

ENV[‘MAIL_SERVER_USERNAME’] = ‘Your mail server name’
ENV[‘MAIL_SERVER_PASSWORD’] = ‘password you have chosen’          # never share your passwords with anyone 🙂

STEP 4  Putting some checks so that not a single email is lost

Create a file mailman.rb in config/initializers and add, to ensure that message get deleted from mail server only    after its being pulled in ruby app.

=====================================

require “mailman”
require “net/pop”
Mailman::Receiver::POP3.class_eval do
def get_messages
begin
@connection.each_mail do |message|
@processor.process(message.pop)
message.delete
end
rescue Exception => e
p ” Exception #{e}”
disconnect
end
end
end

=========================================

This is all required to poll out emails into a Ruby App. Now, if you want  to host app on server and wanted to ensure that this  task should keep on running 24*7 then GOD is a process watcher explained in step 5.

STEP 5: GOD gem to keep a watch on your rake task or scripts. (It’s a ruby file only)

add gem ‘god’  to your Gemfile & bundle install

Create a file mailman.god in config folder.

======================================================================

RAILS_ROOT = File.dirname(File.dirname(__FILE__))
God.pid_file_directory = File.join(RAILS_ROOT, “tmp/pids”)

def generic_monitoring(w, options = {})

w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 120.seconds
c.running = false
end
end

w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = options[:memory_limit]
c.times = [3, 5] # 3 out of 5 intervals
end

restart.condition(:cpu_usage) do |c|
c.above = options[:cpu_limit]
c.times = 5
end
end

w.lifecycle do |on|

on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end

end

God.watch do |w|

w.env = { ‘RAILS_ENV’ => ENV[‘RAILS_ENV’] || ‘development’}
w.name = “mailman”
w.dir = “#{RAILS_ROOT}”
w.group = “mailit”
w.interval = 180.seconds
w.start = “rake execute:mailman”
w.restart = “rake execute:mailman”
w.start_grace = 80.seconds
w.restart_grace = 80.seconds
w.log = “#{RAILS_ROOT}/log/mailman_god.log”
w.behavior(:clean_pid_file)
generic_monitoring(w, :cpu_limit => 80.percent, :memory_limit => 100.megabytes)
end

=======================================================================

Now Start your God watcher using command :
bundle exec god -c config/mailman.god

To stop God watch
bundle exec god terminate

This is it ! Few details:

1. The mailman script used above will handle all Chinese/Japanese & Korean 7-bit characters .
2. There are more options in GOD script you can play around(email notification is one of its kind & useful)
3. If mailman script stops then god will restart it automatically, GOD process is one we need to restart on our own for once.
4. You can start the GOD process in your Capfile at the time of deployment.

And we are Done !! Yay 🙂

Advertisements

One thought on “How To poll out lotus notes(or any other mail server) emails into a Ruby App

  1. These codes are hilarious for me as i m not a person who can code. And just because to ignore such errors i switched to Outlook from rough and tough Lotus Notes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s