I was playing around with Gmail, the gdata api and Lotus Notes/Domino lately. The idea was to migrate some old stuff (obviously private stuff) from my Lotus Notes archives to a separate Gmail account. My company is running Notes and it's easy to have separate archives but I feel like having a backup from time to time and it is also very handy to have access to those things from everywhere without carrying your computer with you all the time. Therefore I need to transfer the archives over to gmail. If you are a Google Apps Premier or Google Apps Education Edition customer, all you have to do is to look at the
provided tools, install them and run your migration. If you have a free account you are stuck. But it would not be google, if you would not find any solution to this ;)
Here is my small howto on migrating Lotus Notes databases to Google free Gmail accounts. If you are trying this, you should have some insights in Java, Notes and MIME/Email concepts already. If not, you will find plenty of service companys out there helping you. You can even contact
Google directly about this. All others go ahead reading :)
Tools, sources and environment
- Get yourself a gmail account for testing :)
- Get yoursel a copy of the Lotus Notes 8.5.1 client (better the designer).
- Have a copy of your favorite Java IDE in place (for me this is still Eclipse) and start a new project.
- If you are not willing to start from scratch, get the gdata-java-client samples.
- Add all required google libraries (gdata-core, -client,-appsforyourdomain,-media, etc.) to your project's build path
- Copy the notes.jar from %LOTUS%\notes\jvm\lib\ext to your project's build path
- Add the Notes install folder %LOTUS%\notes to your project's run and debug-configuration path environment.
- Copy the sample.appsforyourdomain.migration.AppsForYourDomainMigrationClient.java file from the gdata samples to your project
Running the first test
If everything is in place, you should be able to run the AppsForYourDomainMigrationClient for the first time. Running it out of the box requires you to set some program arguments. --username --password --domain. It places 5 new emails in your inbox. If you look at the example in more detail, you will see, that it makes use of the
batch processing api. Basically the code takes a String, parses it, puts it into a Rfc822Msg and this is put into a MailItemFeed which is processed by the MailItemService.batch() method. All the magic happens in the runSample() method. You see, that whatever we are going to push to google, should be a rfc822 compliant message.
Getting documents from domino/notes
The idea is to fetch a message from Notes/Domino, convert it to a rfc822 message and send it via the gdata-api to google. If you are unshure about the notes connection things (they get tricky from time to time .. see links below). Let's go.
// initialize notes session
NotesThread.sinitThread();
Session session = NotesFactory.createSession();
session.setConvertMime(false);
// open your database
Database d = session.getDatabase(null, "folder/oldstuff.nsf");
Now we have to get all the documents one by one and let the magic happen.
DocumentCollection inbox = d.getAllDocuments();
Document doc = inbox.getFirstDocument();
while (doc != null) {
//...
doc = inbox.getNextDocument(doc);
}
That is all you have to do. Now you are able to iterate over the complete document collection in one archive. But: All you have now is an instance of a lotus.domino.Document. Some handy methods are there but it's by far not enough to take it, serialise it and put it on the wire.
From Document to MIMEEntity
First thing you will have to do with a single Document is to convert it to a so called lotus.domino.MIMEEntity. This is the starting point for all further processing.
MIMEEntity mime = doc.getMIMEEntity(); if the mail was already received via smtp this is most likely everything you have to do. If you have original domino Documents at hand, this will not work and mime will be null. Beginning with 8.5.1 you have a
public void convertToMIME(int conversiontype, long options) method at the Document API. This will do the job for you. You can have different conversion types available. I am using the Document.CVT_RT_TO_PLAINTEXT_AND_HTML for multipart/alternative.
From MIMEEntity to Rfc822Msg
Now you are only one step away from a Rfc822Msg. All you have to do now is to build a StringBuffer with all the RFC 822 requirements fulfilled. The tricky part are the headers. To get them you have to work with the lotus.domino.MIMEHeader object.
Vector headers = mime.getHeaderObjects();
for (int j = 0; j < headers.size(); j++) {
MIMEHeader header = (MIMEHeader) headers.elementAt(j);
buffer.append(header.getHeaderName()
+ ": "
+ header.getHeaderValAndParams()
+ "\r\n");
}
If you work through the Document you will see, that it has a child for every multipart part. So you have to make shure, you iterate over each child and add it to the buffer, too.
MIMEEntity child1 = mime.getFirstChildEntity();
while (child1 != null) {
//...
if (child2 == null) {
child2 = child1.getNextSibling();
if (child2 == null) {
child2 = child1.getParentEntity();
if (child2 != null)
child2 = child2.getNextSibling();
}
}
child1 = child2;
}
If you are working your way through attachments, you will find it usefull to have the
public void encodeContent(int encoding) from the MIMEEntity to convert attachements to MIMEEntity.ENC_BASE64. Everything else will not work.
From Rfc822Msg to MailItemEntry
Take your buffer and put it to the Rfc822Msg
Rfc822Msg rfcMsg = new Rfc822Msg(buffer.toString());. Add it to a MailItemEntry
mailItem.setRfc822Msg(rfcMsg);.
Lables and properties
You can apply labels and additional properties to the MailItemEntries. A label will show up in gmail as you are used to. I flaged all migrated emails with a "private" label like this
mailItem.addLabel(new Label("private"));. It's also possible to take the original folder names of the Document and put it as label. You also can decide, if you want the message to appear as unread in the inbox or in the sent folder or whatever if you add additional properties like this:
mailItem.addMailProperty(MailItemProperty.INBOX);.
Sending mail
Use the provided Batch approach but send single mails within a single batch worked for me. Here is the basic approach:
BatchUtils.setBatchId(mailItem, "" + uniqueId);
BatchUtils.setBatchOperationType(mailItem,
BatchOperationType.INSERT);
MailItemFeed feeder = new MailItemFeed();
feeder.getEntries().add(mailItem);
MailItemFeed feedR = mailItemService.batch(domain,
destinationUser, feeder);
You can get a status of the submission
BatchStatus status = BatchUtils.getBatchStatus(returnedEntry);. If something went wrong you will get a status unequal 201.
Further thoughts
You should not try to send messages bigger than 5000000 bytes. Google will reject them. It's best to have some migration/error logging in place allowing you to manualy migrate failed documents. For me it was handy to log the documents
doc.getUniversalID() for later migration. Another limit at Google are the requests per second. I don't know how high/low this exactly is, but you will get batch status erros if you exceed this limit. You should wait >30sec after continuing with your submissions.
Now you have your Domino/Notes documents in your Gmail account. It was fun to try this and you should give it a try yourself if you need too :)
Links and readings
Google Data Protocol Developer's Guide Overview
Java access to the Domino Objects
The Multipart Content-Type
