Tuesday, 14 June 2016

5 Reasons Data Migration is Challenging

Data migration is hard. Granted, it's not something that most people think about, and very few have actually tried it. If you haven't, you'll just have to trust us on this one; it's hard. Also, if you're a software developer who hasn't tried it, it's probably much harder than you think. Despite this fact, there are many naysayers out there who will tell you it's a breeze. They glibly point out that system A has an API and system B has an API, so there you go--easy peasy, lemon squeezy (as my daughter would say). If only it were that simple. So why is data migration complicated, difficult, and riddled with booby traps? This post outlines five of the main reasons.

migrate to Google Apps and Google Drive

Before getting to the list, let's first establish that this post is about the technical challenges. User adoption can actually be the highest hurdle to a successful migration, but that's not in the scope of this post. In terms of the challenges to creating migration software, here are some of the thorniest issues we face.

5. People think it's easy so it always takes more time and effort than expected.

As mentioned above, there is a general misunderstanding about data migration out there that it should be easy. This leads to many problems. One of the most common is trying to build a custom migration solution for a project. I've heard about numerous failed migration projects because the company involved quite often has to seek an off the shelf migration solution after their custom solution fails. There simply is no way to argue with the ROI benefit of licensing an existing product rather than developing your own custom migration solution.

4. Data volume and performance are challenges--especially when dealing with the cloud.

Even under ideal performance conditions (e.g., running both the source and target on-premises), performance is an issue when dealing with large volumes of data. The problem is that an organization can literally spend years putting data into a content management system and then ask for that data to be migrated to a new system over the weekend. First of all, getting all the data out of a system is not the same as getting each piece out. When you try to get all the data, you (understandably) can run into all sorts of bottlenecks and intentional throttling. Even if the system has an API that allows one item to be copied out, asking that API to copy millions of things in one go can quickly go pear-shaped (more about that when we get to API issues).

3. There are a lot of file storage and content management systems out there and every company uses more than one.

The reality of today's IT infrastructure is that practically every company uses some sort of on-premises file shares in addition to a content management system, and cloud-based file storage. Taking the time to write migration software for each of these systems is just not feasible. This means migration Independent Software Vendors (ISVs) have to choose which ones they feel will be the best business case. Choosing which platforms to support can make or break a small ISV business.

2. No one wants you to migrate away from their platform so no API is ever built to easily move data out of a system.

Sadly, no API seems to ever get finished. This isn't because developers won't want to finish things; it's most commonly the reality of dealing with limited resources. However, it's also true that there are times when companies make the conscious decision to not provide API read access to everything in their system. By doing so, they enhance the 'stickiness' of their application. In other words, it's harder for customers to jump ship and migrate to another platform. That's one API issue, but it's not the only way. Even if the API is provided, it's built for customer use cases, not the migration use case. So those of us who try and use it for migration, find that it's either way too slow for a migration, or it simply does not work because it's incomplete or throws too many errors.

1. Data migration is often not apples to apples.

When you migrate from one system to another, there are times when it isn't clear where something should go; and sometimes there just isn't a right answer. You can choose pretty much any two content management systems (or file storage systems), and you'll find data in one that doesn't have a home in the other. The reason for this is clear; every company wants to add value to their software. As each system adds features to try and one-up one another, they create more and more islands of data. For example, as I wrote in a previous post, the Google Drive storage system is a graph. If you try and replicate that graph in a traditional file system, you'll quickly run into a wall. It's just not possible; they are built on different (and mutually exclusive architectures).

Data migration may not be easy, but therein lies the fun. Everyone loves a good challenge and those of us here at AppBridge are no exception. If it was easy, we'd probably move onto to some other project.

Note: AppBridge will be presenting a session on Google Apps migration at the Google Atmosphere conference in Tokyo tomorrow.

Here's the info about our session:
Bring your file servers into the cloud! Covering everything from security to data migration

Until now, replacing file servers by going to the cloud has presented numerous challenges to organizations, such as security concerns and how to migrate critical data. In this session, we'll explain in detail how to use Google Drive as a file server in the cloud, including how to migrate the data from existing file servers on premises. Observe how data migration occurs in real-time by viewing a demonstration of AppBridge's data migration solution.

Tuesday, 7 June 2016

3 Key Ways Google Drive is Different from Other Online Storage

The number is ever increasing, but over 200 million people use Google Drive for their online file storage needs, and because Drive looks like the other SaaS file storage systems, many people just assume that it works the same way. This is just not true, under the hood Drive is quite different than the cloud storage services such as BoxDropbox, or Microsoft OneDrive. Here are three important examples of how Google Drive is different.

migrate to Google Apps and Google Drive

Google Drive is a Graph

When I write that Google Drive is different, I don't mean because Drive isn't just a storage system (you also get Google apps), because it offers the best online co-authoring experience, or even because of the built-in advanced search capability. No, I'm not referring to some marketing interpretation of the different cloud storage systems, I mean the actual file storage mechanism is different. Unlike the other cloud storage systems, Drive isn't hierarchical; it's a graph.

Although the term "folder" is used in Drive, they aren't really folders. Just like Gmail, Drive doesn't have folders in the traditional sense. Instead, files and folders are tagged with metadata "labels" and it's the labels that determine where things appear rather than a folder structure. In traditional file systems (including other cloud services), files are "located" inside folders. (People will commonly say "physical folders" even though the entire system is obviously digital.) Continuing with that terminology, Drive files are not physically inside folders; they can be tagged with multiple labels and therefore they can appear to be in multiple "places." You might think that the labels are similar to symlinks or shortcuts, but again that's not a valid comparison because Drive only has labels, whereas symlinks and shortcuts are used to reference traditional folders.

When you "move" a Drive file or folder, you're actually changing its labels. This is how you can add a file or folder to your drive so easily and freely move it around. It's not a copy, it's the same file/folder that everyone else with rights is seeing, but it can appear in a different place to each user. This affects the way that sharing works in Drive, but we'll get to that soon.

Google Docs are Not Syncing to Your Computer

I know that some people believe that copying Drive folders that are syncing to their PC is the same as copying all their Drive files and Google Docs (sheets, forms, etc), but that's false. Google Docs are handled differently than other types of files and one key difference is they're only stored on the Google Servers--the content is not copied to a user's computer during a sync. This can lead to some unfortunate outcomes. For example, the guy who deleted all his Google Docs files because he didn't understand how Drive works.

What users see when they sync Google Docs to their computer are actually shortcuts to the files. Other file types (e.g., Microsoft Office documents, PDFs, etc.) are actually copied. You can try a simple test yourself to see exactly what happens. Use a text editor to open a Google Doc from a synced folder on your machine and you'll see a reference like this:

{"url": "https://docs.google.com/open?id=1bxxxxaSz-mUda9_RQxxxCrmAPic", "doc_id": "1aCaxxxaxaSz9_RQ94vxxxmAPic", "email": "foo@bar.com", "resource_id": "document:1aCaMDxxxSz-mUda9_RQ94vixxxPic"}

What does this mean for the average user? Not a lot except it's important to understand that Google is backing up these files in their cloud infrastructure. Trying to add some type of extra backup mechanism will not be helpful. Yes, it's possible to download a copy of a Google Doc (e.g., as a Word doc or text file), but that's not something you'd want to do regularly for all the Google Docs stored in your Drive account.

There certainly are benefits to the way Google Docs are handled in Drive. For example, Google Docs are not counted against your Drive storage total. This means that you could take a large volume of Microsoft Office documents, Excel spreadsheets, and PowerPoint presentations, convert them to Google Docs and save money on storage space.

Drive Sharing is Different

Of all the differences, the way Drive handles file sharing might be the most obvious to an end user. The ability to send out a link that provides anyone with the ability to view (or edit) a document in a web interface is a super useful feature. I've seen this functionality used in many different ways including a Reddit writing group called Destructive Readers that openly invites members to tear apart each others' work. Yes, it's possible to do this with other file storage services, but Drive is the most popular solution for the group because it's easy to allow everyone to leave inline comments.

An important point to remember about Drive sharing is that anyone who has "Edit" access to folders can change the parent folder--including the ability to "move" folders or files to folders that don't have the same sharing rights. In other words, it's easy for a user to make files and folder inaccessible to other people by simply reorganizing the folders (i.e., changing the labels).

For more information about Drive sharing, check out this two-part video about Google Drive sharing.
Update: The Google Apps Show just added an episode about Google Drive syncing and sharing.

Google Drive has certainly grown up quickly. In a recent report about online storage solutions, Forrester praised Google Drive for product strategy and performance, "Google Drive delivers high-performing file sharing and storage services, even for very large formats, such as video, that cross into the hundreds of gigabytes. Most other EFSS providers have file-size limits that could impede advanced requirements."

Providing online storage features such as advanced search, access from anywhere on any device, having just one copy of each file, and the ability to organize your files--even shared files--the way you want are integral to the success of Google Drive. However, their modern cloud-based architecture enables some functionality that works differently than other online storage solutions. These are just a few examples, what others have you found?

Wednesday, 1 June 2016

Switch to Native Browser Notifications for Google Calendar

This morning, I once again tried to click on a different tab with a Google Calendar reminder open in Chrome. And I'm not alone, many people have been confused by the fact that these pop-ups act like Windows modal dialogs. In other words, you can't do anything in Chrome until you dismiss the reminder. Fortunately, things are changing. Google has announced that a feature in their labs is now available to anyone.

They are calling the new native browser notifications "gentle reminders" which is a pretty good name considering how abrupt the old version can seem. If you enable this new feature, you'll be able to ignore a calendar reminder and freely click around Chrome (as you can see in their animation, you can also write emails, etc). Some people would prefer to not have that freedom--I'm still on the fence.

From the gentle reminders announcement, "This new feature is opt-in and automatically activated for existing lab users. In order to opt-in, simply enable browser notifications from the settings menu in Google Calendar."