This article is a guest post by Steve Nakata of The Pedowitz Group. It was entered into The Hackies essay contest for the upcoming MarTech conference. Like it? You can register your vote in the contest by sharing it on social media, especially LinkedIn, Facebook, and Twitter.
Is your marketing team having challenges in seamlessly planning and executing campaigns with the tools they use? Does data automatically flow from one system to another, or does your team move data by manually exporting and importing spreadsheets?
In supporting your company’s business objectives, you may realize that your existing martech stack doesn’t meet your current marketing needs due to one or more systems:
- Not having the required capabilities
- Having some but not all required capabilities
- Having the capabilities but are difficult to implement and/or manage
- Not being sufficiently integrated with each other
Whether you need to modify your existing toolset, acquire new technologies or a combination of both, you need to ensure all the platforms are integrated optimally to maximize the effectiveness of your martech stack. While you don’t need to know how to design or build the integrations, you should understand some basics to ensure that you have a cohesive infrastructure rather than a house of cards or a set of application siloes.
Out-of-the-box integration: caveat emptor
Many martec vendors promote that they have built-in integrations to one or more systems. Be sure you understand the use cases that these integrations support.
Do they meet a robust set of use cases or only a few? Do these use cases align with yours, or do you have unique scenarios that may not be possible or negatively impacted with the native integration as-is? Can their built-in integration be customized to support your needs? Will it be able to move all your required data to and from the target system and perform necessary functions?
I recently reviewed the integration capabilities of an online survey platform to an MAP. While the vendor was listed as a partner on the MAP vendor’s web site and claimed having native integration support, I quickly realized that the integration was limited to simply passing a link to a survey for a specific set of recipients so that an email with that link can be sent to them. The integration didn’t support the ability to pass the response data from a completed survey to the MAP, which could provide much more value to marketing.
Another important factor to consider is the extent to which your system has been customized. We have consulted with clients that use Salesforce.com (SFDC) for their CRM, and we were tasked with integrating their newly-purchased marketing automation platform (MAP) with it.
Some clients had lead and opportunity management processes that required them to customize the data models and functionality in their CRMs. This created difficulty in using the MAP’s native SFDC integration as-is.
For some clients, the native integration was customized (e.g., via Marketo sync filter or Eloqua program) to meet their requirements. Others made changes to their processes to align with the use cases supported by the native integration. There also were clients where the native integration simply wouldn’t work due to its adaptability or their limitations in business process changes, and they had to use an integration platform.
Important integration terminology and concepts
Before getting into more integration specifics, let’s cover some terminology and concepts at a high level. There are plenty of articles and discussions on the Internet on these, so if you want more details, just google for them.
Manual versus automated integration
Your team may already be performing manual integration by exporting a list of attendees from your event management platform and uploading them into your MAP. They may be doing the same task between the MAP and CRM.
Automated integration replaces the human factor, enabling the ability to schedule data syncs. Automated integrations can also store field mappings, transform data and allow the transfer of data at scale, eliminating the risk of human error.
Flat-file and web services integration
Flat-file integration involves using flat files (in CSV or text format) to transfer data. Some systems have built-in functions to automatically export flat files to a server, and some can also import files. Note that this approach only supports the transfer of data, nothing more. All business logic is handled within the source and target systems.
Web services integration leverages functionality exposed by a system to the Internet for other systems to interact with it. In addition to transferring data, web services can trigger actions within the system, such as enabling another system to instruct a MAP to run a batch email campaign, without someone having to log into the MAP and performing that task. Web services can open many possibilities for interactions between systems. However, customers are limited to the capabilities offered by each vendor.
With either approach, each system must already have the capability built-in. You can’t add the function directly into the platform.
REST versus SOAP API
These are how one system communicates with another over the Internet for its web services. Without getting into the details or the arguments of which is better (again, just google for more information), let’s just say that REST is newer than SOAP, and the trend favors using REST over SOAP. For this reason, most vendors are expanding their REST web services and planning to deprecate their SOAP web services (if they already had one, of course).
Two other terms that go together with these are JSON (for REST) and XML (for SOAP). These are the data formats used in communicating with these web services.
Lack of integration standards
There are no standards for a minimum set of operations that are exposed by a system, even for well-established categories such as marketing automation. Each system’s API is often unique in almost every way.
If you were to compare the API documentation between Marketo and Eloqua (and I have, extensively and repeatedly), you would see that they differ significantly. Although they offer some similar operations (such as retrieving person data), even those functions differ in structure and technique.
While both Eloqua and Marketo have REST web services, that’s where the similarities end. A CRM integration to Marketo looks very different than a CRM integration to Eloqua due to the differences and capabilities of their APIs.
Going custom: key considerations and options
Let’s assume that a native integration doesn’t exist or can’t be customized to meet your needs. Before diving in and building a custom integration, your team needs to consider several factors. Integration design is a balancing act that accounts for the following:
Timeliness and volume of the data sync
System capabilities often dictate how much data can flow between systems and how often. You need to understand the data availability requirements and anticipated record volumes. This will help determine the best method for integration, whether data flows need to be transactional, periodic or a mix.
For example, an online form submission where the respondent indicated that they want to be contacted about a product should be captured by the MAP then immediately passed to the CRM for follow-up by the appropriate salesperson. On the other hand, a digest of daily activity might be compiled and transferred as part of a nightly scheduled integration.
To handle volume, you must estimate how much data needs to be transferred at any given point in time. Account for scenarios such as a dramatic spike in form submissions due to a new product or service launch or an event list with a significant number of attendees that needs to be passed to the CRM for immediate follow-up. Sending one record at a time (as is the case with a transactional data sync) may not be as efficient as sending in a batch (periodic data sync) to ensure a timely action by the system receiving the data.
Think of transactional versus periodic data sync as analogous to a triggered versus scheduled email campaign. You want certain emails to be sent to the recipients immediately, such as a thank you email when someone submits an online form. Other emails, such as newsletters, can go out en masse at a scheduled day and time. Similarly, certain data should get to another system as soon as possible while other data can be passed at regular intervals.
Technical readiness for integration
In addition to understanding how much data can flow and how often, you will need to determine system capabilities to meet your needs. Flat-file integration functionality can enable you to achieve an automated data transfer with a minimum of time and effort. Web services could provide more options for the integration, but your team will need to determine if the REST or SOAP API is robust enough to meet your use cases, now and into the future.
For example, one MAP system supports flat-file data transfers while another has no built-in capabilities at all. Some offer webhook capabilities, meaning they can send data transactionally to other systems without the need for an integration middleware.
However, just because a system has a feature doesn’t necessarily mean it will meet your requirements. For example, one MAP’s file-transfer functionality is limited to one-hour intervals, meaning it can’t transfer more than 24 files of a given dataset in a day. Most web services APIs limit the number of calls per day (without paying additional fees) as well as the number of records within each call. Some have data insertion and extraction limitations that may require a different integration process than another system, even though it may be in the same functional category such as MAP.
Team readiness: skills and knowledge
In addition to having the technical tools, your company needs to have people with the skills and experience in designing, building and maintaining integrations between disparate marketing systems. Integrations often involve more than mapping data and consuming APIs. They require architected data flows, and they should be able to easily recover from system downtime or errors.
Your team should be able to understand marketing objectives and define business use cases, then design interfaces that meet marketing and technical requirements. They should also have knowledge of data integration platforms and techniques. They should be able to build integrations that can be rapidly reconfigured to meet shifting business requirements.
If you find deficiencies with your resources, you have options. Depending on your budget and timeline, you may train current employees and/or acquire new talent. You can also engage MarTech integration consultants for design advice or to build the integration for you.
Budget and time to value
Budget is a significant factor to the integration design and implementation. You must account for costs other than the integration technology itself, such as training or acquiring new personnel. You may need to do some process re-engineering to better align with system functionality.
An important cost that is often overlooked is data cleanse and normalization. Before any integration can go-live, an initial data load must be performed to ensure the integrated systems start off with the same sets of data. As part of this exercise, the existing data should be cleansed and normalized before the initial data load has taken place. To ensure high data quality is maintained, a data management strategy should be defined and implemented across systems.
Stakeholders may be clamoring to get data flowing sooner than later to ensure they can meet their performance deadlines. If time to value is critical, you should consider hiring outside consultants, as they can quickly implement higher quality integrations given their experiences on a variety of platforms for a broad range of business cases. This may save both time and money compared to a trial-and-error approach.
Integrating in a cloud-based world
Homegrown solutions (“dark integrations” using programming or scripting languages such as Java, C#, PHP or Perl) are things of the past, especially in integrating Software-as-a-Service (SaaS) systems. Unless you’re an integration software vendor, it’s not your core competency; it’s an opportunity cost you can’t afford. Without a significant investment of time and effort, homegrown integrations won’t have the bells and whistles of a vendor solution, such as:
- Operational functions of monitoring, alerting and retries
- Implementation functions of development, test, deployment and maintenance (including monitoring and rollback)
- User interface for managing field mappings or system configurations
Use a commercial integration platform, especially an integration-Platform-as-a-Service (iPaaS), whenever possible. Many provide built-in connectors that abstract away the complexities of interacting with the API of the target system, such as maintaining active communication sessions and making multiple calls to retrieve all relevant data due to API volume restrictions.
Some important considerations in selecting an integration platform:
- Look for point-and-click configuration versus code development for commonly-used functions such as data transformations and field mapping.
- Find out what you can do with the connectors for your platforms. Don’t rely solely on what the vendors tell you what you can do.
- There likely will be gaps between the platform API methods and those made available with the connectors.
- Not all connectors are built alike; some vendors support more API methods than others for a given platform.
- Be wary of vendors that tell you they can build a missing connector right away — this can end up taking more time and effort than you had planned.
- Avoid the Enterprise Service Bus (ESB) mentality. Its time has passed, and it’s not relevant in a SaaS world. (See Maneesh Joshi’s article, Why Buses Don’t Fly in the Cloud: Thoughts on ESBs, for a detailed discussion on the topic.)
Getting help
The exponential growth of marketing tools in the past few years has brought on challenges in connecting them together, especially considering the lack of integration standards. Unless your team is intimately familiar with the integration techniques of martec systems (and, specifically, SaaS-based applications), you may want to consider an experienced partner to assist and guide you in this process.
What did you think of this article as an entry in The Hackies essay contest for the upcoming MarTech conference? If you liked it, you can register your vote in the contest by sharing it on LinkedIn, Facebook, and Twitter.
