Reimagining lead generation for service industries
A few years back I was the CTO and architect for a lead generation project which launched around 2012 after a couple of years of development under the wings of a tiny venture incubator group named Du Molin & Du Molin.
In this article I’ll discuss what we built and how we overcame some of the significant challenges of the day bringing this kind of platform to life. In the end, we found product market fit in a space nobody else had yet figured out how to automate, allowing us to scale up in a space thought to be impossible.
The Problem to Solve
In a space akin to e-commerce giants like Shopify and SquareSpace, how would you reinvent lead generation at scale for businesses who sell services rather than products (dentists, doctors, plumbers, lawyers, finance, etc.)? And importantly, how can this be implemented as a fully automated self-service platform that intrinsically understands each supported industry with pre-designed templates and copious amounts of carefully crafted content — because busy professionals don’t have time to write dozens of articles or learn how to create scores of search engine optimized pages.
Moreover, can this entire process be made as simple as providing a credit card and answering a few questions through a guided onboarding wizard which prompts for things like office hours, services offered, professional credentials, a few photos, and the like? And can we push out millions of pages without upsetting Google?
That was our challenge, and the product was LeadFire — circa 2010.
Our goal was to design LeadFire to attract new high-value sales prospects like a giant Internet magnet.
See our press:
Venture Capital Group Unveils Automated Lead Generation Platform
Proof of Concept — Starting with Dentists
Although our solution was designed so it could be customized for nearly any service vertical, our CEO decided we would first target dentists and prove out the concept. This made a lot of sense for us starting out because it was a high value market and the parent company already had a lot of experience and domain knowledge in that industry through one of its other portfolio companies.
Admittedly, I personally didn’t know anything at all about dentists or dental marketing when I was recruited to lead this project. But I knew platforms. My job at first was to just listen — to talk to all the people in the portfolio company with the domain knowledge and learn how they’d been doing things manually up until then, and crucially, learn what their main pain points were and where things were getting bogged down and causing customer churn.
Parallel to my rapid indoctrination into the world of dental marketing and online lead generation, I also needed to analyze the problem space from the perspective of other service industries. What pieces and logic paths would be common? How could we build an engine that was industry agnostic at the core, yet easily customized at the outer layers for each market we chose to target? And how could we economically scale to over 100 key verticals?
The thinking was that our customer control panel experience should feel natural and include inputs specific to each supported profession. This would help provide some keen differentiation from competitors’ generic solutions.
And then there was the biggest issue of all — given our intent to automatically generate dozens of beautiful pages per website based on clients’ inputs, how could we do this en mass without everyone’s websites looking roughly the same or having nearly identical content?
Our customers didn’t want to create content or spend much time tending to their online presence, yet they all wanted to be unique and stand out from the crowd. To be successful, we would need to solve this problem in a grand way.
Indeed, this was the main stumbling point keeping the big household names from stepping into this space — because hand crafted websites simply couldn’t scale to tens of thousands of customers. Our solution resulted in a patent.
BTW — if you might have already read my companion article on getting a million pages ranked in Google, note that the Inside Stores project came after this one and was inspired by the SEO insights I learned on LeadFire; because here too, we needed to figure out how to get over a million pages ranked using techniques Google wouldn’t discount as being too similar or spammy.
Finally, before diving into the product we created, it might be beneficial to watch the short video below for a high-level overview of the platform.
To better understand what we were looking to accomplish and why it was fraught with challenges, it’s important to understand the state of things at the time. Online lead generation was gaining traction (hence our interest), but creating search engine friendly websites with enough content to capture meaningful amounts of traffic and convert that to phone or form inquiries was a seriously tedious endeavor that required businesses to outsource this work to expensive agencies — frequently costing upwards of $10,000 to get up and running.
Drag and drop website builders like SquareSpace were still in their infancy and WordPress was the predominant technology of the day. Sites were designed using tedious HTML coding after you bought a domain name, set up DNS, web servers, email and newsletter systems, etc. And you still needed to author lots of unique content to get enough pages into Google to have any kind of impact. And after building all this, it needed to be maintained.
Still, there was a lot more to lead generation than setting up a home page on WordPress — things like delivering leads instantly by SMS/text, tracking, keeping up with Google’s algorithm changes and regulatory compliance, and getting listed in key online directories, to name a few.
Leave no doubt, customers wanted to skip all this boring stuff. They wanted a magic pill. I’m busy, just charge my credit card and do all that stuff for me. And for that, they were willing to pay a few hundred dollars a month by subscription, or otherwise just pay a reasonable fee for each delivered lead.
Here is where I spill that the genesis for LeadFire was that the other portfolio company owned by Du Molin and Du Molin was an agency doing all of the above and more by hand. They had nearly 50 people working with text editors to build and maintain a few thousand dental websites (marketing portals) as they received instructions and changes from clients by phone, fax and email. It was profoundly labor intensive and literally impossible to scale up.
Just imagine the effort needed to update a few thousand websites whenever Google made a change that affected the sites’ rankings across the board. The word “painful” doesn’t even come close to describing the situation, and back then, this was a frequent occurrence as Google tuned its algorithms.
My mission was to familiarize myself with how all of this was being done manually, and then figure out how to automate every bit of it. And while I was at it, think way beyond dental marketing — think global SaaS platform.
The LeadFire platform was designed to attract new high-value sales prospects like a giant Internet magnet.
LeadFire was a one-stop all-inclusive solution for generating real-time sales leads through Internet-centric direct marketing venues. LeadFire was delivered as a cloud-based hosted service which could be completely self-provisioned and controlled by ordinary people using a standard web browser.
One Goal — Delivering Leads
LeadFire was a lead-generation technology that primarily used websites to attract local traffic from the search engines. This was sometimes a point of confusion for people who might have mistakenly thought we were in the website-building business — we weren’t. In fact, a typical LeadFire customer might have already had a well-designed professional website, but then they want more leads — lots of them.
LeadFire Sought Out New Customers
LeadFire put businesses into direct one-on-one contact with serious sales prospects through phone calls, emails, faxes, chat, SMS and other forms of personal communication which were initiated by the prospects.
LeadFire Empowered Non-Technical Marketers
The primary goal of this technology was to empower non-technical business owners in vertical markets with the ability to harness the vast potential of the Internet to generate large volumes of organic sales leads generated by geo-targeted local search.
LeadFire Leveraged Organic Local Search
A pivotal element of the LeadFire technology was that it could be highly tailored for geo-targeted local services such as banking, insurance, medical-dental, legal, accounting, plumbing and virtually any local service business where physical inventory is not a central focus (we were not like Shopify).
LeadFire was Completely Automated
The cornerstone of our technology was that it was 100% automated using a patented process designed to scale out to thousands of users in any of the supported industries. LeadFire solved the problem of creating distinct online marketing identities for competing businesses in close geographic proximity to one another through a creative approach to generating content.
LeadFire Could Instantly Generate 1000’s of Websites
LeadFire could also be used to generate thousands of distinctive websites for use by sales reps, franchisees or local branches of large organizations. Top management could maintain strict control over common website elements and legal or regulatory notices, while allowing individual branches or franchise owners to customize their websites within designated parameters.
LeadFire Went to Work within Minutes
LeadFire was designed to get businesses up and running quickly and expressly bypasses any need to author custom content or learn any kind of Internet or website publishing technology. Marketers could begin generating leads within just a few minutes of subscribing to the service. All that was needed to get started was to input some basic business profile information — and a credit card.
LeadFire Opened New Doors
LeadFire’s unique approach paved the way for a new level of scale and economy in lead generation.
LeadFire was the first (and possibly the only) product to automatically generate thousands of websites at Internet scale with prefabricated content while solving the problems of SEO, personalization and duplicate content. I’ve yet to see anyone offer a solution which achieved what we did with LeadFire.
Imagine being able to have multiple geo-targeted websites like those shown below up and running and ready to generate leads within minutes after entering just a few basic pieces of information about your company — all without needing to do any website design. These websites had between 50 and 200 pages each. It was wonderfully effortless, and magical.
Technology Highlights
LeadFire was a highly-automated system which leveraged numerous advanced technologies to transform the process of lead generation.
- Real-time sales lead generator
- Auto-generated web marketing portals
- Geo-targeted local search marketing
- Patented intelligent content generation
- Dynamic Search Engine Optimization (SEO)
- Market intelligence analytics
- Video-enabled enrollment and activation
- Rich industry-centric database for drop-in content
- User-friendly online control panel
LeadFire stood alone. The professional services industry had essentially been left untouched and was generally regarded as automation-challenged when it came to building websites. And the reason was simple — nobody had been able to reasonably solve the duplicate content, SEO and personalization problems at massive scale.
LeadFire was born to scale!
Key Product Features
LeadFire included dozens of features which were carefully designed to help businesses get started quickly and begin generating new leads.
- Integrated national and local directory marketing
- Extreme design personalization
- Integrated social networking
- Smart mobile marketing with fully interactive mobile websites
- Comprehensive multiple-level video implementation
- Call tracking, recording and monitoring
- Integrated promotion and marketing package
- Total lead tracking and analysis
- Facebook integration
- Automated regulatory compliance system
- Instant lead delivery via email, fax, SMS and voice
- Effortless domain management
- Integrated video help and training tutorials
- And much more…
If you want to catch more fish, use a net, not a pole. This was a fundamental thesis behind the design of LeadFire. To us, casting a wide net meant getting as many pages as possible into Google and service directories for each client.
For service industries, Google is very much aware of geographical frictions when consumers search for local professionals. They know you’ll drive only just so far for a dry cleaner, but likely a bit farther for a lawyer.
We took this into account in our design by having clients tell us all the cities, counties, zip codes and regions (local lingo, like East Bay, South Side) they serviced. Our marketing portals would then include extra pages specifically targeting each of these different ways people tend to search — because otherwise, clients might lose out on nearby prospects because they lived just across the town line, or searched using some local geographic reference.
Service Directories
We also used clients’ location information to get them listed in multiple popular service directories — some of which we published ourselves for greater control, like the one for cosmetic dentistry shown below.
Marketing Portals, not Websites
We called the generated websites marketing portals to help eliminate confusion between what we were doing and clients’ existing corporate websites — to help them understand that our lead generation services worked in parallel with whatever they might have for a home page, and that LeadFire wasn’t intended to replace their main corporate site if they had one. And importantly, we wanted to make sure clients understood LeadFire was never intended to be a website authoring tool akin to SquareSpace or WIX.
Our onboarding wizard would gather information like what services were offered, professional credentials, branch locations, and office hours — all simple questions that would be known off the top of your head.
Crucially, we used this information to auto-generate our portals (websites); sometimes as many as five for a single practitioner if they provided a variety of distinct services. Each marketing portal had between 50 and 200 pages that were instantly generated based on clients’ onboarding inputs. This data could later be updated at any time using our client control panel, and the entire set of portals, pages and services directories would instantly regenerate.
Unique to our approach is that we only asked clients for core information about their businesses — we never required them to author content or design pages. Those concepts just didn’t exist in our world. We did, however, allow clients to choose templates, color schemes, and select new photos from our vast collection of stock pre-sized artwork (or upload their own) if so inclined; but most were happy with the randomly chosen defaults intended to ensure no two portals ever looked the same.
Every image in our stock collection was tagged with essential meta data (aspect ratios, kind of content, compatible locations in our templates, etc.) so that whenever clients wished to swap out a photo, we could instantly present dozens of alternatives that were both appropriate and sized to fit for the slot.
Our content database contained copy and photos for common products, professional associations and service descriptions, to name a few. Clients only needed to check a few boxes for what they offered and our site generator took over from there and spun up scores of professionally looking pages which included text and images for the products they offered, logos and descriptions of the organizations they belonged to, and dozens of pre-written articles peppered with search and replace fields to appear as though clients had written the articles themselves.
Everything clients needed was just a checkbox away, because we knew the vertical inside and out and had painstakingly curated every imaginable kind of content that might be desirable for lead generation. That was the key.
Crucial to our mission was that clients should be able to sign up and get started without ever needing to speak to a salesperson or customer service. Of course, we had a team ready to help, but the concept was that a client should be able to show up at midnight, enter a few details along with their credit card, and they’d be up and running right then and there.
And naturally, if something needed to be changed, clients could come back later and update anything they wanted, or fill in anything they skipped during the first pass.
New customers were presented with an enrollment wizard which walked them through entering all the critical pieces of information we needed to spin up one or more portals. They could even preview what their portals would look like on a temporary subdomain without needing to provide a credit card — we wanted to show them how little effort was needed to create a site with dozens of pages that were magically generated and optimized from their inputs. We only asked for a credit card when they approved and hit publish.
Once clients were ready to go live, our back end logic would automatically purchase domain names, configure DNS, provision proxy phone numbers, email, newsletters, push out to service directories, and anything else needed to fire up their portals. This all happened in about a minute.
Notably, because we were building marketing portals and not custom corporate home pages, we decided (after some debate) that we wouldn’t let clients pick their domain names — we’d do that automatically using heuristics we knew to work. Importantly, this removed a bit of emotional friction because people would get hung up over picking names and it interrupted the flow of things. Moreover, it really wasn’t all that important for pure organic search. Of course, if someone truly hated a name, with the press of a button, we could grab a new one. This turned out to be a smart decision.
I previously noted that the key blocker for automating the generation of thousands of marketing portals (websites) with each having upwards of 200 pages was related to content; and specifically, duplicate content. This included copy, search engine meta data, images and visual styles.
No doubt, this is what kept the big boys out of this space.
Outside of the duplicate content problem, the other areas we automated were not necessarily groundbreaking. The other building blocks certainly took a lot of work to complete and get into a state where they all played nicely together in a unified control panel, but I must acknowledge that SquareSpace and WIX had done pretty much the same thing; albeit with much bigger teams and budgets.
Duplicate content problems generally come in two flavors: getting rejected or penalized by Google, and the emotional letdown when a client spots a competitor’s website looking just like theirs with nearly the same template, color schemes, images and text.
In the era of custom websites produced by agencies for thousands of dollars, this wasn’t really a problem in real life unless an agency got lazy and sold cloned sites to multiple competitors. But in a fully automated world, this was truly a brick wall that would doom the project unless solved.
Duplicate Content — Google
It’s pretty common knowledge that Google has tons of rules that must be religiously adhered to in order to rank well; or even at all. They guard their search results carefully in effort to keep out spammy sites as well as ones that look like near clones of each other. For sure, Google is on the lookout for telltale signs of auto-generated pages, because that would be an obvious trigger when looking for pages to penalize or just drop completely.
But as sellers of lead generation services, we had still another big problem to overcome — everyone wants to be first in Google, or at the very least, be listed on the first page. To be successful as a business, our generated sites would need to outperform (out rank) nearly every hand-crafted website meticulously designed by experienced agencies; and we’d need to do this in every city across the country.
Consider how many dentists there are in Los Angeles, New York City, or Chicago. They can’t all be first in Google for their respective markets.
And with our early success came another problem — we’d get half a dozen clients ranked on the first page of Google and the listing titles were identical, so it looked like a text wall of sort. It wasn’t a feel-good user experience. But after first being amused at our knack for picking good titles, we quickly mitigated that by injecting some extra randomization.
Our ultimate solution would need to address all of these issues.
Duplicate Content — Look and Feel
In addition to keeping Google happy, we also needed to keep our clients happy. It didn’t matter how many leads we delivered, clients insisted on their portals being visually distinct from their competitors’ portals — which again is a product of our own success, because this is a problem that only surfaces when we’ve signed up lots of clients in the same geographic area.
Not only would we need to solve this problem, but we’d need to do it in a way that didn’t require clients to do any work to achieve the desired result. Here again, we were competing against the agencies — but for way less money.
The Patented Solution — Infinite Variations
They key was homing in on a methodology which provided for nearly infinite variations between portals — thus eliminating duplicate content by design.
Patents are public knowledge, so I’m not spilling any trade secrets by discussing the solution I came up with below. Central to the secret sauce was a huge database of text, images and videos that was meticulously crafted for the needs of the target vertical — in our first case, dentistry.
To expand into another vertical, 95% of the effort would be dedicated to employing domain experts to populate a corresponding database. From there, the control panel and site generation logic remained pretty much the same across the board, short of a few minor tweaks specific to the industry.
Below is a list of the kinds of information we tracked for dentistry:
- Thousands of labeled sized-to-fit images that were interchangeable for their intended page slots in the stock templates.
- Hundreds of meta data snippets with search and replace fields for the information Google shows in its search results.
- Hundreds of articles on all facets of dentistry with search and replace fields and embedded photos to look like regular long-form blog posts.
- Thousands of paragraphs of copy with search and replace fields covering all of the core pages in our sitemaps; but importantly, crafted and tagged such that the automation logic could randomly stitch multiple paragraphs together without it looking like it was computer generated.
- Hundreds of titles and headlines with search and replace fields that were tagged similarly to the text paragraphs so the automation logic would know which headlines made sense for each situation.
- Fifty different core template layouts, with each having a bunch of sub-variations and color schemes (with colors further able to be customized using our control panel).
- Intrinsic knowledge of 26 different fields of dentistry (yes, who knew there were that many different kinds of dentists?) and the pertinent regulations in each state for what they can and can’t claim in any advertising and websites.
- Logos and copy for all known dental and social organizations across all 50 states.
- Product photos and copy for dozens of common dental products and procedures; all with search and replace fields.
- Hundreds of dental related phrases and bullet points which one might expect to find on a dental website, again, all with search and replace fields.
- Over 100 short videos with search and replace descriptions covering a wide variety of dental areas that were appropriate for embedding into websites.
- A search and replace engine which could take any headline, paragraph or phrase as input and perform fast in-memory replacements using upwards of 100 pieces of data derived from clients’ control panel inputs. These replacements were even nuanced for he/him/his/hers/she/etc because it was crucially important that the output looked completely natural.
Each time we generated a new portal for the first time, the engine would randomly pick a template, layout variation, color scheme, images, videos, titles, text and meta data. In total, there were over 300 random picks that were made for each site. After, clients could optionally override most of the picks in the control panel; but outside of maybe selecting a few different images or uploading their own, most didn’t bother.
Finally, you might remember that I previously mentioned that we regenerated the portals live in real time as clients made changes in the control panel. But of course, it would have wreaked havoc if we randomly picked new content from our database each time — so the solution was to seed a random number generator the first time and remember it along with the rest of our clients’ profile data, and use that same seed for all future rounds as clients turned features on and off which might have required drawing from the database.
And along with that, we’d keep track of any customizations clients may have made so we’d never overwrite their changes with random picks from the database.
Suffice to say, no matter how many sites we generated, none of them ever looked the same. The intrinsic randomization and smart content stitching satisfied all the Google issues and kept our clients happy. Moreover, every page of every site looked like it was crafted by hand. Nobody could ever tell that everything was computer generated. And the credit for that goes to our content team who painstakingly authored, curated, and carefully tagged all of the text and images so our generator logic could pump out great pages.
In the end, we crafted an elegant solution to a problem that initially seemed impossible to solve. Our auto-generated sites ranked right up there on the first page of Google and consistently performed just as well, if not better, than the sites created by pricey agencies.
See the patent.
It’s well known that Google uses inbound links from other peoples’ websites as a ranking signal. After all, if others link to you, you must have something good to say. Moreover, Google also takes into account if the inbound links are coming from random unrelated places, or from websites in the same industry.
But let’s be honest, how many people actually link to their dentist? Probably none, right? And this included our clients as well as their competition. At least it was a level playing field — dentists just don’t get much link love.
Fortunately, we had a secret weapon. We had thousands of dental websites with hundreds of thousands of pages. On top of that, we had many thousands more pages in our online dental service directories. This gave us the opportunity to implement a cross linking strategy between these sites in a way that provided each portal with a measured number of inbound links which slowly increased over time — because that would appear completely natural to Google. And of course, the portals linked back to our service directories to keep giving them more influence in Google as well.
With this kind of linking power available to us, we were in a unique position to give every site however many inbound links were needed to surpass the number of links held by their competitors. This was something the fancy agencies could never do when building sites one at a time.
And yet there was one more important thing we were able to do with our linking strategy, and that was to give new sites a boost by having the more mature high-ranking sites link to them early on. This took advantage of what Google refers to as link authority. Getting an inbound link from a solid mature site is considered more valuable by Google than a link from some website that went live just a little bit ago. And as the new sites matured and gained authority with Google, they in turn would help out the next batch.
Finally, we tracked our ranking performance by scanning Google on a regular basis across all of our portals using a variety of search phrases we knew to be important. This gave us the insights we needed to tune the cross linking and on page factors which worked together to determine who gets to be first on Google.
Below is a list of what we had going on under the covers. Keeping in mind that we started this project around 2010 or a bit before, this was a massive undertaking at the time for our small team.
It’s certainly interesting to look back a decade or so and see what technology choices I made when laying out the architecture. If I had to do it all over again today, I’d surely make a few different choices, but I’m still partial to C#, Visual Studio, and Microsoft’s Azure Cloud on the back end.
Our client control panel was written in Silverlight— it was all the rage at the time, but later abandoned, like Flash. Today, I’m pretty keen on React.
Page Rendering
The websites we generated used ASP.NET 4 with help from jQuery. They were all rendered from memory using a suite of smart custom controls we developed which knew how to fetch just the right data without hitting the disk, so our page loads were super fast. But today, again, I’d choose React.
It’s worth noting here that the technique I used to get our real-time HTML page generation down to a few milliseconds was to use a pseudo compilation step after every data change made in the control panel (which was a statistically rare event compared to reads on the web) and cache the result in server memory. This took about two seconds and could be shown live to customers by using the refresh button on the browser when viewing their generated sites in another tab.
In contrast, platforms like WordPress would run these steps on every page load, which is why it often takes a few seconds or more just to load the initial HTML, let alone, all the images, etc. To me, that kind of delay meant death.
Another popular approach would be to pre-render all the pages in one shot after clients finished making changes, and cache the result to disk, but this method wouldn’t allow for live previews or for dynamic content. Moreover, it would take way too long for 200 pages per portal (potentially upwards of 1,000 pages per client at five portals each) and we’d be back to hitting the disk for each page view once live. This was immediately ruled out.
Finally, there was one more critically important advantage to my approach, and that was being able to instantly regenerate hundreds of thousands of web pages whenever we needed to make a template change; for instance, when Google’s algorithms mutated, or we tweaked a feature. This was all instantaneous because the client data in memory remained the same — we only needed to reload the few files with the changes.
The net result of this creative approach to rendering was that the 50 people previously needed to work with customers and maintain the sites became just two people doing customer service. Moreover, critical updates were live pretty much instantly instead of potentially taking several weeks.
Development Platform
- Microsoft Visual Studio 2010
- Microsoft SQL Server 2008 R2
Programming Languages and Technologies
- Microsoft .Net 4
- C#, SQL
- Silverlight 4
- ASP.NET 4 / MVC
- HTML , Javascript & jQuery
- XML & JSON
Datacenter Hardware and Technologies
- ISWEST Secure Datacenter
- Microsoft Azure Cloud
- 12 Dell Servers
- 2 Kemp Load Balancers
- Windows Server 2008 R2
- SQL Server 2008 R2
- VPN
- DNS Hosting
- Email Hosting
- Newsletter Hosting
- Network Monitoring
Core Software Modules
- Customer Database
- Industry-specific Content & Articles
- Industry-specific Media Database
- Website Publishing
- Website Generator
- DNS Integration
- Domain Registrar Integration
- Video Processing
- Virtual Fax Integration
- Call Proxy and Recording
- Domain Forwarding
- Geocoding Services
- Lead Processing and Notifications
- Email Hosting Integration
- Shopping Cart / Enrollment
- Customer Control Panel
- Administration Control Panel
- SMS/Voice IVR Integration
- Website Templates and Skins (50)
- Datacenter Monitoring and Alerts
- Video-enabled Help and Tutorials
- Reporting and Analysis
- Google Ranking & Performance Monitor
- National and Local Web Directories
- Automated Newsletter System
Now days, we’re all generally familiar with products like SquareSpace, Shopify and WIX. These big hundred million dollar players greatly simplify the onboarding and maintenance issues of the past with extreme levels of automation intended to shield customers from most of the gory details; all except for the content creation part which still requires many hours of work.
And still, none of these popular platforms are actually full-featured lead-gen services with all of the pieces I described above. And none of them help with search engine optimization and other essentials needed to provide a steady dependable stream of business inquiries without constant attention.
Now consider this — we did all this and more with a team of only six. That included myself, a couple other developers, a template designer and content producers. Moreover, we did this well over a decade ago when the tooling and cloud services available to developers weren’t anywhere near as advanced as they are today.
I’ll be forever proud of my team and the great work we accomplished on what was undeniably a shoestring budget. Importantly, we solved the content problem at scale — we made the magic pill customers sought into a reality.
It was a fun project. I learned a lot that I carried forward into other ventures.
https://medium.datadriveninvestor.com/architecting-a-scalable-automated-marketing-platform-for-verticals-c8c57b2c050e