Prove it!

Last week I was talking with some people from the real estate world. At one point the people I was talking with stopped me mid sentence and asked “Wait, we keep hearing about these experiments you run to know if an idea will stick - just routing traffic to a website and checking results. We don’t have any concept of this in what we do. Can you explain that better?” In real estate many decisions are made based on sophisticated models and projections based on past experience, the market, etc., not experiments like we regularly do on real buying customers, and how could they? What are they going to do build an entire neighborhood to see what price they can get out of people so you can then rebuild that neighborhood? Here’s how great tech founders and companies experiment and hone in on what is going to work and what won't. Following that discussion I’ll share how companies from other industries could cannibalize this model to improve new product launches or find the features their clients want most or are willing to pay top dollar for.

The experiment

Like any good experiment when deciding what product to build or feature to add we initially start with an assumption or gut feeling. In our case at Unbill we thought people wanted all of their bills to not only be tracked but also be paid from a mobile app. More specifically we thought that roommates would love this and the rest of the world would like it.

After talking with a few real people in coffee shops and around town I ripped out a website over a weekend showing off the product that I planned on building and asking people to sign up for early access once the product was complete. The idea was to gauge the level of interest by routing some traffic to my website and seeing how many people signed up. I made the call to action as direct as I could and published the website late on a Saturday night.

Once the website was complete I went to Facebook and created 3-4 ads targeting college students between the ages of 19 and 24 - people I thought would be interested in our service. I gave each ad a $50 budget and published them. By the end of day Tuesday (just 4 days after creating my website) my budget was all used up and I had my results. Out of the many people that clicked into the website - I think 150-175 - I only had 4 sign ups. I didn’t feel great about my product idea, but it wasn’t a complete failure.

My co-founder and I created a mockup of the product and added it to the website. We used the same ads and budget then tried again. Eureka! We had 25 people sign up this time - 6x more than the last time. Next we added a video to show off the product a bit more and we saw an even better sign up rate. In two short we knew that if we built that product 30 out of 175 people that came to our website would be willing to sign up.

Looking back we should have known that 30 out of 175 is the worst case scenario. Since then we’ve optimized the process again and again and again. We’re currently converting way better than 30 out of 175.

With this new found knowledge we decided to go ahead and begin development work on this product with out team. We also began to generate a list of early adopters for our product since they signed up to use the product once we released it.

Other examples

We use the same approach when trying determine whether or not a new product or feature will be nice to have - this is typically easier on the web than in a mobile app given the release timelines for mobile apps, but there are some exciting new changes happening on mobile to make this easier.

A year or two into running Comfy - an apartment search platform for college students - we wanted to know whether or not we should build a roommate matching platform. We inserted a simple option into our search “find a roommate” and kept track of how many clicked on it. From there it was easy to find out how many of our customers wanted and whether or not we should consider building the functionality.

People start companies, build products, and add features without testing their hypothesis on real buyers. They expend years of effort and countless dollars. 

How can companies in other industries, like real estate, use this skill set to test out the viability of a product offering without actually building it? A company focusing on building new homes could consider creating the website for the new community far ahead of buying the land or creating the marketing materials for their product offering. After building this website they could buy keywords from Google for searches related to their product offering just like they would if they were actually ready to market their product. They could then run analytics on the website to understand the behavior of website visitors. Which were the preferred floor plans? How many people clicked to call for more information? What brand names resonated better? None of these choices have to be left to one persons opinion. The data can tell us which choice is the best.

If that same company already has a website for a community where they are building homes they could consider extensively testing that website for better conversion before moving on to the next community they build. The knowledge they gain will help them improve every related project they work on. Experimenting like this can be painful. We’ve seen our conversion rates be cut in half one day because of an experiment we ran. The next day we change it back and conversion rates go right back to normal, but it’s still painful.

Some people reading this might be saying, what do you mean? We run surveys on large sample sizes, have fine tuned models, blah, blah, blah in order to find out what features, price point, and amenities to include in our communities. Remember, experiments trump surveys every day of the week. For Statisticians experimental data is the gold mine. Surveys are one of the weakest forms of collecting data.

This approach also raises some moral dilemmas. Is it okay to show a product that may never be built to a person that is really looking to buy? What should the experience be like for customers that click dummy content that you’re testing? All these problems and more will arise and must be dealt with. Companies must find out what they’re comfortable with and what they aren’t. There have been many situations where my co-founder and I have said “it just isn’t the right way to treat our customers” when it comes to some sorts of testing, but typically there is a middle ground that gives you the data you need without the moral cost.


If you’re wondering whether or not experimenting with this could help improve your business consider the following question. What if using this model could help you push home prices by 10%? Would it be worth the experiment? What is the worst that could happen? What if a small product offering or way you present your product improves your conversion rate by 30%? We seen the optimization of our conversion rate improve by 3-4x using these methods on our website and mobile app. With new ways to get so much data we now also have the ability to test more and know more about our customers than ever before. That knowledge helps us give them the right experience and convert them better.

In the next 5-10 years the companies that are most successful will be the ones that are always testing new ways to satisfy their customers without having to go through the development cost of creating the products first (I realize this is not always possible, but it is often possible). These companies have a culture of learning everything they can about why their customers do what they do AND testing ways to get their customers to convert better, buy more, engage more, etc. In the past most good companies have understood how their customer interacts with their products or services. We’re headed into a future where the great companies will experiment with their customer base to see how they can convert customers more efficiently or increasing profits by doing more for their customers than what they’re doing today. What aren’t we doing? Or, how could we do this differently? Will be important questions these companies consider.

In a future article I’ll be discussing the role of gut and instinct in helping people make business decisions. I don’t think we should always only look at the raw numbers. There’s more to it than that, but the numbers and data that are at our finger tips today can reduce our risk in ways that weren’t possible before.