A fellow NYC-based entrepreneur, Vin Vacanti of Yipit.com, had asked twitter what others were doing in regards to metrics.  The question was whether or not people had advice about using a mixture of Google Analytics, Mix Panel, KISS Metrics or built your own metrics solution. I sent him my response in private, but he suggested I share the info with others here.  What follows is an edit of my response to both Vin and Hiten Shah of KISS Metrics.

I LOVE metrics, but I’ve realized while working on my various projects that it’s really easy to get lost in these metrics because there’s just so much to process.  Dave McClure’s famous talk on Metrics For Pirates: Acquisition Activation Retention Referral Revenue (AARRR) is a great start for understanding what is best for the various stages.

Marketing Metrics 4 Pirates (July 2010) View more presentations from Dave McClure.

I’m at the early stage of a products life cycle.  Specifically I am pre-product/market fit. This means that I have an idea I want to build a product around, but I have yet to validate whether or not my product meets the needs of the market (or any market) I am trying to address.

After doing rounds of Customer Discover & Validation I set out to build a simple product that can help me validate and better illustrate BOTH the problem and solution. Part of this was to lay the foundation of data collection to help dive my decision making process about the product.  I learned early on that vanity metrics (unique visits, page views, time on site, etc.) are really just bullshit for a product that hasn’t achieved Product/Market fit. You need to dive down into data that illustrates the behavior of your users.

I’ve specifically been focusing on conversion funnels and cohort analysis.

For conversion funnels I’ve been evaluating Google Analytics, KISS Metrics, and Mix Panel.  All of these tools are excellent, but I will say that KISS Metrics has the best visualization and gets you to what matters quickest.  Mix Panel is good too, but I specifically like the API as it relates to “events”.  Meaning it doesn’t have to rely on page views as Google Analytics does.

For cohort analysis I built a custom system that I’ll describe below.  I know Mix Panel does cohort analysis as does RJ Metrics, but I wanted to be able to crunch the data myself for some custom reports that may prove to be useful.

Data I’m Collecting

I shouldn’t really be saying this, but I try to log as much information about a user as possible.  I may not use it all, but at least I have it for when it may become useful.  I try to not cross the line of creepiness as it relates to privacy, however I do have an idea of how a user came to my site and which features they are engaging with.

When a visitor first visits my site I make note of the channel they traveled to arrive here.  If they click on a link from google, blog, email, or twitter I log the referral in the users session.  If the user activates and becomes a registered, I log that information with that account.  In the future I’m hoping to use this information to identify the channels that perform better as it relates to converting users.

Cohort Analysis

When it comes to the cohort analysis I built a very basic for my project.  It’s mostly based off of what I’ve read/heard regarding metrics from Dave McClure, Fred Wilson, Tim Ferriss, Josh Porter, Hiten Shah, David Cancel, Ash Maurya, and a band of others.

At the core of this I track three primary objects: Entities, Actions, and Change Log.

Entities are unique objects that perform actions.  Typically these are users. I can imagine they could be anything really, but for consumer web users are what I care about the most.

Actions are specific tasks an entity can perform.  For example logging in, posting a comment, inviting a friend to the service, playing a game, etc. I set this up to be pretty amorphous since I knew I would want to focus on the usage of different functionality down the line.

Change Logs are basically annotations I enter into the system to note changes I’ve made to the site.  This is so I can view if there’s a change in performance among a cohort or a set of cohorts I can get a quick view into what may have lead to that difference.

Reports are segmented into cohorts and specific action. The reports are generated on a timed interval and I can configure them to organize cohorts by when they are created (weekly, biweekly, monthly divisions).  For now I’m focusing on time rather than other attributes (region, age, gender, etc) as I don’t believe I have enough users to really make cohorts based on other data meaningful.

When the reports are generated I break it down into trailing count of distinct actions users took for that period. As the cohorts age I record their historical stats. The results look like following:

Cohort: Weekly

Action: Login

Cohort # (qty)  | Week 1 | Week 2 | Week 3 | Week 4 | Week 5 |

Cohort 5 (220) |       26%|

Cohort 4 (190) |       19%|       10%|

Cohort 3 (200) |       15%|         7%|       2%|

Cohort 2 (140) |       10%|         3%|       1%|       0%|

Cohort 1 (100) |        8%|         2%|       0%|       0%|        0%|

I create historical views of previous weeks and calculate the current week on the fly.  Once the week is over I will create a static snapshot to save on doing real time calculations.

The reason I set out to do this is that I want to test specific parts of my application.  I felt I had a somewhat decent activation rate when I pushed users to my landing page, but I had a very low return rate.   Once a user signed up, they would not come back.  They were willing to spend the time with what I was offering, but they did not have an interest in repeating their visit on their own.

I already knew there were a lot of things I wasn’t doing to get users to come back.  I wanted to test specific things such as sending out weekly summaries of their account via email or notifications, but more specifically I wanted to know the impact of those actions.  I also wanted to know if I went from a weekly to a more frequent distribution would it change the repeat visits or provoke users to unsubscribe.

Right now I’m focusing on repeat logins, however I know once I see that improving I’ll want to determine how to increase referrals from within the application as well as content creation.  That’s why I set up actions to be generic.

To create new entities and record actions I use a simple REST service.  It can be called by javascript or server side scripting.  The reporting is pretty barebones right now.  I print out a very simple text based output of the data that I have as part of my application “dashboard”.  I’ve integrated with Google Charts too, but only to show the graph of cohorts overlaid on top of each other.  Most of the time I’ll just export data to excel from mysql if I want more detailed visualization, but largely I want to see the % increase.  I haven’t spent too much time making it more robust as I want to focus on the product I’m building to test out my ideas on growth.

MUST READ

http://500hats.typepad.com/500blogs/2007/09/startup-metrics.html

http://andrewchenblog.com/2008/09/08/how-to-measure-if-users-love-your-product-using-cohorts-and-revisit-rates/

http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html

http://www.pearanalytics.com/blog/2010/understanding-metrics-that-matter-interview-with-hiten-shah/

http://www.ashmaurya.com/2009/11/how-i-am-measuring-productmarket-fit/

http://52weeksofux.com/post/646711369/cohort-analysis-measuring-engagement-over-time

Recently someone posed a question on the NextNY list about what to do after they built their product.  A lot of the initial recommendations were to begin various forms of marketing campaigns to get the word out about the product.  Press release, social media, contests, you name it… straight to customer acquisition mode.

Knowing that this was the first time the product was getting exposed to users there was no way I could agree with any of the advice being given.  But the question remains: “you just built your product (most likely in a vacuum), what do you?”

My recommendation is to get as much feedback as possible. Identify the problem you are solving, find people who have that problem, and talk to those people. Show them your product, listen to their feedback and take lots of notes.

If it looks like people understand the concept of your product and they see a use for it, then start doing some user testing.  Identify the main actions the user would take to solve their problem with your product and conduct in-person user tests to see if there are any obstacles that prevent users from using your product to solve their problem. You aren’t QAing the product, but validating the usability of the product.

If you haven’t conducted any user tests before, a great start is Steve Krug’s last three chapters of “Don’t Make Me Think”.  You can download them for FREE from his site. I’d also recommend getting his books on usability and testing.

There’s even a video of a sample usability test so you can get an idea of what to expect from a session.

I’ve conducted a bunch of user tests for my own products and found them extremely helpful and easy to do without any special equipment.  If you use a Mac I’d recommend trying out Silverback App.  It’s a simple app that records the users screen, audio dialog of the user and tester, and video of the user.

Silverback screencast from Jeremy Keith on Vimeo.

Admittedly conducting your own tests can be a bit challenging if you don’t have a flexible schedule.  Luckily there are services such as UserTesting.com that will conduct user tests for you.  I don’t recommend you start with this because you want to have an understanding of what makes a user test important.  It’s a combination of what you are testing and how it’s conducted.  Performing some of your own tests before outsourcing them will help you get more value out of the remote tests as you will know what to expect and prepare for.  Remember it’s the usability of the product you are testing, not the user.

The Pizza Cones arrive!

Boxes carefully taken out of the travel bag.

Looks promising!

K! K! Pizzacone

Yumalicious from above!

Out of the box! Hmmmm Yummmy!

AWESOME presentation from Jason Putorti

putorti:

One of my favorite scenes from the movie “A Perfect World”.

“Fact” - The difference between a threat and a fact: Saying you’re going to do something is a threat, doing it makes it a fact.

I have great respect for what Matt Mireless is doing with both his business and for the NYC Startup community, however as a Technical Founder in NYC I find some issues with his latest post “The City of Founders Without Hackers.”

In his post, Matt cites a problem that there is a lack of entrepreneurially minded technical talent in NYC. While I don’t disagree there is a proportional disconnect of technical talent and “aspiring” entrepreneurs, I would also like to argue that a core of the problem is a lack of non-technical founders who understand the process of building a technology-based product.

The majority of Non-Technical Founders I run into only have an idea.  I’d say 85% of the Non-Technical founders I meet do not offer much more value to their idea than the concept itself. As we’ve learned over the years, ideas are nothing. A majority of the magic comes from execution and luck.

For an example I recently met an aspiring non-technical founder who is running into a problem finding a technical co-founder. By day they are a personal trainer, but they have aspirations of building an online destination that connects personal trainers to clients.

I asked the personal trainer about his thoughts on aspects of his venture. Things like user acquisition, user experience, distribution, competitors, etc.  His responses were not very thought out.  Turns out this would be entrepreneur has been working on this concept for 2 years.  When I asked what sort of progress he’s made, his response was “I talked with other trainers, gyms, and have been trying to get a prototype built. Everyone loves the idea, I just need to get it built.” This project is going nowhere as he’s showing he cannot execute on his own idea.

I suggested to him that he should put together his thoughts on things that will grow his business.  Things like how he plans to acquire users, how much user acquisition is going to cost, how he plans to make money on those users, and how he plans to measure it all.  What will allow him to validate his business is working or not and if it’s not working, what are the failing points.

After reading Matt’s post and recounting my experience (which happens often at meet ups) I feel that there needs to be a better crop of non-technical founders to match the available technical talent.  Without that non-technical founders will always have a problem impressing someone who is vital to building their product.  If you are a non-technical founder you must bring something to the table that’s more than an idea.  Bring something solid to show you are thoughtful and can articulate a clear vision for your business.  If you can do that I can assure you that finding technical talent will be much easier.

I shouldn’t admit this, but I rarely get up before my alarm rings in the morning.  The snooze button and I play out a familiar battle that I typically win.

However today I actually woke up a whole 20 minutes before my alarm went off. I figure I can take the next 15 minutes of time to get some thoughts down… About anything.

Last night I had a good discussion about web application growth. I’ve been learning that after building an app there are a lot more things to consider than new functionality, beautiful design, and crafty code.  Building an app is really only 20% of building a business (and 20% may be a generous # since I am a developer and product person). There are so many other things you also need to worry about.

The area that really geeks me out lately is user acquisition and revenue. I’ve been spending a lot of time thinking about that stuff. One of my recent thoughts is figuring out the sweet spot of Revenue Per User (RPU).

Say I want to earn $10k per month from my web service.  If I have 20k users that means my RPU must be $6 for the year (at $0.50 per month). For most consumer web products that’s a lot! That means I need to find something I can offer each of my users that returns $6 to me.

One way I am looking at this is by considering the variables in a simplified (and admittedly naive) equation:

# of User x RPU = Total Revenue

RPU = Total Revenue / # of Users

This leaves me feeling that I do not have direct control over the RPU as it is a function of my desired Total Revenue and # of Users.  So what can I control? Really the only thing I can control is what my desired Total Revenue is. I can lower it and thus the RPU becomes more manageable… Or I can do the more challenging thing and find new users to dilute the RPU.

If I increase my # of Users while keeping my Total Revenue the same I can distribute the RPU across those users. For example if I raise my user base to 40k users then my RPU is $3 for the year (at $0.25 per month).

This seems more ideal to me as the lower I get my RPU the more obtainable options I have for creating value from my site. For example placing ads or offers on the site becomes realistic if I lower my RPU to $0.05 to $0.10. However that means I need to have 2.4 to 1.2 million users (respectively)… Yikes!

So this becomes a play on volume of users. How to I find my new users and grow to grow my service? More on that later as my 15 minutes are up!  

I’m an “independent” developer who loves executing my own ideas.  I typically will scope out a minimal feature set and run full force to build something in a short period of time.  Once I’m done, it’s very natural to feel the need to “get it out there”.

When I worked on my original vision of Cupid’s Lab in 2006 (an online matchmaking/introduction system), I was very eager to get the word out to the world.  I was able to grow the service to a few hundred users through my own network of friends, but I was really eager to expand further.  I wanted to get the word out there.

I worked with a small boutique PR firm to write a press and shot it out through PR Web.  Over the next few days I got a few reviews & write-ups from technology related blogs like Mashable.

At the time I felt the release worked well.  Within a few weeks I had grown to a few thousand users and things seemed great, but it wasn’t long until the buzz died off.  With that I also felt the urge to want and send out another release to try and drum up some more buzz. Thankfully I resisted the urge. The process was exhilarating and intoxicating, but I realized that the results were short lived and I wasn’t doing anything sustain the momentum within the application.

Had I known then what I now know, I would have held off on the initial press release.  I feel very strongly that you need to consider your press very strategically.  You may want to get the word out there, but be VERY meaningful about how you intend to execute your press and make sure that you meticulously measure the results.

For me with Plan.FM I’m working towards building a press worthy product, but at this stage I know I’m not “ready”.

When I am ready I need to answer the question: “What is the reason to do a press release?”

As a very new product/company, there are two primary reasons that I will focus on.  Those reasons are User Acquisition and Financing Round. There are many other reasons, but realistically these will be the first two I will focus on.

Before I send out a release that focuses is on User Acquisition I need to make sure my product optimally converts anonymous visitors into registered users and successfully engages users to use the application on a consistent basis. Furthermore I need to make sure I can dilute my user acquisition costs by making sure that my users have the ability to refer other users to the service.  All of these things should be in place before any release goes out.

When it comes to starting a conversation about a Financing Round, I feel a press release can help a lot.  However you need to do it very carefully.

Let’s take one of the companies I’ve been watching in the event space, Plancast.com. (DISCLOSURE: although they seem like a competitor to Plan.FM, I feel that Plan.FM is actually a compliment to their service).  They initiated their press on Tech Crunch with a glowing review (it doesn’t hurt to be a Tech Crunch Alumni either).  You can read their press here: http://techcrunch.com/2009/11/30/plancast/

One thing to note is a the following paragraph:

The Facebook integration is key for Plancast. The service was borne out of Facebook’s fbFund ‘09 initial winners (its codename was Magellen at that point), and they’ve received a micro-seed round from that so far. The next step is to raise a proper seed round, Hendrickson says.

Four months later Plancast.com closed an $800k seed round with a stellar group of angel investors (http://techcrunch.com/2010/03/08/plancast-funding/).

The press was strategically initiated to handle both User Acquisition and Financing Round in a single shot.  And it worked.

So if you’re eager to get your product out there and spread the word, think first about how you can benefit from the release.  Do not do it just to get your product’s name out there.  Do it purposefully, make it meaningful, and make sure you are set up measure the results.