A few days ago I was giving a presentation from my iPad on a projector, the only problem was I had lost the iPad-to-VGA out cable I purchased for my presentation. I had every intent of going to the Apple store during lunch, however as the day progressed it looked less and less likely that I was even going to be able to leave the building for more than 10 minutes.
I had Tweetie open when I see the following tweet come across my screen:
For those who don’t know, Tekserve is an alternative Apple retailer in NYC and Jonathan Wegener (@jwegener) is an awesome techie from the NYC Startup community (who was going to be attending the event).
I sent Jonathan a private tweet asking him if he could do me a favor and pick up the cable I needed. A few minutes later he responded with a confirmation that he purchased the cable and he’d charge me via Venmo.
Seconds later I receive a text from Venmo with a charge for the iPad Cable and I promptly confirm the charges.
Each month I organize and moderate an event called the Feedback Forum. Here is a blurb from the Eventbrite page:
The Feedback Forum is a monthly gathering of NYC entrepreneurs and technology-based startups or businesses in the NY metro area. We focus on a specific company and discuss critical operational challenges in an open environment that creates a learning opportunity which can benefit the broader NYC entrepreneur community.
What I do is find companies that want feedback on their business and then bring other entrepreneurs (seasoned, aspiring, curious, anyone really) to discuss the business we are focusing on. I limit the # of attendees to 20 people so that there is a productive discussion.
It’s a two hour event that’s broken up into segments. The first segment is a round of brief introductions of the attendees and then a round of discussion to get familiar with the business. The rest of the event is broken up into segments to discuss the challenges the business may be facing.
Last night I was a little self serving and held a Feedback Forum for Plan.FM. For the event Min and I put together a deck to help describe the service and highlight some items we wanted to discuss:
The discussion was very open. Here is some of what we discussed and what came out of it:
Usability - The online event space is fragmented with over 70 different event scheduling, personal calendars, and social calendar services. Our mission is to separate the signal from the noise within this fragmentation. What are some ways in which Plan.fm can be an indispensable compliment to a users preferred calendar/scheduling system?
There was a lot of focus on making the calendar simpler, but without replacing a users existing calendar. Users prefer Google Calendar, iCal, or Outlook and there is no need to create an alternative calendaring system. The ability for Plan.FM to synchronize invites and rsvps to a users existing calendar is what people were most interested in discussing.
Simplifying the functionality was another area that the group had directed attention to. There’s a lot of long term vision on how a service like Plan.FM can be a social tool for attendees of events or even coordinating with other people within your social and professional networks, but a lot of that gets lost in the current offering.
Strategy – Does it make sense to focus on certain vertical networks as a growth strategy? Is there value in focusing on networks like Linked In that allows users to find events their colleagues, competitors, and potential clients are attending? Focusing on professionals may also opens up more possibilities for providing lead generation opportunities for industry conferences. Are there any other verticals that we should be thinking about?
“Do one thing, do it well” seemed to be the theme when there was a discussion about strategy. Reaching that 20k user mark will surface who the real target audience is and how they are using the product. This will then help define where to focus user acquisition initiatives.
Branding was briefly discussed on whether or not Plan.FM was the right name for the product. However a larger consensus was that the focus should be on the product right now, not the branding.
Revenue – The focus of revenue has been on sponsorships in emails, sponsored event listings, and affiliate programs (fandango, live nation, eventbrite, open table, stub hub). We’d love to hear the experiences other entrepreneurs had with various similar methods as well as discuss other opportunities.
There were clearly ideas an opportunities about how to generate revenue, however it really depends on the direction the strategy taken. I illustrated how focusing on larger and more expensive events would be more interesting than smaller ticket items like movie tickets.
In general the response to revenue was to focus on isolating the value and focusing on that before focusing on monetization.
More detailed post to come, but I recently completed my migration from AWS to Rackspace Cloud. Here’s a quick overview of what I built out for Plan.FM. It’s a pretty basic setup, but I’d be interested in comments from others on architecting a scalable infrastructure.
One of my Founder Institute buddies sent out an email asking which blogs we read as it relates to entrepreneurship and VC. There is a much larger list, but these are the ones I learned the most from. I also probably don’t have some that I should be reading. What blogs do you read/recommend?
I like to group conversion funnels and cohort analysis into what I call “Positive Influence Metrics.”This is when we look at the reports generated from the data we collect about users to determine the whether they are taking the desired action we want them to.Hence we look for repeat positive signals showing users taking the actions we want them to.These metrics and reports show us whether or not our hypothesis is correct.
However there is something I call “Negative Influence Metrics.” I use it for measuring the actions I DO NOT want the users to take.
This isn’t the same as looking for negative trends in cohort analysis or conversion funnels. You may want to consider low conversions, lack of retention among cohorts, high churn, etc as “Negative Influence Metrics”, but what we really want to do is identify specific actions that a user should not take if they are happy using our product.
For example I want to know when a user disables an account, opts out of email notifications, or revokes permissions from facebook or twitter.Collecting these actions is important, but the real value comes from when you, as a company, take action on being alerted that someone has taken a negative action.
After being alerted that a user took an action I did not want them to, I want to know the reasons why they did.Was it because they were receiving too many emails, were they skeptical about how I am handling their personal data, or did they misunderstand the value proposition and realize that the service wasn’t for them?
It’s a step in opening a dialog with a customer who is heading for the door and gives me an opportunity to learn why and evaluate how to address the problems they encountered with the service. Early on I believe these are going to be some of the most valuable conversations I will have with my users.
A fellow NYC-based entrepreneur, Vin Vacanti of Yipit.com, had asked twitter what others were doing in regards to metrics.The question was whether or not people had advice about using a mixture of Google Analytics, Mix Panel, KISS Metrics or built your own metrics solution. I sent him my response in private, but he suggested I share the info with others here.What follows is an edit of my response to both Vin and Hiten Shah of KISS Metrics.
I’m at the early stage of a products life cycle.Specifically I am pre-product/market fit. This means that I have an idea I want to build a product around, but I have yet to validate whether or not my product meets the needs of the market (or any market) I am trying to address.
After doing rounds of Customer Discover & Validation I set out to build a simple product that can help me validate and better illustrate BOTH the problem and solution. Part of this was to lay the foundation of data collection to help dive my decision making process about the product.I learned early on that vanity metrics (unique visits, page views, time on site, etc.) are really just bullshit for a product that hasn’t achieved Product/Market fit. You need to dive down into data that illustrates the behavior of your users.
I’ve specifically been focusing on conversion funnels and cohort analysis.
For conversion funnels I’ve been evaluating Google Analytics, KISS Metrics, and Mix Panel.All of these tools are excellent, but I will say that KISS Metrics has the best visualization and gets you to what matters quickest.Mix Panel is good too, but I specifically like the API as it relates to “events”.Meaning it doesn’t have to rely on page views as Google Analytics does.
For cohort analysis I built a custom system that I’ll describe below.I know Mix Panel does cohort analysis as does RJ Metrics, but I wanted to be able to crunch the data myself for some custom reports that may prove to be useful.
Data I’m Collecting
I shouldn’t really be saying this, but I try to log as much information about a user as possible.I may not use it all, but at least I have it for when it may become useful.I try to not cross the line of creepiness as it relates to privacy, however I do have an idea of how a user came to my site and which features they are engaging with.
When a visitor first visits my site I make note of the channel they traveled to arrive here.If they click on a link from google, blog, email, or twitter I log the referral in the users session.If the user activates and becomes a registered, I log that information with that account.In the future I’m hoping to use this information to identify the channels that perform better as it relates to converting users.
At the core of this I track three primary objects: Entities, Actions, and Change Log.
Entities are unique objects that perform actions.Typically these are users. I can imagine they could be anything really, but for consumer web users are what I care about the most.
Actions are specific tasks an entity can perform.For example logging in, posting a comment, inviting a friend to the service, playing a game, etc. I set this up to be pretty amorphous since I knew I would want to focus on the usage of different functionality down the line.
Change Logs are basically annotations I enter into the system to note changes I’ve made to the site.This is so I can view if there’s a change in performance among a cohort or a set of cohorts I can get a quick view into what may have lead to that difference.
Reports are segmented into cohorts and specific action. The reports are generated on a timed interval and I can configure them to organize cohorts by when they are created (weekly, biweekly, monthly divisions).For now I’m focusing on time rather than other attributes (region, age, gender, etc) as I don’t believe I have enough users to really make cohorts based on other data meaningful.
When the reports are generated I break it down into trailing count of distinct actions users took for that period. As the cohorts age I record their historical stats. The results look like following:
I create historical views of previous weeks and calculate the current week on the fly.Once the week is over I will create a static snapshot to save on doing real time calculations.
The reason I set out to do this is that I want to test specific parts of my application.I felt I had a somewhat decent activation rate when I pushed users to my landing page, but I had a very low return rate.Once a user signed up, they would not come back.They were willing to spend the time with what I was offering, but they did not have an interest in repeating their visit on their own.
I already knew there were a lot of things I wasn’t doing to get users to come back.I wanted to test specific things such as sending out weekly summaries of their account via email or notifications, but more specifically I wanted to know the impact of those actions.I also wanted to know if I went from a weekly to a more frequent distribution would it change the repeat visits or provoke users to unsubscribe.
Right now I’m focusing on repeat logins, however I know once I see that improving I’ll want to determine how to increase referrals from within the application as well as content creation.That’s why I set up actions to be generic.
To create new entities and record actions I use a simple REST service.It can be called by javascript or server side scripting.The reporting is pretty barebones right now.I print out a very simple text based output of the data that I have as part of my application “dashboard”.I’ve integrated with Google Charts too, but only to show the graph of cohorts overlaid on top of each other.Most of the time I’ll just export data to excel from mysql if I want more detailed visualization, but largely I want to see the % increase.I haven’t spent too much time making it more robust as I want to focus on the product I’m building to test out my ideas on growth.
Recently someone posed a question on the NextNY list about what to do after they built their product. A lot of the initial recommendations were to begin various forms of marketing campaigns to get the word out about the product. Press release, social media, contests, you name it… straight to customer acquisition mode.
Knowing that this was the first time the product was getting exposed to users there was no way I could agree with any of the advice being given. But the question remains: “you just built your product (most likely in a vacuum), what do you?”
My recommendation is to get as much feedback as possible. Identify the problem you are solving, find people who have that problem, and talk to those people. Show them your product, listen to their feedback and take lots of notes.
If it looks like people understand the concept of your product and they see a use for it, then start doing some user testing. Identify the main actions the user would take to solve their problem with your product and conduct in-person user tests to see if there are any obstacles that prevent users from using your product to solve their problem. You aren’t QAing the product, but validating the usability of the product.
If you haven’t conducted any user tests before, a great start is Steve Krug’s last three chapters of “Don’t Make Me Think”. You can download them for FREE from his site. I’d also recommend getting his books on usability and testing.
There’s even a video of a sample usability test so you can get an idea of what to expect from a session.
I’ve conducted a bunch of user tests for my own products and found them extremely helpful and easy to do without any special equipment. If you use a Mac I’d recommend trying out Silverback App. It’s a simple app that records the users screen, audio dialog of the user and tester, and video of the user.
Admittedly conducting your own tests can be a bit challenging if you don’t have a flexible schedule. Luckily there are services such as UserTesting.com that will conduct user tests for you. I don’t recommend you start with this because you want to have an understanding of what makes a user test important. It’s a combination of what you are testing and how it’s conducted. Performing some of your own tests before outsourcing them will help you get more value out of the remote tests as you will know what to expect and prepare for. Remember it’s the usability of the product you are testing, not the user.
Hi!
