Eric Tang's Blog

Startups, Software, Everything Wildcard

Algorithms, Design, Humans, Wildcard

| Comments

Today we are happy to announce Wildcard 2.0 is officially in the iOS appstore. The new Wildcard is the best news reader on the iphone, done through a combination of smart algorithms, good design, and a team of amazing editors. The past 7 months has been a process of constant iteration, and the new Wildcard shows a glimpse of our vision to the future of mobile media consumption.

When we started down the path of creating Wildcard 2.0, we thought along three different axis - speed, content coverage and personalization. A lot of design and engineering thoughts went into this, this post will cover some of our technical considerations.


The smart phone is fantastic. It’s a powerful computer attached to you 24/7. But this incredible accessibility is often limited by software. On average, it takes many times longer to read and understand a topic on the phone than on the desktop. It is quite ironic, that you would ever have to sift through web search results on your phone to find a piece of news.

We approached the speed problem from a fundamental level. Inside Wildcard, every piece of information is organized as a mobile card, which is somewhere between a tweet and a webpage. It is a small unit of content that captures a single use case - whether it’s an image, a video, or a summary and a link to an article. With the combination of a novel random forrest based card creation system (which I will write about in a separate post) and a card-based search engine, we have created what we call “the internet of cards” (IOC). Free from the shackles of the legacy web stack like HTML, CSS and Javascript rendering, mobile cards are represented with structured data and rendered natively. This nice property of mobile cards means we can truly deliver content with incredible speed, and natively change the presentation to any platform, whether it’s iOS, Android, FB, Twitter, or any smartwatch.

Wildcard In Facebook
Wildcard In Twitter

Understanding Design Culture

| Comments

A few weeks ago, I attended a discussion panel about building design culture with Khoi Vinh from Wildcard and Pierre Valade from Sunrise. They are both designers I respect and admire tremendously, and I was surprised by the astonishing difference in their design processes. With so many variables in practice of design, how can we, as founders, try to build a design culture?

“Design is everything engineers don’t do.” — Khoi Vinh

“Design is everything, it’s how we solve problems.” — Pierre Valade

These quotes are much broader than the traditional understanding of design. It touches every corner of a company, and it requires a culture with tremendous empathy and understanding of the design process. So, to me, good design culture means everyone in the company has a certain level of understanding and appreciation of the design process. This is especially difficult to adopt, because shifting the mentality for a group of people takes time and consistent effort.

Design is having ridiculously high standard for product experience. Design is going above and beyond to delight users. Design is being obsessive about every little detail. Design requires numerous iterations and relaxed time constraints.

Predictability of shipping date goes down dramatically when the standard of design goes up. This means engineering needs to understand that 29 out of the 30 prototypes will not ship. This means marketing and PR needs to understand the marketing plan may start 2 months after the initial planned date. This means product needs to understand that design will gather information, get inspired from unconventional processes that sometimes seems like they are “wasting time”. And our job as a founders is to make sure we create enough space to make that happen.

One cannot simply build a design culture without having first-class design DNA. You can’t fake design culture by hosting the occasional customer interviews or getting employee passes to photography exhibits. Design culture is a commitment to the art and the process, and the first step to that commitment is to work with designers who REALLY knows what they are doing.

If you are just starting out with this process, there WILL be a period of time of learning to understanding where the lines are. I have been fortunate enough to work with some talented designers over the years, and here are a few things I have learned:

  • Things WILL get thrown away. Figure out how to build CHEAP prototypes.
  • The design process requires TIME and INSPIRATION. Sometimes it means browsing Dribbble for an afternoon, sometimes it means working in complete isolation for a few days.
  • Don’t expect to validate everything in mockup form. A lot of times (especially for consumer products), you just have to play with the product.
  • Subtle things REALLY matter. Things like typography make a big difference.
  • Over-communicate, try to define a very clear goal at the beginning.
  • Know that design is a never-ending process, sometimes things cannot be measured by metrics.

If you are not a designer, what other things have you learned working with a design team? If you are a designer, what other things do you wish your counter parts would understand?

The Age of Mobile Integration

| Comments

When Tim Berners-Lee created the hypertext (think web links) at CERN in 1989, he forever changed how the internet worked by creating connections between resources. The simple concept of open and sharable references means web experiences can be integrated together in an infinite possibility. Thus the web, from its birth, promotes integration. Mobile (at least the version we know today), on the other hand, was conceived without that concept. This fundamental difference has been one of the biggest limiting factors in the mobile ecosystem for the past 6 years.

Information on the web is open. A large portion of information is openly available, another large portion is hidden behind login walls, but fairly easily accessible. Information is passed around between web apps through plugins, widgets, async apis, links, and many other formats. Information in the mobile world is locked up inside apps. There is a very high threshold for app download, and the majority of the apps on phones don’t get opened. Push notifications are used largely as a triggering mechanism rather than an unit of consumable information. SDKs are generally action-based instead of information-based. This makes the majority of mobile experiences inaccessible to users.

On the web, there are many ways we can discover new content, whether it’s through search, curation, social, or any other mechanism. This is because links on the web are open, and the “control” lowers percieved switching cost. On mobile, there are no good ways to discover new content. App search is notoriously bad, and apps are generally not created with broad discoverability in mind. Social platforms tend to keep people on the platforms, which causes the problem of “hit-and-leave” for content publishers. There is no equivalence of “just browsing the web” in the mobile app world, and hence, users lose the ability to casually discover new experiences.

The lack of accessibility and discoverability is further worsened by the lack of communication between apps. Due to OS restrictions, apps live in very controlled sandboxes. Even with the recent development in deeplinking and app plugins, we haven’t seen information passed between apps in a meaningful way except for a few very specific use cases. The “app constellation” model remains unattainable for most developers. There lacks a communication pipe so that apps are aware of other apps, and there lacks a mechanism for apps to define an “embedded experience” inside other apps.

We all know that mobile doesn’t live on a single device. We live in a multi-device world and the devices are in different environments. Environments are important because their refer contexts and the underlining expectations of users. Other than large companies with proprietary homegrown solutions, most apps are not aware of other environments, and have no easy way to communicate between environments. With the upcoming smart watches and internet of things, communication in a multi-device world is a huge opportunity. Cross device experiences like iMessage is and inspiration for thinking about a user as she is using a service.

In the app-centric world, experiences largely live in silos. The switching cost between apps is high, and the poor navigability of the OS makes information inaccessible. It’s quite ironic that the highly accessible hardware (you can take the phone out of your pocket at anytime) is coupled with an inaccessible software layer. Comparing to the ease of information flow in the desktop environment, smartphones don’t come close. We are handicapped by legacy technology layers and OS-layer restrictions to take full advantage of the internet infrstrcutured built up over the last 20 years. The rigid programming paradigms make development slower and more costly. The lack of access to low-level functionality limits the programmer’s ability to innovate. The existing legacy web makes re-designing for a small screen extremely difficult.

With so many problems in the mobile software layer, many companies (include Wildcard - the company I helped to start) are working on new solutions. Mobile cards, deeplinks, various SDKs for identity management, mobile payment are all solutions that give mobile apps new super powers. With the base layer established and many interesting use cases explored, here comes the age of mobile integration.

Technology Behind Wildcard

| Comments

Wildcard launched our first product a few weeks ago - a card-based mobile browser. It’s a new way of browsing the internet on your smart phone, and it’s just the beginning. A group of incredibly talented people came together, designed and built the app over the past 12 months. I want to share a few things under the hood for those who are curious.


We started our journey with the goal of a thin client on iOS. Instead of making a runtime that interprets programming languages and specification of behaviors, we opted to build an app that recognizes “types” of cards and focus on the data. For example, the browser understands the a “product” card, or an “article” card. We did this because of the speed of implementation, and the risk of increasing complexity for a “card language”.

The effort involved, however, is far from the sound of a “thin client”. It’s carefully designed and built by our product / iOS team. Significant efforts has gone into things like custom animation, custom gesture, local caching, etc.

The Wildcard browser is written in Objective-C. We chose this because of implementation speed and user experience. Cross-compiling technologies like PhoneGap or RubyMotion break down quickly when you want to have custom UX. Even though this means we have to start from scratch for Android, I expect the Android UX to be quite different that code-sharing would be very difficult regardless.


When we started our journey, the mobile card ecosystem was in its nascent state. We started by decomposing webpages into their most essential components - which eventually evolved into our own concept of “mobile cards”. A platform was created to convert generic webpages into cards.

For example, a product page on Lululemon might have a search bar, the product details, a checkout button, and a locator to find the closest store. We would create a “product search” card, a “product” card that allows users to checkout, and a store locator card.

Many iterations have gone into the platform, and it has evolved into a framework to execute commands like “extract image url at this XPATH” or “Parse this piece of string into a proper U.S Address”. Many of the iterations have to do with getting to the right level of abstraction, so that humans can manipulate the data pipeline as well as computers (through layers of abstraction). These commands also became building blocks for more complex algorithms - and we are working on machine learning techniques to make the platform more robust.

The platform acts as a blackbox that turns inputs like an URL or a piece of structured data, into a mobile card. It’s configurable by humans and algorithms, it’s parallelizable across machines, and it’s at the core of our technical infrastructure. We chose Java and Dropwizard for a good trade off between performance and ease of implementation. The renaissance in the JVM stack gives us confidence in our technology choice.

Data Services

Wildcard works on many large-scale data problems. As the complexity of our system grew, we grouped together services that directly manipulate data. This set of services includes things like RSS feed readers, small web crawlers, one-off scripts for data processing, and GUI tools to allow direct data manipulation. This is a high-touch/offline environment, where a lot of “raw data” gets processed into a state that can be fed into the platform through a simple API.

Having an offline environment gave us the ability to move quickly without worrying about the stability of our platform. Things can get a little messy if you are not careful, so following good software engineering practices is extra important here. We use a mixture of different technologies in this system, with the dominant languages being Ruby and Javascript.

Search Engine

As we built more and more cards, we started to realize the importance of search. Search is often the best way to expose a large variety of experiences to users. In our case, it has become an essential component of our product.

Building a search engine from scratch is no easy task. We started out with a vertical approach - focusing on e-commerce first. We used techniques like LSA to map queries onto commerce brands, rank the brands, and automatically kick off search within the brands. This is a powerful feature because users will always get the most up-to-date search results from the brands, and we wouldn’t have to do a massive/costly web crawl.

Since then, we have added many more result types into our search index. We have also built a scalable and configurable crawler (across the brands that use mobile cards). Search is the fastest evolving service in our stack, and we are constantly trying to expand our capacity / use cases.

We currently use a combination of Redis, Elasticsearch and different python scripts for search. As the index sizes continue to grow, more scalable solutions will be needed.

Integration Systems

One of the goals of Wildcard is to make our mobile card platform “writable”. This means any content publisher, retailer, service provider is able to take advantage of the system and control their own cards. To make this as easy as possible, we’ve created a product just for that. It allows anyone to publish to Wildcard, Twitter and Pinterest.

We built a schema translation layer in Java to support this product. We also built tools like this to help developers validate and visualize their own card implementations.

Dev Ops

As our engineering team grew, automation became more and more important. We started out with using Chef, Jenkins on top of AWS, and have since adapted an concoction of automation tools.

At the core, we use AWS as the base layer of our infrastructure. We use Chef for server configuration, terraform for orchestration, docker for resource management, graphite/keen for metrics and monitoring, consul for async jobs (like deployment), github/circleci for continuous integration, hubot to tie everything together, and probably many more tools I forgot to mention.

Automation has been a game changer in our daily engineering lives. It’s a significant investment - and it’s necessary. When done well, it can be a significant productivity booster for the engineering team.

As our 15-person product/engineering team continue to grow, our technology will continue to evolve and adapt to new needs. If you are interested in anything discussed here, shoot me a line and I’m happy to chat more.

The Dirty Secret of 10x Engineers

| Comments

"Taken from"

There is a mantra in the startup world - “we only hire 10x engineers”. This is exactly the arrogant, bullshit attitude that turns your company into a fear-driven, political and unproductive place to work. Nobody is consistently a 10x engineer. Here is why:

If someone constantly works at a rate 10 times more productive than the average engineer, this person is an expert who has stopped challenging himself. This could be due to a variety of reasons, but trust me, the smartest engineers got there by constantly challenging themselves and learning new stuff.

In flow theory, the “flow state” is when the right amount of expertise and right amount of challenge intersect. This is a rare occurrence when you are productive at your highest potential. This is your “10x” state. Everyone can have this 10x state. (for more, read this post by Jeff Dickey)

However, almost by default, you are rarely in the flow state when you are working at a startup. Startups work on problems that have not been solved, and they are usually extremely challenging. You should have enough basic understanding in related topics to gain the expertise, but rarely do you already have the expertise. (Already having the expertise would mean you are working on the exact same problem as the one you solved before, and we have a much bigger problem here)

The mode of operation is usually something like this:

  • Hit your head against the wall for a few days
  • Search google, email friends, read papers / books
  • Prototype a few different solutions, realize they are all flawed
  • Cold email experts in the field (or get introduced by friends), set up coffee/skype meetings
  • Build new prototype with newly gained knowledge
  • Repeat

This process is filled with learning new tools, new terminologies and new ways of thinking. None of this qualifies for the prerequisite of operating at your 10x state.

Will you eventually learn all the things you need to learn to be productive? Of course. But that’s usually short-lived. You will get to be 10x when you are building the last prototype (which will grow into the real product). At this point you are so knowledgable in this particular topic that it takes you 10% of the time compare to the first prototype. This is your 10x state.

At this point, you realize the problem you solved is just one piece of the 3000 piece Seurat Sunday Afternoon Jigsaw Puzzle Now you have to move on to the next problem. And the seemingly never-ending death cycle of prototyping starts all over again. Oh by the way, you have to somehow magically align this effort with timeline on the business side, figure out a reasonable deployment strategy, manage your AWS instances, find time to see your boy/girlfriend, etc.

But there is a silver lining here. The more you go through this process, the better you get at navigating the unknown. You learn to use the right tools, you learn resources to look for, you meet other smart people, and most importantly, you learn to look at a unknown problem from an inventor’s eye. All of the experiences you’ve learned will increase your chances of entering your 10x state, and soon people will start calling you a 10x engineer. But you won’t let that get to your head, because now you know too much to know that you can’t possibly know it all.

So this is OUR mantra at Wildcard - “We hire ridiculously intelligent people and challenge them to constantly get better”. We are working on problems no one has solved, and we will constantly push ourselves to learn new things. May we never become 10x engineers.


| Comments

Taken from frothingslosh

5 months ago, Jordan, Doug and I sat in Doug’s living room in San Francisco for a week, talking about the mobile web and our visions of what it should be. It was an exciting week. Github account was created, AWS machines provisioned, and sketches were drawn.

We were all fed up with the current state of the mobile internet. It is an ecosystem plagued by terrible user experiences and little-to-no common practices. Users are left to their own devices and businesses lost precious opportunities. So we set out to change all that. Our goals are ambitious, our problems are hairy, and on top of that we decided to bootstrap, which means resources are scarce.

In the subsequent months, we moved back to New York, got a few desks at a co-working space (thanks to Lerer Ventures), and went to work. To better understand the problem, we talked to the smartest people we know, and created prototype after prototype.

We are still in the thick of it - changing the mobile web on a fundamental and infrastructural level is no easy task. It’s one of those rare projects that sit at the borderline between a completely crazy idea and a huge opportunity. This type of projects need time and a massive amount of effort. We will create infrastructure that the mobile web lacks today, we will create new modes for people to interact with “the internet” on the go, and we will invent new ways for businesses to reach their customers. The difference between now and 5 months ago is that now we have $3 million bucks, a few awesome new additions to the team, new prototypes to play with, and a new company name.

Our new company is called Wildcard. Jordan describes it well in his blog post about what we are up to, and Doug has posted about some technical problems we are working on.

Drop me a line if you want to learn more, or sign up for early access to the future by putting your email in the sign up form.

Where Is My Robot Scientist?

| Comments


Shot from the Movie AI

Mass production is something that most of us take for granted today. We enjoy the effect of it everyday but rarely think about the dramatic change it has brought to the world. Almost everything we buy today is mass produced.

One thing that’s never been able to be mass-produced is ideas. An idea can be a song, a piece of writing, a software program, or a business plan. The process of idea creation is a personal experience that requires creativity and complex thinking. It seems impossible for machines and science to create ideas like we can.

But times have changed since the day of mechanical, dumb machines that can only handle simple tasks. Recent advancements in AI and data science are giving us signs of truly “intelligent” machines, capable of mass-producing original ideas. How exactly can we do it? To answer this question, we have to understand:

  • Fundamental pieces of a mass-production system
  • Categories of ideas in the world.

Mass Production System

It’s fairly straight-forward to understand a mass-production system. People have been studying and improving it for centuries, ever since the industrial revolution (one can argue it started a lot earlier than that, but you get the point). Such systems usually involve

  • Supply chain to gather raw materials
  • Transformation process to change raw materials into generally useful, multi-purpose forms
  • Assembly lines that assemble small pieces into functional units, again, usually multi-purpose
  • Assembly lines that assemble functional units into final products
  • Quality assurance in each step

There are also distribution and fulfillment systems, but let’s focus on the creation process for now.

Categories of Ideas

Categorizing all ideas in the world is much harder. One can turn to cosmology or epistemology, but that tends to get philosophical pretty quickly. It gets even more confusing when the categories of ideas itself is an idea (hmm… did somebody say M.C Escher?) But for arguments sake, let’s group ideas into:

  • Metaphysical ideas (music, modern art - ideas for ideas sake)
  • Worldly ideas (journalism, a business plan, a piece of computer program - ideas rooted in the physical world)

Mass Producing Ideas

To imagine a mass production system for metaphysical ideas, let’s think about music. Notes/sound are the raw materials. Rhythm, riffs, choruses, movements are the small, functional units. Songs are the final products. The brute-force approach would be generating an infinite number of rhythms, riffs, choruses, create all permutation of these elements and listen to every generated song to find the ones you like. But that would take an infinite amount of time.

The key here is quality control. At each step, we identify the “quality” we care about, and create a filter to only let through what we care about. Machine learning technology can already learn “styles” of music. Rules can be created to structure pieces of music into songs, and music analytics engines + recommendation algorithms can analyze them to determine if you will like the music.

Worldly ideas on the other hand, are harder to create. There are many constraints that are hard to capture using mathematics / rules. How can we generate a piece of original news article? What about a business plan to solve a real-world problem? It seems that we have to teach machines to “understand” the real world before we can generate “analysis” from it.

But short of creating original worldly ideas, we can create derivatives of existing ideas. With sufficient data, machines can “learn” from existing ideas and create similar ideas or other forms of the same ideas. The process varies depending on the algorithm, but usually statistical analysis and natural language processing is involved. For example, text analysis engines like the SRI Internaional are able to take existing text and create short summaries from them. Diagnostic engines are able to use patient data and symptoms to create diagnosis.


The goal of mass production is a very simple economic principle: dramatic reduction on cost and dramatic increase on supply. We are still far away from creating music as good as Bach, or generating news reports worthy of the New York Times, but technologies that create ideas have fundamental impacts on society.

With SRI’s text summary technology, it now takes 10% of the time it did before to achieve the same breath in the past, which means it increased our reading speed by 1000%. Electronic music lowers the bar for music production by eliminating the necessity of learning to play musical instruments (biggest hurdle of music creation). The amount of new electronic music changed the fundamentals of the publishing side of the music industry. Companies like Game Salad try to lower the threshold for creating games, and the effect has been the commoditization of game creation. With the same level resources, my friends are way more likely to create games I want to play than Zanga ever will.

Technology is at a point to challenge the traditional method of idea creation. With the right application, we will be able to create more new ideas than ever before. And maybe one day, we will have a robot scientist making things for us.

Of Flying Cars and Meal in a Pill

| Comments

If you haven’t watched this video of the debate between Marc Andreessen and Peter Thiel, you should. It’s some of the best content I’ve consumed in the past 6 months, and I highly recommend watching it.

I really like the way Andreessen describe the Internet. While the internet alone is not enough to “save innovation”, it will be the information backbone for almost any new innovation going forward. Self-driving cars, wearable smart devices, or even cancer treatment will rely on it. At its core, the internet is an information management / distribution system, supported by governments and some of the biggest corporations in the world. It takes in information and sends it to the people who want it with incredible speed and accuracy. Since almost everything with significant impact needs management and distribution, the internet will be the most important piece of support infrastructure for innovation in the foreseeable future.

One of the downsides of the internet infrastructure as a distribution system is the restriction of wired networks. Since we haven’t figured out how to economically scale satellite data transmission, the next best thing is local/regional wireless transmission. To that extend, mobile will be a larger infrastructure than the internet today. For one reason or another, mobile hardware did not pan out like the open architecture of PCs. This places a lot of trust on companies for an open ecosystem to create enough room and incentive for innovation. The iPhone might not be the best suited for connecting self-driving cars, but they have the best distribution and that’s valuable. It’s too great of a hurdle for someone to create a self-driving car control panel that plugs into the iPhone, when Apple is charging developers ridiculous fees just to connect through the data port. Of course they are not trying to make a buck here, it’s a “more acceptable” way to limit competition and that’s why almost all hardware innovations on top of the iOS ecosystem has been low-data-transmission devices like Square or Kinsa that go through the headphone jack. Open access to widely distributed hardware will be important to the mobile ecosystem and sustaining the rate of innovation.

It’s exciting to think about innovation in the fields of energy, raw material, transportation, and bio tech. Andreessen and Thiel talked a lot about the obstacles in these fields, but I’d like to think about the ideal situation. For example, energy is a highly controlled and regulated field. Most people make money in energy because of the lack of innovation. It’s all about supply chain control and price control. Fracking is the “latest and greatest” in oil, but it was first invented and commercialized in the 40s. It took close to 70 years and multiple wars for it to be as big as it is today. What will it take for there to be as many energy hackers as there are javascript hackers? It’s not because there are less energy / chemical engineers. It’s because of the (negative) hollywood effect and the lack of clear path to success. Being a code hacker means you will at least be financially self-sufficient, with the possibility of creating the next Napster or Facebook and have huge impact. Being an energy hacker means you will probably be jobless, have no prior examples to follow, and eventually be acuqa-hired by one of the 3 giant corps in your industry for pennies while they go on and use your technology to make millions.

Equally as important is the business side of these industries. Not just venture capital, but also “business guy” founders, marketers, community management, etc. It takes a few successful examples and some marketing to set a trend, but it will be hard for these examples to come from within the industry because of its ingrained culture. University research groups / outsiders might be better positioned to create new innovation in these industries, and software startups have created a great model of operation. The key here is for the ecosystem to provide enough support so these young companies can survive on their own. Even though the constraints are different, energy / transportation / bio tech startups can take a lot of the same principles when it comes to figuring out product-market fit, creating communities, human-centric design, etc.

Btw, who do you think won the debate? Personally I liked Thiel’s arguments better - more quantitative and less anecdotal, but I agree with Andreessen that innovation is happening faster than ever before.

Flying cars and meals in a pill are figurative imagination from the past, and we have embodiments of these concepts today. Think Google self-driving cars and new dietary movement (juice, protein shakes, multi-vitamin pills and other meal supplements) With the macro trends like the hardware renaissance and biohacking, I’m very excited to see more entrepreneurs in other engineering disciplines.

Intent and the Internet

| Comments

I’ve been thinking about intent lately. In the world of “big data”, “sentiment analysis”, “behavioral marketing” (blah blah blah…), “we use intent to drive user behavior change” is the party line. It’s a shame that “driving intent” is this black box that “our data scientists created” and no one else understands it. What does it really mean in the context of the internet we live in today?

To Lay the Ground Work

On the most fundamental level, every small action we do on a webpage is triggered by an intent. These are not “want to buy a car”, “want babies” intent. These are more like “click on name link”, “navigate to home page”, “sign up”, etc. Micro-view, super short-term, knee-jerk reaction type of stuff. From that we can define an “Intent” as “the desire of performing an action”, which really is the action itself + some meta data (like time, location, etc).

Now that’s quite a simplistic and lower-value view of Intents. What people really want is to derive “insights” from “intents”, which are the more “meta” intents people talk about in the advertising world (“want to buy a BMW”, “want to go to Daft Punk concert”, etc).

Let’s call the more insightful intents “meta-intents”, and the lower-level, action-representations “simple-intents”.

The Problem Of Predicting Intent

So what does it take to arrive at the “meta-intents” from a bunch of recorded “simple-intents”? When we translate it to “data science” terms, the question becomes “Given a series of prior simple actions, how likely is it for a person to take a particular action I care about in the immediate future?”

Statisticians have long studied this problem. Countless research studies have been conducted, ranging from brand loyalty, purchase behavior, to World of Warcraft and condom usage. Many regressions and latent-models and collaborative filters later, they all have one thing in common: the data has to be high-signal and low-noise. Said in another way, we have to know all of the relevant actions and at the same time filter out actions that are not relevant.

To put it in an example: in a cookie-centric world, an advertiser typically have about 10%-25% of the data. This means if I visited 100 sites this morning, they know about 10-15 of them. How high of a signal is that? How accurately can they predict what I’m trying to do?

In Real Life

Simple intents and meta intents inform one-another. We can derive simple-intents from meta-intents to guide users down to specific paths, and we can observe users’ simple-intents to derive meta-intents. But these 2 operations require different amounts of efforts. “Meta-to-simple” can be achieved almost automatically (mostly driven by algorithms), “simple-to-meta” is much more manual (data scientists or analysts validating assumptions on top of hadoop clusters). On by the way, it’s not obvious what those meta intents are so the assumptions are hard to make. Think the book Freakonomics.

From a cold-start, since you don’t have enough data to study any patterns of simple intents, ‘meta-to-simple’ is the only approach. There might still be room to use public data sets to generate meta-intents, but that’s a timed window as data science becomes commoditized over time. A data-driven product needs to have enough data science DNA in-house to continually experiment with new assumptions, and that’s how you build true advantages that no one else can copy.

There are a lot more topics to think about, like personal profile and short-term vs. long-term intent. We’ll talk about that in another post.