“Do Not Track” Explained

September 20, 2010 at 4:13 pm 7 comments

While the debate over online behavioral advertising and tracking has been going on for several years, it has recently intensified due to media coverage — for example, the Wall Street Journal What They Know series — and congressional and senate attention. The problems are clear; what can be done? Since purely technological solutions don’t seem to exist, it is time to consider legislative remedies.

One of the simplest and potentially most effective proposals is Do Not Track (DNT) which would give users a way to opt out of behavioral tracking universally. It is a way to move past the arms race between tracking technologies and defense mechanisms, focusing on the actions of the trackers rather than their tools. A variety of consumer groups and civil liberties organizations have expressed support for Do Not Track; Jon Leibowitz, chairman of the Federal Trade Comission has also indicated that DNT is on the agency’s radar.

Not a list. While Do Not Track is named in analogy to the Do Not Call registry, and the two are similar in spirit, they are very different in implementation. Early DNT proposals envisaged a registry of users, or a registry of tracking domains; both are needlessly complicated.

The user-registry approach has various shortcomings, at least one of which is fatal: there are no universally recognized user identifiers in use on the Web. Tracking is based on ad-hoc identification mechanisms, including cookies, that the ad networks deploy; by mandating a global, robust identifer, a user registry would in one sense exacerbate the very problem it attempts to solve. It also allows for little flexibility in allowing the user to configure DNT on a site-by-site basis.

The domain-registry approach involves mandating ad networks to register domains used for tracking with a central authority. Users would have the ability to download this list of domains and configure their browser to block them. This strategy has multiple problems, including: (i) the centralization required makes it fickle (ii) it is not clear how to block tracking domains without blocking ads altogether, since displaying an ad requires contacting the server that hosts it and (iii) it requires a level of consumer vigilance that is unreasonable to expect — for example, making sure that the domain list is kept up-to-date by every piece of installed web-enabled software.

The header approach. Today, consensus has been emerging around a far simpler DNT mechanism: have the browser signal to websites the user’s wish to opt out of tracking, specifially, via a HTTP header, such as “X-Do-Not-Track”. The header is sent out with every web request — this includes the page the user wishes to view, as well as each of the objects and scripts embedded within the page, including ads and trackers. It is trivial to implement in the web browser — indeed, there is already a Firefox add-on that implements a such a header.

The header-based approach also has the advantage of requiring no centralization or persistence. But in order for it to be meaningful, advertisers will have to respect the user’s preference not to be tracked. How would this be enforced? There is a spectrum of possibilities, ranging from self-regulation via the Network Advertising Initiative, to supervised self-regulation or “co-regulation,” to direct regulation.

At the very least, by standardizing the mechanism and meaning of opt-out, the DNT header promises a greatly simplified way for users to opt-out compared to the current cookie mechanism. Opt-out cookies are not robust, they are not supported by all ad networks, and are interpreted variously by those that do (no tracking vs. no behavioral advertising). The DNT header avoids these limitations and is also future-proof, in that a newly emergent ad network requires no new user action.

In the rest of this article, I will discuss the technical aspects of the header-based Do Not Track proposal. I will discuss four issues: the danger of a tiered web, how to define tracking, detecting violations, and finally user-empowerment tools. Throughout this discussion I will make a conceptual distinction between content providers or publishers (2nd party) and ad networks (3rd party).

Tiered web. Harlan Yu has raised a concern that DNT will lead to a tiered web in which sites will require users to disable DNT to access certain features or content. This type of restriction, if widespread, could substantially undermine the effectiveness of DNT.

There are two questions to address here: how likely is it that DNT will lead to a tiered web, and what, if anything, should be done to prevent it. The latter is a policy question — should DNT regulation prevent sites from tiering service — so I will restrict myself to the former.

Examining ad blocking allows us to predict how publishers, whether acting by themselves or due to pressure from advertisers, might react to DNT. From the user’s perspective, assuming DNT is implemented as a browser plug-in, ad blocking and DNT would be equivalent to install and, as necessary, disable for certain sites. And from the site’s perspective, ad blocking would result in a far greater decline in revenue than merely preventing behavioral ads. We should therefore expect that DNT will be at least as well tolerated by websites as ad blocking.

This is encouraging, since there are very few mainstream sites today that refuse to serve content to visitors with ad blocking enabled. Ad blocking is quite popular (indeed, the most popular extensions for both Firefox and Chrome are ad blockers). A few sites have experimented with tiering for ad-blocking users, but soon after rescinded due to user backlash. Public perception is a another factor that is likely to skew things even further in favor of DNT being well-tolerated: access to content in exchange for watching ads sounds like a much more palatable bargain than access in exchange for giving up privacy.

One might nonetheless speculate what a tiered web might look like if the ad industry, for whatever reason, decided to take a hard stance against DNT. It is once again easy to look to existing technologies, since we already have a tiered web: logged-in vs anonymous browsing. To reiterate, I do not believe that disabling DNT as a requirement for service will become anywhere near as prevalent as logging in as a requirement for service. I bring up login only to make the comforting observation there seems to be a healthy equilibrium between sites that require login always, some of the time, or never.

Defining tracking. It is beyond the scope of this article to give a complete definition of tracking. Any viable definition will necessarily be complex and comprise both technological and policy components. Eliminating loopholes and at the same time avoiding collateral damage — for example, to web analytics or click-fraud detection — will be a tricky proposition. What I will do instead is bring up a list of questions that will need to be addressed by any such definition:

  • How are 2nd parties and 3rd parties delineated? Does DNT affect 2nd-party data collection in any manner, or only 3rd parties?
  • Are only specific uses of tracking (primarily, targeted advertising) covered, or is all cross-site tracking covered by default, save possibly for specific exceptions?
  • Under use-cases covered (i.e., prohibited) under DNT, can 3rd parties collect any individual data at all or should no data be collected? What about aggregate statistical data?
  • If individual data can be collected, what categories? How long can it be retained, and for what purposes can it be used?

Detecting violations. The majority of ad networks will likely have an incentive to comply voluntarily with DNT. Nonetheless, it would be useful to build technological tools to detect tracking or behavioral advertising carried out in violation of DNT. It is important to note that since some types of tracking might be permitted by DNT, the tools in question are merely aids to determine when a further investigation is warranted.

There are a variety of passive (“fingerprinting”) and active (“tagging”) techniques to track users. Tagging is trivially detectable, since it requires modifying the state of the browser. As for fingerprinting, everything except for IP address and the user-agent string requires extra API calls and network activity that is in principle detectable. In summary, some crude tracking methods might be able to pass under the radar, while the finer grained and more reliable methods are detectable.

Detection of impermissible behavioral advertising is significantly easier. Intuitively, two users with DNT enabled should see roughly the same distribution of advertisements on the same web page, no matter how different their browsing history. In a single page view, there could be differences due to fluctuating inventories, A/B testing, and randomness, but in the aggregate, two DNT users should see the same ads. The challenge would be in automating as much of this testing process as possible.

User empowerment technologies. As noted earlier, there is already a Firefox add-on that implements a DNT HTTP header. It should be fairly straightforward to create one for each of the other major browsers. If for some reason this were not possible for a specific browser, an HTTP proxy (for instance, based on privoxy) is another viable solution, and it is independent of the browser.

A useful feature for the add-ons would be the ability to enable/disable DNT on a site-by-site basis. This capability could be very powerful, with the caveat that the user-interface needs to be carefully designed to avoid usability problems. The user could choose to allow all trackers on a given 2nd party domain, or allow tracking by a specific 3rd party on all domains, or some combination of these. One might even imagine lists of block/allow rules similar to the Adblock Plus filter lists, reflecting commonly held perceptions of trust.

To prevent fingerprinting, web browsers should attempt to minimize the amount of information leaked by web requests and APIs. There are 3 contexts in which this could be implemented: by default, as part of the existing private browsing mode, or in a new “anonymous browsing mode.” While minimizing information leakage benefits all users, it helps DNT users in particular by making it harder to implement silent tracking mechanisms. Both Mozilla and reportedly the Chrome team are already making serious efforts in this direction, and I would encourage other browser vendors to do the same.

A final avenue for user empowerment that I want to highlight is the possibility of achieving some form of browser history-based targeting without tracking. This gives me an opportunity to plug Adnostic, a Stanford-NYU collaborative effort which was developed with just this motivation. Our whitepaper describes the design as well as a prototype implementation.

This article is the result of several conversations with Jonathan Mayer and Lee Tien, as well as discussions with Peter Eckersley, Sid Stamm, John Mitchell, Dan Boneh and others. Elie Bursztein also deserves thanks for originally bringing DNT to my attention. Any errors, omissions and opinions are my own.

To stay on top of future posts, subscribe to the RSS feed or follow me on Twitter.

Entry filed under: Uncategorized. Tags: , , , , , , , , , .

Women in Tech: How Anonymity Contributes to the Problem Facebook’s Instant Personalization: An Analysis of Fundamental Privacy Flaws

7 Comments Add your own

  • 1. anonminer  |  September 20, 2010 at 4:29 pm

    I feel do not track harms society.

    I have done data mining on server logs, which would have been a great wealth of resource and could have deeply affected public policy. However people are just paranoid and wish to wear tin foil hats. We malign data mining by conjuring up the worst possible fears like insurance companies using the data or social engineering. but we forge that the same data is one of the most powerful mirror of our society and mining it could be extremely useful. It is same as assuming nuclear power to be evil because of nuclear weapons.

    I never understand why media always portrays data miners as evil people, humanity progressed because we were willing to share ideas and data. not because we were wearing tin foil hats and assuming everyone else was alien trying to abduct us.

    Also what about other forms of tracking by mobile companies or by transactions.

    WSJ and other News Corp subsidiaries just hate Web, and this is just another attempt to strike fear and malign internet.
    No wonder WSJ did that report.

    • 2. Arvind  |  September 20, 2010 at 4:41 pm

      I don’t think anyone is portraying data miners as evil people; the problem is the lack of transparency and choice. Do Not Track is not a ban on tracking, it is merely a way to allow users to opt-out. Even if you think people opting out are merely paranoid, they should still have the right to do so.

  • 3. Mark  |  September 21, 2010 at 6:25 am

    Interesting piece. While I’d agree that DNT provides what appears to be a simple and elegant method of opting out, I think it is likely to fall down over the issue of trust. The industry has shown time and time again that it will use every trick and underhand tactic known to grab user data, and as far as possible conceal the extent of what it holds and the uses to which it is put. In short trust, trust is breaking down – if it hasn’t done so entirely already – and I for one would be unlikely to take the word of any of the industry players that they were not simply lying through their teeth. Problems notwithstanding, therefore, I tend to prefer the domain list to ensure they cannot acquire the data in the first place.

    The bullishness and arrogance shown thus far (at least till the FCC got interested) mean they will have a long, long hill to climb before any solution that relies on trust will be publicly acceptable. An insistence on transparency would probably kill 50 percent of current practice, and I doubt the major players would be prepared to accept this.

    Your first commenter suggests people like myself are tinfoil hatted paranoiacs. Far from it. But I do know when I am being taken for a ride and getting nothing in return. Until I can properly control the way any PII is used, I will block the lot, advertising and tracking in every way I can.

    • 4. Arvind  |  September 21, 2010 at 5:44 pm


      If trust is your concern then you should definitely prefer the header approach. Putting myself in the shoes of someone who is trying to subvert the domain registry, I can think of any number of loopholes, including tracking that does not use a specific 3rd party domain at all (e.g., 2nd party deploys Panopticlick, silently uses 3rd party API in the background to achieve linkage across sites.)

      As I hinted in the article, regulation that focuses on the tools of the trackers rather than their intent and actions is much more vulnerable to being bypassed, and provides much less of an avenue for pursuing enforcement actions.

      Note that header-based DNT does not necessarily mean self-regulation and taking advertisers “at their word”. Depending on public opinion and other factors, it could even be direct regulation, i.e., periodic audits.

      Finally, I disagree that the industry won’t survive a move towards transparency. A good chunk of consumers either don’t have a problem with being tracked or don’t care enough to take the affirmative step to opt out.

      • 5. Concerned  |  February 25, 2011 at 7:57 am

        It’s my data of my surfing in my browser in my computer, and I get to say if anyone makes use of it.
        And I choose no.
        Simple as that. If someone wants to use my data, then we should talk about money. And if they don’t want to pay what I consider to be reasonable, then tough.

        • 6. Jay  |  July 3, 2012 at 8:01 am

          Try putting that shoe on the other foot. The stuff you are surfing is someone else’s data, coming to you from someone else’s computer. Do you still want to talk about money? Right now, surfing is free . . . but we all know there is no such thing as a free lunch. There needs to be “reasonable” give and take on both sides. DNT allows us to speak softly and carry a big stick.

  • 7. Anonymous  |  April 23, 2011 at 11:05 pm

    I am being “tracked” but for worse reasons. Apparently I’m a victim of involuntary and rather nasty human subjects experimentation. It’s an interventional study to boot. I think that there should be something in the law, in Do Not Track, to address human experimentation too.

    Unlike other more anonymous people, my name serves as a very unique identifier and I don’t doubt it had something to do with my getting put on the involuntary research victim list.

    I’m as annoyed by ads as anyone else, but when the things on my net (constantly hacked at home, evidence of interference in public places) show up, they are often part of the “conditioning” aspect of the experiment. I’m not so interested in US or internet law, but I’d like to see someone put an end to this.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

About 33bits.org

I’m an associate professor of computer science at Princeton. I research (and teach) information privacy and security, and moonlight in technology policy.

This is a blog about my research on breaking data anonymization, and more broadly about information privacy, law and policy.

For an explanation of the blog title and more info, see the About page.

Me, elsewhere

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 265 other subscribers

%d bloggers like this: