How to prepare a technical talk

I used to suck at giving technical talks. I would usually confuse my audience, and often confuse myself. By the time I became a prof, I sucked a lot less. These days I enjoy giving technical talks and lectures more than non-technical ones, and my students seem to like them better as well.

So something had changed; I’d developed a process. The other day I sat down to see if I could extract what this process was. It turned out to be surprisingly formulaic, like an algorithm, so I’d like to share it with you. I’m guessing this is obvious to most professors who teach technical topics, but I hope it will be helpful to those who’re relatively new to the game.

There are three steps. They’re simple but not easy.

1. Identify the atomic concepts
2. Draw the dependency graph
3. Find a topological ordering of the graph

Identify atomic concepts. The key word here is atomic. The idea is to introduce only one key concept at one time and give the audience time to internalize the concept before moving on to the next one.

This is hard for two reasons. First, concepts that seem atomic to an expert are often an amalgam of different concepts. Second, it’s audience-specific. You have to have a good mental model of which concepts are already familiar to your audience.

Draw the dependency graph. Occasionally I use a whiteboard for this, but usually it’s in my head. This is a tricky step because it’s easy to miss dependencies. When the topic I’m teaching is the design of a technical system, I ask myself questions like, “what could go wrong in this component?” and “why wasn’t this alternative design used?” This helps me flesh out the internal logic of the system in the form of a graph.

Find a topological ordering. This is just a fancy way of saying we want to order the concepts so that each concept only depends on the ones already introduced. Sometimes this is straightforward, but sometimes the dependency graph has cycles!

Of the topics I’ve taught recently, Bitcoin seems especially difficult in this regard. Each concept is bootstrapped off of the others, but somehow the system magically works when you put everything together. What I do in these cases is introduce intermediate steps that don’t exist in the actual design I’m teaching, and remove them later [1].

Think of a technical topic as a skyscraper. When it’s presented in a paper, it’s analogous to unveiling a finished building. The audience can admire it and check that it’s stable/correct (say, by verifying theorems or other technical arguments.) But just as staring at a building doesn’t help you learn how to build one, the presentation in a typical paper is all but useless for pedagogical purposes. Having dependencies between concepts is perfectly acceptable in papers, because papers are not meant to be read in a single pass.

The instructor’s role, then, is to reverse engineer how the final concept might plausibly be built up step by step. This is analogous to showing the scaffolding of the building and explaining each step in its construction. Talks and lectures, unlike papers, must necessarily have this linear form because the audience can’t keep state in their heads.

[1] This process introduces new nodes in the dependency graph and removes some edges so that it is no longer cyclic.

To stay on top of future posts, subscribe to the RSS feed or follow me on Twitter or Google+.

How to pick your first research project

At Princeton I get to advise many gifted graduate and undergraduate students in doing research. Combining my experience as a mentor with reflecting on my experience as a student, I’d like to offer some guidance on how to pick your first research project.

I’m writing this post because selecting a research problem to work on is significantly harder than actually solving it. I mean the previous sentence quite literally and without exaggeration. As an undergraduate and early-stage graduate researcher, I repeatedly spent months at a time working on research problems only to have to abandon my efforts because I found out I was barking up the wrong tree. Scientific research, it turns out, is largely about learning to ask the right questions.

The good news is that three simple criteria will help you avoid most of the common pitfalls.

1. Novelty. Original research is supposed to be, well, original. There are two components to novelty. The first is to make sure the problem you’re trying to solve hasn’t already been solved. This is way trickier than it seems — you might miss previous research because you’re using different names for concepts compared to the standard terminology. But the issue is deeper: two ideas may be equivalent without sounding at all the same at a superficial level. Your advisor’s help will be crucial here.

The other aspect to novelty is that you should have a convincing answer to the question “why has this problem not been solved yet?” Often this might involve a dataset that only recently became available, or some clever insight you’ve come up with that you suspect others missed. In practice, one often has an insight and then looks for a problem to apply it to. This means you have to put in a good bit of creative thinking even to pick a research question, and you must be able to estimate the difficulty level of solving it.

If your answer to the question is, “because the others who tried it weren’t smart enough,” you should probably think twice. It may not be prudent to have the success of your first project ride on your intellectual abilities being truly superlative.

2. Relevance. You must try to ensure that you select a problem that matters, one whose solution will impact the world directly or indirectly (and hopefully for the better). Again, your advisor’s help will be essential. (That said, professional researchers do produce massive volumes of research papers that no one cares about.) I encourage my students to pick subproblems of my ongoing long-term research projects. This is a safe way to pick a problem that’s relevant.

3. Measurable results. This one becomes automatic as you get experienced, but for beginning researchers it can be confusing. The output of your research should be measurable and reproducible; ideally you should be able to formulate your goals as a testable hypothesis. Measurability means that many interesting projects that are novel and make the world better are nevertheless unsuitable for research. (They may be ideal for a startup or a hobby project instead.) “Build a website for illiterate kids in poor countries to learn effectively” is an example of a task that’s hard to frame as a research question.

Irrelevant criteria. Let me also point out what’s not on this list. First, the general life advice you often hear, to do something you’re passionate about, is unfortunately a terrible way to pick a research problem. If you start from something you’re passionate about, the chance that it will meet the three criteria above is pretty slim. Often one has to consider a dozen or more research ideas before settling on one to work on.

You should definitely pick a research area you’re passionate about. But getting emotionally invested in a specific idea or research problem before you’ve done the due diligence is a classic mistake, one that I made a lot as a student.

Second, the scope or importance of the problem is another criterion you shouldn’t fret much about for your first project. Your goal is as much to learn the process of research as to produce results. You probably have a limited amount of time in which you want to evaluate if this whole research thing is the right fit for you. While you should definitely pick a useful and relevant research task, it should be something that you have a reasonable chance of carrying to fruition. Don’t worry about curing cancer just yet.

Note that the last point is at odds with advice given to more experienced researchers. Richard Hamming, in a famous talk titled “You and your research,” advised researchers to pick the most important problem that they have a shot at solving. I’ve written a version of the current post for those who’re in it for the long haul, and my advice there is to embrace risk and go for the big hits.

To stay on top of future posts, subscribe to the RSS feed or follow me on Twitter or Google+.

The job talk is a performance

This is the second in a series of posts with advice for computer science academic job candidates.

One shot, one opportunity

The philosopher Marshall Mathers once asked rhetorically, “Look, if you had one shot, or one opportunity / To seize everything you ever wanted in one moment / Would you capture it or just let it slip?”

He added, “Yo.” [1]

I don’t mean to imply that an academic position is everything you ever wanted, but it’s a pretty good life (although not for these reasons). Like it or not, it’s set up so that your career up until this point comes down to one moment. After years of hard work, your ability as a researcher will be judged primarily based on how you sell yourself in the fleeting span of an hour. Of course, you’ll (hopefully) give your talk at many places, but it’s going to be the same talk!

There’s a reason I’m saying this, and it’s not to stress you out even more. Rather, if at any point the level of preparedness that I suggest seems excessive or disproportionate, remember the wise words quoted above.

Public speaking is a performance

My first piece of advice is to read the book Confessions of a Public Speaker.[2] As in, don’t even think about giving your job talk without having read it. You can read it in a sitting; putting it into practice will of course take longer. I cannot overstate the impact this book had on my talk (and my public speaking in general). There are probably other books that capture much of the wisdom in Confessions, and I’d love to hear other recommendations, but if you’re going to read one book it would have to be this one.

There are numerous very useful little details in the book, but it has one central idea that can be boiled down to the phrase “public speaking is a performance.” Job talks are are even more of a performance than public speaking in general, since the audience is specifically there to judge you.[3] This is a generative metaphor — it allows you predict things about your job talk based on what you know about performing. Fully appreciating the metaphor will require reading the book, but here are two such predictions that might otherwise be surprising.

Your first priority is to entertain

Certainly you must both entertain and inform, but the point is that you don’t really have a shot at the latter if you fail at the former. Sitting in a lecture, as everyone who remembers their student days is surely aware, can be excruciating; it’s an extremely unnatural situation from an evolutionary perspective (again, read Confessions to appreciate why.) The chart below from the book What’s the Use of Lectures? shows students’ heart rate over time as they sat in a lecture. It’s only a drop of a few beats per minute, but it translates to an enormous difference in alertness.If you don’t do anything different in your talk and simply present your material, your audience’s attention level will be greatly diminished by the half-hour mark, and by the end of your talk people will basically be comatose. Anything you can do to break the routine, linear, hyper-boring pattern of a lecture will help jolt the audience out of their stupor. (That includes asking questions — I usually asked two or three in my talk.) Otherwise they won’t be excited about you nor remember much after the talk.

Rehearse, rehearse, rehearse

You may have heard “practice, practice, practice.” I’d rather cast it in the language of performance, as there are some subtle differences. For example, when people tell you to practice, they tell you not to overdo it because you’d lose your spontaneity. I disagree. In a rehearsal, everything is practiced down to the last detail. In fact, the apparently spontaneous things that I said my talk were the most well-rehearsed parts.

Rehearsal should include videotaping yourself and watching it. Yes, it’s painful and majorly cringe-inducing, but it’s absolutely, absolutely essential. In addition to all the obvious facets of good presentation style that I won’t repeat, one of the subtle but important things you should watch for is nervous tics or other repetitive behaviors — almost everyone has one or more of those, and they can almost derail your talk by distracting your audience.

The reason rehearsal makes such a huge difference is that when you’re delivering a rehearsed talk, your every word and gesture is subconscious, freeing up your mental bandwidth for observing and reacting in real-time to the facial expressions of your audience. There are never more than 40-50 people in these talks, a small enough number that you can instantly notice if someone looks confused, skeptical, or bored. But this won’t be possible if you have to think through your slides instead. The reduction in cognitive load also minimizes the chance of “hitting the wall,” a phenomenon of sudden mental fatigue that’s a serious danger in long-ish talks and can leave you helpless.

Let me close with an example of a little theatrical thing I did that shows the value of rehearsal and the performance metaphor. One of the goals in my location privacy project is to minimize smartphone power consumption. When I got to that part, I’d say, “those of you with Android phones know how bad the battery life is. In fact I usually carry a spare battery around… actually, I think I have it on me.” Then I’d pull a smartphone battery out of my jacket pocket with a bit of a dramatic touch. Somehow the use of a physical prop seemed to reframe their thinking from “yet another academic paper” to “solving a real problem.” It would also usually elicit a laugh and elevate their attention level.

There is so much more to say about job talks, not to mention other aspects of the job interview. I might do follow up posts on a mathematical model of audience behavior and/or an explanation of why slide transitions are (by far) the most important part of your slides.

[1] This post was written while listening to Lose Yourself in a loop.

[2] If it needs to be said, I have no stake in the book, financial or otherwise.

[3] I hasten to add that teaching is very different from public speaking and is emphatically not a performance.

To stay on top of future posts, subscribe to the RSS feed or follow me on Twitter or Google+.

Embracing failure: How research projects are like startups

As an academic who’s spent time in the startup world, I see strong similarities between the nature of a scientific research project and the nature of a startup. This boils down the fact that most research projects fail (in a sense that I’ll describe), and even among the successful projects the variance is extremely high — most of the impact is concentrated in a few big winners.

Of course, research projects are clearly unlike startups in some important ways: in research you don’t get to capture the economic benefit of your work; your personal gain from success is not money but academic reputation (unless you commercialize your research and start an actual startup, but that’s not what this post is about at all.) The potential personal downside is also lower for various reasons. But while the differences are obvious, the similarities call for some analysis.

I hope this post is useful to grad students in particular in acquiring a long-term vision for how to approach their research and how to maximize the odds of success. But perhaps others including non-researchers will also find something useful here. There are many aspects of research that may appear confusing or pathological, and at least some of them can be better understood by focusing on the high variance in research impact.

1. Most research projects fail.

To me, publication alone does not constitute success; rather, the goal of a research project is to impact the world, either directly or by influencing future research. Under this definition, the vast majority of research ideas, even if published, are forgotten in a few years. Citation counts estimate impact more accurately [1], but I think they still significantly underestimate the skew.

The fact that most research projects don’t make a meaningful lasting impact is OK — just as the fact that most startups fail is not an indictment of entrepreneurship.

A researcher might choose to take a self-interested view and not care about impact, but even in this view, merely aiming to get papers published is not a good long-term strategy. For example, during my recent interview tour, I got a glimpse into how candidates are evaluated, and I don’t think someone with a slew of meaningless publications would have gotten very far. [2]

2. Grad students: diversify your portfolio!

Given that failure is likely (and for reasons you can’t necessarily control), spending your whole Ph.D. trying to crack one hard problem is a highly risky strategy. Instead, you should work on multiple projects during your Ph.D., at least at the beginning. This can be either sequential or parallel; the former is more similar to the startup paradigm (“fail-fast”).

I achieved diversity by accident. Halfway through my Ph.D. there were at least half a dozen disparate research topics where I’d made some headway (some publications, some works in progress, some promising ideas). Although I felt I was directionless, this turned out to be the right approach in retrospect. I caught a lucky break on one of them — anonymity in sanitized databases — because of the Netflix Prize dataset, and from then on I doubled down to focus on deanonymization. This breadth-then-depth approach paid off.

3. Go for the big hits.

Paul Graham’s fascinating essay Black Swan Farming is about how skewed the returns are in early-stage startup investing. Just two of the several hundred companies that YCombinator has funded are responsible for 75% of the returns, and in each batch one company outshines all the rest.

The returns from research aren’t quite as skewed, but they’re skewed enough to be highly counterintuitive. This means researchers must explicitly account for the skew in selecting problems to work on. Following one’s intuition and/or the crowd is likely to lead to a mediocre career filled with incremental, marginally publishable results. The goal is to do something that’s not just new and interesting, but which people will remember in ten years, and the latter can’t necessarily be predicted based on the amount of buzz a problem is generating in the community right now. Breakthroughs often come from unsexy problems (more on that below).

There’s a bit of a tension between going for the hits and diversifying your portfolio. If you work on too few projects, you incur the risk that none of them will pan out. If you work on too many, you spread yourself too thin, the quality of each one suffers, and lowers the chance that at least one of them will be a big hit. Everyone must find their own sweet spot. One piece of advice given to junior professors is to “learn to say no.”

4. Find good ideas that look like bad ideas.

How do you predict if an idea you have is likely to lead to success, especially a big one? Again let’s turn to Paul Graham in Black Swan Farming:

“the best startup ideas seem at first like bad ideas. … if a good idea were obviously good, someone else would already have done it. So the most successful founders tend to work on ideas that few beside them realize are good.”

Something very similar is true in research. There are some problems that everyone realizes are important. If you want to solve such a problem, you have to be smarter than most others working on it and be at least a little bit lucky. Craig Gentry, for example, invented Fully Homomorphic Encryption mostly by being very, very smart.

Then there are research problems that are analogous to Graham’s good ideas that initially look bad. These fall into two categories: 1. research problems that no one has realized are important 2. problems that everyone considers prohibitively difficult but which turn out to have a back door.

If you feel you are in a position to take on obviously important problems, more power to you. I try to work on problems that everyone seems to think are bad ideas (either unimportant or too difficult), but where I have some “unfair advantage” that leads me to think otherwise. Of course, a lot of the time they are right, but sometimes they are not. Let me give two examples.

I consider Adnostic (online behavioral advertising without tracking) to be moderately successful: it has had an impact on other research in the area, as well as in policy circles as an existence proof of behavioral-advertising-with-privacy.[3] Now, my coauthors started working on it before I joined them, so I can take none of the credit for problem selection. But it’s a good illustration of the principle. The main reason they decided this problem was important was that privacy advocates were up in arms about online tracking. Almost no one in the computer science community was studying the topic, because they felt that simply blocking trackers was an adequate solution. So this was a case of picking a problem that people didn’t realize was important. Three years later it’s become a very crowded research space.

Another example is my work with Shmatikov on deanonymizing social networks by being able to find a matching between the nodes of two social graphs. Most people I talked to at the time thought this was impossible — after all, it’s a much harder version of graph isomorphism, and we’re talking about graphs with millions of nodes. Here’s the catch: people intuitively think graph isomorphism is “hard,” but it is in fact not NP-complete and on real-world graphs it embarrassingly easy. We knew this, and even though the social network matching problem is harder than graph isomorphism, we thought it was still doable. In the end it took months of work, but fortunately it was just within the realm of possibility.

5. Most researchers are known for only one or two things.

Let me end with an interesting side effect of the high-skew theory: a successful researcher may have worked on many successful projects during their career, but the top one or two of those will likely be far better known than the rest. This seems to be borne out empirically, and a source of much annoyance for many researchers to be pigeonholed as “the person who did X.” Let’s take Ron Rivest who’s been prolific for several decades not just in cryptography but also in algorithms and lately in voting. Most computer scientists will recall that he’s the R in RSA, but knowledge of his work drops off sharply after that. This is also reflected in the citation counts (the first entry is a textbook, not a research paper). [4]

In summary, if you’re a researcher, think carefully about which projects to work on and what the individual and overall chances of success are. And if you’re someone who’s skeptical about academia because your friend who dropped out of a Ph.D. after their project failed convinced you that all research is useless, I hope this post got you to think twice.

I may do a follow-up post examining whether ideas are as valuable as they are held to be in the research community, or whether research ideas are more similar to startup ideas in that it’s really execution and selling that lead to success.

[1] For example, a quarter of my papers are responsible for over 80% of my citations.
[2] That said, I will get a much better idea in the next few months from the other side of the table :)
[3] Specifically, it undermines the “we can’t stop tracking because it would kill our business model” argument that companies love to make when faced with pressure from privacy advocates and regulators.
[4] To be clear, my point is that Rivest’s citation counts drop off relative to his most well-known works.

Thanks to Joe Bonneau for comments on a draft.

To stay on top of future posts, subscribe to the RSS feed or follow me on Twitter or Google+.

Five Surprises from My Computer Science Academic Job Search

Next in series: The job talk is a performance

I’ve just about settled into a rhythm at Princeton — classes started two weeks ago — and next year’s academic job search cycle is already underway! Indeed, I started my job search in earnest almost exactly a year ago. So I guess surprise number zero from this whole process is how time-consuming it was. There’s been no ‘normal’ or ‘routine’ during this year; each month has been unlike the previous. If you’re starting your academic job search, buckle up, it’s gonna be a wild ride!

There’s lots of advice online about the process; you should read all of it. Instead of duplicating what’s been said, I will focus on the things that surprised me in spite of having prepared as well as I possibly could. So if the rest of this post appears a bit contrarian, it’s just selection bias.

1. You’ll need someone to hold your hand. I can’t overstate how much of a difference it makes to have someone who’s been through the process whom you can talk to on a regular basis during your job search. Whether it’s achieving the right depth-breadth balance in your job talk, or wording emails strategically/diplomatically, or knowing how to best space out your interviews, you can’t figure it out by yourself or by reading online advice.

Typically the person helping you will be your advisor, but if they are busy you should find someone else. The good news is that many people will be willing to help out and pay it forward. It doesn’t have to be one person, you can split it between two or three people. I know that I would have screwed it up many times over if it hadn’t been for my advisor and everyone else who helped me out.

Some candidates networked extensively both to compare notes with other job searchers and to obtain and share privileged information. I avoided this entirely because I didn’t want the stress associated with it, and I’m very happy with my decision. That said, maybe I missed out in some way, I don’t know.

2. It’s not an interview. Perhaps this should have been obvious, but I was taken aback during my first “interview.” People already assume you’re an expert in your subfield, and so they aren’t trying to assess your technical competence. At all. I was tested exactly once in my whole tour — a professor asked me to state and sketch a proof of any theorem (of my choice) from my Netflix paper. Another professor apparently found this egregious, so he later wrote me a rather apologetic email. I found the whole thing rather amusing.

Best as I can tell, what they’re trying to assess is your personality (more bluntly, they want to make sure you’re not an asshole), and whether they can collaborate with you. So everyone was extremely polite to me and never asked adversarial questions or gave me much pushback.

One consequence of this interview style is that no one reads your papers, because they don’t need to. I already knew that no one read my papers in the normal course of things, but before the interview season I thought, “Finally, a few dozen people are going to read my papers!” Didn’t happen. Maybe it has something to do with me, but I think a big part of the reason is that in computer science we don’t seem to have a culture of reading beyond the abstract or introduction of papers (except in reviews, or reading groups, or when directly extending previous work.) Knowing this, authors don’t have an incentive to write in a readable manner, and the cycle is self-perpetuating. But I digress.

A happy side-effect of non-interview interviews was that the process wasn’t mentally exhausting. It was sort of like meeting a bunch of people for coffee and chatting for half an hour with each one. Since all the advice I’d gotten suggested that interviews would leave me dead tired, I was initially worried that I might be doing something wrong ­— maybe I wasn’t having sufficiently technical conversations? I suspect the real difference is that most people get exhausted due to being pumped full of adrenaline; due to a biological luck of the draw I don’t generate any noticeable adrenaline in these situations (including right before talks, which I’ve found a bit surprising).

3. You don’t have to interview them. Everyone else dispensing online advice seems to think, “you’re not just being interviewed; you’re also interviewing them.” I disagree. In one or two cases it was obvious during my interview that the school or department wouldn’t be a good fit for me without even having to ask them specific questions, but absent any obvious issues, I’m skeptical of how much you can determine by asking. Of course you should discuss areas of possible collaboration with people you meet, but to determine things like how good the students are, how effective the administrative staff are, etc., asking directly is not very useful.

The reason is that people will always spin things in a favorable way (this is not a criticism — most of the time they do it because they’ve been there long enough that they’ve adjusted to the situation and they actually see things the way they spin them.) And you’re not experienced enough to parse what you’re told to figure out what the reality is. You should definitely ask them questions lest they think you’re uninterested, but receive all information with a skeptical ear.

Instead, what was extremely useful for me is to talk to people who’d previously been at the departments I was considering, ask them what they didn’t like, then present that information back to people in my interview loop and ask them for their take, and finally try to reconcile the two views.

4. The job talk is a strategic piece of communication. There is so much subtext it blew my mind. For one, your job talk is all about telling people how awesome your work is, but of course you can’t state that directly. You’ll need to humblebrag without being obvious. For example, in my closing slide, I put up a collage of 24 faces, and said, “Finally, I’m incredibly grateful to my amazing co-authors without whom none of my work would have been possible.” That statement was certainly true, but also important was the subtext: “I collaborate like it’s going out of style.” This one was balanced precariously on the obvious threshold — I got called out in one of my talks!

But there’s more. You have to consider every single thing that you say from the point of view of someone in your field, someone not in your field but familiar with it, and someone not familiar with your field. It has to make sense at different levels to all of them. Also, you have to consider how each statement will sound to someone who spaced out for a bit and just started paying attention. And so on.

Overall, this isn’t going to be like any talk you’ve given. I think I spent 3-4 weeks working primarily (albeit not exclusively) on my talk, with regular tweaking afterwards.

5. You will fall sick. Airports and airplanes spread germs, plus you’re much more susceptible to infection when your sleep and diet are irregular, as they likely will be during your tour. Assuming you have a moderately busy schedule, falling sick is just a matter of time. I mentioned being sick to about 4-5 people, and each of them recalled how they had fallen sick during their own job search.

Naturally, then, you should treat proper sleep and diet as a priority. You should schedule your interviews so that you have time to recover when it happens. Also, try not to schedule your two most important interviews too close together. Finally, there’s a lot you can do in terms of symptom relief (e.g., benzocaine cough drops instead of menthol) to minimize the impact on your thinking and speaking during your interviews, so be medically prepared ahead of time.

That’s it for now. There are several topics that I’d like to address in more detail in separate posts, time permitting: 1. how to prepare for and deliver the talk; 2. what to say in your 1-on-1 meetings; 3. travel tips, and 4. suggestions for interviewers from the point of view of a candidate. If you’ve been through this process recently, I’d love to hear how your experiences matched or differed from mine.

Finally, Princeton CS is hiring this year, and our searches aren’t targeted by subfield, so if you’re on the market you should apply!

To stay on top of future posts, subscribe to the RSS feed or follow me on Google+.