How to Organize Your Notes for Optimal Research Productivity

When I first started to write notes for my PhD work, I remember trying to keep notes in a single .txt file on my laptop. As new ideas came in, I just added them to the file. Eventually, that file got huge, making it really hard to find anything useful when I needed it. I eventually abandoned the .txt file, losing all that content in the process. Then, I switched over to writing my notes in composition notebooks. The quad ruled ones were and still are my favorite. My note taking was a bit better in the quad ruled notebooks, but I still had the same problem. How could I find relevant notes after I had written them? Once I’ve filled an entire notebook with ideas, diagrams, and results, how on earth do I find them afterward? I basically had to flip through every single page, unless I knew for sure that what I was looking for was near the beginning or the end of the notebook. After years of failed attempts at note taking, I finally realized that the medium didn’t matter. I had the same problem digitally as I did on paper. I needed a system for organizing my research notes.

I laughed when I realized what the problem was, given that I’m a researcher in data science. It is literally my job to make it easier for other people to organize, explore, and make sense of their data. The same principles apply to my own data as well. After coming to this realization, I made it my mission to develop a note taking system that actually works for me. After ten years of trying, I believe I finally have a system that works. It is magical! In this post, I share my note taking strategy, so you can apply it to your own computer science research.

The Key Idea: Focus on Connections, Not Just Content

My goal with note taking is to build a powerful information system to support my research. But how do we structure such a system? To answer this question, consider the structure of two of the most powerful information systems in the world. One is the brain, which contains an interconnected network of billions of neurons used to pass information from one part of the brain to another. In machine learning, neural network architectures try to mimic how the brain uses neurons to build vast networks of information. Another example is the internet, which is an interconnected network of digital information spread all over the world. You can be on the opposite side of the world from me and through the internet you can still read this post. Within the internet, consider Google search, specifically the Page Rank algorithm that made Google famous. The way Page Rank works is it analyzes which webpages are connected to each other through hyperlinks. The more tightly clustered a set of pages are through hyperlinks, the more likely they are to be semantically related. Furthermore, the pages that are linked to the most are likely to be the most authoritative pages within the clusters. In other words, Google exploits the connectivity of the internet to power its search engine. See a theme here? To apply this concept to research notes, we simply substitute neurons and pages with individual notes. In this case, the content of a note matters, but how it connects to other notes (and our brains!) is even more important.

High-Level Steps to Writing Excellent Research Notes

Here is a quick summary of my key strategies for writing great research notes:

  1. Write everything down
  2. Tag notes by category, concept, or theme
  3. Link related notes together
  4. Create a table of contents for each major project

In the rest of this post, I describe each step in detail and provide some helpful tips to boost the utility of your research notes.

Step 0: Choose Where to Write Your Notes

If you do not currently take notes, then you will need to choose a note taking application to write your notes in. My favorite note taking application is Evernote, so I will be using it in all of my examples. However, you can apply these same principles to any other note taking strategy, including paper notebooks. Check out commonplace books for examples of how people applied similar concepts throughout history. However, having access to hyperlinks and search bars makes navigating related notes much easier. Also, digital notebooks never run out of pages! So I recommend going digital if you can.

I wrote up some helpful tips on picking a good note taking app in a previous post. If you already have a note taking app that you enjoy using and it has the features that I recommend, just start there. Don’t delete anything and don’t make a new notebook. Just use the default notebook for now. The beauty of my note taking strategy is that it doesn’t matter whether all the notes are in one place or not. To keep things easy though, we will place them all in the default notebook. If you are a bit more advanced and want a separate notebook to test things out, then feel free to create a new one and change the settings to make it your new default notebook.

The most important part of Step 0 is to be consistent. Keep all of your notes in the same place. For example, I keep all of my research notes in Evernote. For me, if it’s not in Evernote, then it doesn’t exist. This way, I never have to remember where I put my notes. Do you want to migrate to a new note taking app? No problem, but make sure you copy all of your notes to the new app. If you have hundreds of notes this might sound intimidating, but trust me it’s worth it. If you don’t want to do it all at once, then do the following:

  1. Just migrate 5 notes per workday. The most important thing is to be consistent. Do what you know you can handle on a daily basis and do not stop!
  2. Do not tell yourself that you can just “get it all done on the weekend.” Find a way to do the daily migration on workdays only.
  3. For each note you successfully migrate to the new app, delete it from the old app. That way, you always know exactly what still needs to be migrated.
  4. If you are pressed for time, then go down to 1 note per workday, but do not stop migrating! Then, go back up to 5 notes per workday as soon as you can.
  5. If you have more time, try to migrate more notes, for example 10 notes instead of 5 notes. But remind yourself this is just a temporary boost. Go back to 5 notes the next workday.
  6. As a bonus, start tracking your migration using your favorite habit tracking method. For example, you could use a habit tracking app, check off a daily task in your preferred calendar app, cross days off on a paper calendar, and so on. Do what works best for you, but as Jerry Seinfeld famously recommends: don’t break the chain.

This simple trick of breaking big projects down into easy micro-tasks and doing them every single day is the reason why I started this blog. This strategy applies to everything in research, and arguably everything in life. My goal is to show how to apply this method to any and all aspects of computer science research in future posts.

Step 1: Write Everything Down

To build an information system, we need information and lots of it. Also, with an open-ended discipline like computer science research, we never know which notes will lead to new scientific breakthroughs. For these reasons, we need to write everything down in our research notebooks. However, much like how I can’t just write crappy, disorganized webpages and expect to rank highly on Google search, I can’t haphazardly write disorganized research notes either. The information stored within each top ranked webpage has to be organized a certain way for Google to recognize its value. In the same way, each research note has to have a detailed and consistent structure.

My biggest tip for writing detailed notes is to always write the note as if someone else, for example a new student joining a research project, will need to use the note. An even better way to look at it is to treat notes like webpages that someone else might search for online. Is the note content going to help a random person searching for this information online? If not, then it probably will not help you either, at least once the original context is forgotten. Forgetting the context only takes a couple of months, but research projects are completed on the scale of years. Do your future self a favor and write everything down now. My second biggest tip is to write a descriptive title. Imagine all those Google search results you ignored because the webpage titles were too vague. The same holds true for searching your own notes. Again, make life easy for future you and just write the details down now.

I’ve written an entire post on how to structure individual research notes, and describe four types of notes to focus on: ideas notes, process notes, reference notes, and results notes:

  • My ideas notes capture new ideas in the moment so I never lose them.
  • My process notes record step-by-step guides for critical yet complicated research routines.
  • My reference notes manage external resources that I turn to again and again as I do my work.
  • My results notes record the findings and implications of completed research tasks.

If you want a concrete example of how to write ideas notes in a structured way, check out this Evernote template that I made. It provides all the details you will want to track for your ideas. You don’t have to use the whole template, and can complete each part at your own pace. For example, I typically write the core idea down as soon as possible, then add relevant papers and whatnot over the following days, weeks, or even months, depending on how long it takes me to get back to the idea. If you want detailed tips on how to write all four types of notes, please check out the original post linked above.

Step 2: Tag Notes by Category, Concept, or Theme

After you’ve written a new note, the next step is to help your note app understand how it connects to the rest of your information system. This step matters because keyword search is not always that effective out of the box. Unlike bloggers and website owners, when I write notes, I’m just writing my thoughts down, and not necessarily trying to maximize keyword relevance and back links. This is exactly how notes should be written. However, this makes keyword searches over your notes less effective than online keyword searches over websites. That being said, there are steps we can take to make keyword search work better. One step that will definitely help is to use your note taking app’s tagging feature.

The key idea with tags is to include information that you might not include in the body of the note. To keep things simple, I recommend using the following types of tags:

  • Note Type Tags: for the type of note you wrote (idea, process, reference, result)
  • Project Tags: for the research project this note belongs to.
  • Task Type Tags: for the type of work you were doing as part of this note, such as “paper writing”, “thesis writing”, “user study”, “proposal/fellowship”, “job hunting”, “teaching”, etc.

You could also add tags for research areas, such as “visualization” or “databases”, but I would avoid them since different areas may go by multiple names. For example, “database” research could also be “data management” research or “database management” research, or even “data science” research. Furthermore, this content will likely end up in the body of your note anyway and so will be redundant as a tag. Save yourself the trouble and avoid adding these kinds of tags. For conferences, you will probably put the conference name in the body of the note, so there’s no need to add a dedicated tag. The same goes for course numbers if your note is related to a course.

I often like to include tag names in the title as well. For example, research idea notes I might title as “Research Idea: [rest of title]”. This helps me skim notes while reviewing search results in Evernote.

Step 3: Link Related Notes Together

My absolute favorite feature of Evernote is internal links. This feature is arguably the single most important feature for building a robust information system! I prefer to find relevant notes by clicking a couple of links rather than having to use the keyword search feature.

Never used this Evernote feature before? All you have to do is:

  1. Right click on the thumbnail of the note in the notes view on the left hand side of the Evernote window.
  2. A menu should pop up, like the one below.
  3. Click on the “Copy Note Link” menu item.
  4. Paste the link within a different Evernote note as desired.
To get the internal hyperlink for a note in Evernote, right click on the note. You will see a menu pop up with an option to “Copy Note Link”.

When you paste a note link in Evernote, it will look like this:

Internal note links will appear green in Evernote and show the title of the note.

The link will be green instead of blue, and the title of the note will automatically be used to describe the link. As I work on projects, I try to always include links at the top of the note to other relevant notes. For example, if I am analyzing a dataset three different ways, I may make each analysis a separate note, and add the links for the other two analyses at the top of each note. Here’s an example from my own notes. This note is from an old project where one of the main contributions was a literature review:

An example of how I like to include links to related notes in Evernote as I do my research. This example is from a literature review project.

We had to iterate on the clusters of papers we wrote about, and the text for the literature review itself. I kept track of these previous iterations using Evernote links.

However, you may get to a point in your research project where it becomes a huge chore to copy all of the links over all the time for each new note. Furthermore, not every note about a project is relevant to every other note for that same project. This is where the last step becomes crucial.

Step 4: Create a Table of Contents for Each Major Project

There is actually a fifth type of note that I use all the time in my research notes. However, this note type is more organizational than content focused. This is why I don’t include it in my four types of notes. The fifth type I call a “Table of Contents” note. This note is the master list for all relevant internal and external links for a given project. Here’s an example for an old research project:

An example of a partial Table of Contents note from an old project. I used this note to keep track of later revisions for a conference paper.

For this project, I conducted a user study and had many smaller analyses I needed to do using the study data. I made separate notes for those analyses, and linked them in the table of contents. I also had to submit the paper a total of three times, and wrote a lot of notes on how to improve the paper for each submission. I linked those notes in the table of contents as well. As you can see, I had a lot of notes to keep track of for this one project! Furthermore, I did not start my table of contents until after my paper was rejected the first time, so there are older notes that did not make it into this table of contents example. My recommendation is to save yourself some trouble and create a table of contents note at the very beginning of the project, that way you can just maintain the same note the entire time. If you end up having to spend a year or more on a project, then you will be glad you linked all of your relevant project notes in one place.

When writing papers or generally collaborating with other people on a project, I also write content outside of Evernote, such as in google docs or Overleaf. Table of contents notes are also great for capturing these links. Furthermore, if there are specific resources that I need to reference for the project, such as submission guidelines for a particular conference or award program, I include those links as well. Here’s another example for a (rejected) proposal I wrote last year:

A table of contents note that I created to manage various notes for a proposal submission. I use table of contents notes to track internal and external links in one place.

Something I love to do is to add table of contents notes to my Evernote shortcuts. That way I can always go straight to the table of contents for an active project to find a relevant note. Once the project is no longer active, for example once the corresponding paper is finally published, then I remove the note from my shortcuts.

Bringing Everything Together: Some Helpful Guidelines

I provide specific steps above for creating an effective information system to support your research. Although following these steps has worked well for me in the past, you may want to adapt these steps or even create new ones to better suit your needs. Here are the key principles to keep in mind when extending these steps:

  • Write your thoughts down ASAP: It is difficult if not impossible to remember every detail and every outcome from years of research. Even if you choose to write different kinds of notes or focus on different note details, the most important thing is to write down your thoughts as soon as you have them! Write down as many details as possible to capture the full context. If you do not do this, you will hinder your growth as a researcher.
  • Keep everything in one place: If you are like me, then you tend to lose things when they are not in an intuitive place. My life is much easier when I have exactly one place to look for things. Even if you hate Evernote or prefer paper notebooks, I hope we agree that keeping notes in a single notebook or a single app is easier than keeping notes in multiple places.
  • Cluster related notes: I shared three ways that I cluster relevant notes (keywords, links, table of contents). The key idea is for notes that are related to have an established connection that you can use to navigate easily within the cluster. Even if you don’t remember the exact details of a specific note, you may still remember the general topic or theme and can use these connections to find it.

Summary

Writing good research notes is all about the connections, how different ideas connect to one another across notes. Connecting disparate ideas together is the definition of creativity and the cornerstone of scientific innovation. If you want to accelerate your ability to make these connections, then you absolutely must organize your research notes! In this post, I described my four step process to organizing my notes, and the key principles behind my organization strategy. Check out the infographic summarizing the steps below. If you want more details on how I organize individual notes, check out my previous post.

Four Kinds of Notes to Boost Your Research Productivity

a snapshot of a note written in Evernote. The note contains a list of quotes about data, science, and knowledge.

I am obsessed with writing notes. I document a lot of my personal and professional life through notes. If I come up with a new research idea, I write a note about it. If I come across a relevant paper, I write about it in my notes. If I find a recipe I like and use frequently, then I add it to my notes so I don’t have to search the web to find it again. Every time I read a new book, I write a note about it, which includes any quotes I liked and my review of the overall book. For example, I just finished reading Digital Minimalism by Cal Newport; here’s a quote from my notes: “A foundational theme in digital minimalism is that new technology when used with care and intention creates a better life than either Luddism or mindless adoption.” [2] I love to play disc golf, and I track all of my past plays using my favorite note taking app, Evernote.

Since I am an avid Evernote user, I share all of my examples exclusively in Evernote. If you are looking for a new note taking app, I share some helpful suggestions on how to pick the best note taking app for you in my previous post.

The Key Idea: Write Everything Down

The main theme behind this post is that for anything meaningful that we encounter in our work, we should be writing it down. For example, with an open-ended discipline like computer science research, we never know which notes will lead to new scientific breakthroughs. If we don’t write ideas down as we have them, then we risk losing them and in turn potential innovations and breakthroughs. For these reasons, I recommend writing everything down as research notes.

Although I focus on computer science research in this post, these ideas can apply to any part of our lives that we deem meaningful. For example, I write notes for all of my hobbies in Evernote. In fact, I often learn new note taking skills through the process of writing personal notes, which I then apply to my research notes. Writing good notes is a transferrable skill that I can use in every part of my life.

An Example Note

Here’s an example of a note that I use every single quarter, and it is just 5 lines of content in Evernote:

An example of a note I regularly use, written in Evernote. The note is about five lines and explains the process for creating a Python3 virtual environment.

This note explains how to create a Python3 virtual environment. I have written a templated example on the first line, followed by a concrete example that I frequently use, and finally a link to the original source of the information, in case I want to review the material. In this case, the source is a page from the official Python documentation. Technically, I could easily look this up online each time I want to create a new Python3 virtual environment rather than writing a note in Evernote, but this way if I don’t have internet access I can still look up the answer in my own notes. You might be wondering “don’t you eventually just memorize how to do this?” Sort of, but I never remember the exact details. Rather than guessing how to do it, or relying on my (often faulty) memory, it’s easier to write it down once and look it up later.

The above note is an example of documenting one of the processes I perform as I conduct my research. By knowing how I intend to use a note, I can structure it accordingly to maximize my ability to reuse it in the future. In this case, by documenting a step by step process for creating a Python 3 virtual environment, I make it easier for myself to create virtual environments later on.  Also, if a student asks me how to create Python3 virtual environments, rather than explaining or demonstrating the steps for every student individually, I can just share this one note in Evernote by turning on public sharing for it and sending each student the sharing link.

Writing Notes for Ideas, Processes, References, and Results

However, not every note should be written as a step by step process, depending on what the note represents. I group research notes into four different categories:

  • Ideas: new research questions or ideas I might want to explore in the future.
  • Processes: a detailed, step by step guide to completing a specific procedure that I performed in the past.
  • References: Bits of information or pointers to information sources that may be useful in the future.
  • Results: A description of my findings after completing a specific research process, ranging from 1 paragraph to multiple pages of content.

I think of Ideas posts as being “before” posts, or notes recording my thoughts before I conduct the research. Process notes are what I record while conducting the research, and results notes are what I document after conducting the research. Before, during, after. In this way, I can cover all aspects of the research process. Reference notes I think of as just-in-time notes, for specific situations that come up frequently but not with obvious regularity. To get you started on writing great ideas notes, I’ve created a template anyone can use, available here. I’ve also made an infographic summarizing the four note types, for easy reference!

The four note types I use to boost my research productivity. Although I provide a lot of examples from my research in computer science, the ideas apply to any kind of personal and professional work.

Writing Process Notes: Another Example

In the past, I wrote abbreviated versions of process notes, assuming I could “fill in the blanks” with my memory and past experiences. However, I have found that I often revisit process notes months or even years after I originally wrote them. By then, I have completely forgotten the original context in which I wrote the note, making the abbreviated version useless. My most vivid memories of these kinds of debacles involved installing the database management system SciDB. I had to re-install SciDB four times throughout my PhD, basically once per year. These were in the very early days of the software, when the installation process was difficult and largely undocumented. The first time, I didn’t bother taking any notes. Installing SciDB took several hours! Unfortunately for me, when I had to install it again months later to upgrade the software, I knew I was in for another multi-hour installation session, but this time I actually took notes. They weren’t perfect, but they had a lot of useful information. Each time after that, I refined my notes. Near the end of my PhD, when I installed SciDB on the MIT servers for the last time, I was able to complete the installation in under half an hour. Here’s a snippet of my SciDB note, to give you another example of a process note:

An example of a more complicated process note from my own research. In this case, I was tracking how to install SciDB in my earlier years as a PhD student.

It’s pretty hard to remember exactly how to do something you only do once a year, especially if it’s super complicated. For example, it’s not like people just stop using TurboTax because they’ve been doing their taxes for years.

Writing Great Process Notes: Some Tips

The key to writing great process notes is this: write down every step such that someone who is not you could follow them and successfully complete the procedure you are writing about. Even when you write notes only for yourself, you do not stay the same person year after year. An additional tip is to write process notes as if you are writing them for someone who has only baseline background knowledge, such as someone who is completely new to your research area. In this way, you ensure your notes are usable by anyone who might join your research project in the future.

When you use someone else’s tutorial to complete the steps, then make sure you add a link to the tutorial in your notes. That way, if you revisit the note and don’t remember a step, you can always return to the original source. I link back to the original sources in both of my process example above. However, a risk is that the tutorial or information page will eventually disappear, which has happened to me before. I recommend recording all of the important steps directly in your note in addition to the link. Sometimes I will paste steps directly from a tutorial page into my notes, such as for cooking recipes (not research, I know, but still useful!).

This goes without saying hopefully but write the steps in the order in which they should be performed, ideally using a bulleted list, or at least headers saying “step 1”, “step 2”, and so on. You see in my SciDB installation note that I use numbered lists to track steps. If order doesn’t matter, then mention this in the note.

Another tip is to practice searching your own notes before searching the web. The next time you go to look something up using your favorite search engine, just see what happens when you execute that search in your note taking app. For procedures you regularly perform, if you can’t find the answers in your own notes, then you should probably write a new one or update existing notes. Or if the answer changes and your notes go out of date, then you need to update them so they stay relevant. Developing a regular practice of using and reusing your own notes should help with note maintenance.

Writing Ideas Notes: an Example

Unlike my process notes, my ideas notes are pretty unstructured. Sometimes the ideas notes are just one line, like in this example:

An example of an ideas note. This idea eventually became one of the major research contributions for a conference paper that I published in 2019.

But these days I try to write ideas notes as self-contained thoughts, where I include the motivation, context, and if relevant the original inspiration for the idea, for example if a certain conversation or paper led to the idea. I prefer to write one note per idea, but this is not a strict rule. Here’s another example, which I originally wrote back in 2016:

An example of one of my ideas notes. This note mentions the system I built for my thesis, and some research questions of potential interest to the human-computer interaction and visualization communities. This note eventually became a research project that was also published as a separate conference paper in 2019.

I was at a conference presenting some work I did during a research internship the previous summer, and I was having lunch with my intern mentor. I was lamenting how I thought I couldn’t write a paper on my thesis work that the HCI or visualization communities would care about. My mentor pushed me to brainstorm ideas that were related to my thesis work and was kind enough to share some of his own ideas during the lunch, and I wrote down the best idea that came from our discussion. As it turns out, the idea recorded here eventually became a published paper three years later. The published project would have never happened without that initial discussion, but likewise the resulting project idea would have never come to fruition if it wasn’t remembered afterward (by me or Evernote).

Writing Great Ideas Notes: Some Tips

With ideas notes, the most important thing is to write down the idea as soon as possible so you don’t forget it. Add as many details as you have time for, but make sure you write every idea down, even the bad ones. If you are brainstorming and come up with multiple ideas, then feel free to list them in the same note. Here’s an example list from when I was brainstorming domain names for my blog site:

One of my ideas notes. This note shows many of the ideas I brainstormed for the Bits in Bites website name.

The list was so long that I struggled to get the entire note to fit into a single image! The name Bits in Bites doesn’t appear until near the bottom of the list, which is unfortunately off-screen in the image. As you can see, I came up with a lot of bad ideas 🙂 But what is key in these examples is that I didn’t limit myself when creating my ideas notes. Write as much as you can while the context is still fresh. Give your inner editor a break so your inner creative can shine through and generate lots of ideas and lots of details for those ideas, regardless of quality.

Writing Reference Notes: an Example

Reference notes are probably the most commonly written note in computer science research. You find an interesting bit of information and you want to save it for later, so you write a short note about it. I generally keep them fairly simple. I write a descriptive title about what the resource is, and then add a link to the original source. Here are some examples of reference notes I’ve written in the past:

I also keep reference notes for logistical things, such as my National Science Foundation (NSF) ID for submitting grant proposals, a link to the brand and logo information for my university, or a short description of my dietary restrictions for food orders for easy access.

Writing Great Reference Notes: Some Tips

Honestly, I don’t write a lot of reference notes. I am selective in what reference information I include in my notes. Why? Because reference notes are generally not connected to any specific research idea, process, or project, so they tend to be under-connected and therefore underutilized within my own information system. My general rule of thumb is to only add a reference note for content you have already had to look up multiple times. By the second time, you know that you have needed this information twice, and thus may need it again in the future. Here are some examples:

  • Professors in computer science periodically submit grant proposals to the NSF to raise funding for their research labs, myself included, so keeping a reference note on hand about one’s NSF submission ID makes sense.
  • My lab runs a lot of user studies, which we use to collect data to answer specific questions about human analysis behavior. We often run multiple statistical tests on the collected study data to determine the statistical significance of our findings. When you run multiple statistical tests over a dataset and measure significance using p-values, it is good practice to adjust the p-values based on how many times you reused the same dataset to conduct the tests, hence the reference above to Bonferroni correction.
  • I regularly present my work to other research labs and research groups, who generally ask for information about my presentation so they can advertise it within their organization. I often give the same presentation many times, so I keep reference information on hand that I can paste into an email or google doc for sharing purposes.
  • When attending a conference, daylong retreat, networking event, and so on, food may ordered for attendees. I have strict dietary restrictions so it is nice to have an existing explanation on hand that I can just copy and paste into a registration form or email.
  • When I read books, I like to save quotes about data, science, and knowledge. The last note in the gallery above is a snapshot of some of my favorite quotes so far.

However, there are some exceptions to this rule, since in academic research you never know when information will actually prove useful for a project. My typical exceptions are notes that I write when I learn from other researchers, such as when attending colloquium talks at my university, attending talks or chatting with fellow researchers at a conference, or writing notes on a paper I recently read. The third example above is a reference note for a paper I read on using GOMS models, a certain type of data-driven psychological model [2], to analyze website usability for people with visual impairments.

But again, these notes rarely connect to any particular idea or project in my information system, and as a result just end up floating around among my other notes with no established connections. For this reason, I try to only write reference notes for content I am likely to reference again in the future.

Writing Results Notes: an Example

Results notes are the most important note type in my information system. Basically any activity that stimulates my thinking on a particular research project or research idea I write down as a results note. For example, I use results notes to capture what I have learned from writing code, analyzing experiment results, and reading related papers for a given research project. Here are some examples:

As you can see, my results notes tend to be text heavy. They generally include a synthesis of my analysis findings, followed by my reasoning for why these findings are significant or interesting from a research perspective. The analysis findings themselves may be in the form of a visualization, such as in the first example, or even copy-pasted command line output, as in the fourth example. However, research is not relegated solely to running experiments and writing them up. In fact, a significant fraction of my research involves simply writing down my ideas and finding related research to bolster them. In the second results note example, I mention a paper I read at the time, and how the paper’s perspective on AI is interesting for visualization research as well. The third example is a snippet of an old draft of my doctoral dissertation. I often draft initial paper text in Evernote, then paste that text into a separate LaTeX editor to produce a shareable LaTeX project. This way, my paper text can stay close to my notes while I am still playing with nascent ideas, reducing context switching between applications. Later, when the ideas have matured more, then I may edit directly in a separate LaTeX editor, such as Overleaf or TexShop.

Writing Great Results Notes: Some Tips

I break results notes into two types: results from experiments, and results from thinking about my or others’ ideas. Both types are with respect to my current research project. I’ve written out separate steps that I recommend for each type below.

For experiments:

  1. Explain the motivations and goals behind conducting the experiment at the top.
  2. Describe your experiment as if someone will need to replicate it using your notes.
    1. If your experiments involve programming code, then add lots of comments to your code and include a detailed README.md file explaining how to run it.
    2. If possible, use virtual environments to setup the code, that way you can avoid most installation snafus in scenarios where you have to run the experiment on multiple or different computers.
  3. Write a summary of the analysis findings, so you don’t have to interpret the raw data directly in your notes.
  4. Include a reference to the raw data so you know where to find it. If it’s small enough, paste the raw data directly in a separate note and add an internal link to the results note.
  5. If performing this analysis for a paper, summarize your ideas into a concise paragraph or two that can be pasted into your paper.

For developing research ideas using existing sources:

  • Explain the motivations and goals behind the analysis at the top.
  • Describe the key sources that inform the current ideas and arguments. Include full citations, and even include quotes to ensure the source ideas and attributions are clear.
  • Include links to PDFs of the key sources, for easy access later on.
  • Write your own ideas and arguments in more detail, citing the key sources as needed.
  • If performing this analysis for a paper, summarize your ideas into a concise paragraph or two that can be pasted into your paper.

Getting Started: A Template

To help everyone get started, I created a structured template for writing great ideas notes. The template is available here. A handy summary of the different note types is also available in the infographic I shared above (also available on pinterest).

Summary

My philosophy is that my brain will work better if I don’t have to memorize as much of my work. My brain is like a room, I can make the most of its space only if I keep it uncluttered. To this end, I offload the memory burden from my brain by writing a lot of notes as part of my research and even my daily life. However, it is not enough simply to dump my thoughts into random documents. Imagine what the library would be like if books were haphazardly placed on shelves, and if even the content within a book was random from one page to the next. No one would find the library particularly useful and would probably avoid going there, except perhaps to amuse themselves. The same goes for my notes. I try to keep my notes organized and detailed, so I know exactly what to look for and where to look when I need it.

In this post, I share four kinds of notes that I write as part of my daily work: ideas, processes, references, and results. I provide examples of each type of note from my own database of notes in Evernote, and practical tips for making the most of each type of note. In my next post, I’ll explain how I organize these notes into a powerful information network to boost my research productivity.

References

[1] Card, S.K., Moran, T.P. and Newell, A., 2018. The psychology of human-computer interaction. Crc Press.

[2] Newport, C., 2019. Digital minimalism: Choosing a focused life in a noisy world. Penguin.

Applying the 80-20 Rule to Computer Science Research

A key principle touted in the productivity world is the Pareto principle, which says that 80% of your results come from only 20% of the work that you do. What this means is that you can divide your work tasks into high value tasks that lead directly to results and low value tasks that don’t.  Productivity gurus love this principle because, if it’s true, you can boost your productivity seemingly overnight by substituting low value tasks for high value ones. There are some nuances, for example someone may still need to do a low value task even if you don’t do it yourself, but the general idea is to dump low value tasks as much as possible.

That being said, academic research is different from the business world, which tends to be the focus of most productivity books, blogs, podcasts, etc. We researchers rarely know in advance which of our hunches will lead to exciting breakthroughs and scientific discoveries. As a result, we sort of have to “try everything” to verify whether we can or cannot answer a particular research question with available methods. On top of this, we never know if the next coffee break, group meeting, paper discussion, or other interaction may lead to a new discovery. For these reasons, one may wonder whether the Pareto principle really applies to academic research and other forms of open-ended work. Is there really such a thing as high value and low value research tasks? For example, isn’t it always good to read lots of papers? Haven’t studies shown that group meetings are critical to producing innovative and impactful research [cite]? I argue in this blog post that yes we can apply the Pareto principle but doing so requires a shift in how we think about the practice of academic research.

What Makes for Successful Computer Science Research?

As a researcher in computer science, I study how data analysts and data scientists make their own discoveries and innovations. In my experience, there are three key concepts that we need to distinguish between high and low value tasks in computer science research. The first key concept is that all academic research is a form of knowledge work, basically work that happens mostly in our heads rather than mostly in the physical world (compared to say, construction or house cleaning). Second, the key measure of success for innovative knowledge work is the dissemination of new ideas. For example, the initial successes of Facebook, Microsoft, and Apple were all based on bringing a single new idea to the tech market. The third key concept is that there seems to be a recipe of sorts for generating new ideas, and this recipe can be applied by anyone in virtually any discipline, including computer science as well as academic research.

Where does psychology and cognitive science fit in? Well, to generate new ideas, we need to exercise our creativity, and as it turns out, creativity is fundamental and universal across people. According to the academic literature, creativity is made of two parts: originality, or the ability to connect completely different ideas together to generate new ones, and relevance, or the ability to generate ideas that actually matter to other people [Runco 2012]. If our research tasks cover both of these components, then we know we are producing high value knowledge work.

To summarize what we’ve covered so far: computer science research is a form of knowledge work; successful knowledge work leads to the dissemination of new ideas; research tasks are valuable if they help us generate new ideas; to generate new ideas, we need to be creative; and creativity has two components, originality and relevance. Putting everything together, high value research tasks should help us generate new ideas that are both original and relevant. Now, how we can structure academic research to generate original and relevant ideas?

Generating Original Ideas

Although we are focused on computer science research here, generating original ideas is a universal skill that can apply to any discipline, suggesting that we should look to not only computer scientists but also psychologists and cognitive scientists for answers. Across psychology and cognitive science there is a pretty consistent definition for an original idea. DeHaan provides an excellent summary: “A creative insight, then, is a sudden, unexpected recognition of concepts or facts in a new relation not previously seen.” [DeHaan 2011] Here, DeHaan’s “creative insight” is what I call a new idea. So to come up with an original idea, we just need to take two concepts that researchers wouldn’t normally put together, and connect them in an interesting way.

To shed more light on how this is generally done, I’ll share a slightly different perspective from data science research. In data science, specifically visualization, we study how data analysts discover original ideas as they explore and manipulate a dataset. Existing research posits that a person discovers an original idea by combining ambient information, or information extracted from their environment, with personal experience, or the data already in their brain, in a new way [Chang 2009, Gotz 2006, North 2006, Smuc 2009]. In data science, ambient information generally takes the form of tables, spreadsheets, visualizations such as bar charts and scatterplots, or even the output of machine learning models. However, ambient information is not limited to these data science-specific examples. It can be any information from your immediate environment, such as a quote from a paper, an idea posed in a podcast episode, a scene from a movie, and so on. However, as I’m sure you’ve already experienced, most information you perceive in a day is not going to relate to your research. So interesting ambient information is akin to searching for a “needle in a haystack,” or in general a rare event.

Data science research reveals three important points on generating original ideas. First, we need a robust and structured knowledge base in our brains that we can use to connect to interesting bits of ambient information. Otherwise, ambient information will be useless because we will have nothing useful to connect it with. Second, we want to expose ourselves to large streams of ambient information in an efficient way. If we don’t, then we limit our opportunities for finding ambient information that actually connects to what we currently know. Third, we want structured mechanisms for plucking interesting bits of ambient information from these streams and connecting them to our existing knowledge base. If we do not have an efficient way of recording and organizing new connections, then we will forget them, lose them, or become overwhelmed by them.

Psychology and cognitive science add nuance to these three points by emphasizing the need for connecting disparate concepts. For example, I never would have fully understood the science behind creativity if I only ever read computer science papers or data science papers. I had to start reading psychology papers and cognitive science papers to gain the full context. Taking this example a step further, I only stumbled upon DeHaan’s definition of creativity while reading a book completely unrelated to my research: The Autistic Brain by Temple Grandin and Richard Panek [Grandin 2013]. What is even more interesting is that I probably never would have gotten the idea to study creativity in the first place if I had not read the book Where Good Ideas Come From by Steven Johnson [Johnson 2011].

As it turns out, in academia we have a built-in measure of relevance for our new ideas: published research papers. You know that phrase “pics or it didn’t happen”? Well, in academia you could say something similar: “published paper or it didn’t happen.” This is basically just another form of the famous saying “publish or perish” in academia. I admit that the academic review process has its limitations, but the principle benefit of a published research paper is that you have proof that your new idea has been vetted and deemed relevant by your own academic community. Even for graduate students in data science, a typical doctoral dissertation encompasses multiple papers worth of content. Thus, a tangible deliverable for most researchers is a published research paper describing an original idea. Of course, one can quibble with the details, but in computer science, a published conference paper is the standard unit of success for generating a new idea.

That being said, we may not want to wait until we’ve already written an entire paper to find out that our original idea isn’t relevant, or worse isn’t even a good idea! To avoid these worst case scenarios, I recommend getting regular feedback on your ideas at every stage of the research process, from the initial pitch for a new idea down to the full paper draft you submit to a conference. There are lots of different kinds of feedback you can receive, such as feedback on clarity (can someone understand my new idea as written?), viability (does this idea seem achievable and will it produce meaningful results?), audience (would this idea be considered valuable in my research community?), and narrative (is this idea presented in a compelling way?). For example, friends and family can assess clarity. Junior peers can assess clarity and audience. If your university has a dedicated writing center, the writing center can assess clarity and narrative. Research mentors, advisors, and senior peers can assess all of the above. I highly recommend getting into the habit of sharing written documents for feedback rather than just presenting ideas verbally, even in the early stages of a project. This one habit alone will dramatically speed up your paper writing and lead to more published research, as long as you stick to it.

Identifying High Value Research Tasks

Applying both components of creativity (originality and relevance) to research, we can say that high value research tasks increase our ability to publish research papers on original ideas. Taking things a step further, I break this idea down into four specific ways that high value tasks can improve our research performance:

  • The task helps us generate more ideas in less time
  • The task helps us prune bad ideas quickly
  • The task helps us turn ideas into papers faster
  • The task helps us publish papers faster

Now, how do we actually apply this definition of high value tasks to our daily research work? Start by asking yourself whether the task will help you achieve the four points above. To help with this, I reframe the points above as yes or no questions:

  • Will this task help me generate more ideas?
  • Will this task help me prune bad ideas?
  • Will this task help me turn my ideas into papers?
  • Will this task help me publish my papers?

In my experience, the value of a research task is proportional to the number of yeses. Ideally, a task is high value when I answer yes to at least two of these questions. A task with four yeses is one of the most valuable tasks you can work on, and I would put it at the top of your todo list. Keep in mind that to generate new ideas, we need to expose ourselves to steady streams of ambient information as well as build a robust knowledge base in our brains. To publish new ideas, we need to turn them into written documents that can be reviewed by family, friends, peers, mentors, advisors, and eventually conference paper reviewers.

Common High Value Tasks to Build Into Your Research Routine

Whether you are a professor helping a research mentee, or a student researcher looking for tips, here are some common tasks that I highly recommend everyone incorporate into their research routines. However, even the most seasoned researchers did not do all of these things starting out. If these are new to you, try to incorporate just one task at a time into your research routine. Keep the ones that give you good results, and drop the ones that don’t.

  • Share a written document with a peer, mentor, and/or advisor to get feedback on a new idea.
  • Brainstorm with peers or your research group to generate new ideas.
  • Synthesize a template for how to write a certain type of paper (or section of a paper) based on relevant papers from the top conferences in your research area.
    • Example: Create a template for how to write a great visual perception study paper, based on existing visual perception papers published at IEEE VIS, ACM CHI, and EuroVis.
    • Example: Create a template for how to write a great database benchmark paper, based on existing benchmarking papers published at SIGMOD and VLDB
    • Example: Create a template for how to write a great introduction to a CHI paper, based on papers that have won the CHI best paper award or honorable mention award in the last five years.
  • Write up your current progress on your research project
    • Example: Write the introduction section
    • Example: As you read related papers, write up the relevance, significance, and limitations in a tentative “Related Work” section 
    • Example: If conducting experiments such as user studies or benchmarks, write the experiment protocol as a tentative “Experiment” or “Evaluation” section
    • Example: If building a system, write the planned or implemented architecture for the system and adding a snapshot of the current prototype
    • Example: If analyzing data, create an RStudio, Jupyter, or Observable notebook to organize and track your analyses. Use markdown to record your reasoning and approach to your analysis methods as you go along. (I adopt a similar structure with code files and GitHub).
  • Give a 10-15 minute presentation with slides on the motivation, approach, and results for your research project thus far.
  • Give feedback to a peer on their new ideas and/or writing
  • Review your notes for a project or topic that you haven’t been working on for a month or longer
  • Pick a major conference to submit your project to, and complete as many of the submission logistics as soon as possible:
    • Example: Migrate your current paper draft into the official conference paper template.
    • Example: Write a tentative paper title and abstract
    • Example: Create a submission on the submission website, including the paper title, authors, abstract, and keywords
    • Example: Create an OSF project with an anonymized link for sharing supplemental materials
    • Example: Anonymize the current set of supplemental materials and upload them to OSF
  • Consume content from outside of your research area/discipline and brainstorm ways to incorporate it into your knowledge base
    • Example: Read a non-fiction book that interests you and write down any quotes that resonate with you.
    • Example: Attend a networking event and write down contacts for organizations, such as inspiring non-profits, that you may want to work with in the future
    • Example: Attend a distinguished lecture in another department, and brainstorm ways you can incorporate the speaker’s technical ideas and/or research approach into your own work.
  • Attend a weekly accountability group with peers where you share your professional goals and group members hold you accountable for making progress towards these goals every week.

Notice how revising and editing your own work is not listed as a high value task in itself. Revising is a necessary task but it only becomes high value when done in preparation for more feedback. In other words, people have to see your work in order for it to get published so revising only matters if someone else sees the changes. For this reason, always give yourself a revision deadline, even if you are just showing your work to a friend.

Common Low Value Tasks to Avoid

Some of these tasks are inevitable, but I try not to start my day with them and spend as little time doing them as possible.

  • Reading and writing emails/social media posts.
  • Scheduling or attending meetings with no clear agenda and no clear goal
  • Scheduling or attending meetings that consistently fail to produce meaningful ideas, action items, and/or tangible writing deliverables.
  • Developing infrastructure and running experiments without writing down the objectives, rationale, and/or algorithms behind them in your paper draft first.
    • Example: writing code before writing the objectives, rationale, and algorithm/pseudocode in your paper draft or research notes beforehand.
    • Example: conducting an experiment without writing out the objective, hypotheses, protocol, and analysis methods beforehand (basically without pre-registering the experiment somehow).
  • Reading a research paper, attending a talk, or generally participating in some work event without taking any notes on how to incorporate what you learned into your knowledge base, such as:
    • How the techniques presented could be applied to your research project or research discipline in a new and interesting way
    • If it is directly relevant to your research, how your project provides something new over this work
    • What you like about how the material is presented, and specific steps to apply the same writing/presentation techniques to your current project
  • Passively consuming content without actually incorporating it into your knowledge base
    • Example: Listening to lots of podcasts or reading lots of books without taking any notes on what you’ve learned or find valuable. How will you keep track of what you’ve learned in the past so you can connect it with new ambient information?

Note that if you find yourself thinking “there’s nothing that I want to write down” during a task, then either you need to try harder to find something interesting to add to your knowledge base, or the task is low value and you probably shouldn’t be doing it.

General Habits to Boost your Research Productivity

These habits have enabled me to build a strong foundation for my research success. Stay tuned for a future post on how productive personal habits can enhance professional performance!

  • Write everything down, preferably in a centralized note-taking application with an effective search feature.
  • Give yourself plausible but ambitious deadlines and commit to them, even if you are just showing your work to a friend.
  • Participate in hobbies and creative pursuits outside of work to further exercise your creativity during your free time.
  • Meditate. You can start with just 5 minutes a day.
  • Take frequent breaks so your mind can refresh and recharge for the work ahead. Just like how we need daily, weekly, quarterly, yearly, and lifetime goals, we also need daily, weekly, quarterly, yearly, and career (e.g., sabbatical) breaks.

Summary

Great research is innovative research. Creativity is the cornerstone of innovation and creative ideas have two interlinked components: originality and relevance. Original ideas come from combining ambient information, or information from the environment, with your personal knowledge base, or the network of information in your brain, in an interesting way. Original research ideas are deemed relevant when they have been successfully vetted by the research community, initially by your peers and mentors but ideally by conference paper reviewers. Applying the components of creativity to computer science research, we can say that high value tasks enhance our ability to: (1) generate original research ideas, (2) prune the bad ones, (3) turn the good ones into research papers, and (4) get regular feedback on our written ideas prior to publication. High value tasks should cover at least one but ideally two or more of these points, and the more points that are covered, the higher value the task. I have also created some concrete examples of high and low value tasks, please feel free to share the infographics. Check out the infographics below for easy to remember highlights from this post.

References

  • [Chang 2009] Chang, R., Ziemkiewicz, C., Green, T.M. and Ribarsky, W., 2009. Defining insight for visual analytics. IEEE Computer Graphics and Applications, 29(2), pp.14-17.
  • [DeHaan 2011] DeHaan, R.L., 2011. Teaching creative science thinking. Science, 334(6062), pp.1499-1500.
  • [Gotz 2006] Gotz, D., Zhou, M.X. and Aggarwal, V., 2006, October. Interactive visual synthesis of analytic knowledge. In 2006 IEEE Symposium On Visual Analytics Science And Technology (pp. 51-58). IEEE.
  • [Grandin 2013] Grandin, T. and Panek, R., 2013. The autistic brain: Thinking across the spectrum. Houghton Mifflin Harcourt.
  • [Johnson 2011] Johnson, S., 2011. Where good ideas come from: The natural history of innovation. Penguin.
  • [Newport 2016] Newport, C., 2016. Deep work: Rules for focused success in a distracted world. Hachette UK.
  • [North 2006] North, C., 2006. Toward measuring visualization insight. IEEE computer graphics and applications, 26(3), pp.6-9.
  • [Runco 2012] Runco, M.A. and Jaeger, G.J., 2012. The standard definition of creativity. Creativity research journal, 24(1), pp.92-96.
  • [Smuc 2009] Smuc, M., Mayr, E., Lammarsch, T., Aigner, W., Miksch, S. and Gärtner, J., 2009. To score or not to score? Tripling insights for participatory design. IEEE Computer Graphics and Applications, 29(3), pp.29-38.