How to think about data

Issue 211 | February 17, 2023
23 min read
Capsid and Tail

This week, Jan goes meta about data. Everyone throws around the term data, but what is it exactly? Jan explores the different kinds of data, how to work with data, and shares some ideas for keeping data neat and tidy.


Targeting Phage 2023 Conference banner

The 6th World Congress on Targeting Phage Therapy 2023 will take place in Paris, France June 1-2, 2023.

The conference will highlight the recent innovations and clinical applications of phages, validation and limitations. Targeting Phage Therapy 2023 will also cover how phages will transform the medicine of tomorrow. A discussion will be held on how to overcome barriers between experts, academics, industrials, and regulators. Phages will be the hottest topic not only to combat infection and antibiotic resistance, but also to modulate microbiota.

What’s New

In this new paper in Virus Evolution, Eric Olo Ndela (Université Clermont Auvergne, France) and colleagues report the discovery of sixteen complete microvirus genomes that represent two novel groups of microviruses, tentatively named Reekeekeevirinae and Roodoodoovirinae.

Add a Comment

Research paperTaxonomy

In a new paper in Cell, Brianna Duncan-Lowey (Harvard Medical School) and colleagues determined the cryo-EM structures of the RADAR defense complex, revealing RdrA as a heptameric, two-layered AAA+ ATPase and RdrB as a dodecameric, hollow complex with twelve surface-exposed deaminase active sites.

Add a Comment

Research paperStructural Biology

A new paper by Pablo Quirós (Universitat de Barcelona) and colleagues published in The ISME Journal reports the isolation of the first phage infecting some Nitrosomonas species. This polyvalent virulent phage (named ΦNF-1) infected Nitrosomonas europaea, Nitrosomonas communis, and Nitrosomonas nitrosa.

Add a Comment

Phage characterizationResearch paper

Soo Khim Chan (University of California San Diego) and colleagues have developed peptides which bind specifically to Pseudomonas phage Good Vibes, which may enable real-time analysis of phage in vivo and monitor the efficacy of phage therapy.

Add a Comment

Phage monitoringResearch paperPhage therapy

In a new paper in Biotechnology Advances, Wajid Hussain (Huazhong University of Science and Technology) and colleagues review genetic engineering approaches for bacteriophages, with a focus on their potential applications.

Add a Comment

Phage engineeringReview

Latest Jobs

Cell-free expressionPost Doc
The National Research Council of Canada is seeking a Postdoctoral Fellow to develop a cell-free expression system for rapid and cost-effective host-independent production of therapeutic and engineered phages for multidrug-resistant bacteria. The successful applicant will leverage NRC’s expertise in bacteriophage biology, genomics, proteomics, fermentation and downstream purification.
MicrobiomePost Doc
The University of California San Diego is seeking a Postdoctoral Fellow to investigate the role of bacteriophages as members of the human microbiome. The work will be conducted under the mentorship of Dr. David Pride, MD PhD, Professor of Pathology and Medicine. Please send CV and the names of 3 referees to Dr. Pride at [email protected].
ModellingPost Doc
KU Leuven is looking for a motivated candidate with experience in systems biology and bioreactor process modelling to fill their Postdoc position on advanced modelling of phage-host dynamics in bioreactors.
Central European Institute of Technology of Masaryk University (Czech Republic) is seeking a postdoctoral scientist to use cryo-electron microscopy and tomography to determine three-dimensional structures of viruses and bacteriophages and characterize their infection of cells.
EndosymbiontsMicrobiomeResearch Technologist
The Bordenstein Laboratory in Penn State’s Department of Biology seeks a Research Technologist 2 to support rigorous research in host-microbe symbioses and the microbiome sciences spanning insect endosymbionts, bacteriophages, and various animal-associated microbiomes including humans and insects.
Post DocLysins
Postdoctoral position available at Rockefeller University working on phage lysins. Only considering applicants currently in USA. PhD or MD. Proficiency in protein chemistry, molecular biology, and protein purification a must. Send application to [email protected]
Gut phageMicrobiomePrincipal Research Scientist
The Singapore Center for Environmental Life Sciences and the Saw Swee Hock School of Public Health at the National University of Singapore (NUS) are recruiting a full-time Principal Research Scientist / Senior Principal Research Scientist to work under the supervision of A/Prof. Yann Felix Boucher to lead isolation of phages targeting bacteria from the human gut microbiome.

Community Board

Anyone can post a message to the phage community — and it could be anything from collaboration requests, post-doc searches, sequencing help — just ask!

Save the date! The Evergreen International Phage Meeting is back this year (and back on campus)! Confirmed dates: Aug 6-11th 2023. 500 people max. Details to come!

Add a Comment

ConferenceEvergreen Phage Meeting

The 28th Biennial Conference on Phage/Virus Assembly will be held at Shrigley Hall in Macclesfield, UK from 18-23 June 2023.

Add a Comment

ConferenceMolecular biology

On Feb 28 at 5 PM East Africa Time, the Africa Phage Forum will be hosting a talk by Dr. Evelien Adriaenssens (Quadram Institute; Chair of ICTV Bacterial Viruses Committee) on the recently announced changes to phage taxonomy, including the abolishment of Myoviridae/Siphoviridae/Podoviridae families.

Add a Comment

TaxonomyVirtual Event

How to think about data

Profile Image
Product designer and co-founder of Phage Directory
Co-founderProduct Designer
Iredell Lab, Phage Directory, The Westmead Institute for Medical Research, Sydney, Australia, Phage Australia
Twitter @yawnxyz

Bioinformatics, Data Science, UX Design, Full-stack Engineering

I am a co-founder of Phage Directory, and have a Master of Human-Computer Interaction degree from Carnegie Mellon University and a computer science and psychology background from UMBC.

For Phage Directory, I take care of the product design, full-stack engineering, and business / operations aspects.

As of Feb 2022, I’ve recently joined Jon Iredell’s group in Sydney, Australia to build informatics systems for Phage Australia. I’m helping get Phage Australia’s phage therapy system up and running here, working to streamline workflows for phage sourcing, biobanking and collection of phage/bacteria/patient matching and monitoring data, and integrating it all with Phage Directory’s phage exchange, phage alerts and phage atlas systems.

“Data” is one of those words that everyone uses regularly. But ask two people “what is data” you’ll get vastly different answers! This is because “data” is a broad, catch-all term for various kinds of information.

Previously, I’ve written about data from a higher level, from organizing your biobank data to designing a phage data system. I’ve also introduced the field of data analytics, and hinted at writing about data analytics in biology. Before we do that, let’s break down and define all the different types of data, and explore what they’re used for.

You might have heard of “Big Data” and “data-driven organizations.” The trap many of these orgs fall into is that “the cost of keeping data around is less than the cost of figuring out what to throw away,” so they record as much data as they can. Unfortunately, most of this is noisy and uninformative and rarely creates new insights, and often only exists to confirm what the leadership has already suspected (this is unfortunately too common in both academia and industry). Even Elon is guilty.

The different kinds of data

We should be very intentional about what data we collect and generate, and the kind of insights we want our data to generate. Broadly, I think of data as a tool to both accumulate and communicate organizational knowledge. I tend to categorize data in the following buckets:

Measurements are quantitative pieces of information (inches of rainfall, number of plaques in a plaque assay, PFU/mL) usually recorded by hand or by a machine. These can be either discrete or continuous, and can usually be graphed into trends, compared side by side, and manipulated with math. 3/10/23 note: The general category is “Tabular Data” and also includes any kind of information that fits into tables of rows and columns: this includes lists of names, events, measurements, etc. These are usually represented with spreadsheets and CSVs.

Artifacts are bundles of information that generally need to be extracted and processed before we can make sense of them. I think of these as photos, images, and files (photos of plaque assays, screenshots of a table or graph, PDFs, FASTQ, Excel and Word documents). Some artifacts are easy to extract data from (CSV files), and others are much harder (# of plaques in a plaque assay photo). Some are proprietary files (GraphPad Prism files) that requires the right app or license; others are URLs. Generally, most artifacts are hard to read and parse with computers.

Presentational data are technically artifacts, and are meant convey and communicate insights, findings, and observations from measurements and artifacts. These include tables and human-readable spreadsheets, pictures of tables, slideshows, graphs, and infographics, preprints and manuscripts, and sometimes websites (Airbnb shows a reduced, user-friendly view of their data). Presentational data is meant for people to understand and is focused around communication rather than completeness.

Records are pieces of information that describes a thing or event, like logs (the package was scanned and shipped at 2pm; 5mL of Pae7 was used; 50 users visited our homepage on July 1), transactions (Heather paid $25 to Mary on July 1), and “objects” (Driver’s license, license plates, DOIs, SKUs, and the information they describe, commonly also called “documents”).

Records that have relationships are called “relational data” (Heather has Driver’s license number 123456 and license plate ABC123; Mary used 5mL of Pae7). Directional relationships can exist too (a transaction showed that Heather gave $25 to Mary; Heather and Mary love each other; Mary is annoyed at Heather). Records can also describe measurements, artifacts, and metadata.

Metadata are pieces of information that describe how things are described. Technically these are a type of record data, but I like to single these out. Metadata usually tracks “meta-information,” which means “information that describes information” (How many versions of a manuscript exist within a lab, and where are they? How many of those are named _final_final_final.doc, and how many versions of those exist? What are the differences and similarities between all those files, and when were they all edited?). Not everyone does this, but I think of metadata as processes (How do we publish Capsid & Tail every Friday? How is the team doing plaque assays? Which lab members don’t use sterile tubes for plaque assays?), settings, and configurations as well (what type of media was used, what settings were used for a sequencing run).

Raw or cooked?

We use “raw” to describe data that’s close to the source that generated it. For example, a temperature sensor that collects data per second will store a lot of temperature data. If you place these sensors all over a city, then you’ll have a lot of temperature data. Consequently, “rawness” ends up describing how easy the data is to work with. Here’s how I think about “rawness”:

Raw data is usually information collected straight from a sensor or machine, like temperature per second, or BAM/SAM/FASTQ files from a sequencer.

Processed data can be filtered or reduced data (e.g. daily temperature), or in bioinformatics, the data output from each step of the pipeline, from assembly to BLASTing. Processing data makes it easier to use and for humans to understand. It can always be processed further (turn second-by-second temperature to daily, weekly, or monthly temperatures), or combined with other data (weekly temperatures and humidity). Once you have the appropriate “resolution” you need of the data, you don’t have to go back to the raw data. If all you need is weekly temperature, you don’t have to re-process it unless you need to get second, minute, or daily data. As data is processed further, it starts to lose “resolution.” You can’t un-bake a cake. Ultimately, processed data helps us see patterns, get insights, and communicate findings much easier than raw data.

Presentational data, like slideshows, graphs, or tables (or screenshots of tables) is the most processed kind of data. These are usually reduced from dozens or thousands of data points to just a few, meant to communicate insights and findings. Because presentational data has lost so much resolution, we shouldn’t make a habit of re-processing, combining, or relying on it as primary evidence.

A potluck data party

Data is at the center of any research lab or clinical trial work. We collect and interpret it to gain insights, make decisions, and build evidence for our work — but it’s almost always consumed individually, and rarely shared among the lab. What if data was shared more often? What would that look like? Let’s use another food analogy!

In a potluck dinner party, everyone brings a dish to be shared. The food is placed easily within reach, and guests can choose to eat whatever they fancy. Everyone wins!

Sharing data is kind of like a potluck. Everyone brings a single dish — no need to share everything in the fridge. All the dishes are easily within reach — you don’t have to ask someone to pull their dish out of a bag every time. If a dish isn’t placed on the table, it’s assumed not ready to be shared (or doesn’t exist in the first place). In data science terms, the table is considered the “single source of truth.” The best part is: we only share what we’re proud of, and only when we’re ready!

Great data parties require great data housekeeping

If you run a lab or do any kind of data science or bioinformatics across team members, you’re now the (un)official host of the data party (lucky you)!

A kitchen full of hungry guests gets messy quickly! Setting up lines, placing plates and cutlery, and separating the food and drinks stations will help you direct traffic. As the host, cleaning up spills and picking up empty cups after guests is also crucial to a great experience. As a party host, your job is to plan and curate all the elements for guests to have a great time.

Hosting dinner parties is hard. Hosting data parties is harder! As the data party host of Phage Australia, here are a few pointers I’m trying to follow, in no particular order:

  1. Create the communal space: This is the table where all the data (or food) goes, and it should be the single source of truth for the lab. Every piece of data in this space is “real,” “official,” “verified,” and ready to be shared, re-used, combined, and audited. Any published data and publication data should be committed to the communal space. The space itself can be anything — a Notion, Airtable, MS Access or Filemaker project, a Trello board, Google Drive or Dropbox, a network drive or a single lab computer, or another document management system. There are a ton of options — just make sure to use one and stick to it for at least a few months before making a switch.
  2. The space should be easily within reach: Regardless of what tool you use, make sure everyone knows where the space is, and that everyone has access — either physical access or account/password access. Help those out who lost the link or the password or are locked out — if this happens often, create a set of instructions and email them out or put them on a wall somewhere in the lab.
  3. Help your guests if they need anything: Like a responsible party host, if guests need anything, you should help them. If people repeatedly need the same help, create a process document or poster and email/print it out, or make the process easier (e.g. if people keep forgetting the link, email it to everyone; if people keep forgetting their passwords, create a Bitwarden account for secure passwords for the lab). Helping guests out helps you keep things tidy in the long run.
  4. Build clear processes: Make sure everyone knows how to create, access/read, and update the data. Make these processes easy, and communicate them clearly. If you expect people to provide data, make sure it’s clear how they do that. Do they add their spreadsheets to a folder? How should they name their files? What should they upload? What data should they upload, in what format? This is tricky to get right, and something we’re still struggling with at Phage Australia. We’re building processes and SOPs that include better data handling practices, including templates, forms, and QR code systems.
  5. Create convenience: use templates and forms! Is someone tracking a patient case, or submitting a catalogue of new phages and bacteria, or putting in a sequencing request? Use forms for simple data, and Excel / Google Sheets templates with properly labeled columns for complex, tabular data. If a person can’t access a computer, as a last resort try using paper forms (and at a later date, save the image and scan using OCR). Use checklists!
  6. Use schedules, reminders, and checklists: On many occasions, the data will fall behind reality, especially for busy teams. Just like a cleaning schedule, make sure to check if the data is behind and get it up to date. Create checklists to track what data needs to be updated, and the cadence at which they need to be up to date. Use checklists to remind people what and when to record all the things — and make sure you know who’s responsible for what!
  7. Assign Ownership: Hand out responsibility to different lab members. Complex projects rely on contributors with different skill sets — make sure the experts share their knowledge and data back to the others. The very bare minimum data you should collect is who contributed what — so they can be chased up to fill in the gaps when the data is missing or expired.
  8. Lower expectations through the floor, but strictly enforce them: Make data super simple to submit; do a lot of the heavy lifting, even if means you have to clean up after others’ data spills. However, make sure they report the minimum data required of the lab or clinical trial. While people need freedom to operate, they still need to meet the bare minimum — don’t let anyone bring a half-baked cake to the party!
  9. …but also, don’t ask for too much: Most of the data we think we need is actually just “nice to have”. Requiring too much data becomes a burden, and people end up sharing less data. It’s a tough balance between asking for too little or too much, and it’s a dance that never stops. (Side note: they also don’t respond well when you say “well, it’s your job…”, so don’t try it!)
  10. Data has expiry dates too! Data can expire in the same way milk expires… if it’s past the expiry date and not sour yet, maybe keep using it? Data needs to pass the smell test too, and someone (usually the host) needs to smell the data. Some data can last longer than others: expired data means that the data doesn’t reflect reality anymore. If your Airbnb’s canceled, your app better reflect that news! The same goes for tracking phage therapy patients in the clinic, queued up sequencing jobs, or isolate shipping requests. Often, you’ll have to chase up the right people to confirm the data still reflects reality, and if not, they’ll have to update it. Set schedules to remind you to smell the data once in a while!
  11. Be iterative and experimental: Start early, start small. Try collecting one small piece of data first and grow it from there, and adjust as you go. You’ll be surprised how much you’ll learn from the team if you sit back, observe, and adjust the processes.
  12. Get everyone on the same page! Just like setting some house rules of a party, remind everyone that a clean, single source of truth for data makes the party more fun. (Also, that it’s a requirement for clinical trials). Yes, it involves more work for everyone. After all, no one likes cleaning the kitchen, but everyone enjoys the outcome!

If it’s not in the source of truth, it doesn’t exist!

Keeping the source of truth up to date is time consuming, and it will always lag the real world, and the data collected by individuals. It’s ok for people to work “in their own kitchens,” off their local spreadsheets and images and data — this keeps them productive. Just make sure to periodically remind everyone to share a dish back to the potluck! Because we can’t just turn people’s computers over to auditors, if it’s not in the official source of truth it doesn’t officially exist!

A plethora of tools and bases for a plethora of data

Recording and tracking data is hard already for individuals. We are limited by our tools: We use Word for written text, and Excel for numerical data, but Excel can’t work well with relational data (if Heather gets a new license plate, it’s tricky to update every location her license plate appears in the spreadsheet). Most of our tools don’t have “multiplayer” features like track changes and version control. It gets even harder for a team to record and track data together — it’s a massive communication challenge.

Unlike a dinner party, there isn’t a table you can just place all the food on and call it a night. There isn’t a single database that can collect and store our collective data: Excel and Sheets store numerical data; Airtable, Notion, and SQLite/Postgres store relational data and records; REDCap stores patient and trial data; loggers and key-value databases store event logs. Every researcher’s laptop (and phone) stores all sorts of images, PDFs posters, preprints, presentations. If you want anything shared, you can set up a Dropbox or Google Drive — but these don’t store record data and aren’t API-friendly! You could use AWS S3-compatible bucket, like S3, Cloudflare R2, or your own Minio instance, but they’re not user friendly…

With all these ways and places to store everything, how on earth do we keep track of all these places that store our stuff, or track who’s responsible? Do we know for sure that the one file is the actual, final-final version? And how does anyone in the lab find anything? What happens if someone leaves, or if a new person joins the team? Is there a handbook for all the rules? Where can you find it?

It would be nice if we had a tool that would let us work on a meta scale:

  • Let us create easily accessible, versioned documentation, like tutorials, how-to guides, explanations, and references [according to the Diataxis framework]
  • Compare and merge versions of files (when authors insist on emailing separate Word files instead of using track changes in the existing Google Doc).
  • Search for that exact piece of information across all our shared Excel sheets, CSVs, and Google Sheets.
  • Allow us to create relationships between a hodgepodge of PDFs, Word Docs, graphs, database records, and images, and search, filter, and extract data from them.
  • Upload successive versions of files to replace older ones for when we re-run our bioinformatics pipelines against newer databases.

In the next couple of issues in the Data series, I’ll explore some of the underlying ways to think about and structure data for us to be able to create such tools for ourselves. Strap in, as it’ll get more technical!

Capsid & Tail

Follow Capsid & Tail, the periodical that reports the latest news from the phage therapy and research community.

We send Phage Alerts to the community when doctors require phages to treat their patient’s infections. If you need phages, please email us.

In collaboration with

Mary Ann Liebert PHAGE

Supported by

Leona M. and Harry B. Helmsley Charitable Trust

Crossref Member Badge