Illustration by The Project Twins
Every laboratory scientist has a horror story. The five-minute step they didn’t know they needed, which ended up costing them five months — or five years. Maybe it was swirling the plate as crowded cells were split between culture dishes. Or maybe the published protocol said to wash your sample once and heat thrice but meant the opposite, so that following the printed instructions destroyed the sample.
More than 60% of respondents to a 2016 Nature survey said they had tried to repeat other scientists’ experiments and been unable to do so. A poll of members of the American Society for Cell Biologists similarly found that more than 70% had been unable to replicate a published experimental result, with incomplete detail in the original protocol given as the most common explanation.
That’s no surprise to Tim Errington, director of research at the Center for Open Science in Charlottesville, Virginia, who managed the Cancer Biology Reproducibility Project (CBRP), which launched in 2013 to replicate results of prominent cancer-biology papers. Errington sees two overarching reasons for poorly explained methods. One is a lack of incentives or training for better descriptions. The other is that researchers assume everyone works the same way and fail to recognize what details are crucial. The word ‘standard’ is a red flag for this type of thinking, he says. “There is no such thing as a standard protocol: it’s only standard in your lab.”
But things are improving, says Elizabeth Iorns, who helped to launch the CBRP and is chief executive at Science Exchange in Palo Alto, California, which provides research-outsourcing services. “There’s a lot of progress in documentation and tools that are available to people to use.” A move towards automated experiments and data collection is also helping, she says, and people are more aware than ever about the need to describe reagents precisely.
Such tools will move the needle on reproducibility only if they are widely used, Errington warns. “Even though we have them, it’s not systemic yet, or rewarded.”
Communicating methods more accurately, research-improvement advocates say, requires changes to both mindset and workflow. Here are some strategies that can help.
Document as you go
If nothing else, consider how you’ll share your experimental methods from day one. In the past five years, several publishers of high-profile journals, including Nature, have strengthened requirements for reporting experimental details in their papers. And many publishers encourage depositing methods in a repository, such as Nature Portfolio’s Protocol Exchange, protocols.io or Bio-protocol. But waiting to gather these details until manuscript submission will leave researchers scrambling to find information that could have been right at their fingertips, says Lenny Teytelman, chief executive at protocols.io in Berkeley, California. Logging information as the experiments are done, he says, “makes your life easier, rather than harder”.
Write less, show more
In 2014, Kornelia Polyak, a breast-cancer researcher at the Dana-Farber Cancer Institute in Boston, Massachusetts, showed just how tricky sharing methods could be (W. C. Hines et al. Cell Rep. 6, 779–781; 2014). Her collaborators in California were getting flow cytometry results that were very different from hers, even when they shared samples and reagents and discussed their methods extensively. Visiting each other’s labs revealed the reason: one group disaggregated tumour tissue by vigorous shaking for 6–8 hours, whereas the other used gentle rocking for 18–24 hours. When both teams switched to the same, slower method, results aligned. Now, Polyak’s lab makes informal videos that new lab members can watch before certain experiments even start.
The Journal of Visualized Experiments, based in Cambridge, Massachusetts, lets researchers submit text descriptions of methods for peer review. If accepted, the journal sends in a film crew for a day and then edits experiments into short videos, usually 8–15 minutes long. Researchers do this both to get credit for their work and to persuade others to adopt their methods and expand a field, says Moshe Pritsker, who founded the journal in 2006.
Timelines and diagrams can also be valuable, says Errington. For one replication study that he coordinated, the original authors, reviewers and replicators consulted intensively using written documents to agree on a replication protocol, only to find that mice became too sick to complete the experiment (K. Eaton et al. eLife 7, e34364; 2018). It turned out that a vague description of time points meant that the replicators started taking measurements weeks later than the original experimenters had. A flow chart would have eliminated that confusion from the start, Errington says.
There are tools that can help, he says. For instance, Experimental Design Assistant from the UK National Centre for the Replacement, Refinement & Reduction of Animals in Research is a web-based platform to help researchers plan animal studies. It captures reporting requirements before work begins and displays them in a clear diagram. However, researchers tend to leave out the diagram and focus on written paragraphs, he says.
Errington urges researchers to think less about lengthy protocols and more about incorporating online tools while they plan and conduct experiments, so that the information can be shared. (A bonus, he says, is that this can show what didn’t work, such as commercial reagents that failed to perform — information that is otherwise hard to come by.) Anything used to share data can also share methods, he notes: electronic lab notebooks, or repositories such as Zenodo, GitHub or Figshare. It is just a matter of recognizing the need to do so.
Link, template, standardize
Electronic lab notebooks (ELNs), in particular, could be just the ticket for linking to lab protocols, reagent details, related experiments or other essential details as experiments proceed, says Ingo Przesdzing, the ELN programme leader at the Berlin Institute for Health. ELNs can be much more efficient at capturing crucial details than conventional written documentation is, he says, and tools to tag whether or not experiments worked can help with both sharing and troubleshooting. But the benefits come only if people are trained properly — otherwise, he says, they use the ELN like a paper book, without the advantages of a digital solution.
Electronic laboratory and research notebooks can also help to standardize methods and record-keeping while providing experimental flexibility, says Alastair Downie, head of information technology at the Gurdon Institute in Cambridge, UK, who helps lab groups there implement ELNs. Having appropriate templates at the ready makes it much easier to communicate what to do, he says. “It becomes a huge advantage; everyone talks the same language.”
Deposit and document reagents
Countless experiments go wrong when a colleague gives a scientist the wrong cell line or reagent. Now, organizations such as the ATCC in Manassas, Virginia; Addgene in Watertown, Massachusetts; and the Jackson Laboratory in Bar Harbor, Maine, can act as gatekeepers, supplying cell lines, plasmids and genetically engineered mice, respectively, that are quality checked and distributed to other researchers. Many journals (including Nature) ask researchers to deposit research materials in the repositories of such institutions. This spares labs the burden of distributing them and ensures that other scientists will receive quality materials in a reasonable time.
Also available are tools to document commercial reagents in a standardized way. The Resource Identification Initiative, for instance, assigns reagents unique identifiers that persist even if a company goes out of business or modifies its catalogue numbers; more than 600 journals (including Nature) encourage their use. The identifiers can be used to populate fields in a lab notebook, for example, or exported into in-development manuscripts to ease study write-up, troubleshooting and reproducibility.
One often-overlooked strategy for enhancing reproducibility is to understand the rationale behind every step, says Polyak. That way, researchers can troubleshoot situations that might be unique to their lab. Off-the-shelf kits undermine this ability to reason, she says. When it comes to reproducibility, a good protocol “gives you enough details”, says Luciano Martellotto, director of the single-cell core facility at Harvard Medical School in Boston. But a better protocol, he continues, helps researchers understand every step, and why one RNA inhibitor, for instance, is used over another.
Communicate
One of the most powerful aspects of emerging methods-sharing tools is how they enable conversations that share tips and extensions, says Martellotto. He uses protocols.io to host a group, called Single Cell Ninjas, to help other researchers get single-cell studies working. Updating, correcting and interacting about methods is becoming a scientific norm, he says.
Perhaps because these protocols are seen as less formal than papers, other researchers are more likely to ask questions, says Benjamin Schwessinger, who studies plant–microbe interactions at the Australian National University in Canberra and helped to launch an initiative called Reproducibility for Everyone to help researchers share methods and data. Schwessinger says many more people approach him about his protocols than about his papers, even if only to thank him for posting them and explaining the modifications they’ve made for different organisms. He gets alerts when people comment on his protocols at protocols.io, and also on social media. “It comes naturally for people to interact,” Schwessinger says. “You just need to build a community around it.”
Five keys to writing a reproducible lab protocol - Nature.com
Read More
No comments:
Post a Comment