Life after the PhD and the future of science

I haven’t written here in a while. Finishing my PhD took a lot out of me, and since then I’ve been finding my feet and working on a new project – prediction markets for science. (I have a little website at sciencebetting.com, and you should check out the open-source SocialPredict software I’m using. Please star the repo if you like it!)

Prediction markets for science sound like a fairly niche area. To be fair, prediction markets in general are still fairly niche. There’s a number of reasons for this – they sound scary and technical; they appeal to a comparatively small audience; they can run afoul of gambling rules and regulations. I am more bullish than some commentators that we can get past these hurdles, but I also grew up in the UK where gambling is fairly commonplace.

One area where I think prediction markets in science would be helpful is in deciding what to fund. Optimising research funding is fiendishly difficult and has been for decades. The individuals who make the decisions about what research to fund and why are eminent scientists, but as science and expertise become increasingly fragmented it becomes increasingly difficult for funders to make good judgements. This is problematic because funding bodies have a limited amount of money to award, and money spent on nonreplicable research is money taken away from other promising studies.

Where I think prediction markets could help is by aggregating many different kinds of judgements from many different experts, giving those experts an incentive to make correct predictions, and presenting these aggregated judgements clearly and legibly. So this is what I’m banking on. More to come in the future, but for now let’s talk about AI.

A solution in search of a problem

The arrival of generative AI that can create passable (if not actually outstanding) outputs offers mundane utility but also creates headaches. Workplaces are adjusting to generative AI in their workflows, skilled creatives are split on whether to use AI or resist it as a set of technologies that deskill workers, people are using AI for companionship…generative AI can be used for many things, but not all of those things are sensible. 

(There are also good questions about whether we should be using generative AI at all. My instinct is that generative AI is useful for reducing the executive function overhead of administrative tasks and for helping people to think through complex tasks.)

Two areas where AI might be helpful are encouraging people to make better predictions, and making predictions themselves. Some exciting research has come out showing that using LLMs prompted to provide high-quality advice does help people to make better predictions. And AI does have some applications in prediction markets. So as a society, we can use AI to help us make better decisions about the kinds of science we’d like to fund.

Automation for the people

Last year, this absolutely fascinating paper from NBER came out. The authors note that hypothesis formation in social science isn’t formalised – i.e. researchers generate ideas through reading and thinking. On a personal scale there is nothing wrong with doing this; I’d argue that identifying ideas is an incredibly important component of being a researcher. On a systematic level, trying to do hypothesis formation with everyone reading and thinking is vulnerable to a couple of different pitfalls:

• Availability bias. People are better at recalling certain things and tend to form hypotheses based on what they can recall easily.
• Popularity. Different academic disciplines have fads and fashions; absolutely no discipline is immune to this. Certain turns of phrase, methodologies, or ways of writing become popular. There can be tremendous pressure to work on something “popular” or “sexy”, distorting the kinds of hypotheses that academics form.
• The file drawer effect. Also known as publication bias, we know that studies confirming a hypothesis are more likely to get published than studies disproving a hypothesis. Since academics can generally only work with published papers or “grey literature”, rather than unpublished work, they are drawing on a heavily biased sample of papers.

So a bunch of stuff gets left uncovered. This is bad and negatively impacts hypothesis formation. Maybe if we had help from automated hypothesis generators, we’d be able to cover more stuff, including stuff that our biases miss.

You could then also run prediction markets, perhaps with AI involvement, on which hypotheses might produce replicable results or lead to interesting avenues of inquiry. So you’d be able to do hypothesis generation, as well as hypothesis evaluation, much more quickly.

Bugs in the system

So, is our grand automated future in the making? No, it’s not. Here’s why:

• Garbage in/garbage out. Large language models are wonderful at synthesising large quantities of data, but they are not magic. If they are fed bad, biased data, they will produce bad, biased outputs.
• Algorithmic bias. Some algorithms end up privileging some groups of people over others. This is bad when you’re trying to do science that should describe lots of different groups of people well.
• Lack of transparency. We still don’t know exactly how models produce their outputs, and they can’t explain their processes to us. This also means that we have trouble determining how good they are at producing correct outputs.
• With a human in the loop, still subject to availability bias. Again: some concepts are simply more psychologically available to people than other concepts.
• With a human in the loop, still subject to popularity bias. Again: some hypotheses are still going to align with more popular or sexy fields than others. A human could still gravitate towards these sexy, popular fields and reproduce the biases in hypothesis generation that we wanted to avoid!

Automatic hypothesis generation might have its place, but it’s not a magical fix-everything technology and shouldn’t be treated as such.

Science thrives on serendipity and the unexpected connections made by human minds. There is a risk that over-reliance on automation could stifle innovation and reduce the diversity of scientific exploration. 

There’s also perhaps a deeper set of questions to be asked here: science is an area of human activity that thrives on craft-like skill, tacit knowledge, and all the same things that creatives value. How much of that gets deskilled when we have automated hypothesis generation? How much of that is deskilled when we also have automated peer review by large language models? How much time and space will there be for fact-checking? How important is it that science is a set of activities primarily done by humans rather than a set of activities primarily done by machines?

I don’t have the answers here. Nor would I claim to. But as machines crunch more and more data, and generative AI develops ever further, I think asking the questions is important.

Scientists are once again faced with choices about the future of scientific disciplines: how we know things, how much human involvement to have, how fast we choose to make science, what we lose and what we gain. And we should take the time and space to be clear-eyed about those choices.