How AI Can Cure Diseases Never Before Possible | YC Paper Club

[music] How's everyone doing today? [screaming] Uh, lots to talk about tonight. YC AI for Bio Club. Very exciting night tonight. Some of my favorite uh, speakers and a couple of my favorite um, folks in the in the lab that are coming and even some of my favorite folks um, that I worked with at YC are coming tonight to speak. So, I'm super excited. But before that, I kind of have a crazy idea. I need everyone to take out their phone and scan this. And then I'll tell you why in a second. Okay. So, I had uh kind of an argument with um Greg at Arc AGI who we had this dinner and he was kind of arguing that the Turing test is solved. Specifically, uh Alan Turing said in about 50 years in 1950, so in the year 2000, uh it'll be possible to program computers to make them play the imitation game so well that an average interrogator will not have more than 70% chance of making the right identification after 5 minutes of questioning. And I think that we have not passed it because every time I talk to, you know, a chatbot on some website, you know, some customer service bot on Air Canada, I'm like, I know this is AI. Like, I'm pretty sure this is not a person. And so, I don't know. I don't think I I think I have better than 70% chance. Let's find out. So, what we have, I have about uh a bunch of people who are on this website ready to chat with you. And some of you will be paired with those people and some of you will be paired with an AI and it's up to you to decide. Uh, and I vibe coded this in the last 12 hours. So, there's like 0% chance that it's going to actually work, but we're going to try it anyway. Um, okay. So, you all have it open. Start the chat. Go for it. And we'll set the timer for 2 minutes. So, Astra won, Astra, then Claude, then GLM, then Gemini. Sorry, Google man. [clears throat] Gez, Google. Then Grock and Kimmy are tied. And here's the interesting one that I also want to include on the the false negative false posit. Uh 33% of humans were actually mistaken for AI. And so [laughter] that's just how I talk is what I heard. Um, okay. Well, I don't think we're there yet. So that I think and I I really did try really hard to make it seem like I gave it. So what actually So let me actually tell you what I did. So half of you got paired with each other and the other half of you got an actual agent. And so that's why this is so much higher. we have a hundred over in the human um cuz you got paired with each other versus over here. And I tried really hard to give it a persona of like a 32-year-old that uh is um male and is sitting in the room with you and give it the whole context. I tried really hard to bolster the the the the pro touring case and even then we're at like 30%. So, you know, anyway, not that I won, Greg, but we're pretty close. Okay. Anyway, let's go back to here. [snorts] Also, kudos to Cloud Code for coding that in the last like 12 hours. So, good job. Thank you. Good job, Boris. Okay, so why AI for bio? Um, I sit next to Yasa in most lab meetings and he just tries to uh convert me to uh AI for bio constantly and tell me that when you get cancer, you're going to wish that you spent a bunch of time on AI for bio. Um, and that's his main argument. And he said he's like, you know, I tell this pitch to everyone, it never works. And but it always tries. But anyway, I'm getting more and more pled. I'll I'll help you out now. I'm working on [laughter] >> so um my hope is to also bring you along in this in this uh in this journey. Uh more data than ever. Uh just the EVO 2 was trained on 9 trillion base pairs. Um alpha and and the next one is being trained even larger. Alphaold as Dennis said a couple months ago at YC is now being used at some stage in every drug brought to market which I've I I almost I I had that had to be hyperbole. And I've talked to like 10 people I know from Janentech to PhDs and they're like, "Yeah, that's probably sounds right." Like, "That's really really crazy." Um, 10 to 100% success rates of AI protein uh uh binders on this Minecraft benchmark. AI developed phases are actually going into patients with uh radical numeric, a company out of our lab uh run by Eric Ngoyen. Um and then uh Divi is going to be presenting this as well. But just with uh four to 64 ICL in context learning examples, you can really steer uh bio design. And then we also started to have really good benchmarks uh to hill climb on CASP being one of them. Virtual cell challenge which may have some challenges but is getting better as I'm as I've heard and I think everyone's going to be presenting this slide here. But the cost of sequence keeps going through the floor and that results in exponentially more data that we're uh that we have access to and there's huge investments going on uh from Lola to chai to isomorphic and the the most exciting part in my mind is that we really have figured out it's expensive but we figured out some RSI procedure to go after any RLVable task. Um, and so if you can do the ver verification of it either in silica or in the real world, then we know how to RSI it. If it's in silica, it's super fast. Just scale it up like unit test on a codebase or if it's in physical, it takes a lot more time and effort and it costs a lot more money. Um, but it's still possible. The same procedure would still work. And if you apply that to you know the the star paper otherwise to biology we can samp we have these foundation models that we can sample some drugs we can select some top end we can actually spend money to go test them we'll measure the effect efficacy we will retrain the models loop loop around around and it's just going to work I really do believe it's going to work and so it's just going to cost a lot of money and time and so we have some of the coolest speakers I could think to assemble here um Alfredo I worked with um at YC uh and his wife Renard amazing uh founders built this company called AMA um curing cancer. Uh before that uh Alfredo was a PhD at UCLA in cancer cancer bioinformatics. Uh then we have uh Yousef who works on AI models for cell state uh did his PhD at Stanford uh started started the ML team at Arc Institute now head of science at Intology uh and does where they do autonomous AI research for science. Uh Garrick works on foundation models for biology and some of the biggest foundation models at that from he led the EVO2 team and now this uh Manurva project uh also PhD student at Stanford and then Divia is in our lab at Hazy as well uh also co-advised by Brian He uh works on genai models for uh bio design before that did structure uh based methods like alpha fold at MIT. All right. And the sad part, this is my last YC paper club. It's been a great honor uh building this uh community uh for all of you and you guys have been such a great uh part of my life the last you know six months. Um we've gotten over a million views on these past uh nine that we've done. This is the 10th one. Um and I'll be leaving uh YC uh on Friday unfortunately um to start my next venture. Um but yeah, if you ever any need anything, please reach out. um and excited to to keep involved with the community. [applause] Okay, first up, Alfredo. [applause] >> Thank you, First of all. >> Thank you. Thank you. Um and thank you first of all for everything you've done for the community. I've been, you know, watching a lot of these YC paper clubs and just the beautiful mindmelt that has happened over the last couple of months and just the exchange of ideas and the growth of the community is absolutely fantastic. Um, well, thank you for having me. Uh, my name is Alfredo. I'm one of the co-founders of AMA. We help patients beat their cancers by building a personalized research lab around their specific disease. Uh, as France mentioned, my background is in cancer bioinformatics. because I did my PhD at UCLA where I built computational pipelines for processing tumor normal genomes at scale. I also built one of the largest rare saroma tumor data sets in the world and also my work focused on these rare end of one cases with the idea of you have a patient come in whether it's a rare tumor or a normal tumor how quickly can you go to getting the genomics to being able to actually influence the patient outcomes and uh before that I was at the broad institute uh helping with the validations of the very early crisper screens that were being done in cancer cancer cell lines that's where I met my co-founder Vernad who led a lot of the drug repurposing projects there. Uh and before that I was at Harvard studying uh early action transcription factor. So point being is that you know cancer and research on it has been my life and I'm very passionate about where we are today because it's very exciting time. So when we say solving cancer with AI, it's not a hyperbole. We're there. We're right at the cusp and we're at the point where we'll be able to help so many of people with all the efforts going on in AI and what we've seen the previous uh YC paper clubs. So without further ado, uh for this YC paper club, I wanted to touch on three specific topics. One is how cancer itself is now becoming a data problem. Two is how more on the applicability side, real world for those that are interested on the actual real world application of these things uh and how we're actually building personalized research labs around each specific patient. And three is just a couple of notes of the future and which actually I think will be great intros to some of the amazing speakers that we have coming up. So Francois already mentioned and I think you have have seen this at a couple of other presentations before specifically if you have any backgrounds in genomics and informatics. It's just the cost curve and essentially how the cost of sequencing has just plummeted. It is actually probably more affordable to get your actual genome sequenced than it is to have dinner at a lot of places in the Bay Area right now. And and the great thing about it is that actually sequencing for cancer patients is actually covered by insurance including Medicare and Medicaid for most advanced tumors and recurring cases. Right? As a result, we are just generating more data than ever before. Right? And to contextualize this, right? We can look back at one of the landmark papers from 10 years ago, right? The cancer genome atlas tcga, probably one of the greatest achievements of humanity at that point is we essentially were able to sequence for various discrete tumor types a couple hundred patients. And at the time 10 years ago, that's all that we could do. So because you have a limited number of patients, you have a limited number of genomes, what you would end up having to do is you would have to look for averages. you would have to look for similarities between them and ultimately this wasn't really applicable to the individual patient. But where we're at today with just the ability to just generate these gigantic data sets, right, we can actually go ahead and instead of looking at it and looking at that same data set and instead of just looking for summaries and averages, you can look at the individual patient and use that as its own individual data set. Right? So when I say cancer now is a data problem, right? What does that mean? Well, if you actually go ahead and you look and each one of those maps is an each line is an actual patient, right? And you expand it and then you actually think about the whole actual genome, not just similarities of the top couple genes. That means you actually have about a 20,000 gene data set data points for each specific patient. Right? You expand that to whatever cohort you want to use if you want to actually get rid of the discrete labels for tumors. Right? Now you have a multi-dimensional data set expanding further towards copy numbers, RNA expression, uh alternative splicing, cell states, uh the actual patients own background, right? So as you can see this itself is now a gigantic data set and what we were looking at before from uh you know academic perspective was like.1% of the data. But now the question is now that we can generate this data set for everyone, how do we fully utilize that to improve patient outcomes, right? How do we actually mine this data set because the infrastructure currently is still not there? So one of the things that you know one of the paradigm shifts that's essentially happening real time is instead of looking at the similarities of looking at the top 50 genes and their commonality of mutations is treating that data set as its own independent point to mine and with the idea of instead of trying to generalize something for everything is how can you leverage the data to solve cancer for that one specific patient. So that's one of the questions that we started with specifically for OM. And then the second question that we started with is if you're familiar with these gigantic data studies, if you've been in any genomics lab, you would know that a couple of years ago probably was most most of these landmark labs were about 20 30 plus people. Uh because that's how many people were required to be able to do these studies. But now thanks to artificial intelligence agents uh being able to create all this infrastructure, what if we took that same 30 40 person team and instead of applying it to looking for similarities across different you know no samples what if we took that to mine that specific data set in real time to improve the outcomes for that patient. So that's moving to applicability. That's where we are today and that's one of the things that we're solving at Omante is how do we build that personalized research lab for every patient right so I mentioned that you know in the past this is what actually would look like to get these landmark studies you would have to have somebody that was an algorithms expert to figure out the optimal algorithms you would have to have uh somebody working on infrastructure you would have to have somebody that maintained the actual cluster somebody working on the variant callers a geneticist to interpret the data somebody wrangling and putting together the actual visuals uh an actual uh uh MD to like interpret some of of the findings. That's what was kind of necessary and as a result that's why a lot of these landmark studies came from uh places like the broad uh um Sanger Institute because those are the only places that could afford to have these gigantic teams. But what if again if you took that team now in the age of AI where you know everybody I think has heard about software factories at this point and multiple people in this room they have actually have set up their own software factory but the next version of that in biology is how do you set up your own personalized research lab right so instead of thinking of the agents as their own software engineers thinking them of them as these individuals that were the bionirmatics experts that were necessary before to process all this data wrangle this data and get to the actual outcomes but applied in real time for patients. So what is now feasible and what we can actually do is essentially aenticize that whole discovery pipeline right so where we can mine that data in real time and not only that have this connected to the greatest corpus corpus of data that's being generated right so if there's something that gets discovered on Tuesday by Wednesday morning if it's something that's feasible applicable to the patient is something that we can pull down to be able to help and improve that patient's outcome so again shortening that timeline line of years to real time and something that can actually help the patient. Uh and then setting up an infrastructure where you have instead of having you know to rely on waiting for human input having your own specific agent that is an expert on trials that is an expert on evidence and safety that is an expert on your own uh background in biology uh as far as toxicity aspects as well. So that's kind of where we are kind of going towards today and I'll touch on a couple of quick case studies of some of the patients that we've actually been able to work with and help uh get to remission. Uh so historically speaking some of the only tumors that could be studied by pure sample size were these fairly generally common more common set of tumors which is why you would have these umbrellas. So if your tumor was a rare tumor, which is actually a significant number of the population that get their tum their tumor happens to be a rare tumor, which it just means it's just not too as common. Uh ultimately you would kind of be in like a an un ideal situation because your clinician unfortunately is doing their best working off limited data. But if you essentially get rid of that category and now that you can take that specific patient, even though there may not be a lot of samples within that specific category to compare it with, biological processes don't necessarily care about these discrete categorical variables where we place uh each sample. The same pathways that could be driving a neuroblastto could be driving a lung adino carcinoma. Right? So for one of our patients, it was a fairly rare uh brain tumor and we were actually able to see when we clustered it across the space of the of various multiple data sets uh that we were able to produce for them, we actually see that their specific tumor even though it was a brain tumor, it was actually more uh closers to tumors that were from the meeny so uh messenymal tumors and as a result we were able to even though there was nothing clear on the literature as far as what the next steps are based on their actual pathways and was activated as far as like their surface markers and what was actually driving the disease. We're able to find therapies that was not feasible before. So that's for patients that are in the rare case where there's essentially no other information that's been that's fairly available. Other cases where we have is actually just improving outcomes. Right? So in the case of a patient that unfortunately has gone and failed first line second line therapies right for for some of our patients this is an actual patient that we work with who were able to actually track their evol tumor evolution by looking and going back and analyzing some of the additional samples right and again all this automated through the setup of setting up all this infrastructure to to to do this automatically and for this specific patient we were actually able to see that unfortunately The last three lines of therapy that they had received were not ideal based on their actual underlying biology. But unfortunately again hospitals and clinics are not yet set up to fully mine this data set. They were they were doing their best. They did their best guess. Uh they went down the line of the next set of therapies and unfortunately selected three therapies that were chemotherapies and cytotoxic. But ultimately all the data supported based on the actual heterogeneity of the tumor that all the data support that it was not ideal for the specific patient. So we were able to kind of correct the course and figure out something that actually works for them in real time. So so you know I'm I'm running short on time but uh moving on to kind of what the future holds right. So I gave you a couple of glimpses into how we're working with patients in real time. Um but so this is kind of working on the idea of stuff that's already available but what about making stuff in real time in the future right so um I know that some of the uh speakers will speak on on EVO as well which again incredibly landmark uh aspects uh but in the future we'll essentially are going to be able to build proteins in real time right we already have seen some success rates of this uh essentially But it kind of goes further than the actual protein uh design as well. So you can think about actual delivery mechanisms right uh thinking about uh peptides done in real time. So we have already seen a couple of examples specifically from the bened paper that we were able to actually see that we were able that that you could actually successfully cut the design timeline by 10 years to be able to design these proteins. Right? So the next 50 years uh 80 years ago, you know, we're still getting up to speed with the the DNA double helix, but the future will be this signing proteins in real time and designing everything so that way for each specific patient, we'll be able to improve the outcomes. And I'm short on time, but uh the one thing I wanted to close off with is sensor in Mountain View specifically uh was one of the one of my favorite papers from the past year, which was co-scientist uh from uh it was from Google Deep Mind uh in part here in Mountain View. And uh so we've heard a lot about self-improving hardnesses over the last couple of paper clubs and uh self-improving agents, but what about self-improving biology models or self-improving agents in bio itself? Right? So this is one of the first features that we have been able to see specifically over the in recent times of that applied in the bio space and what we can show is that the actual models ultimately did not extensively outperform the humans but they were close enough that ultimately you're saving thousands and thousands of hours at scale of research time. Right? So again, instead of having a research lab, writing a grant, starting up your own lab, recruiting your PhD students, embarking on this 10, 15, 20 year journey to get to actual discovery, getting that paper published, we can compress that timeline. The future will be compressing that timeline in real time. So that way we can actually improve the patient outcomes in cancer. So, um, I think I'm out of time, but it's been an absolute pleasure and thank you first off for having us today. [applause] Okay, so we have time for two or three questions. Um, hi, great talk. Uh, could you tell me more about like you said I saw like organoids as one of the data types that you could see is like something you see in the future. Um, and I'm curious how think is that scalable for a personalized lab for every single patient? >> Great question. So I think I actually contributed to a couple of organoid projects during my PhD. Um and it's so there's actually have been if you look back at history right uh there's been about two waves of organoid companies that kind of have started. There was a one about seven eight years ago and there's been a more recent one of companies that try to do organoids in real time. And from a scientific perspective feasible like incredibly feasible. We could get data that's meaningful. We can essentially generate more tissue that we wouldn't have before which in the case of rare tumors absolutely fantastic. And you can actually do perturvation screens like this is something that we've been able to do in organoids. It's feasible. The main limitation actually there is uh more systemic as far as the uh the way that tissue is handled just in most hospitals, right? So the way that tissue is handled in most hospitals because just the way it is it kind of gets dumped in formulation. So it gets put in fixitative essentially eliminating the capacity to do organoids. So from like a organoid perspective, you can treat a lot of that data as like uh almost as you would at a met metastatic sample, right? Something that is similar to the primary but slightly biologically different and then you can account for the perturbations. But the actual limitation there unfortunately is the uh infrastructure that's not necessarily set up for it. So but great question. >> Hello. Um thank you for the talk. uh really excited about the work that you're doing for cancer patients. I wanted to understand even when you're able to decode the underlying eeology or biology of that tumor. Uh oftentimes you have to be able to and you're able to make a recommendation for a treatment. How are you able to bridge that last step where a patient can actually get a physician who will understand the report that you created and prescribe the treatment or in the case of maybe having to develop a NF1 trial for that patient. So what is the solution around that space? A great question. So essentially the question there is uh how do we build out and part the infrastructure to be able to allow clinicians to fully utilize this, right? Um because in a sense, you know, if you build all this out, right, there's a communication that the the clinician ultimately has the final say, right? The clinician is the person that's driving care. They're the person that's responsible for it. And a lot in our experience so far is that clinicians have been very happy to have all this infrastructure and support. Uh in part because you know clinician time is fairly limited. Unfortunately they're already incredibly overworked and kind of working incredibly long hours. So they necessarily don't have the uh capacity to spin up their own cluster and do real-time analysis for the patients and mine that data. So getting that actual output and in a way that's communicable to them, that's been incredibly helpful for them. But as far as like going to the actual um selection and the actual uh outcomes, the NF1 trial space and setting that up, that's like its own different like that's its whole other beast. But if it's something that's already available, right, and that's you have all the rationale and specifically if you failed your first line and you failed your second line and at that point you're doing your kind of best guess. Having that data packet to make that decision is something that most clinicians in our experience have been very happy uh to make that decision because have have to make that decision. So >> thank you. >> Hi, great idea especially the personalized lab for each patient. Wonderful. So I want you to comment on the initiative of cancer reversal, the reversal of ma malignancy. What do you think about that approach? Uh you heard about ka one Korean company is claiming that they are you know using their organoid model they do signals that they can reverse the malignancy. Um so I am not too familiar exactly with their claims and I think it it all depends in a way how you define malignancy right and realistically how you define that is going to be very different depending on the tumor subtype that you want to you want to select um and the progression itself and it might even vary from patient to patient but uh yeah I think that's a fairly difficult question I think I would want to see more data supporting that uh number one how are you defining it and number two I guess extended patient outcomes So, >> so my question is around uh ensuring safety. So, when we speak about the end of one development and having this AI lab, personalized lab, how will you make sure that this is not going to accidentally kill someone or create um unwanted consequences? >> Fantastic question. So, um so, so there's two ways to like break that down into, but focusing primarily safety like safety is number one, right? And actually this is something where it actually increases your safety profile if you think about it. So looking at the the reason I like to use that specific patient for the clonal evolution analysis is because if we had been part of that journey or this infrastructure was set up for that patient from the beginning they could avoided about three different chemotherrapeutic lines, right? Like that's something that significantly worsened their day-to-day life and outcomes, right? So there's so if you're thinking about it from like real time, there's always the selection of stuff where of of compounds and therapies where we do know the safety profile, right? Because we're not we're not necessarily building for some of these everything from scratch. For some of these you already know the safety profile and you kind of understand how that progression is. So you can almost stop it. Now, if you're thinking about building it completely from the ground up, right, that is something where um you can actually pull from the the way that clinical trials are done kind of today in aspect of it where you can essentially start a dose escalation, right? There's ways to kind of test it to try to compensate for that safety because at the end of the day, the goal is improve the outcome of the patient, right? Not necessarily hurt them. So, great question. and it's always at the top of mind specifically on the the signing from the ground up. So, >> all right. Thank you so much, Alfredo. >> Thank you for having us. [applause] >> All right. Can you hear me? All right. Great. Uh, thank you so much for uh to the organizers for giving uh giving me the opportunity to speak here. Um, my name is uh Ysef Ruhani [snorts] and I was told to speak about virtual cells. Um but I wanted to add a little twist to the story and talk a bit also about uh an area of work that I've been involved in the past couple of months which is um autonomous language model post training and also draw connections between that domain and how we can build better virtual cells. All right. So before we can get started I think it's useful to just talk about what even this term means and what are we like what exactly is the problem we're trying to solve. If we think about biological discovery um very much like science in general, it can be thought of as a search problem where you have a large space of potential hypotheses that you need to test over experimentally um with the goal of essentially identifying the most informative outcomes that lead you to um you know a specific answering a specific scientific question you might be interested in understanding a specific phenotype. And I would argue that one of the big advantages of artificial intelligence for u science and biological discovery uh broadly has been accelerating the search. Um so a great example of that is something like alphafold um or say ESM or one of the other famous molecular machine learning and foundation models which are able to map out for you this kind of broad landscape of possible outcomes. Um and you can do this without actually running most of these experiments. So you can actually um predict these threedimensional structures directly from sequence and and by doing so you can now essentially prioritize experiments and interventions that would be more interesting to perform or more valuable to perform without having to spend all the cost and time associated with that with that experiment. So a a grand vision has um within this within this space broadly has been how do we bring these capabilities to levels of abstraction that are perhaps more physiologically relevant um to levels of abstraction that connect more cleanly to human phenotypes to disease phenotypes and and this is broadly taken on the term um virtual cell research but I think more precisely the way to define it is how do we use AI to actually model biological systems at levels um such as the cell level the tissue level and all the way up to the organism level. What we've seen so far is tremendous capabilities at the level of sequence. Um so if we look at protein sequences, DNA sequences, wonderful talks coming after me that describe the state-of-the-art in that space, we're seeing enormous capability shifts, right? We're able to do things that we couldn't even imagine a few years ago. But what's missing is how do we take these benefits and how do we take them to that next level of abstraction the cell which is a complex and very messy bag of molecules that is constantly in flux constantly changing their whole other cohort of variables and considerations that need to be accounted for. And so how do we do that effectively? That's broadly I would say the subject of my research and and also the research uh area known now now more generally as uh virtual cells. One way to think about it is if if we're able to build these um these models effectively, we would have essentially a hypothetical space like the one shown here on the right where essentially you can map out um different cell states both healthy and diseased and identify interventions and experiments that can help you transition from like more disease towards potentially more healthy states. Why would this be impactful? Um so one of the biggest reasons for failure in the clinic is actually clinical efficacy. Um and one way to interpret that is we have we are very bad at predicting what is going to work um in a human being. Um and the reason for that is we just don't have very good models that are able to translate to actual human phenotypes to actual biological phenotypes. We've gotten very good at um designing molecules. We've gotten very good at binding to targets once they've been identified, but we are still not very good at identifying the targets themselves that we should be going after in order to um modulate a specific phenotype or specific outcome. All right, so how do we go about building models like these? Um so here's an example that that we built. It was called the universal cell embedding model. Uh so it was just published in nature earlier this year just to give you a sense for what these models look like. So when we're talking about cell state um essentially we can measure cell state in many different ways. One very common approach for doing this is using something called gene expression. So over here on the left what you see is is a cellby gene matrix. Um so every row here corresponds to a cell and every column corresponds to a gene and the value there corresponds to the number of transcripts that you measured for that specific gene. So if we take the standard language modeling approach, you can mask out um certain genes in each of these cells and try to predict what those are. And if you do this at scale, the goal then is that using something like self-supervised learning, you can learn some kind of a general representation for cells that can bring you to that vision of a, you know, this landscape that you can map over and predict outcomes and interventions over. What's great is that we're not the only people that have thought this way. So biologists um a hundred years ago proposed this idea called the Wington landscape which is a really cool concept in developmental biology which states that cells don't actually take on arbitrary states. They actually have um they they actually lie along something like a lowdimensional manifold. So so there are states that are much more likely to exist and there are others that are almost impossible just because of um transcriptional epigenetic um rules and constraints. And so in a way for us um I assume many of us here come from an AI background. This is actually very similar to when we think about representational spaces and lowdimensional manifolds along which your data might lie. It's kind of very similar idea. Um and it's kind of like an intersection between the way biologists think about cell biology and the way um machine learning scientists do. So we trained a model to do this um but with like 35 million cells which is nothing these days. Um and we were able to create um a pretty um mapap which is evidence that we have um a universal um a shared representation across all of this data. Um and the different colors correspond obviously to different types of biology and you can now study this and try to understand new relationships and new connections. If you look more deeply into that representation space, what we found was that there was structure emerging that was um in line with what we've seen in in the in the literature. So we can see that broad cell types are clustering together and then within those broader cell types known um finer cell types also form structures that u you know are well wellknown well studied. So all this is great we didn't provide this information to the model. We were able to get this out without uh without doing anything but self-supervised learning. That's that's great. How do we take this to the next level? What did folks do in natural language processing? Um we essentially went and tried to find what is the equivalent of the common crawl, right? let's go and find the internet equivalent of cell cellular data and essentially try to train a model on that. And so the way that we did that and this was a paper published earlier this year in cell we were um one of the first labs to use agents to go into the NIH repository of all recorded raw transcrytoic data. Um this was back in 2025 when agents were very new and cloud code was actually not uh the norm and it was it was a it was a very new idea back then. Um and essentially we had these agents run for a month and essentially reprocess all the data they could find um for cells. And so on the left over there what you see is the kinds of numbers that we got. So we went from 36 million to an order of magnitude higher close to like 500 million plus cells. It was great. We were very excited. But then when we started to look more deeply at the data that we gathered, we found that um the vast majority of data that we have is essentially um immune cells. And in within that by an order of magnitude um T- cells are over represented and and so that you can see this the the x-axis here corresponds to cell types and the y- axis is tissue and and the color bar is in log scale and and and the reason why this is um surprising is because you can essentially go scale up and mine all the data that is accessible publicly but you're going to get very little diversity very little um breadth of understanding um of the biology right and so this is something that the field has grown to realize um more and more. So how do we solve for making real foundation models for capturing cell state? So I would argue that the solution to this problem is to actually start thinking from a loop perspective to discovery as opposed to a linear perspective. Meaning that we don't have to necessarily scale up uh upfront and build as much data as we possibly can and then train our model. Instead, it's probably um more effective to now make use of models such as Claude and and NGBT that already have a really good understanding of this biological landscape through text and use them as a means for optimizing the data generation process. And so what I I like to call this as scaling AI guided experimentation as opposed to scaling um data generation. And what what makes this exciting and this is where I come to the title of the talk is when we start thinking about this process in the form of a loop. There's actually a lot of parallels that we can draw from other fields that are emerging very quickly such as automated um ML research or recursive cell improvement whatever words you want to use. Um they also actually um are attempting to solve um problems that take on a very similar nature as what we are trying to do in biology. So one way in which we like to think about it is we are actually increasing the time and the cost of the evaluators and the size of the loops but the the structure of those loops is actually not very different. And why do I say that? Because even if you're actually using cloud code or an autonomous research system to optimize your machine learning model, you still have to deal with costly experiments, noisy evaluations when measuring progress, and you still have to search over vast vast search spaces over experimental hypotheses. So, so what we what we theorized is there's a lot that we can learn from what's happening in the frontier labs as they are trying to automate their machine learning research. let's try to bring those insights into um biological discovery. So in order to do this I recently joined a startup inlogy um about six months ago and essentially the goal for this was a pure AI research startup which was focused on general purpose automated AI research systems. Most notably, they had a result a few months ago where they were leading the the leaderboard on post train bench, which is um a benchmark used by Frontier models for evaluating how well can claude and GPD do on autonomously building um on autonomously post-raining Quinn models, right? And so you can see that um they had built a system that was actually doing better than a lot of the common coding agents. And and and this is actually not rocket science. The reason why they were able to do this is because the the the autonomous systems that we use right now, the agents that we use right now for research are not really optimized for research. And anybody working in agents can tell you this that there are a lot of things we can do to optimize their ability to search and discover new um new ideas, new concepts. And in our case, we wanted to use it to test can we start to understand new biology. So what we did was we made use of the same system uh we call it locus and we applied it to molecular design and we started to ask um can we actually take foundation models which are in some ways a simulation of some aspects of the physical world. Can we can we essentially adapt those models for um tasks like property prediction and and can we actually search over them to um identify and arrive at more interesting scientific hypotheses better than if we were using agents off the shelf. So this is these are results from actually just yesterday. Um and indeed we were able to show that um the system that was optimized on language model post training was also able to do well in this scientific setting. So I'm not going to go into too much of the details because we're short on time but essentially we try to evaluate how does our validation performance increase on predicting molecular binding efficiency um as we scale AI R&D compute. And we find that like the system that was actually optimized for these discovery tasks in a machine learning setting was also able to do well in these scientific settings. As we scaled compute, we can see that there is a significant improvement over the coding agents. Um we also looked at more generative tasks. If we are to sample from the foundation models um representational space, so it's similar to how you perform experiments while you're sampling from a space. Um can we do that more efficiently? And yes, we could. And we were able to do it much more efficiently than um existing models including bolts gen codeex and and others. Of course these are incilico results. These have to be validated in in a lab. But this is really exciting because it shows that there's a lot of intersection with where the um AI research field is going in in terms of pure AI research and how we can connect that to um scientific discovery. Um there are more results in the paper. We also sorry in the blog where we also look at um are we able to handle evaluation noise evaluator noise more effectively. What that means is in many of these settings we have very small validation sets. This is very common in in science. You have very noisy means of evaluating um a physical system. Can you build um robustness into your discovery system such that it can still do well in test set um when it has access to small validation sets? And indeed we were able to show that we could we could do that um for this specific task. Okay. So just kind of summarizing and coming back to the bigger picture here. Um the idea here was to essentially build confidence that there are some shared primitives that underly um discovery research in pure AI that we can apply to scientific discovery. And and the vision there is that we can start to build loops um that connect with actual physical labs and that can actually optimally generate data within physical labs that are optimized for specific questions, specific biological and scientific questions that are of interest to us. And and I would claim that this is a much more effective and perhaps um accelerated approach to get to a usable and high utility virtual cell model than [snorts] um first generating a large amount of data and finding out later that perhaps it's not the most optimal for getting at you know answers to questions that we're interested in. Okay. Okay, so just just ending um on this line. This is actually the first line in the textbook uh the the leading textbook in the field of cell biology. Um in many respects, we understand the structure of the universe better than the workings of living cells. We know the full genome sequence of many species, but we still cannot predict how a cell will behave if we mutate a previously unstudied gene. It's exciting to think that this research is actually pushing towards this very central question. And um and there's a lot more work uh from my own uh previous research group that you know I'm happy to talk about since this is the paper club. Um and uh yeah I want to thank everybody involved in this work and happy to take questions. [applause] >> Okay, we got time for two or three. >> Hi there. U great talk. Um this is a two-part question. The first part is you mentioned training. Um what are some of the techniques uh you use for your post training? Is it RL or um do you see um you know some of the challenges that we see in RL and language or vision like defining judges defining um you know rewards or do they translate here? Second part of the question is you mentioned virtual cell. Have you seen a gap between your virtual simulation with physical reality and how do you bridge the gap like what are there metrics to measure them and ways to transcend that gap? Thank you. >> Yeah. No, thanks for your question. Um so I'll take the second one first. Uh yes, for sure. I think um all the evaluations that we perform, we are essentially testing how well are we re how well are we able to recapitulate reality or physical experiment. Um the reality is that itself is a very uh challenging thing to do often because every time you measure a cell even if it's in the same conditions it there's a lot of fluctuation is a lot of as I mentioned noise in the evaluators. Uh but yes that that is the goal is to try to um simulate or emulate the the physical system as much as possible. to your first question. Um, yes, of course. I think I think the idea uh behind some of the last part of the slides is that there's a lot that the scientific community can learn from the post the automated post training and the RL communities there. There are a lot of shared challenges. So you correctly identified uh challenges such as staying on your task for long periods of time, long horizons, making sure yeah you have judge agents, other kinds of agents tracking progress. Um making sure that we carefully think about the ideas that we generate that we account for the cost of experimentation when we're generating ideas. Uh we're strategic about how we um rank and process those. I think these are aspects that are overlooked under like you know the standard coding agents that people normally rely on and so that's something that we tapped into when we were doing the machine learning work and we found that it actually translates to scientific problems as well which is quite exciting. Yeah. >> Hello. Yes. Yeah. Hi. Great presentation. Thanks. Um quick question on the uh x-axis on those charts at the end of your slide. You you say that H100 compute are we talking inference or are we talking postraining as in RL and how do you know what anthropic and open AI are doing if it's RL? How do you know how many H100s they're using? >> Right. So um so it is it is GPU compute. Um and what we are evaluating in that slide is how much GPU compute did we use. So essentially the cost of each experiment is measured in GPU compute hours. And so how did we make use of our GPU compute hours to improve the performance of this model at this specific property prediction task or this ability to search over like um a foundation models latent space and then we compared to like anthropic and open AI's off-the-shelf agent tools. So they were so everything every every um agent or research system was working in the same sandbox with the same GPUs and we were just measuring who's using um compute more effectively. >> So token generation >> what's that? >> So token generation over time >> um it's not tokens it's a compute because we train models. So we are um running the we are running that training on GPUs. >> So it's not uh it's not it's we're not measuring token costs um if that makes sense in that plot. >> Yeah. But but it's a good point that you do have to account for it because in some some ways there is a trade-off between the GPU compute and well not trade-off but but both of those kind of impact uh uh how how costly an experiment is but most of those were using the same amount of tokens. Yeah, >> I'm super curious to learn of the architecture for Locus and uh how it works and uh yeah. >> Yeah. Well, I I can't share a lot, but uh it is um it's it's essentially um it's built on top of LLM. So, it makes use of a language model under the hood and uh we've essentially built uh clever engineering around it um with different strategies. Uh I think that's like maybe a high level perspective. Yeah. Yeah. >> They use they use Python. >> We do use Python. That is Oh, you gave it away. [laughter] >> Um last question from me. So almost a lot of the gains in uh like a lot of the issues when you when it costs a lot of money to get a sample to evaluate a sample versus if it's free you care more about sample efficiency and so you generate a a drug trial how much can you learn from it you want to you know you're you're much much more likely to do the con spend infinitely to train something to learn as much as you can from that sample most of the gains in the LLM land have been from things like continual harness, not actually updating the weights at all, but doing a different type of learning. Uh, have you seen you I've seen a lot about updating the the the parameters. Have you done a lot on uh continual harness type stuff, the harness on top and having it be more in context, updating the context and things like that? >> So, uh, I agree with that sentiment. I do think there's a lot of value in the harness. Um I would say for a lot of the work we've been doing for especially for um uh autonomous uh agent guided like research and experiments there is definitely a lot of lowhanging fruit that you can get from the harness first before you do any kind of post- training. So um so I guess the answer is yes for sure. Uh is that answering your >> I mean I guess is the right answer going to be a mix of both? You're updating the parameters but then you also have to update the update the harness as well. Yeah, I think I think I mean there's a lot of debate on this online, but like uh in many in many cases, yeah, I think the harness helps make more general like harness engineering helps you make more generalizable solutions without having to tap into the weights directly. Often if you really need to bake something for fast retrieval, it makes more sense to perhaps do something like post training. But we've been very interested in general purpose solutions that work across domains cuz that lets us test to make sure that we're not overfitting and just like benchmaxing. Uh so yeah, we've favored less model weight fine-tuning approaches. >> Divia just gave me five bucks to to say ask that question. So to tee up her talk. So what's that? Divia asked pay me five bucks to tee up her talk. Awesome. Thank you so much everyone. Thank you. Good job. [applause] So how much can we learn from DNA? Uh language models are really powerful and they're trained really stupidly, right? With next token prediction and then a lot of stuff on top. We've created these amazing models that are superhuman in many ways that all of us use every day. But humans can read language and we can read code and we can read math. We can't look at DNA and understand what it's saying. In some ways though, DNA is a language. Although it was written through this process of natural selection. So DNA, this genotype of an organism, it encodes the RNAs and the proteins that go on through complex biological mechanisms to give rise to complex phenotypes like whether you're sick or how tall you are. And then natural selection acts on phenotypes, changing what DNA survives and what is observed when we sequence large amounts of genomes. So this is a rich data source that can tell us a lot about ourselves and about biology. And as you've heard, it's growing really quickly. So just in my PhD, the amount of genomes on NCBI has doubled in the last three years. And we'd have no way of making sense of all of this vast growing data and finding the most interesting things in it to follow up scientifically or using it to build better and smarter models. And part of that has been because genomes are pretty hard to model. If you think about the DNA base pairs, there's only four. So that can be your vocabulary size. And the evolution is not just acting at one level. It's acting at every level. So you have parasites within parasites within parasites within organisms. They're copying themselves all over the genome. And so it's a highly redundant degenerate sequence. And it's extremely multi-cale. So you might have a single gene that spans hundreds of thousands of base pairs that needs to be processed. And in that just one base pair has a mutation that causes you to not survive. So we need to account for all of these things if we want to actually be able to model genomes and understand biology. And we start by being very simple. We train a next token prediction model just like GPT but on DNA. And it's a it's a simple generative pre-training across all life. And we're hoping to learn some of these higher order interactions. In our case, we use a hybrid architecture of attention and convolution to get to longer sequence lengths. So something that's really important is data. And we collect all the public data we can get and then throw a lot of it away. So we only take one representative per species and then we throw away DNA data that's too similar to others in that data set to get to this 9 trillion base pair data set that we train on. But we actually go further than that. So in the pre-training phase, which is most of the training time, we actually focus on this augmentation of the data that's focused around annotations. So parts of the genome when where there are genes or RNAs or other evolutionary conserved elements, these are more likely to have less of that repetitive junk that I was talking about. And we focus training on that more. So we're actually throwing away trillions of tokens of of data. And we train the model in two phases. So we first start shorter context length focusing more on those genic regions with more conservation. And then we extend the context length up to 1 million base pairs. So you can imagine this is learning what's in DNA and then how to compose it at longer and longer sequence lengths. We do this at two sizes, the 7 billion and 40 billion parameter models. These are dense models. This was 20 late 2024. And we let it rip. And so the loss goes down. Awesome. We can predict the next DNA base pair. That's really exciting. Uh what does that actually buy us? So okay, so I'm showing you that when I give it more DNA in the context, it's predicting that next base pair better. when I do a bigger model, it's predicting the next base pair about better. But what does that actually buy us? So, we now have this DNA language model, input DNA, output DNA. Uh we we need to figure out ways to use it. We need to figure out ways to uh ask scientific questions using it, combine it with other methods. So, broadly there's different ways of interacting with a model like this. either you know through the input and output such as by doing a a prediction of the likelihood or generating more sequence just like a language model where you're prompting and generating English or you take intermediates of the model like certain embeddings or using mechan to find interesting neurons. Um this is important because we can't just talk to it in in DNA, right? We we need to figure out how to understand what it's telling us. And so one way right away that we start using the model is looking at the likelihoods. So this is a generative model trained on lots of real DNA. So it has a good sense of what's likely in DNA. And when you give it a mutation and it says it's unlikely, that correlates with that mutation causing a disease or causing a loss of function of some kind. And since it's trained on DNA, it can do this for proteins, for RNAs, for non-coding sequences. This is basically a decision model. Um, so EVO2 is a decision model. Uh, and we can benchmark this on different data sets that exist from human clinical data to experimental data sets. Here I'm showing you one result which is on BA 1 uh a gene that causes cancer risk and we're comparing the correlation with mutations that cause loss of function u between EVO2 and many other models. So for coding, non-coding, and combining these, Evo2, which is just trained on all of these different species and their genomes, you only seen one human reference genome, is competitive with or better than supervised models that specifically have humanpecific training and data. Um, this is only possible thanks to that data augmentation step. So it turns out that if we had just left in, you know, 50% drawing from random parts of a whole genome, you actually go.16 lower in your Spearman correlation for that benchmark I was showing you. So that's from the difference between being state-of-the-art and bottom of the pack was just the data mixing. And in fact, uh, EVO 2 there there is a weakness which is these distal enhancer elements. And recently another group has figured out a data composition to boost performance on on that. So really you know how do you change what are you trying to predict? How do you change the data to to learn it? The other way of interacting with the model other than just taking the likelihoods that come out is interpreting what it's learned inside by looking uh at its embeddings. And so we train these sparse autoenccoders and pick up features that EVO2 has learned by comparing with annotations that humans have made. And this is one very interesting feature that EVO2 learned which is a fage associated feature. So on the left uh what I'm showing you is a the inner circle is the proage annotation of this E.coli genome very wellstudied genome. Um and you'll see that the outer circle is where this feature this set of neurons in EVO2 is firing. And so through just training on predicting the next DNA base, it's firing on different proofages. It's also firing in other places. But when we actually look at a lot of those other places, there are some proofages that weren't annotated by other methods. There are crisper arrays which contain uh tiny banks of of fage DNA that the bacteria uses to defend against phages. There's individual genes that have been repurposed from the phages by the bacteria. And so this is a a very diverse set of biological functions all connected uh in in you know this concept of a phagee but completely different in their sequence that just training on DNA has taught the model are related and and we see this for for different features as well. But it is also a GPT style model. So you can put in a prompt and generate more DNA and we can actually compose this with other models because DNA you know encodes RNAs encodes proteins. We can compose a generative model of DNA with function uh with with classifiers with function prediction and uh use it to achieve more and more ambitious biological design. One of the most exciting of which was the design of new fax-like bacteria phagee with which are viruses that infect bacteria. And so this is the solved crystal structure of the capsid of one such fagee which was generated by fine-tuning evo2 viruses and then running it through uh a design pipeline. And as you heard this kind of design is now making its way into patients. So by training on DNA, we're able to get a sense of of these dependencies and and this uh language that we're not able to read as humans that underlies a lot of different levels of biology and we can learn to use them for different predictive tasks. So uh one one such task that's really exciting is discovering new important or interesting systems from genomes. Right? A lot of our most powerful biotechnologies were evolved and not made. And so what we can do uh with with this new language model Manurva, we can predict actually the interaction between different parts of DNA which helps reveal things like non-coding RNA structure or regulatory DNA structures and helps us discover new things like uh crisper-like RTS. So this is a a new model that patches a a problem that EVO 2 had, which was it struggled with predicting this kind of co-evolution between different parts and how they interact. So by training a birectional model, we're able to predict co-evolution more accurately, which allows us to rapidly scan different genomes and identify interesting parts for follow-up. And so here I'm showing just the prediction of base pairing, which is a type of interaction across this bacterides genome. It lines up with known annotations for interesting systems like crispers and RNAs. But there's also a lot of uh unknown stuff and in fact in most even the best studied bacterial genomes we have hundreds of non-coding RNAs that are revealed now that are missed by biostatistical methods or previous models. And one of the coolest things that we validated in the lab was this UG27 reverse transcriptase which has a crisper-like non-coding RNA array. So, it's this thing that viruses carry that has this programmable array that goes from DNA to RNA back to DNA creating these little hair pins. And we we still don't know what they do, but we'll try to find out. So, I hope that I've convinced you, you know, training on DNA helps us learn interesting capabilities, many of which we as humans don't have. And these models can be tools for human scientists, for AI scientists, or you know, integrated in in creative new methods to train on multiple types of modalities together. Science is of course a team sport and I've showed you work from from a lot of different teams across different institutions that make this possible. Thanks Nvidia for helping us train Evo 2 on their GPUs. And I'll now take your questions. >> [applause] >> All right, we got time for two or three. >> All right, so it looks like you tokenized at the nucleotide level. So I'm curious if you tried out different tokenization methods and and why you landed on the nuclear tide level. >> Yeah, I I'm definitely a fan of of you know bite level tokenization eventually end to end as much as possible. um in research that that we did at smaller scales, it did seem to help with some of these benchmarks we had to tokenize at that level. I think there has been some progress um uh at different levels also. So you know definitely more research needed but so far it seems to help with especially the mutation prediction to have it at that resolution. Um I have a question. Um so uh in physical AI uh people have been playing with these two tower transformer models where one tower can uh of transformers can uh deal with uh visual data and then the second tower can deal with u uh tokens or something like that. I wonder if you've uh considered trying something like that with this data to extract like human language interpreted um insights out of the DNA. >> Yeah, great question. Uh there's I think there's different competing approaches right now. It's not clear which one will win. I do think that you know Frontier Labs are sprinkling in some DNA and just uh training their models on on text and DNA. Some researchers have tried using yeah using language models to to embed DNA and combine it with a human language model. It's not clear yet which one of these approaches will will work best. >> Thank you. >> Uh hi great talk. Um I'm curious of like how do you thoughtfully assemble like the pre-training mixture? Um because you guys like made the choices where you threw out some sequences and you augmented some. Um, and I'm sure you guys use a lot of judgment for that, but is there a way that you can find the optimal pre-training mixture? >> I think especially in this case where we're writing the book, right? There's not like all of these different pre-prints we can look at and we can compare hyperparameters like there was a lot of uh eval being written at the same time as the data was being processed and the the scaling analysis was being done. So, so I think you know with more with more time like as the eval mature you can you can definitely do more uh guided based on what's lacking and so on. Uh uh thanks for the talk. I was wondering um you had a slide about the interpretability and I was wondering based on what you found from there if you try to steer some of what you learned from EVO 2 to maybe I don't know design DNA that can you know bind to cancer cells you know for for you know therapeutic purposes or any other applications uh based on uh inter learn features. Yeah. >> Yeah. So just uh last week we put out some new preprints on how we're prompting and steering EVO. It uh right now I'd say the the mechan approaches did not work as well as these other approaches which you will hear about in the next talk but it's definitely a active area of research and it's a great great question. >> Thank you. >> Um and I'm to your right. Hi >> did I get it correctly that you uh train on one organism only in your prefilter? Uh so you separate out one >> it's one genome per organism. >> Exactly. Are you not missing quite substantial variance within those organisms then? And yes because then I got confused because you're also able to predict cancer mutations. >> Yes, exactly. That's right. >> How how how does that work? >> Yeah, great question. So, and this is also true if you use a statistical method where if you line up a 100 different primate genomes and you just align them with a statistical method, you will be able to predict which sites are more conserved because they've changed less over longer time scales. So, if I were to train on everyone in this room, there's a lot of things I would learn which while interesting, would not tell me very much about cancer. And there are parts that would tell me a lot about cancer. So I think going forward we definitely want to use that information but it's just it is like very redundant between different humans while if you have a fixed token budget you want to focus on very diverse conserved signals and that seems to be the case but yeah it's definitely a source of information we're missing there's also a lot of multiomic data that you know is not in EVO2 that tells us about the function of DNA and so especially for human specific uh things we have a lot of population data we have a lot of cell data and and new models are bringing all those together. Yeah, spectacular work. Thanks. >> Thank you so much for your talk. Um, I really liked it. Uh, I've been following EVO 2 as well. Uh, I would I would be really curious to know what would postraining EVO 2 would look like because again this is kind of like you are pre-training and mid-training on huge amounts of genomic data carefully filtered and curated. Uh, but I would be really interested to know what would postraining look like? >> Yes. So, I'll give a two two-part answer. And I think that we're starting to see this the conversion of these models from just foundation models to actually being post-trained for for specific task for being uh state-of-the-art for helping make biological discoveries that you couldn't make before. I'd say one approach right for for the discovery specifically in in Manurva what we do is we pick out specific parts of the model that through pre-training have learned these distal uh interactions and then we we use that to project out these structures. So, so the model in the attention it turns out that it's it's picking out interactions and we just do a very light regression just mech interpreting it and this is like not like language model post- training. Um, but I I I think that's a sign of getting towards like molecular structure and interaction from a model that's purely you know DNA trained on EVO 2. There's some really exciting results that we just put out two weeks that you're going to hear about in the next talk. basically by by using the model iteratively I think we will be able to construct uh you know things to post train on. >> All right thank you so much [applause] hi everyone uh thank you to the organizers for inviting me to speak today. uh excited to share some recent work on a new paradigm for efficient biological design um which involves prompting your genome language model and uh follows on to a lot of the ideas that Garrick introduced in his talk. Um before getting started I want to acknowledge this is work done by an incredible team. Much of this work was led by Alex who's a postoc in in Brian Heath's group as well as Garrick who you just heard from and many others in the HEAB as well. So it's probably not a surprise to anyone in this room that fot prompting has radically shifted the way that we use LLMs. um without any weight updates task specific fine-tuning we're able to provide examples of a task in context for example email is written in a particular style and the LLM is able to infer the particular task and task constraints uh from the prompt and is able to complete that task and this fshot prompting paradigm is enabled by an emergent property of language models in context learning or ICL and it's a property that emerges from large scale pre-training and has enabled everything from multi-step reasoning to real world applications of agents. The paradigm in biological sequence design traditionally has been quite different. It's relied on data in intensive task specific fine-tuning. So if we look at some examples from the EVO 1 and EVO 2 papers for example, EVO1 was fine-tuned on tens of thousands of crisper cast LOSI um in order to generate uh functional uh cast 9 systems um in a paper from this year Evo 2 was fine-tuned on thousands of micro verirday genomes to generate a functional bacteria phase genome and while this is possible and has been effective in some applications we often don't have you the the amount of labeled data that we need to fine-tune for many design tasks in biology. So this paradigm of fshot prompting can be quite powerful for this reason where we might not have um you know tons of labeled data. We might only have a handful of examples and we are able to prompt EVO 2 with a set of examples share some properties and are able to design a novel sequence that has those properties. And this fshot prompting paradigm in EVO 2 is enabled by a surprising finding that this model actually has capabilities that emerges from its pre-training. And this is surprising to us because this emerged from just large scale next token prediction on nucleotides um which has not been widely documented to occur in regimes other than natural language pre-training. So using this ICL paradigm, we will I I'll show you some applications where we design systems from the scale of around 100 nucleotides to 100,000 nucleotides. And if we look at kind of what EVO2 is doing in when when we prompt it, it's able to leverage many examples in its prompt and seems to be learning algorithms to recombine or mutate them in order to efficiently design novel novel things um that are predicted to be functional. So going through a few examples where fshot prompting has been was useful. We started with natural proteins. Here we tried to design land modulins which are a class of proteins that bind rare earth metals and have been revolutionary for mining applications. There's a certain class of land modulins where there's only 14 known examples in nature. And when we prompt with those 14 examples, we're able to generate novel proteins that are predicted to be, you know, land modulins of this family uh with, you know, just the the very low number of examples that we had. In uh antibbody design, we're able to prompt with hits from a structure-based generative model and without any explicit structural conditioning. Uh with EVO2's ICL capabilities, we're able to generate novel hits to all of the targets that we tested. We also uh revisit the crisper cast design setting and just prompting with four crisper cast lossi in context. We're able to generate crispercast 9 systems with the crisper array, the tracer RNA and the cast 9 protein. And finally, we're able to take this to the scale of whole phagee genomes where we're able to generate things which have the predicted arch gene architecture that we're looking for at a fraction of the compute cost of if we were to fully fine-tune the model. One of the design settings that we focused on to actually validate in the lab is the design of cell type specific enhancers. So enhancers are elements where um you can express proteins in in particular cell types which is important for things like drug delivery. And one thing that we found when um when doing this design application is we can actually improve fshot capabilities through post trading. So um coming to the question raised last talk we can actually employ a reinforcement learning loop with predictive models as our reward function. In this case, the reward model was a sequence to function model where we it can take in a designed enhancer and predict its specificity. And we can improve uh we can update the weights of EVO 2 and actually it's able to leverage its incontext examples better after post-training. And with the post-train model, we fought prompted it to generate a novel enhancer. And Alex in the lab actually tested this in the in the mouse brain. And as you can see by the fluoresence in the particular regions shown here, it was actually um known to it shown to be active and specific in the mouse brain which was really exciting. So where does this new paradigm of prompting your genome language model leave us or where can that take us? I'm particularly excited about new approaches for for data efficient conditioning as I think it can enable lab in the loop style design where maybe we have some initial prompt pool from natural sequences or existing generative models. We prompt EVO 2 in order to diversify or uh design novel things that meet those constraints. We test the most promising designs in the lab and we're able to update the prompts um in creative ways in order to create this loop type procedure. Also with the with the paradigm of fshot prompting you can imagine some of the um the ideas that have been explored in the LLM world of uh different inference time techniques uh or harness engineering techniques. Uh we can start to think about how might that might apply in in the biological design setting. So with that uh thank you so much for listening. Uh the work that I talked about today is covered in these two preprints that we released last week. Uh the left one on the enhancer design setting with uh the wet lab validation and the right exploring the protein and and fage genome design applications. So thank you and happy to take any questions. [applause] >> Uh thanks for the talk. Um I'm super curious on uh kind of like if you could give like an overview of the um kind of like the RL and like the uh environment design or like specifically like the post-training uh mechanism. >> Yeah, that's a good question. So here the the setup was essentially you prompt EVO2 with some um natural examples of enhancers that are active in a particular cell type. you um Evo2 generates um a set of novel sequences and we use GRPO um in order to update the model and specifically the the reward function used here is a sequence to function model like things like alpha genome if you're familiar but it essentially takes in uh regulatory elements and predicts um what what cells it would be active in the oral environment is like quite standard um essentially doing this doing this in a loop and there's some other reward terms that we used to avoid uh reward hacking um things to make sure that the entropy of the DNA sequences is reasonable um the GC content is reasonable things like that. >> Uh that was a really excellent talk. Could you maybe give some principled explanations of um why this incontext learning would be happening for example you know biosynthetic clusters or things like this would make a lot of sense but I don't understand exactly why for antibodies we would expect this behavior. >> Yeah that's that's a great question. And I think one of the things that we found surprising as well, but given that um this emerged in in pre-training and it wasn't explicitly trained in, I think there's there's several like theories as to why ICL emerges. Um one of the prominent ones is that the longtail um like kind of power law properties of your pre-training data distribution kind of gives rise to these properties. And a lot of natural data including genome data has these these longtail properties where you have certain cursors that only appear in rare contexts or in certain contexts. So I think the the one of the main hypotheses is the burstiness of this data actually leads to leads to ICL. There could also be some um like like you allude alluded to some like semantically related data or like parallel structures in genomes that that might lead to this and mechanistically there's some analysis of this in the pre-print but it does seem like um a similar like induction head mechanism that we see in language models um also appears to be driving the ICL in in EVO 2. >> Super interesting talk. I guess I'm kind of curious on maybe a similar note, how much of the I guess in context learning plus RL working specifically for enhancer design uh can maybe be attributed to I guess the ability to replicate like uh specificity patterns that we already saw versus how much of it like I guess maybe a better way to say it is is it able to learn like very general patterns and so able to construct enhancers that are specific uh in certain combinations of cell types where we don't see that pattern of like specific expression in the body or is it more uh able to like say okay I've seen this pattern of specificity before uh or something relatively similar and so I can you know look in context or in my like set of things I've been trained on for that. >> Yeah, that's a good question. And I guess um maybe you're kind of alluding to the trade-off between what it has learned in weights versus what it is able to pick up in context. I think it's a combination of both um where perhaps what's happening is um in the prompt examples there might be certain motifs that um kind of tell the model that maybe we're we're interested in specificity in certain cell types or activity in certain cell types and the model is able to um kind of preserve those important motifs and recombine them. Um, but of course like it is also likely seen some of these motifs in pre-training. I will say that for enhancer specifically um they're they're a little bit sparer in EVO 2's pre-training distribution. So I think that while it does kind of rely on it its pre-trained prior, I think a lot of this is um what it's reading from its prompt context and recombining the the motifs. >> Thank you. Thank you so much Devia. [applause]