Ian Maurer

PangoPup

Sat, 01 Aug 2026 00:00:00 GMT

PangoPup is an open-source, GPL-licensed Rust service built on the Pangolin model. It provides fast, local splice predictions for GRCh38 variants. Bioinformatics and clinical genomics teams can run it on their own hardware and keep variant data inside their walls. The project is active.

Most AI Agents Are Toys. These Ones Treat Cancer

Wed, 06 May 2026 00:00:00 GMT

Introductions

Dev: Hi everyone and welcome back to Devin the Details, the show where we try to cut through the AI hype and focus on what it takes to build and deploy intelligent systems in the real world. Today’s episode is a little bit different and I’m really looking forward to it. When we talk about agents, LLMs and enterprise workflows on this show, uh what usually happens is, you know, we work focus on how different enterprises and organizations are thinking about their production deployments of AI, the security and risk profiles around them. But today we’re going to be talking about what happens when you take those same ideas and apply them to something that’s even higher stakes, like cancer treatment.

I’m joined today by Ian Maurer, CTO of GenomOncology. Ian has over 25 years of experience building complex systems and today he’s leading a team working at the intersection of AI, genomics and real-time clinical decision support. What stood out to me though is uh about Ian’s work is just how practical it is. It’s not theoretical AI, we’re talking about systems that pull together drugs, clinical trials, safety data, research and is looking to be able to give clinicians a full picture and the ability to actually scale this out as well via AI agents.

So in this episode, we’re going to dig into what it takes to build agentic systems in health care and how reliability and trust also operate on higher stakes domains and what the future of AI looks like when it’s not just optimizing a workflow but actually now in the loop in a more sensitive field like health care. Ian, really excited to have you here. Thanks so much for joining us today.

Ian: Yeah, thanks for having me.

Dev: Awesome. Ian, tell us a little bit about GenomOncology in your own words. What’s the mission of the company and where has it found success?

What GenomOncology Builds

Ian: Yeah, great. So yeah, GenomOncology is about 14 years old 14 years old now, uh based in Cleveland, Ohio. We’re a precision oncology company. Our main product and go-to-market is around pathology, molecular pathology. So it’s helping clinicians, molecular pathologists, take data from devices, whether it’s next generation sequencing, IHC, FISH, these are some older modalities, taking that raw data and then turning them into actionable insights. So, our software helps them understand the quality of the data they have and then interpret that information and then generate recommendations for helping a patient downstream. And so, we generate reports, but we also have to integrate with back-end systems. So, that’s, you know, these laboratory systems and the EHR and and other back-end systems.

Our software is all on prem, so we’ve been, you know, we since the beginning, uh where, you know, I’d say 90% of our clients were actually on physical hardware that the client owns, slowly moving towards, you know, virtual private clouds that they manage and control and we’re cloud agnostic, so we can support, obviously, Amazon and Azure and Google and anything else that people have. Uh and then a few clients that where we do all the soup-to-nuts hosting for them, as well. And so, that’s our main product line. It’s really, you know, how do we help a molecular pathologist do their job but do it faster and trust the information.

And then since then, we’ve been getting into like, basically, growing our business around that center core competency that had that had its root is really, you know, good old-fashioned AI. So, we have a knowledge graph, right, where we have diseases, drugs, genes, mutations, all organized in in what’s called an ontology or a graph of of information that we can then make recommendations off of. What we we called inference, right? It’s it’s a different type of inference, but you’re basically given a patient record, the patient’s the query, do inference off this graph and generate a set of set of recommendations and insights. And so, that that thing has been around since, basically, 2013 when I designed and built the first version of it.

And that that one capability, which we call match, uh we’ve used in a variety of different other modalities. So, there’s a there’s a exercise called a molecular tumor board or tumor board at a at a major cancer institution where the hardest cases go, right? They’ll review, you know, 10 cases a week of the the hardest cases that doctors can bring their patient to tumor board, and then our software helps them organize the information and make presentations so that they can actually understand what is state-of-the-art and what’s available and what are the clinical trials for it. And then we have analytics, right? You know, sort of like a Tableau or, you know, a BI tool, but it’s genomics-aware, right? So, it actually understands how the, you know, genome works and it understands all these ontologies.

And And we also do what’s called clinical trial screening. So, helping, you know, helping the matchmaking process, right? Where you have a, you know, a thousand patients a month coming into your clinic and you have a hundred clinical trials that are ongoing. How do you do that matchmaking? Uh it’s a sticky problem that we’ve helped with for years using our that same knowledge base. And now we’ve got agents. Um and with agents, it’s really what’s next. Uh how do we take agents and make all the stuff that we do that’s really powerful, but just make it easier, right? Make it faster, more accessible, and and really drive the results at places where they don’t have the staff necessarily to maybe integrate a complex solution uh into their into their daily workflows.

Harder Than It Should Be

Dev: This is really fascinating. I mean, there’s a few different areas that it sounds like you’ve been invested in. Like, I heard around analytics and dashboarding and reporting, clinical trial matchmaking, which I know is actually a large business in and of itself as well. And then being able to support the new kind of AI agent infrastructure. I want to talk about BioMCP in a few minutes. Before jumping into that though, I’m curious across the different essentially almost um offerings inside of the product, was solving one of them more challenging than you initially anticipated? Or did it all kind of flow naturally within the initial knowledge graph?

Ian: Yeah, I mean, it’s it’s all more challenging than it should be because healthcare just hasn’t kind of moves at a bureaucratic speed, which is fine. You know, we’re we’re we’re we’re trying to be that change agent helping folks do that. Um you know, that initial generating a report, you know, 13 years ago, understanding the different variants. We thought we had the interpretation that it was going to be, “Oh, you look at the variants and then you just make a report with all the variants on it.” And we realized that basically has no economic value. What you need to be able to do is think about the patient holistically, understand their disease, prior interventions, and other information about them. Yes, you need to know what the variants are and what the interpretation of those variants are, but you that then need to like look at it holistically to then say, “Oh, here are the Here are the therapies that are available, right? Here are the guidelines and here are the trials.” It doesn’t make any sense to tell somebody about um you know, you can tell them, “Hey, they have this this variant that’s not a great variant and there’s no drugs and there’s no trials.” Right? That’s That’s doesn’t leave the person with a lot of options. So, you want you really want to uncover all the information for that patient.

And then what we realized quickly is “Okay, we’ve got this knowledge base, we can match against it, but we need to know what the patient’s uh record is.” And, you know, think back to 2015, the patient’s record is a PDF or a set of PDFs or some text that a doctor typed out. How do you actually get that text into something actionable? So, we really invested a lot of time in the 2015 to 2020 era of how do we use OCR? How do we do basic NLP? So, I was using technologies like spaCy and and and and other things to do named entity recognition and and and linking to our knowledge base. And that was really hard. I I I looked at these reports and thought, “Oh, yeah, I can find all the entities. I know what this is.” And then you try to do it in practice and you realize, “Oh, wow.” You know, people are excited about 30% accuracy and that’s obviously not going to fly in the face of you know, trying to help a patient. So, we had to build a bunch of human in the loop systems for helping our clients you know, label this information or extract this information, but do it at a at a speed that actually was cost-effective. And that was that was a major that was a major challenge.

And now with LLMs, I’m seeing every single point along the way, right? At you know, starting with GPT-3.5 Turbo, trying to take that thing and say, “Okay, can this thing accelerate what we did with named entity linking and named entity recognition?” Uh and it’s that has been a a slog as well. And it’s really it’s kind of like everything else, you could really see how it’s turned a corner with first the reasoning models, and then kind of the Opus 4.5 and and on moments where you can really see, “Oh, yeah, these problems are all going to fall away.” Which is great. I I mean, I don’t want to be in an NLP business, right? I was in an NLP business for 5 years. It was a terrible business. Nobody’s happy. I want to be in a I want to be in a world where NLP is a solved problem, and now I can actually do the genomic insights and and take advantage of all that great structured data that we can now get for our for our clients.

Why NLP Was a Terrible Business

Dev: Why was NLP such a terrible business? Uh and I’m actually really curious about the practical reality of starting an AI-oriented company in 2013-2014, pre-jerk of AI, but post-deep learning. Um and so you had libraries like spaCy and some early transformer architecture models, and maybe like that intermediary {slash} beginning period of when you were probably um jumping into this. I had a guest on uh 2 weeks ago who’d also started a company, actually I think, you know, around the same time in customer support. His experience was like at a certain point basically had to throw away everything that they had been developing for years, and really move towards like the net new stack built on top of LLMs. But tell me a little bit about why like building, you know, in an NLP space was not the best business, and then what was the practical reality? Did you have to rip and replace, or did you some like incremental add-ons with LLMs?

Ian: Yep. So, yeah, so it’s hard. NLP is hard just in general. And then NLP in healthcare is extra hard. And then in NLP in healthcare where you do what we do, which is not trade software for data. So, one of our fundamental premises of a company was we’re tools company. We help our clients make solutions that make them a learning organization that help their patients. And that at the end of the day, the data is really the patient’s data, and the caregiver of that data is the hospital. We have competitors who go in and take the opposite approach, which is like, “Oh, uh we’ll give you this $0.05 of value, and we’re going to take your data, and then we’re going to go sell it for $10 of value behind your back.” And that to me has always just been uh gross. So, so we didn’t do that. We weren’t in the business of we’re going to do this thing, harvest your information, and then try to build a product off of it. And never did that, never would do that. So, and then on top of all that, HIPAA and other regulations that are actually good, right? Cuz they help protect patients’ data and protect them from payers knowing about what their sicknesses are, and therefore denying them coverage.

Dev: Mhm.

Ian: Those are all good things. So, but it makes our job really hard cuz what you realize real quickly is that NLP is a labeling problem, right? You really need to What you need to have is a strong model that’s not too expensive to run, right? So, the BERT-based models are still actually really good at named entity recognition. Like there’s a zero-shot library called Galiner, which is fantastic. Like you give it some labels, and you say, “Here’s some text, go ahead and do your thing.” And I was using Ludwig back in the day, right? Like, “Hey, how do I take this data, and like how do I train it using, you know, some some fabricated information, or can I train the data off of publicly available PubMed articles and uh clinical trial documents, and then make a model that actually can parse a clinical note?” You can’t. Like it it just it You certainly couldn’t in 2018.

Dev: Mhm.

Ian: Um so, it was just a really hard problem. So, we would basically, the way we did it was an ensemble approach. And the ensemble approach was uh we’re going to take the best models we can, right? The best BERT-based models we can, do as much as we can with them, but then we’re also going to build out um basically, for lack of a better term, rules-based approach. And the rules-based approach was, okay, we’ll have a set of regexes over here, right? Patterns for things like measurements, invariants, and other things that kind of follow a pattern. Great, dates. Yeah, we can we can go parse the dates. Obviously, you have to deal with European dates versus American dates, but other than that, you can kind of get something somewhere with those types of regexes. But beyond that, anybody trying to do regex, cuz I tried, you’re you’re you’re it’s a trail of tears.

And then on top of that, what I would do is, you know, take ontologies, right? There’s there’s this thing called UMLS, which is basically a dictionary of all the terms, right? All the drugs, diseases, etc., and all their synonyms. And then what you try to do is you take that, and then you basically make uh what you know, what’s called uh uh I don’t remember what the term is now, but there’s a algorithm called Aho-Corasick, where you basically take text, and then you can go find all the terms in the text. And you can make a really fast one, and it can find all the terms, and then you realize very quickly that you either have a false negative problem or a false positive problem, where you either accept too much, and you’re just generating too much noise that you have to filter through, or you have a or you’re missing too much. And so basically, then you have OCR on top of it. And OCR back in the day with Tesseract and some of these other technologies, they’re just a mess, especially when you’re dealing with a scan of a fax. Tesseract in like three.

Dev: Yeah. I feel like at least. At least.

Vision Models Change Everything

Ian: Right. And so now, you flash forward So then we basically built an ensemble thing, where we take all these different technologies, kind of merge them all together as best we can, and then show it to a human, and then try to build a user interface for the human to just do their job as quickly as possible to get rid of all the stupid mistakes, and then now you’re now you’re cooking, and and you have a process. And you can, you know, but once again, nobody wants to pay a lot of money for this because it doesn’t have a that much economic value compared to all the work you just did. Uh at least not at smaller scales.

But now you flash forward and now you have vision models, right? There’s obviously chat GPT and Claude and all these other guys. You give them a PDF and they’re just like, “Yeah, here’s the markdown of it. It’s perfect.” Uh it just depends on how much you want to pay. Uh and then there’s also these smaller models like uh the ZAI company in China. There’s a GLM model that they have a vision model and I was just like blown away. I have this I have this uh path pathology report that’s in the public domain that’s terrible. I’m just like, “How did they even make this thing so bad?” Like it’s like a facts of a copy of a something and you barely can read it. I can’t read it. You give it to this this little model that basically runs for millions of a penny and it can it generates perfect markdown. So that problem is solved, right? That was a problem I had to deal with for, you know, weeks, maybe months. That’s just a solved problem and now you’ve got the thing where you’re saying, “Okay, great. Go take this text and turn this into a little graph, a little knowledge graph of triples and that strategy for me works the best. So if you’re trying to do this at home, convert it to triples and now you got Now you’ve got a little knowledge graph of that document, whatever it is, and now from there you’re off and and cooking.

And so then our basically our value add on that now is just the linking. How do I link that to those underlying ontologies for there’s these these all these coding systems in in the medical domain, ICD for diseases and SNOMED and and all these other things. So So once you’ve normalized and you’ve linked it to these underlying things, now you can actually understand what the patient is and then you have to think about it longitudinally cuz the notes are kind of a mess, too. You have to figure out, “Okay, I gave this patient this drug at this time and then later on I give them this other drug at this other time.” You have to infer, “Oh, when did I stop this other drug?” And why did I stop this other drug? Those types of problems still remain, but man, the large language models are really good at it where an NLP engine just broke down right off the bat, right? Because they could never tie together this event over here and this event over here having anything to do with each other. And so the large Go ahead.

Rip Out or Keep?

Dev: Yeah, in practice what I was going to say is like the system that you were describing kind of at the beginning, Tesseract OCR, you know, combination of different models that may or may not work that well on NER tasks, and then probably needing to be able to fine-tune them, which is maybe where you were using Ludwig’s compositional model interface or you know, needing to potentially multi-shot them. Like that sounds brittle and painful, right? And I think people forget like 4 or 5 years ago that was like the absolute state of the art for really what existed.

One thing I’m really curious about is from a business perspective though, 4 or 5 years ago that sort of pain was some of the technical value and differentiation in terms of like what you created, right? Like you had built the ensemble of like what rules need to apply on top of what models. Now it’s feed it to Opus or Sonnet and like get some of those same underlying results like earlier. I think a lot of questions that people ask is are companies that are doing this kind of work now more valuable because they are able to actually operate more quickly than they would have, you know, previously and they can therefore serve a larger market or less valuable because they had an expertise in a developed system that sort of got upended with the new transformation wave. And obviously you’d have a stake in this, but I’m curious like how you think about it having built in the space and probably made the decisions about which parts to rip out and which to keep.

Ian: Right. Yeah, I mean never wanted to be an NLP company, did it because we had to to win some deals and to and to help some clients actually get their data cleaned up so they can actually use the stuff that we’re actually good at. So from my perspective, I didn’t really ever want to be quote unquote an NLP company. So and we had to compete with folks that, you know, they weren’t necessarily tied to this domain. They didn’t have, you you they did NLP across domains and the just happened to be one that they put, you know, in a drop-down on a website. So, we were competing against them regardless.

What we want to do is help folks, help patients maximize the the value of the molecular information cuz to me that’s the big bet. The big bet to me is that healthcare we’re kind of like walking around kind of blind still on how to help patients. And the reason why we’re blind is cuz we don’t really know how it all fits together. We don’t really know how uh you know, we have AlphaFold fold which knows how to predict a folding mechanism, but it still doesn’t know why it’s folding that way, right? Like we have all these mutations and we can tell you that yeah, that BRAF mutation’s really bad and that and oh, now we figured out a drug to help that patient. But in the future we know we’re going to with AI’s help, we know we’re going to figure out what this genome means and and you know, all the other types of omics data means to really help patients and that’s what I’m very excited about. And NLP is just something I’m happy to just put behind me and not think about ever again.

I’m also a software developer, right? I’ve been coding since I was 13 and I love software, I love building software. I don’t care if I ever write code again either, right? Like that wasn’t why I’m doing this. My goal is to help people and and the to help people is we can build software. Software is just one means of doing that. And that software can be NLP and it can be other things. What we’re trying to do is help people with you know, terrible diseases, right? Terrible diseases live longer and be with their families. So, that’s always been kind of like the division of what I’m trying to do with GenomOncology.

Healthcare at Bureaucratic Speed

Dev: And I want to ask you one question unrelated to AI, but actually relates building technology in the healthcare space for a minute. I’ve always been curious. I think a lot of my friends that have spent time uh especially in like core machine learning roles or others, they look at healthcare as a domain and they see vast amounts of unstructured data. Um you know, really highly economical like high high economics basically in the end value of what gets created. Right? Like medical code billing uh sorry, medical coding for billing is like some of the classic use cases where there’s a large amount of breakage and dollars lost just because of like coding mistakes or other things that could have gone, you know, clinical oncology things have been one where like people have been looking at working radiologists for quite some time to be able to see like can we augment or otherwise replace some work. So I feel like technologists from the outside of healthcare looking at healthcare as a domain, not experts, often look at it and they’re like, “This is a great place to be able to apply machine learning AI.”

At the same time, some of my friends that I think have started in this space hit some of the same things you mentioned, which is it’s not quite move fast and break things in my you know, in the Silicon Valley parlance. It’s not like a fast it’s bureaucratic I think is the word that you mentioned. There are rules for very good reasons that get placed, but oftentimes those are a little bit you know, in culture orthogonal to the way that a lot of technologists like to operate, which I don’t know if this was you, but I feel like when I was running code it’d be like, “Does it compile? No, okay, try again.” It’s just like, you know, guess and check is like kind of like the outcome and that’s less possible to do here. What was what’s been your experience I think as you think about like technology in healthcare? Would you say like it’s overblown like this concern around like the bureaucracy and the slow down of tech? It’s or is it underblown? Like where have been your experience in the velocity of what you’ve been able

Ian: It’s probably it’s probably blown, right? It’s probably right right in the middle. I mean it’s I think it’s correctly priced. Uh so yeah, healthcare’s tough. Like don’t get into health like we’ve been doing this for 14 years. Uh we’ve got a very good business now. I’m very I’m very happy with you know, the the the level of business we’ve got. We’re growing healthily, but it’s not going to be we’re not going to achieve entropic numbers anytime soon, right? So um now of course we do have that challenge coming up and we can talk about that, right? So open AI and and Claude they’re going after, you know, open evidence, which is a you know, pseudo competitor to us.

Uh so the really the question becomes like yes, is healthcare slow and bureaucratic? Yes. It’s actually beneficial to us, right? Because we’re installed at a large portion of the academic medical centers. We’ve got some major reference labs and other institutions that are our clients. We’re directly integrated into their workflows, so we got their eyeballs and we got their process and we’re there to speed them up every day. Like, hey, can I shave a minute off of their work process every quarter? If I can keep doing that, they’ll keep me around, right? Like, that’s the goal. The goal is for them to save time, because time is money and time is helping patients. So, so there’s negatives to it, but but the positive is once you’re in and have a trusted reputation and a trusted brand and you don’t blow that brand, right? You want to keep that that that brand brand strong, uh, it’s a it’s a good business from that perspective.

Frontier Labs Come for Healthcare

Dev: Hm. And tell me a little bit now, I think on the same topic you brought up, which is like the Anthropic opening eyes to the world. Opening I will announce ChatGPT health, um, and then I think, you know, Open Evidence has taken a large market share in terms of like the physician usage of LLMs, specifically with like a domain adaptation then towards health care. Um, and I think you see this kind of same type of paradigm in other fields, too, right? In legal, you see leading companies like Legora, Harvey as like vertical applications, plus like you then have Anthropic like trying to take on more of like the legal market as well. What do you think will be the most difficult for these foundation model lab companies to get right in order to be able to build a large scalable health care AI business?

Ian: Um, yeah, they’re never going to grow the Anthropic business and the Open AI business at the the run rate that they have now if they focus on my business, right? My business isn’t going to grow They’re not going to get that that growth rate. So, I’m not worried about them coming directly after me. Uh, Open Evidence seems like a great company, right? They’ve got, um, lots of great adoption. They’re kind of like the iPhone model, right? iPhone was, oh, IT companies IT departments would ban iPhones, but people just brought them into the door and what what are they going to do? So, you got to start adopting it. So, now the they’ve got the physicians. Open AI’s trying to catch up. Anthropic’s trying to catch up to them. Uh I’m I’m sure they’ll be very successful. They’ve got all the agreements with the publishers, right? So, one of the main problems with uh publishing in our country and in the world is you we pay for the science and then the science gets trapped behind paywalls. Whatever, that’s that’s something I you know, so BioMCP my I can talk more about BioMCP later, but BioMCP does access all the publicly available information. So, we can kind of compete with Open Evidence there, but once again, they have the licenses to these to these uh published published papers. So, open Open AI and Claude and Anthropic will need to be able to get access to those as well.

And then the question really becomes where do the IT departments make their stand, right? These tools are so valuable that the IT departments are going to have to let some of it in. And so, the the bet I’m trying to make is okay, what I want to be able to do is one, be a good consultant to these to these groups, right? We have trust, we have brand, we have partnerships with folks. So, I want to help them solve these problems, right? Come to GenomOncology, we will help you figure out we’re we’re vendor-agnostic. I’ll work with Open Evidence, I’ll work with Open AI or Claude. We work with all the major labs, we work with all the major device makers. We That’s not our our goal. Our goal is to help you make your organization work.

And so, what you need to be able to do is take publicly available information. You need to take these large language models, whether they’re the big frontier models because you your your use case demands it or oh no, this is a repetitive process, right? This is a prior authorization or this is finding a trial or a drug or whatever. And really the most important thing there is access to the right information and a really small model, like a Gemma 4 fine-tuned on on your that specific thing or even the Med-Gemini models are pretty good. Um you know, you can get that that small model running it locally pretty cost-effectively, tying it to tools, right, in an agentic loop, it can now do that that task, that job to be done. Given the inputs, do the job to be done. And it’s better than what our system is, right? From a from a flexibility perspective.

Our system’s great because I can not only is it uh explainable, I can tell you why I’m giving you the answer, and repeatable, I give you the same answer every single time, no matter how many times you ask me, you’re going to get the same answer. An LLM will won’t do that. No, even if you set the temperature to zero, that doesn’t You’re going to You’re going to get a different answer every time. But what you need to be able to do is figure out how do I tie this with tools so I can get that explainability, get that repeatability, and then how do I do it on prem so I’m protecting my data, I’m I’m protecting my patient information, HIPAA compliant, and and how do I integrate it with all my systems, right? EHR, LIMS, CTMS, whatever these different acronyms are that I’m sorry that I’m dropping on you guys. Uh that information is core to these businesses for getting their jobs done. And and we, you know, we love helping our clients do that stuff.

Agents, Tools, and BioMCP

Dev: Help me understand I know that you just talked a little bit about being able to connect some of these models to some of our some of the smaller variants towards tools. That’s often times how we define agents. Like in, you know, you ask four different people an agent definition, you’ll get six definitions. I often just think about like agents now as models with access tools so they can go ahead and like take an action. It’s not just purely like generation or synthesis, it might actually be making like a function call at the other some parameterization. Like I’m curious like A, how do you define agents? And then B, um where have you seen the greatest lift from agentic workflows? Cuz part of what you were doing, you know, here is basically a better faster mouse trap. Like you had certain versions of models probably starting in 2016, 2017, and now you can do that work with much less training, you know, with the with the latest like the large language models today. But that introduction of like tool use for models is different. So where have you seen the biggest lift from models accessing tools in healthcare?

Ian: Yeah, it’s I mean it’s coming, right? So I’ve been talking about agents for a for a while. Everyone has, obviously. And and I’ve been trying to figure out how do I blend these things with my tool set uh from the beginning. And so our system is API first. So I built all these systems, they’re all they all have APIs to them. All the user interfaces all sitting on top of the same APIs. Uh so when GPT plugins came out in April of 2023, I was using those. I was using custom GPTs which could plug into your open open API swagger based uh API. I was using those in custom GPTs. And then when MCP came out in 2025 now? I don’t even 2024, sorry. Time’s evading me. End of 2024, MCPs came out. I was kind of like, “Eh, I don’t know if I want to use these.” Just cuz I’ve been been through that rodeo. But then you realize, “Oh wait, Sonnet 3.5 actually works with this thing. Like it can actually know how to call the tool.” And that was kind of the great unlock.

So that’s when I invented the Bio MCP open source project. So Bio MCP is basically a connector to all these open source data sets from PubMed articles, all the variant information, genes, diseases, drugs, pathways, all the biomedical information. If there’s something that’s out there that I haven’t added, send me an email, I’ll add it to it. So I invented Bio MCP. At first I thought it was an MCP model context protocol. And I was like, “Okay, this is going to be the kitchen sink model context protocol.” And I basically added 35 tools. And then I realized, “Oh no, this like loads up your context and slows you down.” And context right now is the most important thing. Like you got to have the smallest context possible while giving access to the tools. So I actually refactored the whole thing recently and now it’s a CLI first, so you can plug it into Claude code and and it’s still MCP, but it’s just one of the CLI calls to start up the MCP. And that works great. So, if you’re using cloud desktop, use MCP. If you’re using cloud code or using codex, use the CLI version. Or if you’re using Pi, cuz Pi is actually the best coding agent, uh you plug it in with Pi and it works great, too.

So, okay. So, what’s an agent? An agent is, as you said, an LLM in a loop calling tools with a goal, right? And a set of skills now. So, you give it a set of skills and you give it a goal and you give it a set of tools. And so, the abstraction layer that I’m working through is So, I have my coding agent, which helps me with all my projects that I’m working on, and it has those set of skills. I have a marketing agent, right, that gives has me helps me make slides and stuff. And it’s got its own set of skills and its own set of tools. And now I’m building biomedical agents as well. And starting with here’s a set of here’s one set of skills for doing trial matching, right? Helping a pa- given a patient, how do I find a trial for them? Or given a trial in a database of patients, go find me patients that might be eligible for that for that trial. So, that agent, right, has is encapsulated with a set of skills, right? The skills tell you how to use the tools effectively. And the tools are, you know, tools for accessing bio MCP or accessing our knowledge base, really, right? Locally for trials and therapies and and things like that. Tools for how to understand biomedical information, right? Molecular data, accessing clinical data from an EHR. So, we have a tool that can talk fire, uh which is a biomedical or it’s a medical standard for talking to EHRs and other systems.

And so, that agent is really the encapsulation of a subset of tools, cuz you don’t have you don’t want to give it everything. You want that agent to be focused on that task or set of tasks. And and so, you’re minimizing the amount of context that they have to like devote to getting up and running. But the awesome thing is you can give them a task and then they can figure out it’s not a workflow. So, one thing that some people keep going back to is, “Oh, it’s got to call this step and then call this thing and then call that thing. That’s not what you’re supposed to do with agents. If you want to make a workflow, make a workflow and you can use an LLM in that sense, right? Where basically you’re calling one system, putting it into a LLM and getting the results out or maybe you’re doing rag where you’re basically giving it a database and and it’s just a question answer machine. That’s not the power of an agent. The power of an agent is I have a task to do. It’s not clearly defined in that I don’t know the actual steps we’re going to take ahead of time to actually solve the problem. And then it’s going to call tools and a tool in this case is for the most part means it’s finding some information, searching or getting some information and then taking some action. And that action just might be write a report, right? Here’s the report. I want to give my patient this test or I want to give my patient this drug. Please create an authorization letter. So I can give it to the insurance company so the insurance company can say, oh yes, that drug or that test makes sense for this patient. Please go ahead and do that.

Code Goes to Zero

Dev: What’s an agent use case that you’re excited for somebody to build with BioMCP that doesn’t exist yet?

Ian: The one I don’t know about. So I’ve made BioMCP a marketing device, right? So in my opinion, all code goes to zero. Sorry to the coders out there. The value of code itself by itself is zero. The value you add as a software developer is that you stand behind the code, you can help drive the code and that you understand how it all fits together and that you’re solving a domain problem. So code goes to zero and public information goes to zero. If it’s out there on the web, Claude will go ahead and build a tool that knows how to do curl and grab it from the web and and then answer it. The reason why BioMCP exists, and this is what the paper is going to say, the reason why you use BioMCP is because it’s going to use less tokens and it’s going to do it faster and it’s going to be more reliable and it’s always going to it’s going to give you more consistent results because it’s using a known set of of of known resources. So you use BioMCQ rather than letting Claude go figure out, “Oh, I found clinicaltrials.gov and now I’m going to whip up a clinicaltrials.gov client right in Python.” Yes. Well, why do we need to do that a thousand times a day, right? So, just use BioMCQ. It can do all the searching on clinicaltrials.gov and it can find those fields that Claude’s going to skip over.

So, all the public information though is basically worthless effectively from a business perspective. It’s great for a patient perspective. There’s lots of value that’s going to be unlocked for patients and doctors and stuff like that. But from a software company, I’m not going to make any money from these public sites. So, that’s why I’ve given these things away for free. But what that unlocks is latent demand, right? Okay, I’ve used all these things. Hey Ian, you made BioMCQ. It’s great, but now we want to use it with my private data or I want to use it with my EHR or I want to use it with whatever or your knowledge base that we’ve hand curated. That’s the unlock. That to me is the the the use case that I want to see. Uh and really what I want to see is, you know, how can we either if it’s public, right? I’ll add it to BioMCQ. Or if it’s private, let me help you do agents at your organization, agents that you can trust, agents that have repeatable answers and are explainable and all that other stuff. So, that’s the goal for me. And then really honestly, the end goal is uh I mean I want to make money. I want our company to be successful. I want to help people. Like I just want our company to be a a a source of good in the world and help uh help in this fight against cancer.

Labor Displacement and Caregiving

Dev: On that idea, I know that this is uh a really popular direction people like to take things when it comes towards agents, but I’m curious about your opinion on like labor displacement specifically in healthcare. Cuz if you say like code goes to zero and we say public information goes to zero, it’s hard for me to listen towards like what BioMCQ could enable plus like the rest of, you know, the prior authorization use case there that you kind of mentioned, you know, medical coding as an example. There’s lots of different I think inefficiencies that also probably mean jobs today. Like these are things that have to be manually done. Um and I think you’re seeing this kind of disruption in a lot of different markets. If you were to take a medium-term view and I’ll think I’ll take think of the medium-term as like over the next 3 to 5 years. What are the domains within health care that you think will have the most staying power through AI? And are there some one or two areas that you think, oh, in 3 to 4 years like we’re going to see 20% employment of what it is today?

Ian: I mean it’s it’s caregiving, right? That’s what health care is. People go into health care to be caregivers or to do science. Like there’s a whole branch of health care that they don’t talk to patients, right? They’re You know, I’d be that person. I’d be going over there Let me not help the patient directly. Let me help the person that’s helping the patient. And to me there’s it’s infinite demand. There’s infinite work that needs to be done to help patients, right? Cancer is There’s There’s been amazing progress on the battle against certain types of cancers, less against others, right? Pancreatic cancer has been a terrible disease. My grandfather passed away from it. It’s basically incurable. It seems like it’s it’s extremely aggressive. It’s not very easy to treat, right? With pancreatic cancer.

Dev: Yep.

Ian: And there’s you know, one set of genes. There’s the KRAS gene for instance has been notoriously undruggable. And there’s a recent drug that is making a lot of headlines, right? Doubled doubled the overall survival length, right? From 6 months to 13 months or whatever it is from a median survival. There’s former Senator Ben Sasse that was on 60 minutes this week talking about his experience. And it’s just been great for him, right? It’s decreased his pain. And it’s adding months to to to his life that’s been greatly shortened. He’s He’s basically said I would be dead by now if it wasn’t for this drug. So, that is That’s why we’re here, right? And I think that there’s infinite work there.

Now, is there going to be displacement? Yes. I think I think that just in general for AI, I think everybody I guess developers are kind of here first cuz we’ve seen how these models work, how they can actually improve your work. But the other thing is developers also are very good at software development. They’re not necessarily good at economics and how diffusion works and I think it’s going to take time, right? So I think I think these models are going to make us more productive. I think we’ll have time to react to it. I think that there’s always unfortunately always ups and downs with the economy and that plus AI plus a plus an election, all those things will probably mean that there’ll be some job displacement and then it’ll get blamed on AI. I think in healthcare it’s one area where I I have no compunction. Like I’m very excited about working on AI for healthcare cuz I’m helping people directly. Will Will somebody who’s doing medical coding right now be out of a job in 6 years? Maybe, but from what I understand, they can’t hire new medical coders. Like for whatever reason, I think that’s actually a job that’s like has aged quite a bunch. Like they used to be 30-year-olds, now it’s 50-year-olds. And so getting new coders, I don’t even know if that’s like a job that people would necessarily want. I don’t think it’s super exciting to do, to be honest. So I think that that’s a a great job to actually try to automate. And how can that same How can that same dollar and that same person be used in a way that helps patients more directly? And I I think AI can help us there.

Overhyped, Underhyped, and the Jevons Paradox

Dev: Ian, you know, before I started Predibase, which was an infrastructure company, one of our investors asked me, “What do I think is an overhyped and underhyped area in AI infrastructure?” Now, I’m going to ask you the same thing in healthcare. Like in healthcare AI, there are so many different types of applications towards AI as well as like tooling layer, like types of applications. What do you think are Like what do you think is one place in healthcare AI that you think is overhyped and one place that you think is underhyped? If you were to put almost like an investing hat on.

Ian: Oh, man. Um I think that that’s I don’t know if I have a good answer for that. I don’t know if I have I don’t have a good answer for that one. Let me think. Give me a minute here, Dev.

Dev: No problem. I thought it was really challenging. While you think, I’ll actually just give you a little sense of like how I thought about it in the past. I thought one thing that was overhyped at the time was these like wizzy wig low code like no code I should say no code model builders. You know, this all the rage in 2019 2020. I thought they were a little bit overhyped because it’s hard to figure out how to be able to plug these into your actual workflow once you had a UI kind of build your model. It’s like did you know how to be able to use it downstream. An area I thought I was that was underhyped in 2019 was data labeling and data infrastructure. I thought it was an unsexy beast that every organization had to do to make a machine learning that actually worked and I think it’s, you know, consistently is true with AI today. I will say you don’t need to give me like necessarily one of both, but I’m curious if either on the underhyped or overhyped side you have a tick on something that you think is getting more attention than it should or not as much attention as it deserves.

Ian: Yeah, so there’s a couple different areas. So I think the one so we’re in this business of clinical trial screening. Right? So and there’s I’ve seen at least 20 papers if not more and there’s a lot of companies that are doing clinical trial screening similar to what we do. And the the idea that people have is oh, what we can do is we just read the the LM can read the clinical trial and then it can read the patient and basically say can this patient go on this trial? And the main way they do that is or one of the strategies I see is take this document the eligibility criteria which is still unstructured which is annoying. Like like hopefully the and you know, NCI or the clinical trials website maybe we could I don’t know maybe we can help some. If someone is there that needs our help, let us know. Instead of collecting the information in an unstructured format and then turning it into a structured set of questions that you can then ask. That’s kind of the strategy people do and there’s a lot of papers, there’s a lot of you know, there’s a lot of people with PhDs looking for papers to write. So they write a paper about this. And the the and which is nice, right? So hey, you take this trial and you ask the patient can this patient go on this trial? It’s just missing a few few things. Well, one if you have a thousand patients and a hundred trials, you’re going to do that a hundred thousand times. It’s going to be cost cost ineffective.

The way our system works is we kind of read the trial, codify that to our knowledge graph, and now and we then we take all the patients and we codify them to our knowledge graph as well. And then what you’re doing is now we’re you’ve turned that LLM problem into a search problem. And then and then now you’re doing it at sub-millisecond queries, right? Like, okay, go here this patient, go find all these trials and bring them in. The other thing you’re overlooking is the fact that there’s this data problem. The data problem is you have to be integrated with a CTMS cuz the clinical trial management system, cuz you don’t actually know what the trial’s recruiting status is because clinicaltrials.gov doesn’t have that. Um so there’s all these other technical problems as far as why uh matching isn’t just this pure you know, easy button thing with LLMs. So that’s one thing.

Uh the other thing is I get investors and folks that are investment in in GenomOncology sending me things like, “Hey, here’s this new test or this new modality or this new thing.” And it’s like, “Great uh great, you know, blog article, great paper. That thing’s going to take a while for it to get FDA approved. And once it is FDA approved, it’s already going to fit into our model, which is, you know, you’ve done this test, you’ve got this new way of recognizing a biomarker. Well, the biomarker is a biomarker is a biomarker in my system. We then just take that and then use it downstream.

Um so those are just two areas. And I guess the last one that that’s jumping out at me is the radiology comment from Geoffrey Hinton, right? Everyone thinks about this Everyone talks about this, but 10 years ago or so, Geoffrey Hinton’s like, “Don’t let your babies grow up to be radiologists,” right? Because there’s not going to be any radiologists. And the the problem is now there’s more demand for radiologists. And that’s just showing you, right, that someone as brilliant as Geoffrey Hinton, who’s so smart about AI, uh doesn’t understand necessarily that, okay, what we’re doing is we’re taking this person’s job, right? A radiologist. And they’re were at one narrow part of it where they’re looking at the pictures and trying to find out where the cancer is, and then extrapolating, oh, the AI is going to do that. That means their job goes away. Well, no, there’s so much other parts of the job from from, you know, before and after that one task that now that one task that was very cost costly now goes to zero, and now the demand for their services go way up because we can do it in a more cost-effective way. So, it’s the Jevons paradox brought to brought to healthcare. Um so, I think that that’s still something very relevant to to the medical industry. You know, I actually think that the same thing is actually going to happen in a few other disciplines. I know engineering we’ve all been talking about like the decline of software engineering. I think software engineering will change, but I actually think the aggregate number of roles that are available for people will grow. Um cuz I think now the same kind of builder mentality, if you if you were a good engineer, that’s kind of like one of the core things you wanted to do is you wanted to problem solve using a set of tools. It’s not like because you went from like low-level abstractions and like writing in ASCII or C to like, you know, higher-level programming languages, that the demand for this role uh declined. Then I think you’re going to actually see the same thing with some of these um coding agents as well. But, it’s, you know, great

Dev: Agreed. I think what’s happening in um in kind of the healthcare side of it with respect to AI as well. You know, I really want to thank you so much for uh you know, coming onto the show today and being able to walk us through both what it took in order to be able to build a company in a healthcare AI pre-generative AI. I’d say through the deep learning wave that happened in the late to uh 2017 to 2020 era. Then, of course, with the advent of like, you know, these generative pre-trained transform models and everything else that came through. It’s fascinating to hear your um perspective and also to hear, candidly, you want to make money, but you want to help people. That’s like a huge uh you know, motivation for why you’re in this game. I’m going to check out the BioNLP project in more detail after this, and I can’t wait to revisit with you in 3 to 6 months and actually find out what was one of the best things you’ve seen someone build with the tool. So, thank you so much for I think uh joining us here at the conversation today, and really look forward to seeing where the rest of the future goes with health care and AI.

Ian: Thanks, Jeff. Thanks, Jeff.

BioMCP: Biomedical Model Context Protocol

Thu, 23 Oct 2025 00:00:00 GMT

Timestamps refer to the original recording, not the published cut. They will be rebased against the YouTube video once it is posted.

Introduction: GenomOncology and precision oncology

[00:07:07] but let’s get into it. so, i’m ian. uh, i lead a company called GenomOncology. we’ve been around for about 13 years. uh, my colleague annab [00:07:18] is actually on the call, too. she’s a senior developer that’s actually now working on BioMCP pretty much full-time. so, you’ll see her name a lot in the github repo, which is great. um and then you know what we do is really we [00:07:29] generate molecular reports right so about 70, 000 this year and so those reports have knowledge on it up until now we’ve been doing stuff the old school way what they call good oldfashioned ai with knowledge graphs [00:07:40] and you know basically generating reports that say you know what are the variants what do they mean what are the uh clinical trials and therapies etc and our software is actually onrem we don’t take data rights we don’t do any of that [00:07:51] kind of weird trading of data for software uh and you know uh and obviously large language anguage models are a part of our strategy long term and i can talk about that at the end. so in [00:08:01] general what’s why is this important right now? uh large language models, they’re generative ai. they’re next token predictors, right? and i’m going to get into like, you know, next token prediction versus pre-training versus [00:08:13] post-training versus reasoning. um but basically, you know, especially in a complex domain like precision oncology, which is my focus, right? and i’m just going to say precision oncology. um is [00:08:24] that they’re they’re good, but they’re just making up what sounds like a good answer, right? and it’s pretty good, but it’s better when they use a tool. and this paper from june uh in nature talked [00:08:35] about you know integrating gpt4 which is actually like an old dumb model at this point. um integrating it with a a tool set just greatly improves its accuracy and we’re seeing that across the board [00:08:46] and and i’m going to talk about coding tools as well. you might have heard of claude code. i’m sure people have heard of cursor. uh the new coding agents that are coming out and there’s actually literally one every week that comes out. [00:08:56] uh the coding agents are actually just a really powerful paradigm for how to do software development and i’m actually seeing exactly how that’s going to translate to my work and my work is you [00:09:06] know structuring unstructured data uh you know using that data to then you know drive decision support drive trial enablement and a bunch of other downstream use cases for our clients and you know i think that that clawed code [00:09:18] kind of you know agent in a loop or llm in a loop calling tools i think has a lot of legs and u and BioMCP is is kind of my onboarding uh tool set for people [00:09:29] to kind of get get their feet wet, but then to do this stuff seriously, we’re going to we’re going to need a new set of tools, and i’ll explain what i mean by that later. so, this is why i’m excited, right? llms plus tools. uh and [00:09:40] i’ll talk more about my history with like my attempts at this uh over the last three years. and so, i’m going to go over pre-training, then post-training. i’m gonna, you know, do a deep dive on model context protocol, [00:09:51] what it is, what BioMCP is. um, and then coding agents and biomedical agents kind of in this kind of this history like my errors tour effectively of of large

The pre-training era

[00:10:01] language models generative ai. so the pre-training error, right? um, this this happened basically three years ago. why did it happen now, not 10 years ago? it’s really because of these four [00:10:11] things, right? we have the scaled compute. we got the gpus from nvidia. we’ve got the data. we’ve been collecting data on the internet for, you know, a couple decades at this point. the transformer architecture came out in [00:10:21] 2017 or so. and then the self-supervised learning uh modality, which i’ll explain in a minute, uh really unlocked the ability to train these models at scale. uh and then people just saw the pattern [00:10:32] and then they threw a bunch of compute and a bunch of dates data at at compute and got you know first it was gpt 3. 5 and then four and then we were off to the races. so you know transformer is [00:10:44] basically the technical architecture that underpins all of this. it’s a neural net architecture deep learning architecture. i’m not an expert at this. i do understand the difference between an encoder and a decoder, right? i i was [00:10:56] using encoders a few years ago to do classification uh you know named entity recognition and basically the way that you train an an encoder as you do this masking right [00:11:07] you’re basically hiding words and then you ask the during the training exercise you’re asking the model to kind of predict the the missing word and then based on whether it gets it right or wrong it you know changes the the the [00:11:19] model’s weights with gradient descent decoders are basically on the other end the generative part and they they are basically trained using next token prediction and all that really means is [00:11:29] you know if you have the the lazy dog and you know you ask it to predict the next word it’s going to predict jumped right or whatever and if it gets it right it gets you know it gets a point it gets it wrong get loses a point [00:11:39] that’s not technically how it works but you get the gist um and the basic deal is they basically predict the next token for every word on the internet or every yeah every token on the internet and at [00:11:51] the end you have this pre-trained model using using uh the using the internet’s uh corpus and what that unlocks this just this capability of predicting the [00:12:01] next word has just unlocked a a crazy amount of different capabilities right there’s you know obviously you see it with like translation and in context learning and and that just allows you to [00:12:13] then do uh prompt based tasks right before when you wanted to make a mo like to actually do something with ai quote unquote or machine learning you had to get a bunch of data you had to label [00:12:24] that data and then you train the model and then that model did that thing really well. with large language models, you kind of don’t need to do that, right? you get a smart enough model and you can write a good enough prompt and there’s other tools as well to like tune [00:12:36] the what’s called tuning the prompt. you eventually you get to a point where then that model is basically doing any task for you. you just need to uh figure out a way to uh point it at a bunch of data and then have a way of improving it [00:12:48] improving that prompt and now you have a you know a task specific um pipeline and so one of those prompting techniques that was figured out early on and and this is probably even before chat gpt [00:12:59] was chain of thought prompting right so chain of thought prompting is the idea that if you just say uh i have a question and give me an answer and even if you show an example right in this [00:13:09] example we’re do we’re actually showing an example where we say here’s here’s the question, here’s the answer, another question, here’s another answer. and now, obviously, this is with older, much smaller models, it might get it wrong. [00:13:20] now, we wouldn’t get it wrong. um, but but what they figured out was if you give the model the ability to kind of like think it through, quote unquote, think it through, like explain itself, explain its logic, uh it just does a [00:13:31] much better job of like of of answering the question. and the reason why is the same reason why when we solve a hard harder problem is we break it down into smaller problems, right? that’s how you do it. you you you take big hard [00:13:42] problems, break them down into smaller problems. that’s what i do all day making software, right? i don’t try to write the whole uh big giant software at once. i try to figure out what’s the smallest component that i could that i know how to build and then i go build [00:13:53] that and then i assemble them together. uh and then another technique is called fshot prompting. and that that idea is you’re giving it examples, right? so you can either say zero shot, which means you don’t give it an example, you just [00:14:04] ask it for an answer. and then there’s one shot where it’s like here’s an example. uh now give me give me an answer to my question and then there’s fshot where you’re giving it multiple examples once again this is just you [00:14:14] know basic what they call prompt engineering and there’s tons of uh articles and and documentation online on that and that’s the idea of like how do i take this how do i take this chatbot [00:14:25] and make it do a specific task for me um and so there’s lots of you know lots of great uh prompting techniques that are available great for your everyday use with a chatbot that you you know whether [00:14:35] it’s chatgpt or claude or whatever one you prefer uh and i highly recommend you you you kind of learn this as a skill. like it’s it’s definitely a skill for for 2025 that’s still relevant. people say that it’s prompt engineering’s out. [00:14:46] no, you still need to know how these large language models quote unquote think. once again, sorry for the anthropomorphizing. you need to learn how they think so that you can use them effectively and get get the results that you’re looking for. um, another

Prompting techniques and RAG

[00:14:58] technique that folks talk about is rag. so that stands for retrieval augmented generation. this is a great paper that you know was specific to the the medical domain where they showed uh how you take [00:15:08] a question, you take that question, you turn that into either a semantic query or a keyword query or you com combine them a hybrid query and you retrieve the [00:15:18] the relevant information and then you put that relevant information right in the llm and and as an output and this was really important in the days when the context windows were like 4, 000 [00:15:29] words or 8, 000 words and now it’s you know you 750, 000 words or what have you for some of these bigger models. and what that allows you to do is just basically stick the whole pdf in something. before you were basically [00:15:40] limited to like 10 or 15 pages of a pdf. and so what people would do is slice things up and and try to feed it to the lm. and the reason why you do this is because the large language model just [00:15:50] does way better when it’s been kind of grounded in some in some text. and there’s definitely some interesting theories online on like why this is. it’s but it’s basically, you know, kind of shaping, you know, if you think of [00:16:02] what these these models are, these uh just there just an infinite number of of combinations that can happen with these these models and the weights. what you’re basically doing with by putting [00:16:13] stuff in the context is you’re kind of shaping those weights so that the next tokens that they’re predicting are gerine to the content that was provided. it’s just just how they work. and and once again, a lot of people don’t really [00:16:24] even know really or i say lots of people, we don’t really know how they work, right? like we have guesses. there’s mechanistic interpretability. people are, you know, tweaking these things all the time to to show that they [00:16:34] can break them or show that they can make them think that they’re the golden gate bridge. there’s there’s lots of weird things that you see online, but the basic gist is sticking stuff in the context, right? putting stuff in the [00:16:45] chat window itself just makes these models way better at answering questions. and that’s what rag is doing. it’s doing it in an automated way where you’re asking a question, retrieving content, putting in the llm, and then [00:16:55] spitting out the answer. and then over here you even see the calculator. so the calculator is giving it tools. like this was an early version of someone giving something tools in this case like the med md calc, which is a really great [00:17:05] website uh where they’re kind of integrating it all together. and then there’s the idea of fine-tuning, which has kind of gone in and out of fashion. uh it’s kind of coming back a little bit now that uh reinforcement learning is is a big deal. [00:17:17] and i’ll talk about what reinforcement learning is, but this is the idea of taking a large language model. and this is really, you know, what people were hoping would work really well a couple years ago with small models like these small models that came out called llama [00:17:27] from meta and and a few others. mistral was a a group company out of france that had some small models that people were fine-tuning on. and the idea of fine-tuning is you take the base model and then you give it examples, maybe a [00:17:39] thousand if you have them. and you can even create those examples with a big smart model like gpt4 at the time. and you would basically have inputs and outputs and then you’d fine-tune the model. and then what happens is that model just gets better at your type of [00:17:50] thing. and it was really good for things like format, like if you wanted it to generate a certain kind of json output, man, it was really good at like making sure that it generated the right fields and all that stuff. and it just cut down [00:18:00] on the amount of context you needed to provide to the model uh so that you could save your space for other other stuff that you were actually working on. this kind of phased out because the the context windows got bigger. so people [00:18:11] didn’t need this as much and the models just got smarter and and and there’s there’s always this tension where the models just keep getting smarter and smarter that you that you’re kind of like throwing away this work. so that’s why i think that they it it fell out of [00:18:23] favor. it it might come back with reinforcement learning. uh we’ll see. but as long as there’s kind of continuous improvement of the models, i’m actually down a little bit on on fine-tuning as being an approach. i’d [00:18:33] focus more my efforts on, you know, creating the the scaffolding and the structure uh that actually let your uh lets the kind of the big bad models actually do their work. um and then this [00:18:47] was all premised on this idea of the this paper called chinchilla paper, but it’s called the scaling laws. and there’s a great book called the scaling laws by um uh sorry this podcaster guy, [00:18:58] his name evades me, but there’s a scaling uh the scaling era is the name of the book. so you can google that. um, but basically it talks about the idea that there’s this magic triangle of [00:19:08] flops, the amount of compute, right? the number of gpus that you’re throwing at something and the the number of parameters in your model, like how s how the big the model is, right? is it a 7b [00:19:18] model? is it a 100b model? is it a one trillionb model? and the number of tokens. uh, and before there were, you know, new things like mixture of experts and and the reinforcement reasoning [00:19:30] stuff. this was basically the iron triangle that everything was stuck to where you were basically trying to figure out what’s the the maximum or the ideal number of tokens to train a model with the given flops and given [00:19:41] parameters. and it just showed that if they could just throw more data at more compute and have a bigger model, you’ll just get smarter results. that’s basically what happens. and that continues to be true. the problem is [00:19:52] there’s diminishing returns just due to the cost, right? just and because there might be, you know, shortage of data as well. and so pre-training kind of hit a wall. so about a year ago, a little bit [00:20:03] over a year ago at this point, i was actually a little bit down. i was like, well, i think this llm thing might have just peaked. and that’s okay. like i think i can do a lot of good with what we’ve got, but doesn’t seem to be

Post-training and reasoning models

[00:20:13] getting better in 2024 seemed like kind of a down year. and then uh then basically what happened is the post-training era started. and it started with a model called 01 from uh [00:20:23] gpt from the um from openai and then deepseek happened in like december or january and everyone kind of freaked out. so what is this thing? what what [00:20:33] are we talking about? so these are reasoning models and what is a reasoning model? basically the chain of thought trick from earlier where you ask the model to explain itself and then you [00:20:43] take those those trace what they’re called traces of chain of thought and then you use those to basically fine-tune the model to post-train the model and what’s happening is they’re using math and they’re using code and [00:20:54] the reason why is because there there’s a lot of verifiable data sets that are out there and they can obviously you know verify math uh separately so you can basically generate a lot of training data and code obviously can run and and [00:21:06] pass unit tests. and by using those types of examples, uh we can then figure out what chain of thought leads to the best answer. and they do other tricks like, okay, yeah, these these two traces [00:21:18] both ended up with the right math answer at the end, but this one did it in like half the steps. so, let’s choose that one because if they’re if they’re thinking it through and they’re thinking it through more efficiently, that’d be [00:21:29] better for the model uh to be more efficient. so, that’s the basic process, right? you’re basically collecting these chain of thoughts and then you’re fine-tuning the model to do that step and they’re actually making it separate [00:21:39] steps. they actually have a thinking step and then they have the output step. and some of the models show you what the thinking is and some of the the open source models uh definitely show you that the thinking process. some of them show you like a summary of the thinking [00:21:50] process. but it’s that thinking process that actually just got us a big boost uh in performance. and there’s a you know a deepseek uh paper that so basically what happened is uh 20 2024 there’s a bunch [00:22:02] of vague posting and other things online where people were kind of talking about strawberry was the was the keyword that opening i was throwing about throwing around and sam alman got fired for a [00:22:12] minute and that was all about this 01 model that came out in september october and 01 was was kind of cool and kind of impressive. i i played with it and got access to 01 pro and i saw that that was [00:22:24] definitely a big leap and then deepseeek came out and the the great thing about deepseek it’s a model from uh china open source is that they kind of explained how it all works right like they they used a technique called grpo for doing [00:22:37] their uh fine-tuning but they basically opened up the recipe and then all the other labs basically copy the recipe and so everybody has a reasoning model now and so the reasoning [00:22:47] models you know just broke through right so gp gpt40 was kind of the best pre-train only model and then you could see here even 01 was was jumping right right ahead of gpt40 and then since that [00:22:58] time we’ve had 03 and we’ve had now gpt5 which is kind of a an amalgamation model of of all these things and and everything’s just kind of gotten way better um uh since that time period. so [00:23:09] now and and we’re talking about 2025, right? so basically on the right is all the pre the non-reasoning models. you know, i put the gpt 3. 5 in there. and these are this is artificial analysis. [00:23:21] so, take these with a grain of salt. these are all just what are called benchmarks if you’re not familiar with this. the benchmarks are are kind of like an iq test for uh llms, right? and you can see that the that they’re, you [00:23:33] know, they’re not perfect, but they’re pretty good indicative. they’re pretty indicative of like the quality of the model. um, and so you’ll see here that, you know, the the non-reasoning models hit a limit, and over on the left is all [00:23:44] the reasoning models, and it’s just kind of going up. and i’m showing just the openai ones because, you know, you can kind of see the over time. but there’s other models from the other providers. so there’s gpt5, grock, which is the, [00:23:55] you know, the twitter guys, uh, model, and then claude 4. 5 and gemini 2. 5. uh, and you, and once again, i kept the 40 and 35 in there just for context. [00:24:06] and so the other big benefit though of these reasoning models is they’re actually making them smaller. now the the labs don’t necessarily tell us how big these models are, but we can tell that they’re that they’re smaller because they’re faster and they’re [00:24:17] cheaper, right? those are the two things that the the the so when you you train a model and then you run the model or in what’s called inference of the model. inference is the actual act of chatting [00:24:28] with it. and one of the big downplays on llms is that when you’re chatting with these things, they don’t actually learn right right then. you have to actually collect the data and then retrain uh later. so they are collecting the data, [00:24:38] they are retraining. so, make sure you know who what models you’re using and what the agreements are with that. so, you you know, make sure you’re aware of that. um, some some companies are better at uh disclosing what they do and some [00:24:49] are worse. uh, but you can see here that the prices are just getting way cheaper. so, gpt4 here, this is just the most expensive one. this is the old model and the reason why it was so expensive is because it was more expensive for them [00:25:00] to run, right? they want to make these things as cheap as possible because they’re in competition with each other. and gpt5, right? this is a really good model. you can see here that the it’s just way cheaper. now, you’re going to [00:25:11] pay a little bit more than what this says because there’s the reasoning tokens as well. so, you know, as it, you know, when it’s doing that step where it’s explaining the math to itself to then give you the answer, those are [00:25:22] called reasoning tokens. those are priced i think just the same as regular tokens. but you can see that the prices are getting better and the speed is getting better. right? once again, o you know gpt4 compared to gpt5, right? the [00:25:34] speed is just completely uh different. and that once again, it’s just because the models are smaller and the fact that they’re smaller and don’t lose any smarts, if not gain smarts, that to me [00:25:44] is the the the major light bulb as to why 2025 here has been a a really successful year as far as getting getting better around these tools. and then this is kind of what they call the [00:25:55] paro um frontier of like what the best models are from a price perspective. and these guys are just jumping each other all the time, right? anthropic is uh they’re kind of like the artisans of of the llm [00:26:06] world. you know, people people pay a little bit more for them. uh and maybe their models don’t necessarily get represented by the intelligence as well as how they perform in the field. um and they always just charge a little bit [00:26:17] more. uh gemini, which is the the google model, and openai’s models, they kind of just leaprog each other. so, we’re expecting gemini 3 to come out and it’ll be, you know, up here uh in the next [00:26:28] couple weeks. so that’s kind of just some background of like where we are from a pre-training and post-training and and it’s just critical to make sure everybody understands that stuff at least at a [00:26:39] high level to understand that there’s this thing called reasoning models and the reasoning models are good at what’s called tool use and and if you and i know that this is a fact they weren’t good at it two or three years ago [00:26:49] because i’ve been i’ve been trying this stuff myself. um, and i’ve been excited about the idea of tool use since the beginning when chat gpd came out because this paper was was pretty early in llm

Knowledge graphs meet language models

[00:26:59] days. and it’s basically talking about knowledge graphs, which is what my company has, and large language models and how they complement each other. and you can see it right here in the pros and cons, right? where, you know, [00:27:10] knowledge graphs are great. like you, a knowledge graph, if you’re not familiar, is the idea that you have these nodes and they’re connected to each other through relationships. so in our system we have diseases and drugs and genes and biomarkers and trials and therapies and [00:27:22] eligibility criteria that tie them all together with complex boolean logic. right? that’s our knowledge graph. and i can actually run a what’s we call an algorithm we call match where we say go ahead and take a patient the patient’s [00:27:33] the query match them to trials or therapies. and that bas that algorithm basically takes the patient makes a little mini uh graph graphs that onto [00:27:44] the larger graph and then does a a re uh quotequote reasoning in the old school way of you know walking through the graph and then generating results. [00:27:54] awesome. like it works perfectly. it’s white box. i can tell you exactly why i matched what i matched. um the challenge is it’s expensive honestly. right. we have a team of of of folks that are with phds like reading the clinical trials, [00:28:05] reading the fda, reading the nccn esmo guidelines, too. we do all the european guidelines. we’re reading all those guidelines and codifying them in our knowledge graph. it’s awesome. i can tell you exactly why and i can highlight [00:28:16] even the sentence in a trial, for instance, why you match this trial or you don’t match to this trial. the problem is it’s expensive and it’s not complete. like there’s a long tale of [00:28:26] lab results and you know other con conditions and coorbidities that i don’t have time to curate like our team can’t do everything. uh so what we have to do is we have to do the 20 % that matters [00:28:37] right and so by curating that 20 % we’re able to do a you know a really good job of making great reports and helping people with trials. the problem is there’s a gap and that last gap and that gap you know you see it with the [00:28:48] self-driving cars going from 80 % to 90 % is kind of easy. then you go every percentage point above that is re basically gets twice as hard and then [00:28:59] eventually you get to the point where you’re like trying to add 0. 1 % uh reliability and you’re doubling the costs. so you know that’s just from a human effort. same thing on the large [00:29:09] language model side but the large language model side is i i get the benefit of these large labs pouring billions of dollars into making these models and the large language models have capabilities my graph doesn’t [00:29:20] right. it has this general knowledge and has language processing where in my language processing world i’m you know i’m using old school bert models and i’m doing reax and all this other ugly stuff that is hard to maintain a large [00:29:31] language model will you know basically do that stuff uh easily very easily at this point but the problem is it’s it can hallucinate it it is a black box we don’t know exactly how it works but if [00:29:43] we can make those two things work together that to me is the goal right the goal is let’s use large language models as intent processors as language processors. you know, the thing from [00:29:53] star trek, right, where you say, ” hey, computer, go do the thing. ” the computer’s just interpreting what the question is and then bundling that into a a structured query and then exercising these these downstream tools. so, that’s [00:30:05] the goal. that’s what i’ve always been excited about. and so, tool usage looks something like this, right? i got, you know, a user asks a question about the temperature in new york. uh, even vaguely, you know, do i need a jacket? [00:30:16] and now it turns that into a get temperature call because it knows get temperature is one of its tools and it can chain together multiple calls or it can even chain together you know you know a workflow of calls right where [00:30:26] it’s getting a temperature at a location from a hotel name. so like th that’s tool that’s what i mean by tool calling. and now one thing to be aware of is the large language models don’t actually [00:30:36] know how to call tools. it’s not how it actually works. what what they know how to do instead is generate structured outputs. and usually it’s json, right? it’s a format with curly brackets and quotes and stuff and they’ve been [00:30:48] they’ve read bajillion lines of json. so they know what json is and they can generate json. so what you do is you say i have a set of tools. here’s their [00:30:59] names. here are their arguments. go ahead and you know if you want to call one of these tools just generate the json and then then the the controller right the thing that’s actually calling the llm can parse that json out and then [00:31:11] turn that into a tool call get the result and then stick it back into the into the chatbot and that was actually a pattern called react back in you know two or three years ago and so tool calling has been around since the [00:31:21] beginning it’s the problem is they weren’t good at it they just weren’t good at it because they just weren’t smart enough to to recognize how to call those tools or generate that json and i know they weren’t good at it because i’ve been trying. so, chatgpt plugins [00:31:33] came out in march 2023. i built a plugin for our precision oncology platform back in the day trying to get the thing to call the right stuff to make, you know, matched trials or do therapies or even [00:31:44] variant analysis and it was just super hard. uh i tried i tried lots of different things. the other big problem with htp plugins was that the technology was flaky. there was, you know, not a [00:31:54] lot of support from openai and they actually ended up canceling that thing. uh there were function calling which is what i described earlier which is basically generating json and one of the key outputs of that or you know outcomes [00:32:06] of that was actually people figured out that they could use that to generate structured outputs uh and i actually have a a github awesome list that you know is still valid i guess but not as [00:32:17] interesting that you can actually you know do this with basically all the models and all the there’s a bunch of libraries that do this for you. so you can, you know, give it some unstructured text and it’ll spit out some json and [00:32:28] like, you know, one of my favorite libraries is called instructor. you just define a model in a language called py uh in a library called pyantic. you call instructor and it gives you back a json [00:32:38] uh output that gets turned into a python object for you. so function calling’s cool. it it it kind of sort of worked, right? especially with gbt4, it just was expensive and slow. um, [00:32:50] but then having it figure out how to then chain multiple tool calls together was still difficult. there was an another initiative called gpt, custom gpts. openai is really bad at naming [00:33:00] things. i i i have trouble keeping track of it all, but custom gpts is this idea that you can uh create a gpt, give it a name, uh give it a, you know, custom instructions, maybe some pdfs, and then [00:33:12] you can tie it to what are called actions, which actions are basically, you know, the, you know, model context protocol v0ero effectively. it’s a it was a wrapper around the open api spec [00:33:22] and you could actually have it call actions and i had it working like i could do a demo where it would call my precision oncology platform and bring bring back trial recommendations. once again it was you had to like coach it

Model Context Protocol

[00:33:33] though through the process and a normal person uh probably just wouldn’t have the patience to actually do that. but then model context protocol comes out in november 2024 and i’ll be honest i saw [00:33:44] it and i was like yeah that’s great. it’s open. it’s an open standard. anthropic is not making it so you have to use anthropic to to use model context protocol. so i was like that’s good. that’s a big win. but it hadn’t really [00:33:55] dawned on me yet that both claude sonnet with 3. 5 which had just kind of come out and the 01 tool calling would actually make this better. like it just didn’t [00:34:06] click to for me. um, and so what ended up happening is like a month or so later, people were like getting really excited about model context protocol on twitter and i was like, ” okay, maybe i [00:34:16] should look at it again. ” and then i had a colleague ask me about it and i was like, ” okay, if this guy’s knows about model context protocol, maybe i should really think about this. ” and so i dove in and i said, ” okay, let me let me think about it. ” and at the time i was [00:34:29] considering building an mcp around GenomOncology’s stuff, but but honestly, we’re so early. um not everyone’s ready for this this type of work yet. so i thought wouldn’t it be more interesting [00:34:40] to do something kind of open source and let people play with it and understand and learn what’s going on here and why this is actually exciting. and so that’s what you know the brainchild was called BioMCP. uh so BioMCP is what’s called an [00:34:52] mcp server and it calls backend servers right and actually BioMCP is really kind of the collection of all these things right and in this case it’s actually all [00:35:03] of them are not local all of them are all remote so BioMCP is basically a collection of remote servers clinical trials. gov fda, pubmed. i’ll talk about i’ll talk about all the specific servers [00:35:14] in a moment. but basically what happens is you create a server and the server represents the data uh behind it. and the server has tools and that’s what you know model context protocol has more [00:35:25] things than tools, resources and prompts. honestly, nobody uses anything else like they just use the tools. and i actually just saw a conversation about this on twitter yesterday which is yeah tools work. nobody wants to do anything [00:35:36] else. none of the clients support anything else. it’s really all about tools. so that’s fine. i i actually think the rest of the protocol is interesting. it’s just not being represented by what are called the [00:35:46] clients. so the client in this case is the chatbot. so you’re you know you’re using or an agent and i’ll get into what an agent is later. uh but a chatbot has [00:35:56] model context protocol and you know all these different servers that can it can connect to and you can actually in your chatbot you can connect to a bunch of different mcp servers. if you go to this mcp, there’s a bunch of like mcp [00:36:08] directories. i’ve seen at least 5, 000 of these mcps, so they’re out there. now, just be aware if you’re running this thing locally and you’ve got an mcp server you don’t trust, don’t do that. [00:36:20] like, you need to fully trust that mcp server. so, i’m here talking to you. hopefully, you can you can write to me and hopefully you trust me. you can look at the code. uh the code is what’s up on pippi, which is the way we distribute [00:36:31] the code. um you know and we’re just basically uh proxying data to these backend servers. if you don’t trust any of those backend servers like clinical trials. gov etc. don’t use bcp either. [00:36:42] the reason is those things are sending back content in these large language models can get tricked. it’s called prompt injection and i can get into what prompt injection is if you’re [00:36:53] interested. but the idea is the large language model can be used to then exfiltrate information where it can you know especially if you’re giving it access to your local server like you know your local computer like cloud code [00:37:04] it can actually look for tokens or you know api tokens or whatever and send them up to uh remote servers. so, it is it is a security risk. uh, and that’s another big reason why i’m we’re going [00:37:16] to be we’re investing time right now into building local uh tools, local mcps and and command line tools for llms. can you guys can hear me? okay. still, [00:37:27] i’m just taking a a quick break. yep. great. um, anybody completely lost? happy to like take a take a beat and [00:37:37] like answer any pressing questions. all right. so, more slides. we will have plenty of time at the end for questions. i’ll i’ll i’ll answer all the questions i can and i’ll stay late if you guys [00:37:48] need me to. um, okay. ai research assistant, ai research agent. i just wanted to define these two terms real quick. these are my terms. there’s a lot [00:37:58] of different terms out there. you know, people people will call agents lots of things. it’s it’s kind of kind of crazy how many different definitions of agents i’ve seen. but in my terms, ian’s terms, [00:38:09] assistant means it’s a synchronous thing. you’re talking to a chatbot effectively, right? it can invoke tools. uh, usually asking for explicit permission, right? cloud desktop will always ask me, no matter how many times [00:38:20] i say always allow, it’s just going to keep asking me, do can i use BioMCP? that’s fine. um, but it’s basically one single conversation and it’s back and forth. where an agent is more asynchronous, [00:38:32] meaning you give it a goal and it goes off and does the does a thing, right? it might take a minute. it might take 30 minutes. it may take 30 days. it’s not not 30 days yet, but i’ll talk more about what i mean about that in a in a [00:38:44] moment. um, but they manage the context and state over long running tasks and they’re using reasoning to kind of to do stuff. so, those are my definitions. so,

What BioMCP is

[00:38:54] an assistant and an agent can be given BioMCP. you know, they’re basically a set of tools. and what BioMCP is is it’s a library, a python library that has a [00:39:05] bunch of python functions that have been decorated with a tool decoration is what it’s called, but it basically says, hey, this is a BioMCP tool. expose it as a [00:39:15] public interface. and that tool has a description and it has arguments, meaning like here’s what it expects, like here’s the input uh fields i expect and then here and then hopefully the [00:39:26] description tells you why you’d want to call it. and that is then used dynamically by the assistant or the agent to do the right calls. right now, it’s different than writing programs. [00:39:37] when you’re writing programs, you’re just saying, ” i got these functions. i’m going to call these functions in this order and get the answers back. ” well, that’s not an agent or an assistant. that’s just programming. BioMCP is meant [00:39:47] to be dynamic in that you’re going to ask it a a question or give it some data or whatever it is and then it’s going to figure out through an iterative looping process what calls to make and then by [00:39:59] reading those results make more calls. right? that’s the that’s the whole beauty of this whole llm based uh way of programming and it’s kind of a mind bender if you’re not used to it. but so [00:40:10] bmcp is really, you know, a pro what i call a proxy to backend systems and there’s way more now. these were the first three i did and now annab’s on the team and she’s knocking out uh new apis [00:40:20] all the time. if you have apis you that you know that exists, uh send them to us on github and we’ll we’ll put those in as as well. we we’ll take code contributions, but we’re honestly going to be very selective about that because [00:40:31] of the security issues that i’m concerned about. um so, you know, we got clinical trials uh. gov, right? right. and i’ll i’m going to do a deep dive into clinical trials. gov. my variant [00:40:41] info was the first api that we did for genomic variants, but we got more now. there’s c bio portal. there’s ankle kb. uh, and there’s more. uh, my variant info actually has like 20 or 30 databases embedded in it, right? db and [00:40:54] sfp, which actually is another 20 or so databases inside of it. cad and clinvar. it has an old version of cosmic. uh, a bunch of stuff like that. and then there’s research articles like uh we use [00:41:05] a service called pubator, which i’ll talk about. and there’s 35 tools, maybe 40 tools. i actually don’t even know what the number is. i need i need an updated number from annabye. but, you know, these tools are all made available [00:41:17] to the llm as part of the mcp server. part of what you need to do is you actually need to turn on and off tools based on how you want to use them to be honest because they take a lot of context. the context meaning context is [00:41:30] the working memory of the lm. it’s the number of tokens that it understands. claude, for instance, is a 200, 000 token window, i think. and i think even that probably gets bumped down to 175, 000 [00:41:40] based on the system prompt. so then you’re that’s your that’s your ram basically, right? in the old school programming days when you’d actually care about ram, uh that’s how much ram you have. and now your context window is [00:41:53] going to get not polluted, but it’s going to get uh absorbed by the the functions and their arguments and their description. the bigger those are, the more there are, the more context you’re [00:42:03] kind of you’re squeezing out. and that becomes important especially if you’re trying to do a long conversation where where you ask multiple things that you’re going to get that warning at you’re gonna get that error basically at the end that i i always regret which is [00:42:15] i’m i’ve ran out of space. go ahead and start a new chat. it’s like ah i i just started to get to the thing that i cared about. so just be aware. so you turn on and off tools. you can do that in claw [00:42:25] desktop. i’ll show you that in a moment. so claude desktop it’s uh you just download the app. it’s, you know, i think i think there’s still cloud free version, but you don’t get very much [00:42:36] usage if you have a free free account. if you have the $ 20 a month account, i’m pretty sure you can use claw desktop and you can turn on BioMCP. you can turn it on locally using the BioMCP uh program [00:42:48] from github. uh there’s steps on how to do that. and then there’s our remote server as well. and so let’s go through a quick example and what why this is powerful and why this is useful, right?

Demos: trials, literature, variants

[00:42:59] and the first is clinical trials. gov rest api. so, if you’ve ever been to clinical trials. gov, i have quite a bit, um, you’ll know that they their their website, their search engine, their [00:43:10] content is immense, right? and the website’s way better than it was 5 years ago. uh, it’s it’s a really good search engine. uh, as far as search engine goes from a power perspective, right? they [00:43:20] have so many fields. if you look at the api, you’ll see how many fields there are. uh, they have the not only do they have all these fields, but then they have these expression syntaxes, right? where you can do complex kind of boolean [00:43:31] queries. you can do things like latitude and longitude stuff. they have enumes like these um you know these flag based values and that’s this is just the four first four fields. there’s probably you [00:43:43] know 10 pages of this stuff. um, and basically what i did was me and i think it was chat gpt at the time, uh, we worked through this whole thing and figured out what was the critical stuff [00:43:53] that we wanted to expose and then how did we want to shape a query so that we could do this in a simpler way so that the lm didn’t get overwhelmed, right? i could have certainly just given the llm [00:44:03] this full rest api, but man, that would have consumed a lot of um, that could have consumed a lot of context and it might have confused the llm to actually know what to do. so that was part of what bomcp is about. it’s about curating [00:44:15] these tools so that they’re effective and useful and that they don’t spit back so much context that the llm gets overwhelmed and you can’t even uh use it for any uh reasonable tasks. so there’s [00:44:26] the rest api. there’s a picture. i actually drew it in powerpoint or google google slides a few months ago. uh find active pero trials like i was making a little demo slides and then i said, ” oh, [00:44:39] actually let me just copy and paste this thing rather than typing it and paste it right in the thing. ” and and go ahead and see if it’ll actually respond. and at the time that was really cool because images weren’t really a thing yet. and i [00:44:51] was like, wow, it actually parsed the picture and then turned it into this query. so, let’s look at the query active. active is a keyword and it gets translated by the llm. i don’t i don’t [00:45:03] have a mapping, right? there’s no python dictionary or anything that says active means open. it figured out that oh yeah, this means open and pembro is pembroluzamab and kitruda and oh yeah there’s a there’s an experimental name. [00:45:14] i didn’t know that experimental name and it’s not certainly in a database that i’m exposing this thing to. uh we do have um as a database commercial uh and that’s part of you know some of the tools we built for agents. um so [00:45:26] interventions right it mapped pembro to this and then it mapped conditions nclc to non small cell lung carcinoma and then also threw in the acronym as well and then latin long right it figured out [00:45:37] cleveland ohio you know good enough let’s you know pick a pick a point and we’ll do 100 miles by default and so then it comes back and it brings back you know it posts that query and then BioMCP is what responds back with as [00:45:50] tight of possible text right i tried to do a little bit of like indenting and stuff so it you know is it can understand the structure of the output but i tried to do it in such a minimal [00:46:00] way that i’m not consuming tokens recklessly where i’m you know basically consuming everything and then the other trick i had to do was actually for clinical trials because they are so big and there’s so much content i split it [00:46:11] up into multiple queries so there’s a searcher which basically returns back you know search results like you were you know searching for trials and you got back a page of 10 or 50 or whatever the number is uh here’s the search [00:46:22] results and then you can deep dive right. oh, let me go ahead and deep dive on the locations based on the nct ids that i’m seeing from uh the search and then it can look [00:46:33] through the locations and and and get the answers out from there. there’s other optimizations we could do to bmc, right? if you were to build if i were to build something that was like more production ready that i had a narrow focus on, i would then make the tool [00:46:46] filter for me, right? i would basically say, ” oh, yeah, go ahead and don’t show me this thing in alabama. i know that i want the i want only things in ohio, right? so these are like little optimizations i know we could add and [00:46:57] maybe we’ll add that one. that’s a pretty simple one. but you know over time um what you want to do is you just want to minimize the amount of information that comes back but minimize [00:47:08] the number of tokens but you also need to have the other tension here is we can’t have too many functions, right? because then if we have too many functions then the llm gets confused on what to do. and i’m kind of explaining [00:47:18] this to you guys because the strategy by which i’m built i built BioMCP. you might be wanting to build your own set of tools for llms, right? and that’s part of what we do as well. so if you’re, you [00:47:28] know, interested, i can certainly help there. um, and then it comes back with a report. and so the report is like, hey, here’s here’s the active trials. i didn’t tell it how to make this report. it came up with it. it read the data and [00:47:39] it decided this was a good report. now, i could say, oh, well, give me a table or show me something else. and that’s the great thing about large language models that they’re kind of infinitely adaptable in context of a of a chat [00:47:50] experience. and then so that’s clinical trials. that’s just one of the tools. the other uh key tool is uh pubmed articles. and for pubmed, i’m a big fan [00:48:00] of this one researcher. he’s done a lot of great work um at the ncbi or nih. and one of his projects is called pubtater. and pubater is basically a semantic [00:48:14] search engine for pubmed. what does that mean? means they’ve taken all the abstracts from pubmed. i even the full text articles from the pmc, the free versions, right? the pubmed central uh [00:48:25] versions of the documents. they’ve taken all that text and they’ve done named entity recognition on four entity types, genes, uh drugs, diseases, variants. and [00:48:36] so they’ve highlighted all those. so all these like little boxes here, these all tie to one of those four entity types. and when you do a a search in this tool, you’ll notice that it actually does an [00:48:47] autocomplete and it then turns it into like an entity based uh query. so this little at symbol chemical is the is the label and it’s now doing an entity based [00:48:57] search and it just improves the relevancy of the search. so i like that so much that i actually baked it right into BioMCP. so BioMCP behind the scenes you don’t do this uh behind the scenes [00:49:08] uh you ask a question uh the BioMCP actually does have four buckets it has a bucket for uh genes drugs varants whatever uh and a keyword one so if if [00:49:19] one of your keywords doesn’t fit in the four buckets it puts it in keywords and then what i do is i take those four buckets and quickly par in parallel i think you know call the the entity uh [00:49:30] resolver from pubater get the four entities back and then i use those in the search to pub to pupater and it just dramatically makes the the search engine better. it just brings the the higher [00:49:41] relevant stuff to the front. uh because you really want it to be in those top 10 or 20 results. otherwise, you’re going to get, you know, you know, a long tale of garbage. and in pubmed’s paper, they’ll even say, ” hey, this thing’s [00:49:52] like uh 90 % precision. ” and precision in this case means like how many of the top 10 things showed up in the top 10. uh and it’s just better than pubmed itself and and google scholar. [00:50:04] so, those are, you know, some of the original ones and my variant info is out there as well. uh, but we added uh anko kb this week. uh, thanks to annab. uh, you can see here that it’s and it was [00:50:14] excited. claude was super excited that it was added because i i actually was trying to get it to work on my computer and i said, ” try again. try again. try again. ” and then it worked. i was like, ” oh, okay. ” because you have to hook up the right things inside of the inside of [00:50:25] the json. and it finally was working and it was super exciting. um, and so this only works for three genes. uh bfe, uh tp53, and ros one i think are the three [00:50:35] demo genes. um the rest of the genes you need an kb license. i’m sure you guys have a an kb license. i don’t because we’re a competitor. um and then there’s oh g [00:50:51] is another collection of tools. so we’re adding we’re basically uh mimicking those tools. um i don’t know how we’ll give them credit because we’re not i don’t think we’re using gget in our code. um, but we’re we’re we’re replicating some of their their tools [00:51:02] that they have. and there’s, you know, uh, and then there’s another paper called gene agent that has like another 15 tools uh, that i want to replicate. and so, we’re just going to keep adding more and more tools to it. uh, once [00:51:13] again, because i don’t think this is about curation. i’m not trying to curate the perfect 10 tools. i’m trying to just basically show, you know, here’s 50 tools or 100 tools, whatever we can get in there. it’d be great. and then people [00:51:24] can use what they want, right? choose the tools you care about inside of claude or choose the tools you care about inside of your agent, do win your hackathon, uh prototype something, proof of concept, and then you then take that [00:51:35] information and say, ” okay, great. let’s do this for real. ” and to do it for real, we’ll either host something behind our firewall, which is what we’re working on, or you can just, you know, uh copy paste from bmcp and choose just [00:51:47] the the parts you care about, um because it’s all open source. so, let’s do a quick demo. so, let’s see. uh just because i don’t want to fail, i’m going to show you real [00:51:58] quick. um you know something that h right now there’s a there’s these things called instructions which aren’t working in cloud desktop and i don’t know why but you know what does bomcp uh say that [00:52:09] uncle kb says about bref600 and i can see here that it you know this is what i was doing yesterday right and it kind of brings the results back. so what’s happening first? you can see that it [00:52:20] does a search, right? in this case, it’s doing a search uh and it’s bringing back search information. and then i’m doing a get and it’s bringing back more detailed information. you can see all these different codes, right? like all these [00:52:31] codes are from um sorry, i’ll make my screen a little bigger just because my eyes aren’t that good. um yes. oh yeah, i just did that. is it better now? can you guys see? okay. um [00:52:44] yeah, it looks good. all right, great. um, so you can see here, let’s see, it’s actually figured out. okay, here’s the variant id. so that’s the variant id for my variant info is the the proper g dot [00:52:54] or i guess that’s maybe not the proper g dot, but it’s close enough. um, and you can see that it brings back cad, right? brings back uh all these different fields from cad. uh, another database [00:53:08] that i’m not familiar with, another one i’m not familiar with. but once again, these are all up on uh my variant info and you can look up where these sources are from um and what the information is, [00:53:18] right? so there’s, you know, position information, there’s prediction information, there’s frequency information, you know, all the typical stuff. um different scores, etc. and polyfen. oh yeah, here’s a here’s, you [00:53:29] know, here’s a prediction, right? deliterius. obviously, we know this a famous gene, so everyone same famous variant, so everyone knows what this stuff’s about, right? but you can see down here uh [00:53:40] sifting through it. i could i couldn’t find it, right? i don’t see anything about uncle kb. and this is where i’m in the background like now, you know, you know, fixing my uh it was actually i think the the root cause was i didn’t [00:53:51] install the latest version of BioMCP. i had to like i had to like clear it out and then reinstall it. um and then and and oh, here’s a funny thing. [00:54:02] i asked it for uncle kb stuff and it says i can’t find anything about uncle kb, but let me go do a web search, right? because i had the web search turned on. so, it’s going off and doing the web search. so, these are things that these chat bots will do uh that you [00:54:14] might not expect, right? so, you have to know if you want to just be using BioMCP and you just want to be using my variant info, you have to turn off the web searcher or else it’ll go rogue and and start searching stuff. uh hopefully i [00:54:25] don’t swear at it after this. um can you try again? right, this is me just trying again and once again i think it failed again. okay, try one more time. i just updated the server and now it gets super excited. right, so that was funny to me. [00:54:36] um, and you can see down here that, you know, somewhere is the anko kb stuff. i’m not going to make you all look through it, but you can tell then it goes through and gives you the results of an kb. um, and you know, other [00:54:47] information. let’s see. i now i typed this in before i left. let’s see. uh, okay. now, it’s going to look for tp53 variant searcher. great. [00:54:57] uh, variant getter. okay, it found stuff. so, it found ankob classifications for tp53 and it found a ros one, right? right. so, those are the three demo genes. so, [00:55:07] and uh we’re working through with one of the developers at memorial sloan kettering how to make sure that the the tokens work. he couldn’t get the tokens to work last night. so, hopefully today we’re going to try to figure it out [00:55:18] together uh what the issues are. so, so that’s kind of like how you use it in in cloud code. um if i had the oh, wait. [00:55:29] there’s so one of the big things i want to show is this idea of like a deep research thing and so i got a half an hour left and only a few more slides.

The biomedical research assistant

[00:55:40] um, but there’s a there’s resources here and one of the resources i’ve created is a is a research assistant and this is basically a prompt and i’m actually working on a system uh that i’ll be open [00:55:52] sourcing in a couple weeks that does this more um more deliberately and you’ll see what and you’ll i’m excited to show it to people and i’ll i’ll send [00:56:02] it to the team of folks i’m i’m emailing with when i when it’s done. but the basic gist is this unlocked a lot of of of capabilities that i wasn’t didn’t really impact me until i played with [00:56:14] this in april, i believe, uh, and made this work. and basically what the the biomedical research assistant does is it’s really a giant prompt that says, ” here’s all the tools. use this thing [00:56:26] called sequential thinking, which is another mcp that we actually just baked right into BioMCP. ” and what it does is it forces the llm to think through and use a thinking process, step-by - step [00:56:38] process. and you’ll see when it’s using BioMCP that it actually might say, ” oh, i’m going to do i’m going to think for five steps or i’m going to think for 10 steps or think for 20 steps. ” and as long as you don’t run out of context, it’s going to keep it’ll keep doing the [00:56:48] steps. it’s basically a noop that doesn’t really do much other than say, you know, keeps track of how many steps you’ve done. and and then what this deep research prompt does is it kind of shows [00:56:59] you what’s possible, the art of the possible with regards to, you know, a prompt based program here. and what i what i’m doing is saying, hey, first take the user’s question and then first [00:57:09] come up with a framework, right? figure out what’s the right way to answer this person’s question. and then it does that by saying, okay, yeah, if it’s a therapeutic question, use pico, right? what have you? and then here’s the [00:57:21] different tools. and then it says go ahead and do a broad web search and then do focus searches right and then it’s asking and it’s basically trying to bring back relevant information and then it synthesizes it and then you know [00:57:32] self-critique right once again these large language models are better if you ask it to like critique itself like just you know point out what’s wrong and then fix it and then make a and then make a [00:57:43] brief right and here’s the the rules around the brief if you were to go to um yeah, i have i i have another BioMCP [00:57:53] examples website uh github repo and i’ve only done one example. uh i probably should do more, but this example i’m saying what are the emerging treatment strategies for head and neck cancer and [00:58:04] so at the time uh when i first did this uh chat gpt had already come up with deep research. gemini had actually beaten them and had a deep research first and then there was an ai thing [00:58:15] called manis and then claude had just come out with their deep research. and so i was really into like this idea of deep research. and the deep research is basically it’s going off searching a bunch of stuff and then synthesizing it [00:58:25] and analyzing it. and so i said, well, what if i could, you know, how do i make it to compare BioMCP using my prompt? how does it compare to these other four things? and and basically i use that [00:58:37] information to then eval. so i got the prompt after like 25 iterations of this thing. and i was like, it keeps getting better, so i’m going to keep iterating. um it finally [00:58:48] came up with you know and said oh yeah this this version you know BioMCP plus sequential thinking plus web search gives us this output and you can see [00:58:58] here that it’s basically a very robust you know whether it meets your needs you know uh tbd but you can always change the prompt um and it just goes through and and synthesizes a great report and [00:59:09] hopefully i’m not going to lie and i think that there’s um you know there’s charts and all this tables and stuff and then there’s references, right? forcing uh claw to get it to print out the [00:59:20] references was the hardest part actually. um but they’re all references in here. and obviously what you want to then do is give it another agent or or actually a pure python program or whatever language you like and have it parse out these references because they [00:59:32] will hallucinate references, right? even though they’re doing web search, even though they have a url and they’re bringing back content from the url. um, and i should have forced it to put the url in here if i could. but another [00:59:44] thing you could do is use that, you know, uh, doi or whatever these are called and search for them and make sure that the title matches and the authors match and that it’s actually a valid thing. uh, that’s just another great way [00:59:55] of like getting getting some corrections around this stuff. and then once again, that’s not this isn’t a finished product, right? what you want to do is you use this as a researcher to save you time, right? save you five hours of [01:00:06] googling or 10 hours of googling and now it’s a a much better launching point for for whatever research question you have. so, okay. so, and then oh lastly, um

Coding agents and what comes next

[01:00:20] BioMCP also, so here’s my so this is like the predecessor to my whole cloud code thing that i’m going to talk about in a moment. i actually have this is actually codeex which is the opening [01:00:31] product on the left and on the right is a demo slide. well, once again, demo, sorry, demo markdown that i asked uh codeex to make for me. i said, ” hey, i’m [01:00:41] doing a demo uh today about BioMCP. can you go ahead and make a demo? ” right? and i and i told it, make sure that all the clis work right in the command line [01:00:51] tool, generate the output, and then make a markdown file for me. uh and and and so i can show people how it works, right? and i can give you guys this demo file if it’s interesting. um, but basically it, you know, walks you [01:01:03] through like how to do help, right? and then it walks you through like how to do health check. oh, so there’s a health check part, right? if you’re using BioMCP and something’s not working, uh, [01:01:13] it can give you back, uh, an error mode, right? so, hey, hey, annab, i don’t know what’s going on with clinical trials. gov, but i’m getting 403s here in the health mode. maybe maybe check that out. um, and then you can see, but here [01:01:24] it actually works. so, there’s something goofy with the health check, probably. um, so you can go, you know, do a clinical trials. gov gov query and in this case it shows BioMCP command line tool trial search condition melanoma [01:01:38] status open page size 5 and you can see that it gives you back results and i asked the snippet so it wasn’t too long so you can see it here so the point is there’s actually three modes for using BioMCP you can go look at the code and [01:01:49] use the python library so we tried to design it in such a way that the python library itself is pretty useful so you can now import bmcp and then use the trial fetcher or the variant searcher [01:01:59] whatever they’re and then there’s the command line interface. and i honestly built the command line interface for one reason only to make automated testing easier, [01:02:09] right? i didn’t want to try to figure out how to do automated testing against um mcp tools. i was worried that that was going to be hard. so i said, ” oh, i’ll just make a cli as well, thankfully i did because it’s quickly turning out [01:02:21] that cli might actually be the new mcp where people are very excited about command line interfaces because the these large language model um things know how to use clis really well as you [01:02:33] can see i made this demo in a few minutes um and then here’s some more right get a specific trial in this case it’s actually specifying the format of json right so naturally but you know uh [01:02:44] the default output not natural the default output is markdown but you there’s a flag in here for json if you want to get json format. uh, and then there’s searching across pubmed, right? and then once again, i’m it’s asking for [01:02:55] json and you can see the different fields. um, getting getting articles from BioMCP, you get the full article, uh, variance by rsid. so, searching of [01:03:06] variance, searching for variants, retrieving variants, fda adverse effects, right? so, the fair’s database is in here. uh, you can search them for pembro and it gives you back uh, the results there. so this is once again [01:03:18] just another way of interacting with BioMCP and and then the last way is really using these agents right like i can actually build um you know whether you use lang chain or langraph or [01:03:29] pyantic ai or one of these you know one of the ai type safe uh sorry typescript ai platforms or use my platform which is going to come out in a couple weeks you’ll be able to build agents and give [01:03:40] it BioMCP and you can tell it to use BioMCP as an mcp server quote unquote or you can give it to as a cli i and you just say here’s the cli and then claude code actually now has a new thing called [01:03:51] skills and you can create a skills uh file. oh and this would be fun to build a skills thing for BioMCP. so you can say hey let’s go ahead and um add a BioMCP skill and you can then use BioMCP [01:04:02] to do all these different things whether it’s you know investigate for trials or what have you. so uh hopefully that’s enough of a demo for you guys. um, but i’m happy to come back to to to to more [01:04:13] demo mode later. and just to wrap up my slides, i got a few more slides. uh, so coding agents are a thing. and the reason why is because all these [01:04:23] labs have really put a lot of time into improving coding skills. and the thing that’s really interesting to me is what you know, once again, these indices are are are kind of whatever. um, but claude [01:04:34] 3. 5 sonnet was really the first model that everyone was freaking out about their coding skills. and that’s when cursor became a big deal. everyone was like cursor is so great. well, not really. claude 3. 5 sonnet was so great. [01:04:45] like that was really the trigger point for for why coding became a thing. uh and then the models are just getting better and better. like all these companies are now heavily investing. uh even the there’s a free version uh from [01:04:57] xai uh that you can use with an open - source um cli called open code uh which is really nice little uh cli tool. so, if you want to try ajetic coding for real and you have permission to do it [01:05:08] and or your personal computer, like i’m not trying to tell people at st. jude what to do. um, you can install open code and use grock 4, i think is what it’s called, or grock coding fast. it’s [01:05:19] a free version from a from a a platform called open router. uh, and you can go burn some some of elon’s tokens for him. uh, they will train on your data though, so just be fully aware of that. uh, and [01:05:30] then there’s a deep seek model, which is one of the open source ones. and then i just signed up for this glm company from zai. i’m watching them. like i hadn’t heard much about that, but their library, their models are are really [01:05:41] good and they have really cheap coding um uh plat uh options out out there as well. so, you know, i’m honestly spending way too much money on these things. i got codeex, i got claude code. [01:05:52] uh it’s super fun. uh there’s a platform there’s a library called ader which is kind of like the original old og chatbot uh that in the command line and they’ve been tracking how good these models are [01:06:04] and you can see that it started at like you know a four 4 % on these 225 most uh difficult coding skills that would it’ll cost you 32 cents all the way up to now [01:06:15] 88 % accurate on getting these 225 most cod uh coding problems done that cost you almost 30 bucks and if you use one of the most expensive models it’s like almost $ 150. $ 50 just to solve these 225 [01:06:26] problems. um, and you can also see that this ader project is doing a nice job of keeping track of how much code is being written by ai versus the the person or team that uh creates and maintains the [01:06:36] ader. and then coding, you know, when it started was really like this ghost tag, this ghost coding stuff, right? you go into cursor or, you know, copilot or all [01:06:47] these different uh wind surf and it would do this ghost to code where you can do autocomplete. this is powered by an llm. the llm is predicting the function you want based on what you’ve [01:06:57] seen ahead of time, what the code it’s, you know, in context of, and then you hit tab or whatever the key keystroke is for your editor, and it generates the code. that was kind of like, you know, [01:07:07] uh, you know, state-of - the-art uh llm coding a year ago. and then uh it got got so good that this guy andre caparth who’s a really big deal in in the ai [01:07:18] space. i’m sure you all know who he is. uh invented or or coined the term vibe coding. and now there’s a book called vibe coding which i have on my desk here. uh and basically what vibe coding [01:07:29] is is you know the idea that you’re not looking at the code. well that’s not really what people are doing that are serious about that. you can vibe code like prototypes, but really what you’re doing is you’re using the agents to help [01:07:41] you craft the code. and it’s just working at a higher level of abraction. sort of like how we went from, you know, hand coding machine code to using assembly to using c to using python, [01:07:52] right? like we’re now at a new level, which is how do i tell an llm how to create my c or my python? and cloud code came out maybe in march or april. uh i [01:08:02] didn’t use it right right away, but i was intrigued. well, i did use it right away. it cost me like $ 8 to do one thing. and i was i freaked out. i’m like, i i can’t be spending this much money. uh but then they offered it for [01:08:13] for for free with your $ 20 a month account. and then at the end of like an hour or two of using it, i was like, ” okay, i’m in. ” and so i had to go, you know, get approval to get the full uh max account. um and so but basically it [01:08:25] can run for 20 or 60 minutes, you know, if you if you structure it right. so, i have this whole uh process, you know, i have like this whole thing where i’ve got a developer and an architect and, you know, a command line thing and i [01:08:37] have like this whole sdlc. i wrote it down for myself just just so i could show you guys, but basically, you know, i’m i’m using i have a product owner with claude and we’re building up sprints and then i’m giving the sprints [01:08:47] to my codeex developer and it’s running all the code and running all my tests and then i look at the code and i yell at the developers and then i end up throwing away all of it and then starting over and i’ve done that on one project a couple times now. and that’s [01:08:58] okay. like that’s, you know, it’s actually a more efficient way of kind of building some of these things than trying to do it all by hand uh where you end up throwing it away anyways, right? i’ve i’ve i’ve had many times where i’ve [01:09:09] thrown away projects and started fresh once i’ve actually learned some stuff. uh yeah, and vibe coding is this book. that’s the one on the right. it’s pretty good. oh, yeah. vibe engineering is what simon willis, who’s one of the bloggers [01:09:20] that i i read all the time, uh that he that he he’s trying to coin, which is the idea of like you’re using agentic clis to code, but just more efficiently, but not but you’re still looking at the code. all right. and then tomorrow, so [01:09:32] this will be hopefully real brief. the tool calling has also gotten better, right? coding’s gotten better. tool calling’s gotten better. that’s really what comes into what making an agent is is tool calling. and so and and one of [01:09:43] the other things we’re seeing is that these agents or these, you know, these loms are getting better at staying on task. so this is really the real exponential that i’m interested to see if it continues to play out where every [01:09:54] seven months they get more and more accurate and they and they stay on task longer and longer. and so, you know, gpt3 was able to do a task for a half a second and it stopped. and then gpt4 was [01:10:04] able to do it for, you know, maybe a minute or whatever. and now we’re up to, you know, 20 25 minutes with 80 % accuracy. that’s the future, right? so, it’s like, how do you construct these agents to do tasks, right? they don’t do [01:10:17] jobs, they do tasks, right? we we now need more radiologists than we ever have. uh, there was a big freak out 10 years ago about radiologists. well, that didn’t pan out. now, we need more. um but we need to figure out how do we use [01:10:28] these llms put them in a loop with tools and have them have them do uh useful work and uh one of the challenges is going to be context windows. so that’s what emerged this phrase called context [01:10:40] engineering which is really just the idea of like if we’re going to have an llm do work how do we design the system in such a way that the context window doesn’t become a a limitation and that’s [01:10:52] the other key thing that my little framework that i’m working on uh solves. um, and then there’s computer use. so, this is brand new. you know, people are using uh, you know, they’re actually [01:11:02] giving the llm’s vision so that they can actually read the whole screen and click the browser and do stuff. it doesn’t work at all right now. i assume in a year it will work really well and it’ll be super fast and people freak out and [01:11:14] that’s that’s just how things are going. um, and then there’s this idea of deep agents, which is really just the idea that an agent is an lm in a loop that makes a plan and then does the plan and [01:11:24] has access to files and reads and writes stuff and then has access to sub aents. sub sub aents are just a strategy for context engineering. you’re breaking up a task and you’re doing smaller and smaller chunks uh in the sub aent that [01:11:37] consumes tokens and then return the result. the result is just, you know, the smaller piece of information. and then this is this is kind of the enthropic view view of what an agent is, right? a human asking an llm to go in a [01:11:49] loop using with an environment using tools, right? they’re doing tool calls and then they stop once they make uh stop making tool calls. this is kind of how i’m doing it, right? which is i got a goal. i’ve got a step checklist and a [01:12:03] schema. you ask the lm to do the thing. it gets the results back. it rejects the output if it doesn’t validate against the schema. and then you stack them. that’s what i’m building. this is this is a preview of the uh open source thing [01:12:14] that i’m building. um and so basically an agypt is really just these steps. um and they work in a sequential manner. it’s really a workflow, but that’s okay. whatever term people like.

BioMCP

Tue, 01 Apr 2025 00:00:00 GMT

BioMCP is a focused biomedical tool layer for language models. It exposes structured, well-described tools for clinical-trial search, biomedical-literature retrieval, and genomic-variant lookup, designed so a model can act effectively without overwhelming its context window.

Three ways to use it: as a Python library, as a command-line interface, and as an MCP server connected to a chat client or agent.

Precision Oncology, Convergence of Genomics and AI, and Improving Cancer Care

Thu, 27 Jun 2024 00:00:00 GMT

This transcript is unedited YouTube auto-captions; proper nouns corrected, otherwise expect missing punctuation and mis-heard words.

Cold open and introduction

[00:00:00] at the end of the day all of healthcare is going to be molecular I want GenomOncology to be a core part of that at least from a knowledge perspective if not from a bioinformatics and Analysis perspective I think we we have the software and the the people and the expertise help people do that and then the goal is how do we make it so the provider right that’s the industry lingo

[00:00:21] for the oncologist or the treating physician how do we reduce the administrative burden the technological burden Etc so that they can spend their time thinking about the patient and bringing empathy and reasoning and their judgment to helping that patient as quickly and painlessly and effectively as possible that’s the goal like I’ve lost lots of folks to cancer in my life

[00:00:46] that I love I want to help be part of that solution to [Music] help let’s discover what people are building in the Greater Cleveland Community we are telling the stories of Northeast Ohio’s entrepreneurs Builders and those supporting them welcome to the lay of the land podcast where we are exploring what

[00:01:11] people are building in Cleveland and throughout Northeast Ohio I am your host Jeffree Stern and today had the real pleasure of speaking with Ian Maurer the chief technology officer at GenomOncology and co-lead of the cancer informatics for Cancer Centers AI working group group with over two decades of software experience he and his LED GenomOncology a

[00:01:35] cleveland-based software company in their pursuit to provide the Healthcare Community with datadriven insights to improve Cancer Care and strengthen Precision oncology programs by transforming valuable but unusable data into actionable oncology treatment options and strategic insights founded back in 2012 and backed by Cleveland local investors like jumpstart North

[00:01:57] Coast Ventures and zap as capital GenomOncology was early to understand the implications of the convergence of genomics and artificial intelligence as it applies to oncology and Ian has been pivotal in architecting and creating their Precision oncology platform to this end this was an incredibly insightful and informative discussion Ian lays out the state of precision

[00:02:20] oncology and Cancer Care today and how the field has progressed over time the founding and evolution of GenomOncology to amarate this entire space solving hard problems Building in Cleveland the everchanging frontiers of AI and genomics exponential technology and a whole lot more so please enjoy this awesome conversation with Ian Maurer after a brief message from our

[00:02:46] sponsor lay of the land is brought to you by John Carroll University’s bowler College of Business widely recognized as one of the top Business Schools in the region as we’ve heard time and time again from entrepreneurs here on lay of the land many of whom are proud alumni of John Carroll University success in this everchanging world of business requires a dynamic and Innovative

[00:03:07] mindset deep understanding of emerging Technologies and systems strong ethics leadership prowess acute business Acumen all qualities nurtured through the bowler College of Business with four different MBA programs of study spanning professional online hybrid and one-ear flexible the bowler College of Business provides flexible timelines and various class structures for each MBA track

[00:03:31] including online in-person hybrid and asynchronous all to offer the most effective options for you including the ability to participate in an elective international study tour providing unparalleled opportunities to expand your Global business knowledge by networking with local companies overseas and experiencing a new culture the career impact of a bowler MBA is

[00:03:52] formative and will help prepare you for this future of business and get more out of your career to learn more about John Carroll University’s bull NBA programs please go to business. jcu.edu the bowler College of Business is fully accredited by aacsb International the highest accreditation a College of Business can [Music]

Exponential cost declines in genomics

[00:04:14] have so I was thinking about where a good place to start our conversation may be and whenever I I think about genomics my mind goes to the exponential nature of technology which for a long time has been a a curiosity of mine really stemming actually from the the investing world but there’s this woman Kathy Wood who runs an investment firm called ark and that’s actually not so relevant but

[00:04:42] what what is is so she uh I think a lot of people are familiar with you know the concept of of mors law which right speaks to the whole observation of you can double transistors every two years without increasing cost and and uh so you ultimately have this exponential growth of computing over time that that we’ve gotten to to experience over the last 60 years and I I think we’ll we’ll

[00:05:04] probably even talk about that as we talk about AI later but she introduced me to this concept of of rights law which is kind of a kindered observation a Cory to to mors law a little bit that speaks more generally to cost falling constantly for doubling of production and and I always think about genomics when I think about this because when I first was probably aware or introduced

[00:05:27] to the whole Human Genome Project in early 2000s I always think about how it you know cost about $3 billion and took 13 years of computing power to to complete on this like order of magnitude of capital and resources that is akin to like a literal moon mission you know yep at a government level and really an incredible achievement but but not at a mass scale uh you know even if you cut

[00:05:54] that in half and it was one and a half billion it wouldn’t become much more accessible to people but I think what’s amazing is over the last 20 years of cost declining 40% year-over-year you can now sequence a genome today in under a thousand dollar and in a few hours and so we’ve now been able to sequence many millions of human genomes not just an aggregate but even at an annual you know

[00:06:18] clip now and so all that is to say I I’ve long been interested in understanding what are some of the implications of this exponential unlocking in practice and what company can do who are plugged into to what what is now possible and so I’m very excited to hear your story more about the work you’re doing at GenomOncology and and that it’s happening you

[00:06:43] know here in Cleveland because I I think that that particularly is awesome yeah I’ve seen the charts I think ni IH had one where they were showing that Mo’s law going down at an exponential you have costs or whatever per P flop or however they were measuring it and the same thing with the human genome right and it was starting with that 3 billion price point for the first genome and

Founding of GenomOncology

[00:07:01] going all the way down to a th000 and eventually $100 and eventually free right like that the goal is to have this stuff basically so easy to do that it’s basically free to do right and that that actually is the kind of the founding of the company was back in 2012 2011 at the end of 2011 there were a couple papers in nature where they you know they described you know how could we use

[00:07:22] genomic analysis and other you know OMC analysis to analyze a cancer patient’s tumor and then figure out what to do based on that information to then help treat the patient unfortunately it took six months to actually do all the bioinformatics and and the the technical work needed to do that analysis and so when our founder read that those papers it triggered oh I’ve been doing

[00:07:44] bioinformatics since the 70s before it was actually even called by informatics and we could go ahead and we could solve that problem in here in Cleveland with with folks he knew so that was really the founding of the and the joke at the time was really you know $100,000 genome and $100,000 analysis right so yeah great you can do that you great you can get the AC’s G’s and T’s off of a off of

[00:08:05] a Next Generation sequencer which was the devices that kind of emerged in the mid 2000s uh led by alumina and thermofisher but actually doing the technical work to understand what those mean that’s a good technical Challenge and we could we could work on that and that’s really the founding of our company so we we were talking before uh we we pressed record here about our our

Chasing the hardest problems

[00:08:28] mutual friend akos who used to work with you at at GenomOncology and I had asked him you know what question or topic should I be sure to to ask you about and uh he he brought up your philosophy around explicitly opting to Chase and follow the hardest problems and to me that felt like a great place to kick off you know understanding a bit more about what motivates you and how

[00:08:55] you came to be doing the the work that you’re doing and uh you know as a as a seg to to genome which we’ll get to but really in reflection on on your whole journey yeah that’s great uh yeah the hardest problems are definitely kind of my North Star it’s what I enjoy working on I enjoy working on the hardest possible problems I honestly get bored if it’s something that it’s kind of like

[00:09:14] oh make another website or what have you and and so in the 2000s I did a I did do some hard problems around e-commerce which was basically like how do you get money from a credit card or how do you ship something to a person and that those were hard problems in the 2000s they hard problems anymore right Shopify and and Amazon or whatever kind of solve so I was getting I was getting a little

[00:09:35] bit restless so when my friend manual started this company and told me about it I I approached him and said hey I gotta I gotta I gotta join you I got to work on this because this sounds like something interesting and something fun to do and so when it comes to building a company you know creating value software is an unbelievable you know marginal utility

[00:09:54] right so you can create something once and and then sell it unlimited amount of time which is great but to sell something in software that’s actually useful and something that actually solves a really hard problem that’s how you differentiate yourself that’s how you have a mo of any kind right and so the hard problems are really what I gravitate towards because

[00:10:13] I know that that’s where our clients are or our clients are going to need us so when we go into a new client and we help them with their existing problems it’s always looking around and talking to them and trying to figure out what are they struggling with what other challenges do they have that are adjacent to the things that we’ve already solved for them because I know

[00:10:30] that that’s a good direction to take our product so that because if they’re having those problems I’m sure other folks are having those same problems so let’s go ahead and try to tackle those and solve them because I also know that all these institutions that are not software development organizations by Nature right they’re they’re either cancer centers or reference Labs they’re

[00:10:50] busy solving they’re busy doing their work which is helping patients we can help them by creating software that’s easy to use and integrates within their their environments and makes their jobs better it makes their jobs better by letting them do more or do do more to higher quality so I think entrepreneurship almost by definition involves hard

[00:11:14] problems but I think you could find hard problems outside of Entrepreneurship and so I’m curious you know where your draw to entrepreneurship specifically as a as a means to to tackle hard problem stems from I think that that’s a patience problem of mine I don’t I’m not a very patient person so I I don’t do well with

[00:11:36] bureaucracy I don’t do well with meetings like if I’m going to be in a meeting I want to be in a meeting with like the people that matter for that particular problem and like let’s solve that problem and get out of the meeting right like uh so my my Mantra at one point was no meetings no bugs right like let’s not create bugs because bugs are failure demands that cause us our team

[00:11:56] to slow down and meetings or once again not they’re not useful in their own sense you do do them as a necessary evil and when it comes to hard problems I’m also very focused on essential complexity not accidental complexity right essential complexity is like this is a hard problem because it doesn’t exist in the world nobody knows what to do like That’s essential complexity but

[00:12:20] once you’ve solved it now it’s no longer essential it’s now whatever complexity you have left is how do we build the software how do we test the software how do we deploy the software how do we educate the people all good problems but you got to think about it in a way that you can put the thing to bed right like how do I solve this problem so it never shows up again and fortunately for me I

[00:12:41] I have a job where I can solve problems once and then they kind of go away I know that there’s other folks and other lines of work where it’s like you wake up every day and you have to solve the same problem every day it’s just new people or new types of spins on the same thing I’m just not wired to work in that kind of environment so entrepreneurship is perfect for me because it’s like

[00:13:00] small teams doing hard things moving quickly and I did that in the Consulting world the problem with the Consulting world was you do that for nine months you have a successful launch with your client and then they give you a handshake and say hey thanks for your work get out of here because you’re too expensive like we want we’re gonna have this done by our internal team or we’re

[00:13:19] gonna offshore it or something like there was no there was no ever re reaping of the reward of the of the value of the work that we created so with geology I build a product in 2012 and we’re still working with it 12 years later and and I know that that that the company is getting value from that that same thing and so the effort that I can put in i’ I’m getting the benefit of

[00:13:41] software which is that that marginal utility of write it want and use it forever yeah that’s real agency you know and and ownership and all the all the benefits of trying to solve these hard problems I want to do a little bit of of stage setting before we we get into the the heart of it but but kind of paint a picture for us maybe of of what Precision oncology looks like and cancer

What precision oncology means

[00:14:05] therapy and and bioinformatics and just the the space that that GenomOncology is operating within you know may maybe around the time even just before it was founding and some of the history that you know created the opportunity for for this whole undertaking in the first place sure so obviously cancer has a long history I’m not going to go into that what we do for cancer patients at

[00:14:28] this this point right are with when you have a tumor right it’s like you cut the tumor out you radiate the tumor or you poison the tumor right the cutting it out’s good hopefully you can cut the whole thing out and not have Mar you know the margins are clear or whatever what have you radiation’s fine because it’s targeted but once again you’re doing something damaging to your your

[00:14:49] body to do that and then chemotherapy is basically a poison and you’re the tradeoff is I’m going to try to PO poison you in a such a way that it poisons the cancer fast so than it poisons the rest of your body obviously not a great trade the promise and of Target what’s called targeted therapy Precision oncology and now immunotherapy as well those are all interesting topics

[00:15:11] subtopics that I can kind of unpack a little bit okay so first genomics fundamentally right there there can be Nuance to this but fundamentally cancer is uh a disease of the genome right you have some breakdown right there if you sunlight hits your skin in a very specific way and changes a DNA base pair from a t to an A on a certain chromosome on on you know the seventh

[00:15:36] chromosome at a specific position it’s going to change the mutation of a given Gene uh from from basically a e to sorry from a v to an E and this is this is a specific type of mutation that’s happened that’s going to cause that cell to continue to reproduce and not die so now the question becomes okay so you take that tumor you can actually cut the tumor out and then you can sequence that

[00:16:00] that that tissue you can do the DNA sequencing on it and understand okay we do see that change that goes from a v do here on that particular chromosome of the the of the cancer cells well there’s targeted therapies for that right there’s actually FDA approved drugs that will help people with melanoma uh fight that that particular type of of cancer yeah so that’s the Precision part right

[00:16:25] you’re looking at the data for a specific patient and then you’re giving a drug that’s targeted at that specific patient’s exact disease type not just class of disease another thing to recognize right is cancer is not one disease it’s thousands literally thousands of diseases that are in in in these ontologies of medical ontologies okay so that was targeted

[00:16:45] therapy we talk a little bit about sequencing genomic sequencing and there are different versions of this but genomic sequencing can actually look at small areas of the human genome just look at specific hot what are called hotspots areas of of Interest based on cancer that we know there are genomic tests or you know where you actually just look at

[00:17:05] the genes of specific cancer diseases and then there are whole xome where you’re looking at the protein and coding sections of the genomics and then the whole genome processing eventually we’re going to get to the point where we’re just doing testing everything because it’ll be so cheap and you might as well just test it all and then by testing it all you’ll also have a a baseline where

[00:17:24] you can actually say okay that person’s you know germ lines here and now as things evolve over time we’ll we’ll be able to learn how your genome and transcriptome and all these other types of omix evolve over time we’ll be able to measure it and we’ll be able to help treat patients uh more effectively and then the last part is the immunotherapy immunotherapy is really the idea of you

[00:17:47] know can we you know cancer is good at tricking your body right not only is it tricking you know causing it itself to grow but it also tricks your immune system to not fight it there are ways of like kind of amping up your immune system we’re giving it information to then have it more effectively fight the cancer so that’s another tool that has has really evolved over the last 10 or

[00:18:07] so years and so GenomOncology is really in the business of of helping providers um know what the options are providers are you know clinical oncologists and molecular Pathologists help their patients uh better and I can I can tell you more about the ecosystem of our tools and and knowledge that we have in our system no absolutely I i’ love to hear about those maybe as a a

Early days and first products

[00:18:32] way to approach those I love to understand though at the onset you know what were it feels like the the possibility the the breath of things that you could have possibly chosen to tackle is enormous at the beginning under you know this this umbrella problem space how did you know which problems to to start with on

[00:18:58] the Anis side yep and and and what did that actually look like at the beginning yeah so what it looked like at the beginning was what data was available so we actually were able to there was a project called the Thousand genomes project we downloaded all that data which was basically the full genomic you know makeup of a thousand patients and and then we actually built a prototype

[00:19:18] solution that we called our research application and basically what we were able to do was in 2012 on a Mac Mini and these Mac minis are way better than they were 12 years ago on a little Mac Mini you know the four $500 machine you can buy it an Apple store we were able to ingest the Thousand genomes and build a research application that let you you know not only look at each of the

[00:19:40] genomes that were in there but do these crazy you know what called set theory type analyses where you’re like okay show me all the variants that are in this subgroup of folks but not in this subgroup of folks and you know in and of itself that research app on top of you know open source data wasn’t that interesting but we were able to show that to

[00:19:59] Folks at different Convent you know these conferences that people go to and and eventually we were able to land three engagements as a small you know four person company where we were behind firewall working with the teams at Ohio State and Roswell Park in Buffalo and and pit in Pittsburgh and the theory was okay we’ll get in there we’ll work with really smart key opinion leaders and

[00:20:22] we’ll help them solve their problems like we’ll just figure out what their problems are and we’ll help them solve them and then hopefully over the next couple years we believe that people will start thinking about this stuff earnestly and how to use this information to help treat cancer patients that was kind of the that was kind of the bet or the approach was

[00:20:38] let’s go do something that we know how to do which is you know analyze data quickly because you know we’ve got that skill set and then use that to to basically make friends and then we had friends and and those friends would then tell us you know what they were really struggling with and the theory was that you know people would start using genomic information in my 2015 15 or

[00:20:59] 2016 to actually start making reports and treating patients that actually happened way faster than we expected so we ended up building our first commercial product in 2013 which is what we call our clinical workbench work pathology workbench which is really a high performance reporting engine and what it does is it basically at these labs and we’ve got about 50 Labs across

[00:21:21] the country that work with us they have these devices from alumina and thermofisher where they’re sequencing the tumor of data which generates the DNA of that information which is all just AC’s G’s and T’s what our software then does is take that information and help them quickly interpret it and report on it so the interpretation is first validating

[00:21:45] the quality of the run right did the sequencer do its job did it actually understand AC’s G’s and T’s and a and did it report because these things aren’t perfect did it report the information accurately yes it did great Next Step for each variant is this a good variant or a bad variant is it benign or is it pathogenic and we have a database that

[00:22:05] we curate and and an augment that has that information and then once they’ve made those decisions then they then it uses our knowledge base again to then say okay what are the therapies what are the clinical trials what other information can we tell this doctor or this pathology pathologist to about that that particular patient in their case so that was the that was the goal so at the

[00:22:27] very beginning was let’s build something that’s useful make friends and then try to figure out where their next problems are and hopefully be ready uh to take this thing and make it useful in in clinically helping patients it’s such a a sound strategy you know it I feel like people over complicate the the process left but yes

[00:22:49] if if you ask people what their problems are and they tell you and then you help solve them and you build trust correct that was uh it resonates as as my as part of my own Journey as well that’s that’s where it came from it’s just asking people for sure yeah and Paul gr I mean Paul I used to read the Paul Graham essays all the time and it’s just about

[00:23:11] getting out there and talking to people you know I’m a software developer I’m more introverted than most and you know going out and talking to folks is is not your not necessarily my first choice on on what to do every day but that’s the best way to learn it’s like find out what people what their problems are and then try to help them with their problems and that’s how you can actually

[00:23:27] make something that’s commercially viable with that being said did you feel there was a an over riding Vision at the beginning of what genome could become kind of the the North Star Beyond you know your personal one of of finding the hardest problems to solve but of what the kind of company genome could become and the impact it could have could be yeah I mean we’ve always had the

[00:23:51] vision of you know medicine is going to be it’s going to be completely rolu uiz by molecular data right the fact that we don’t use molecular data in almost every decision is is kind of silly at this point and one of the fundamental problems right now is that the EHR the the medical health record systems they’re mostly very good at like billing and they’re very good at like

[00:24:16] reimbursement type stuff and and insurance tracking but to make them useful in a clinical decision setting they have to be more om scale they have to actually scale to handle data like GenomOncology does so fundamentally we believe you know Healthcare is going to be molecular being able to do the the basic analysis of that information is is you know it’s a tricky computer science

[00:24:43] problem which makes it kind of fun and so you need to be able to design your system so that they scale and then from there then need to then figure out what are the different pieces parts that are going to be needed for a end solution and you know the thing that wasn’t necessarily obvious at the beginning was how much knowledge was going to play a part into this and right we’ll get into

[00:25:03] the AI stuff later but the knowledge that we’ve curated into our system which we’ve spent at least 100,000 man hours I don’t know the exact number of hours of you know building a knowledge base that powers these reports that we do we run these things called tumor boards and then just a lot of other different use cases where you know understanding with High Fidelity and high accuracy what are

Reducing the burden on doctors

[00:25:26] the best next steps for a patient that’s really what the GenomOncology system is designed to do well I’d love to you had mentioned doctors and the administra burden if you will of of the health record systems MH to to me it seems like perhaps the biggest risk would be that doctors maybe get lost in in all this and and kind of the inundation of information you know from from my time

[00:25:52] ahead at at my own company you know you come across this stat but it’s it’s very much grounded in a reality that doctor spend you know maybe 2third plus of their time on some administrative task and not clinically working with patients you know a shockingly high percentage of the time which always kind of felt a little ridiculous and ultimately unacceptable to me

[00:26:14] especially given this period of distress that specifically practitioners have been under since the the pandemic but I I’m curious how doctors are supposed to keep up with the vast amount of information that you know you’ve mentioned related to all these things that they need to keep track of from the the genomic tests to the clinical trials to new drugs you

[00:26:34] know in the wake of of thousands of different types of cancers and mutations can you speak a little bit to the world of of doctors and where GenomOncology has positioned itself best to navigate you know actually serving patients who are who are going through this stuff yeah so I described our pathology workbench which is our first product and one of the key things we do when we sat

[00:26:58] Des a new client up is help them design their first their report right so they’re doing this test they’re going to test 500 genes or 800 genes or the whole XOne or whole genome really at the end of the day their product the thing that they’re actually putting their label on which is a white label system we don’t put GenomOncology all over the report it’s their report the first thing they

[00:27:15] got to do is figure out what’s the report look like and there’s many considerations like one consideration is just completeness of information and Regulatory Compliance and other things that they need to do and and that’s why the reports can be like 20 Pages or 80 pages long right but really the most important thing is what do you put on that first page because the first page

[00:27:33] is actually what the doctor’s definitely going to look at right they might dig through it but they got they’re only gonna they might only have time and they might only have understanding of like how to read that first page so really designing that first page is critical for those new engagements in genology actually spent a few years back we spent a lot of time building our first you

[00:27:52] know redesigning our kind of default report every one of our clients gets their own copy of our software own copy of our report we’ve designed our system in a way that it’s like automatically deploys and and it handles all these customizations that I just talked about even though it’s you know at all these different clients but they can all have their own report but

[00:28:09] we have a default report that they can accept and I think I think like half of the clients just take our our default report and they kind of just had their colors and logos and stuff to it and the reason why they do is because we put a lot of thought into making that first page really good and that first page has to be really good from like you know what are the most critical information

[00:28:25] well most critical information is what biomarkers did I find what biomarkers of Interest did I find and then the other thing that’s really interesting is what biomarkers of Interest did I not find because there are specific biomarkers that people are looking for because they know oh I can go give this immunotherapy or this targeted therapy so you get to tell them up front you know they didn’t

[00:28:44] have this Gene this Gene was clear so making sure you know that about your client and what their you know what their goals are and their intent is is really critical you don’t want them having to read an 80-page report to figure out what didn’t what didn’t they find and so that that that’s a great lesson right because that because it gets back to the whole goal of you know

[00:29:02] once again making this stuff as as fast as possible for the ordering oncologist so they can help patients same thing on the molecular pathology side for the folks using our software to make this report we’ve really worked really hard to understand you know what slows them down what what information do they want at their fingertips so they can make the important decisions because at the end

[00:29:21] of the day we don’t make any decisions we’re decision support software we are not decision- making software our clients are the ones that through their electronic signature are the ones actually making the final decision and and signing out the report and so that’s that’s a key thing so then your other part of your question was you know how do we reduce the burden on doctors so

[00:29:41] we’ve had this whole EHR thing over the last 20 years now this experiment that started in good with good intention yeah and the problem is it’s now you know you go to the doctor and you sit there and you can see that the doctor is spending half the time looking at the computer and typing right like that’s the worst possible solution and I think everybody recognizes that and there’s there’s even

[00:30:01] more burnout from doctors lots of lots of articles in the Atlantic and New Yorker about how doctors are all burned out so I think AI has a lot of opportunities there I’m not in this space at all this part of AI space and that that part of the AI space is what’s called ambient technology right so basically listening to the doctor and the patient having a conversation

[00:30:20] summarizing the information and turning that into some notes and then that those notes can then be edited right in dra from draft mode edited by the doc and and sent into the EHR which is a great thing and then there’s other honestly other like billing and other types of of push-ups these docs have to do where instead of like having to write a letters of necess of medical necessity

[00:30:41] or whatever the letters are called you know having them write those themselves they can you know basically ask chat GPT okay I have this patient here’s the situation write me a letter right and here’s the letter for you and now send it off so that’s another you know another area but now we got the now you’re going to have the the bots on both sides of the thing you know

[00:30:59] basically reading and rejecting automated notes yes for sure lay of the land is brought to you by impact Architects and by 90 as we share the stories of entrepreneurs building incredible organizations in Cleveland and throughout Northeast Ohio impact Architects has helped hundreds of those leaders many of whom we have heard from as guests on this very podcast

[00:31:23] realize their own visions and build these great organizations I believe in Impact architects and the people behind it so much that I have actually joined them personally in their mission to help leaders gain Focus align together and Thrive by doing what they love if you two are trying to build great impact Architects is offering to sit down with you for a free consultation or provide a

[00:31:44] free trial through 90 the software platform that helps teams build great companies if you interested in learning more about partnering with impact Architects or by leveraging 90 to power your own business please go to ia. lay ofthe land. FM the link will also be in our show [Music] notes so I’d love if you I I definitely

Evolution of the product line

[00:32:08] want to unpack the artificial intelligence side of this with you and and maybe where a lot of this is headed but take us through kind of the the evolution of of GenomOncology as a as a company throughout throughout this kind of product offering so we started in 2013 with mythology workbench and we grabbed some early adopters right and we saw real quick uptake signed a lot of

[00:32:30] new deals and I think once we actually hit the the you know Clayton Christensen whatever the the valley is between your early adopters and and your late adopters I I forget the those those terms from that book you know we we hadit a little pause there in the middle of of the 2010s and we started looking for other opportunities like how how else can we take this technology to

[00:32:50] Market and and where else could we be of use and so the cancer center was a was an obvious next stop so one of the key things that I ended up doing to rebuilt this pathology workbench it does reporting there’s a knowledge base that’s part of that so we actually have a system to let them curate or let our team curate knowledge to then populate that report we had folks that were

[00:33:12] coming to us and saying hey we we just want that knowledge we don’t actually want to do a we’re not a Pathology Lab we don’t need to do reporting can you just give us access to that knowledge and I realized oh there’s another product here like we can’t help them right now but what’s that product so the first product I built was what what we call our API or API Suite application

[00:33:30] programming interface which means a computer now can just talk to our knowledge base and ask questions to it and what that unlocks is our clients can now build solutions that now have the GenomOncology inside of it right the Intel old Intel Inside Mantra where they can actually use our knowledge base using whatever bioinformatics you know database they have or knowledge that

[00:33:52] they have or whatever use cases they have they can now use our knowledge base to power whatever solutions they have and then we have quite a few clients that do that and then as well uh we have now also built our own uh Solutions on top of that API so we have a solution for what’s called a tumor board and this came out you know the late 2010s the tumor board is what

[00:34:14] happens at a you know large Cancer Center where there the really hard cases are discussed maybe every week or every month or so depending on the size of the the cancer center and so what happens at a tumor board is usually it’s a cross disciplinary uh group surgical pathology that kind of thing all kind of all on one web X or one zoom and they’re all talking about hard cases well our

[00:34:39] software helps because it can basically do all the Google searching for everybody right rather than having everybody on their phones Google searching our software is there driving the presentation you know dealing with the information making high level you know recommendations of content and then the the folks the experts in the room can

[00:34:57] then look at that information dismiss information highlight selct information and make a report make a recommendation for the treating oncologist and then we also have treating tools for treating oncologists and clinical trial folks less than 5% of cancer patients end up on clinical trials and that’s bad for a couple of different reasons one that means we’re not furthering along the

[00:35:19] knowledge collectively on what which of these drugs could could be helping cancer patients quickly yeah um and also this an access problem right that the folks who do get on clinical trials more often than not have you know privilege be you know relative to the folks that don’t get on clinical trials so we need to open up access to to these clinical trials our software can actually help

[00:35:43] with clinical trial with called Clin clinical trial matching and Recruitment and enrollment and AC cruel through a variety of different ways matching a patient to a trial is actually a very difficult task because it’s not only is the trial open but is the patient have the right disease type is the patient healthy

[00:36:03] enough is the patient have these specific biomarkers has the patient not had or has had some prior treatment all those types of criteria need to be evaluated and matchmake against the actual trials themselves and you can have more than one trial at a time keeping that information in your head is not really possible what our software does is actually does that problem it

[00:36:23] actually can solve that problem so we can and we can do it at scale meaning if you have a you know Cancer Clinic where you’re you know taking in 100 patients each week and doing the genomic testing our software can be running in the background analyzing patients and identifying potential matches hey this you know we have 20 trials ongoing we got these 100 patients of these you know

[00:36:45] whatever 2,000 combinations let’s go ahead and we’ve highlighted you know 15 things to go to go look at so rather than having to have a human do that manually they can use our software to kind of do that matching and so that was the next evolution of our software is really getting into the clinic and using our knowledge base to power these different uh use cases and you know

Measuring outcomes and efficacy

[00:37:05] there’s we’re continuously challenged by clients with you know new things that they’re working on and and how molecular information and biomarkers can uh be used more effectively to help patients and care and care of the patients how do you think about efficacy in in outcomes like ultimately you know relative

[00:37:28] to where a doctor might be without the ability to use GenomOncology what is the effect of introducing what you guys have built into the equation great question problem is we don’t have access to all that data right so the that’s the challenge so it’s so we’ve done you know trials guess not necessarily the right words not not typically a clinical trial but it’s

[00:37:51] basically a research initiative with clients you know using our software and understanding you know how does our software help in the ecosystem of decision support and and it does right it’s you know makes their jobs doable to a degree right we we’ve done backwards analysis where it’s like oh we you know we did clinical trial matching for a year before bringing in GenomOncology

[00:38:15] now with GenomOncology we we would have identified you know 50 or 100 more patient than we could identify just by hand so there are there are you know discussion points like that but unfortunately it’s really difficult at a Cancer Center to really understand the Final Destination right tracking progression free survival is is the terminology that they use in clinical

[00:38:37] trial without a clinical trial because that’s the mechanism by which um you know these things can get FDA approvals for drugs for instance unfortunately genology can’t afford to right now afford to run a clinical trial at that kind of scale but we’d certainly be interested in in working with institutions that are are looking forward to doing Precision oncology

[00:38:58] based software interventions and doing it in a systematic way right because you kind of have to do it in parallel where you do here’s you know here’s a thousand patients that we’re going to S you know serve this way versus a thousand patient that we’re not going to and to actually get that to that raw data to to do that analysis but in general you know we look at our success as being you know we

[00:39:19] don’t lose clients right so you know 98% retention rate of clients and really high net provider scores I guess what they’re called NPS scor so we we look for those types of of metrics uh to help guide us and make sure we’re doing the right stuff uh and then we’re always also interested in doing you know deep studies with with folks that are interested in that so

Implications of AI

[00:39:42] what are the implications of AI on the the work that that you’re doing great question it’s where I spend a lot of my time thinking right these days uh so back in November 2022 right chbt 3.5 came out I said oh crud do I have a business anymore like let me go figure out what this thing actually knows and I figured out pretty quickly that it doesn’t doesn’t uh it does fall down on

[00:40:05] on complex stuff like ours and I’ll explain what I mean in a minute and then March came and that was when GPT 4 came out and I had the same same worry and I I was relieved to see that this thing’s not going to put us out of business anytime soon but I did realize oh this is a really good coding assistant so I use it every day for coding and I use now I’m using Claude anthropic as well

[00:40:26] every day for coding so that’s good so the question is in our business why can’t chat GPT or its equivalent GPT 5 right that’s going to come out in a few months we think why can’t it do what genology does and it really comes down to a couple different problems first the way that they’re trained so GPT right generative pre-trained model is created by predicting the next

[00:40:54] token which is a really neat trick like the fact that that works at all is amazing to me I wouldn’t have guessed that uh so they do predict the next token on on all the words in the internet right whether they had the copyright right to do it or whatnot I’ll let the courts decide that and now you’ve got this this thing that can predict the next token and that’s what

[00:41:12] it does it basically has a model of the world and can tell you what the next most plausible word is based on all the previous words and through that it has knowledge incorporated into it which is which is cool the problem is one it’s a it’s a black box we actually don’t know what it actually knows we don’t know what the data was trained on the people who have created it can’t tell you why

[00:41:38] it does what it does it hallucinate right it will just make up things it doesn’t know when it’s lying because it actually isn’t lying it doesn’t know what the truth is it just knows how to create the next token you cannot build a clinical decision support system off of something that’s a black box that hallucinates hallucinate Ates the the word that they use for making stuff up

[00:42:00] um they have bias encoded into them right the world has bias the internet has bias therefore you know no matter how much they work on trying to kind of clean up the bias there’s going to be bias in those Solutions that’s understandable they’re not up to date right so they have a cuto off going at some point they stop training it because they have to get it to Market and the

[00:42:20] way that they work is you can’t just be like oh let me go add some more stuff I’m going to go add February to the January data it doesn’t really work that way and then it’s not genomic scale meaning yes it has all the text of the internet but it doesn’t know all the text of the internet and it certainly doesn’t know all the variants in the human genome that are possible in what

[00:42:38] they mean so at that point those are the kind of like the the foundation of why they don’t replace what GenomOncology does but they are still super useful and how they’re most useful in my world the way I’m thinking about them and you know this might change this is April 9th 2024 so I you know keep that in mind when you’re listening they are really you know one

[00:43:02] phrase that I’ve heard is word calculators right they understand what people are saying they understand my intent they know what I they know and if you can get good at prompting them which is called prompt engineering by some folks you can get good at prompting them what is prompting you’re basically telling them clearly and concisely what you have and what you want and if you

[00:43:22] can tell them that those two things it can kind of fill in the middle for you not even tell you why but you might want to actually ask it why like have it explained to you its thought process and it will do that and it’ll get you’ll get a better result actually if you do that and then it will generate an answer and so that’s called prompt engineering and I’ve just and by

[00:43:40] saying what you have and what you want you’re giving it examples in the the lingo in LM in the AI spaces few shot learning you’re basically giving it a few shots of of of examples of what you want and then you’re asking for it to to give you an answer and by having it explain itself where you have it explain its thought process you’re giving it a chance to kind of work out the Kinks of

[00:44:04] its thinking just like humans do right I’m kind of talking right now and I’m kind of trying to express myself well giving the large language model time to express itself that’s called Chain of Thought where you’re basically having it kind of talk out its thought process and then it actually generates a result you get much better results when you do those things so you know that’s kind of

Building an oncologist assistant

[00:44:25] the background and what I’m doing is I’m actually integrating it with our API and so what a lot of folks do is they do a thing called rag or retrieval augmented generation which is the idea of the chatbot is given text from a database right you basically you know if someone ask a question you use that question to go find relevant content from your database you stick that content in your

[00:44:46] prompt and then you have the large language model reason over that text and then response it just does a better job when you do that another technique that you can do is what’s called tool usage which is you give it a tool you say okay here’s a tool or five tools or 10 tools you can use those tools as you need to just remember large language model you’re not good at math you’re not good

[00:45:07] at knowing what the weather is you’re not good at knowing what genomics are you tell it that through prompting what it’s bad at and then you say okay you’re bad at those things but you’re really good at understanding what the the person you’re talking to wants so try to understand what they want and then use these tools to help them that’s your whole goal and if you explain it to the

[00:45:26] to the chat bot like that and then you give it the access to the tools you can actually get something that’s that’s almost a product right it’s still honestly it’s early days a lot of these things are demos that’s why chat Bots are kind of still the the best product because the human in the loop is responsible for for figuring out what to do with it and so that’s what I’m

[00:45:46] building I’m I’m actually building provider you know for an oncologist an oncologist tool let them have a conversation with the chat bot that has access to our API to both retrieve back you know trials therapies and other information for a patient and and that’s the kind of the long-term vision of of of where we see you know our knowledge base and I think AI is actually a great

[00:46:10] compliment for genology in that this stuff’s hard building user interfaces that can deal with all of the clinical information the bioinformatic information trials the therapies all that stuff it’s very overwhelming so think about the most complicated user interface you have have to use at work or whatever and you you can see you can see what I mean having an experience

[00:46:35] that where that complexity is kind of hidden away from you but can then be selectively retrieved into chat experience or you know eventually it won’t be just chat it’ll be more Dynamic and you you know uid driven having this agent or assistant help you kind of navigate that that world would be so so great for GenomOncology because the thing our our our desired and clients

[00:47:00] they don’t have time to learn another software product they don’t want to go learn how to use some complicated uh you know it’s like saying oh you have to use Adobe Photoshop to do something to do your job it’s like good luck I’ve never I’ve tried to use Adobe Photoshop it’s way too complicated for me I couldn’t I couldn’t use it so doing the same thing with a doctor is really challenging so

[00:47:19] I’m excited that you know large language models especially you know maybe the next year or so will really let us let our our clients truly take advantage of all the knowledge that we’ve curated over the last 12 years no I mean it resonates very deeply I mean ultimately from any product’s perspective right the user is trying to do a job they’re trying to solve a

[00:47:42] problem trying to do a task and their their job is to accomplish that task and not to become an expert in in whatever it is that that you’ve built and they want to learn my software and I don’t blame them I would not want to learn learn our software either the molecular pathologist has a very important job which is like make a report in 15 minutes if they didn’t use our software

[00:48:02] it would take them six hours so the trade-off to them is night and day where in the oncology space it’s well I could not you learn that software and then basically just keep doing what I’m doing or I could spend six hours to try to learn the software and make my and make my decision- making a little bit better well that’s that’s a that’s probably a good trade to make

[00:48:23] sometimes but you don’t know which software to do that with right so yeah you basically spend your whole they don’t have enough time as it is right the job is not to become an expert in the software the product must solve a problem exactly well and what’s really interesting to me about about all your your perspective there is I mean you mentioned at the at the beginning you

Convergence of sequencing and AI

[00:48:42] know we talked a little bit about mors law rights law the exponential cost declines of this technology and maybe the the trit saying in in your industry that you know we’re down to $100 sequencing now but it’s thousands of dollars for analysis it feels like the convergence of the the technology from the the sequencing side and the AI you know the the Confluence of those two

[00:49:04] things might be very powerful here yeah no agreed and and one thing if people are going to remember one thing large language models don’t search you feel like they’re searching like you’re asking a question it feels like they’re searching information to like respond to you it’s not how they work fundamentally they’re not searching they’re predicting the next token so our system is really a

[00:49:26] search Eng engine it’s just a very complicated search engine that lets you search by a patient like basically a patient and all their DNA and all that stuff is basically a query and now you’re quering our knowledge base bring back the relevant content and now the large language model can then reason over that content what that large language model is doing is basically

[00:49:44] replacing learning how to do our search in our software does that make sense yeah maybe I think so who what on the horizon in the space is most exciting for you when when you think about you know genome in the future and and and success and the kind of impact that you hope to have looking back in retrospect what do you see coming you know given all all

[00:50:10] these things and how and how in reflection actually you know you you even mentioned earlier as well the the progress is is maybe even faster than than we’ve expected it to be well you know I think at the end of the day all of healthcare is going to be I want GenomOncology to be a core part of that at least from a knowledge perspective if not from a bioinformatics

[00:50:33] and Analysis perspective I think we we have the software and the the people and the expertise help people do that and then the goal is how do we make it so the provider right that’s the industry lingo for the oncologist or the treating physician how do we reduce the administrative burden the technological burden Etc so that they can spend their time thinking about the patient and

[00:50:55] bringing empathy and reasoning and their judgment to helping that patient as quickly and painlessly and effectively as possible that’s the goal I’ve lost lots of folks to cancer in my life that I love I want to help be part of that solution to help reduce that pain and suffering in the world so so that’s that’s the that’s the real goal the way I’m tackling that goal

[00:51:22] is through what I’m good at right I’m not good at the stuff that oncologists do good at the stuff I do which is solving her technical problems and and I think that the things that GenomOncology can do is just continue to scale because it’s going to become a scaling issue there’s so much data when it comes to genomics and other other stuff so you have to you have to have the right

[00:51:40] algorithms honestly to be able to parse and process and and index and search this information and then how do we leverage these new amazing and sometimes scary artificial intelligence tools to be an amplified and an augmenter of expertise right we’re all experts at something every human right is an expert at something and these things if designed and used appropriately can be

[00:52:08] all our assistance and make us all 10x better at whatever we are an expert at it doesn’t have to all be about you know generating images and silly songs and and other stuff that nobody really wants it’s it’s really about helping helping Humanity you know solve our most difficult problems and I think uh and I think that’s that’s where you know GenomOncology is going to be a major

Advice for cancer patients

[00:52:33] contributor to you you had mentioned your your one takeaway for for AI I’m curious if you were to offer a a similar takeaway for oncology and cancer you know having spent the last decade working in this in this field what is something that you wish more people understood about it that maybe we don’t well if you have a loved one that has cancer here’s my number one advice for

[00:53:00] you find an expert not to you know bmer folks that are uh working in the community but if your doctor if you have melanoma and the doctor you saw you’re seeing just saw a cancer patient before you and a patient with gastro intestinal cancer after you they are not an expert in your disease and as of right now the best thing you can do for yourself is to go find the

[00:53:27] person who is an expert if you can afford it that’s the number one piece of advice the second piece of advice is make sure you’re getting and asking about genomic testing not everybody does it not all the doctors do it you think that it would be standard of care but it’s not insurance companies don’t necessarily pay for it but there’s there are angles around it call the advocacy

[00:53:47] groups for your disease type we work with a group called The Pancreatic Cancer Society and they will help any pancreatic cancer patient using our software find a clinical trial find a doctor and they will help you analyze the information that’s that’s presented in your genomic reports so you know you have to kind of you have to be your number one Advocate to yourself

[00:54:11] unfortunately and then as far as genomics and bioinformatics and and and that information and that it’s a great it’s a great field I think that that’s um you know I think that there there’s still lots and lots to work on and so if you’re a young person that’s that’s looking for you know a direction in life and you’re interested in really hard problems look into systems biology look

[00:54:35] into bioinformatics look into these different Technologies because I think uh just being a computer scientist like myself I’m a computer engineer I don’t necessarily think that that’s the the right idea for the future I think you’re going to want to have an intersection with you know two or three other skill sets whether it’s being able to sell or being able to write software or being

Lessons from company building and Cleveland

[00:54:55] able to you know manage and understand genomic data those are all good skills to have and being at the intersection of something is a good way to have you know kind of uh being more distinctive in the marketplace and standing out I I’ll pull on your advice to a younger self in in the spirit of of that reflection what would you say is kind of the earned

[00:55:17] wisdom that you have from the entrepreneurial process and the company building side and the the business side of this so many different things be patient right things take a long time we’re still I mean I still think of myself as being in a startup we’re 12 years old now at this point we’re you know we’re still I still feel like we’re startup we’re very entrepreneurial and

[00:55:36] and and willing to fight thing fight for the the next deal and and get stuff done very collaborative environment so don’t expect it to to to work overnight you startups only fail when somebody finally gives up right that’s that’s how they actually technically fail so I think that that those are good lessons to learn but I think right now now there’s never been a better opportunity right

[00:55:59] with if you’re ambitious and you’re you know looking to solve hard problems and you’re looking to learn new things that’s a great way to to go if you’re just interested in making money or what have you probably should figure out a big corporate job and that’s that’s probably a smarter path but from an entrepreneur perspective there’s going to be that people talk about like the

[00:56:19] three person billion dollar company right there’s going to be more Instagrams than there you know mid Journeys and all these other small companies that just have enormous scale so those are those are those are interesting things and maybe stay out of healthcare right Healthcare is a tough a tough business to crack so maybe you want to stay out of healthcare and go

[00:56:37] more B Toc or something yeah unfortunately because we there’s a lot of problems in healthcare there’s there’s a lot it’s it’s tough I mean the nice thing about when you’re in software and Healthcare takes a long time to get in the door but once you’re in the door it’s usually harder for them it’s harder for them to kind of like say oh no we don’t Journey right like you’re

[00:56:58] integrated you’re you’re you’re part of their ecosystem you’re delivering value you know a lot a lot stickier of a business well what do you feel is is left unsaid I mean I’m sure there are many things but you know in reflection on your your personal journey and and building GenomOncology that a lot Cleveland’s a great town so we didn’t talk about Cleveland

[00:57:20] at all I do love Cleveland and and I you know I hope I hope to see more investment more interest in technology and Innovation within Clevelands come to the Cleveland and Big Data meetups I think that those might be emerging I heard a rumor at least so there’s a you know good local development Community technologist community so even if you’re not a software developer and you’re

[00:57:44] interested in those topics uh look them up on meetup.com and otherwise you know find me on social media and and let me know if you have any questions perfect well I think we can we can bookend it then with our traditional closing question which which is about Cleveland MH for for a Hidden Gem in the area you know something that that other people should know about that maybe they don’t

[00:58:04] so Lakewood in general so I moved to Lakewood 25 years ago and I love it you’re I know you’re Ohio city which is which is pretty solid I might have to might reconsider that but book book Brothers in Lakewood that’s my that’s my go-to bookstore so they actually have a it’s like a used bookstore on Madison and he’s got he’s curated it like the the owner of the the store store is

[00:58:26] definitely curated you can tell that it’s not your average you know kind of Half Price Book type store it’s it’s got interesting stuff interesting reads that you wouldn’t find anywhere else oh I love it that’s perfect uh will and I just want to thank you again for for coming on for sharing your story it’s uh yeah it’s very it’s long been fascinating to me this kind of work and

[00:58:49] I love that that you guys are are building it here that’s great thanks jef really nice to meet you if people had anything that they wanted to follow up with you about where where’s the best place for them to do it so I’m on LinkedIn last name is Mo m a r e r um I’m also on Twitter and you can also just email me at Ian GenomOncology.com

[00:59:09] anytime perfect that’s all for this week thank you for listening we’d love to hear your thoughts on today’s show so if you have any feedback please send over an email to Jeffrey lay ofthe land. FM or find us on Twitter atlay of theland or at Stern he J Fe if you are someone you know would make a good guest for our show please reach out as well and let us know

[00:59:33] and if you enjoy the podcast Please Subscribe and leave a review on iTunes or on your preferred podcast player your support goes a long way to help us spread the word and continue to bring the Cleveland Founders and Builders we love having on the show we’ll be back here next week at the same time to map more of the land the L of the land podcast was developed in collaboration

[00:59:52] with the up company llc at the time of this Rec recording unless otherwise indicated we do not own Equity or other Financial interests in the company which appear on the show all opinions expressed by podcast participants are solely their own and do not reflect the opinions of any entity which employs US this podcast is for informational purposes only and should not be relied

[01:00:13] upon as a basis for investment decisions thank you for listening and we’ll talk to you next week

Building GPT Actions with FastAPI and Pydantic

Tue, 16 Apr 2024 00:00:00 GMT

Episode Introduction

[00:00] Do you know what custom GPTs are? They’re configurable and shareable chat experiences with the name, logo, custom instructions, conversation starters, access to OpenAI tools, and custom API actions. And you can build them with Python. Ian Maurer has been doing just that and is here to share his experience building them. This is Talk Python To Me, episode 456, recorded January 22nd, 2024.

[00:26] Welcome to Talk Python To Me, a weekly podcast on Python. This is your host, Michael Kennedy. Follow me on Mastodon, where I’m @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org. Keep up with the show and listen to over seven years of past episodes at talkpython.fm. We’ve started streaming most of our episodes live on YouTube.

[01:01] Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode. This episode is sponsored by Sentry. Don’t let those errors go unnoticed. Use Sentry. Get started at talkpython.fm/sentry.

[01:17] And it’s also brought to you by Neo4j. It’s time to stop asking relational databases to do more than they were made for. Check out the sample FastAPI project and see what Neo4j, a native graph database, can do for you. Find out more at talkpython.fm/Neo4j. Ian, welcome to Talk Python To Me.

Catching Up Since 2018

[01:39] Hey, Michael. Good to see you again. Yeah, great to see you again. It has been a little while. It seems like not so long ago. And yet, when I pull up the episode that we did together, sure enough, it says March 7th, 2018.

[01:54] Wow. Years are short. Years are short. They go by really fast. They sure do. So back then, we were talking about Python and biology and genomics.

[02:04] And it sounds like you’re still doing genetic type things and still doing Python and all that kind of stuff. For sure. Yeah, definitely. We work for a company called GenomOncology. We do precision oncology software, helping folks make sense of genomics and trying to help cancer patients. That’s awesome.

[02:21] There’s different levels of helping people with software. On one level, we probably have ad retargeting. On the other, we’ve got medical benefits and looking for helping people who are suffering socially or whatever. So it’s got to feel good to write software that is making a difference in people’s lives. That’s right. I did spend a lot of the 2000s making e-commerce websites and that wasn’t exactly the most fulfilling thing.

[02:48] I learned a lot, but it wasn’t as exciting as what I’m doing now, or at least as fulfilling as what I’m doing now. Were those earlier websites in Python? I was all Java for the most part. And finally with this company, I knocked out a prototype in Django a few years ago. And my boss at the time was like, you did that so fast, you should do some more stuff in Python. So that’s kind of how it evolved.

[03:10] And now basically most of our core backend is Python and we use a little bit of Svelte for the user interfaces. Beautiful. It’s easy to forget, like five years ago, 10 years ago, people were questioning whether Python should be something you should use. Is it a real language? Should you really use it? Is it safe to use?

[03:27] Maybe you should use a Java or a C# or something like that because this is a real project. It’s interesting. You don’t hear that nearly as much anymore, do you? I grew up with Boston sports fans and it was like being a Boston sports fan was terrible for the longest time. And now it’s like, okay, we don’t want to hear about your problems right now. And same thing with Python.

[03:44] It’s like, I like Python. It’s like, yeah, great. So does everybody else in the world. So yeah, it’s really not the issue anymore. It’s now it’s not the cool thing to play with. So now you got to go to Rust or something else.

[03:53] You know what? Shiny. LLMs are shiny. LLMs are very shiny for sure. Yeah. We can talk about them today.

How Ian Uses LLMs

[03:59] Yeah, that sounds great. Let’s do it. First of all, we’re going to talk about building applications that are basically powered by LLMs that you plug into, right? Yep. Before we get into creating LLMs, just for you, like what is, where do LLMs play a role for you in software development these days? Sure.

[04:18] So, you know, like everybody else, I mean, I had been playing with, so I do natural language processing as part of my job, right? So using spaCy was a big, a big part of the information extraction stack that we use because we have to deal with a lot of medical data and medical data is just unstructured and has to be cleaned up before it can be used. That was my exposure. I had seen GPTs and the idea of like generating text, just starting from that didn’t really make much sense to me at the time. But then obviously like everybody else, when ChatGPT came out, I was like,

[04:47] oh, I get this now. Like this thing does, you know, it can basically learn in the context and it can actually produce something that’s interesting and you can use it for things like information extraction. So just like everybody else, I kind of woke up to them, you know, around that time that they got released and I use them all the time, right? So ChatGPT 4 is really what I use. I would recommend if you can afford the $20 a month, it’s still the best model that there is as of January 2024.

[05:10] And I use that for coding. I don’t really like the coding tools, the co-pilots, but there, you know, there’s definitely folks that swear by them. My workflow is more of, I have a problem, work with the chatbot to try to like, you know, think through all the edge cases and then think through the test case, the tests. And then I think through the code, right? And then the actual typing of the code, yeah, I’ll have it do a lot of the boilerplate stuff,

[05:32] but then kind of shaping the APIs and things like that. I kind of like to do that myself still. I’m kind of old school, old school. I guess I’m old school as well because I’m like right there with you. But for me, I don’t generally run co-pilot or those kinds of things in my editors. I do have some features turned on, but primarily it’s just really nice autocomplete.

[05:53] You know what I mean? Like it seems like it almost just knows what I want to type anyway. And that’s getting better. I don’t know if anyone’s noticed recently. One of the recent releases of PyCharm, it starts to autocomplete whole lines. And I don’t know where it’s getting this from, and I think I have the AI features turned off.

[06:10] At least it says I have no license. I’m guessing that means they’re turned off. So it must be something more built into it. That’s pretty excellent. But for me, I find I’m pretty content to just sit and write code. However, the more specific the unknowns are, the more willing I’m like, oh, I need to go to ChatGPT for this.

[06:27] Like, for example, like how do you use Pydantic? Like, well, I’ll probably just go look at a quick code sample and see that so I can understand it. But if it’s I have this time string with the date like this, the month like this, and then it has the time zone like that, how do I parse that? Or how do I generate another one like that in Python? And here’s the answer.

[06:48] Or I have this giant weird string, and I want this part of it as extracted with a regular expression. And I want to… Regular expressions, I was just going to say that. Oh, my gosh. You don’t have to write another one of those. Yeah, it’s great.

[06:59] Yeah, it’s pretty much like, do you need it to detect the end of a line straight to ChatGPT? Not really. But, you know, it’s like almost any level of chat, a regular expression. I’m like, well, I need some AI for this because this is not time well spent for me. But yeah, it’s interesting. Yeah.

[07:12] One big tip I would give people, though, is that these chatbots, they want to please you. So you have to ask it to criticize you. You have to say, here’s some piece of code. Tell me all the ways it’s wrong. And you have to also ask for lots of different examples because it just starts to get more creative, more things that it says. It really thinks by talking, which is a really weird thing to consider.

[07:31] But yeah, it’s definitely some things to keep in mind when you’re working with these things. And they do have these really weird things. Like if you compliment them or if you ask it, you sort of tell it, like, I really want you to tell me. It actually makes a difference, right? It’s not just like a search engine. Like, well, of course, what does it care?

[07:46] You put these keywords in and they come out. Like, no, you’ve kind of got to, like, know how to talk to it just a little bit. I’ve seen people threatening them or, like, saying that someone’s being held ransom or, you know, I like to say my boss is really mad at me. Like, help me out here, right? And, like, see if it’ll generate some better code. You’re not being a good user.

[08:02] You’re trying to trick me. I’ve been a good chatbot and you’ve been a bad user and I’m not going to help you anymore. Yeah, right. That was actually basically a conversation from Bing in the early days. Yeah, the Sydney episode. Yeah, that was crazy, right?

[08:14] Super funny. How funny. All right. Well, I’m sure a lot of people out there are using AI these days. I think I saw a quote from, I think it was from GitHub saying over 50% of developers are using Copilot. For sure.

[08:26] Which is crazy, but, I mean, not that surprising. 50% of the people are using Autocomplete. So, I guess it kind of, kind of like that, right? They’re great tools. They’re going to keep evolving. There’s some other ones I’m keeping an eye on.

[08:36] There’s one called Console, which just takes a different approach. They use some stronger models. And then there’s a website called Find, P-H-I-N-D, that allows you to do some searching, that they’ve built their own custom model. Really interesting companies that are doing some really cool things. And then Perplexity is like the search replacement that a lot of folks are very excited about using instead of Google. So, there’s a lot of different tools out there.

[08:57] You could spend all your day just kind of playing around and learning these things where you got to actually kind of get some stuff done, too. Yeah, you got to pick something and go, right? Because with all the churn and growth and experimentation we got, you probably could try a new tool every day and still not try them all, you know? Just be falling farther behind. So, you got to pick something and go. And go, yep.

Choosing an LLM API

[09:15] Let’s talk about writing some code. Yeah, the next thing you’re going to do after you, you know, use a chatbot is to, you know, hit an API. Like, if you’re going to program an app and that app is going to have LLM inside of it, large language models inside of it, APIs are pretty much the next step, right? So, OpenAI has different models that are available. This is a web page that I just saw recently that will actually, you know, compare the different models that are out there. So, there’s obviously the big guy, which is OpenAI.

[09:41] And you can get that through Azure as well if you have a Microsoft arrangement. And there’s some security reasons or HIPAA compliance and, you know, some other reasons that you might want to talk through Azure instead of going directly to OpenAI. I’d defer to your IT department about that. Google has Gemini, which they just released the Pro version, which I believe is as strong as 3.5, roughly. That is interesting because if you don’t care about them training on your data, if, like, whatever you’re doing is just, like, not super proprietary or something you’re trying to keep secret, they’re offering free API access, I believe 60 words per minute, right?

[10:17] So, basically, one a second, you can call this thing and there’s no charge. So, I don’t know how long that’s going to last. So, if you have an interesting project that you want to use in a large language model for, you might want to look at that. Yeah, especially if it’s already open data that you’re playing with. Exactly, right. Or data you’ve somehow published to the web that has certainly been consumed by these things.

[10:36] And these models are going to train on it, right? That’s the trade, right? They’re trying to get more tokens, is what they call it, right? The tokens are what they need to actually make these models smarter. So, everyone’s just hunting for more tokens. And I think this is part of their strategy for that.

[10:48] And then there’s also a Claude by Anthropic. And then after that, you get into the, you know, kind of the open source APIs as well. There’s some really powerful open source ones out there. Yeah, so this website, yeah, this is DocsBot for people listening. DocsBot.ai. And is it sole purpose just to tell you price comparisons and stuff like that?

[11:07] Or does it have more than it? I assume this company’s got some product. Unfortunately, I don’t know what it is. I saw this link that they put out there. And it’s a calculator. So, you basically can put your tokens, how many tokens.

[11:17] There’s input tokens and there’s output tokens, right? So, they’re going to charge more on the output tokens. That’s for the most part. Some of the models are, you know, more equal. And then what they do is, if you can figure out, like, roughly how big a message is going to be, both the input and the output, how many calls you’re going to make, you can use

[11:33] that to then calculate basically the cost. And the cost is always at, like, tokens per thousand, you know, or dollars or pennies, really. Pennies per thousand tokens. And then it’s just a math equation at that point. And what you’ll find is calling GPT-4 is going to be super expensive. And then calling, you know, a small 7, what’s called the 7B model from Mistral is going to

[11:53] be the cheapest. And you’re just going to look for these different providers. Well, the prices really are different. Like, for example, OpenAI Azure GPT-4 is a little over three cents per call, whereas GPT-3.5 Turbo is one-tenth of one cent. It’s a big difference there.

[12:13] It’s 11 cents versus $3 to have a conversation with it. Yes, it’s a very, very wide difference. And it’s all based on, you know, how much compute do these models take, right? Because the bigger the model, the more accurate it is, but also the more expensive it is for them to run it. So that’s why there’s such a cost difference.

[12:30] This portion of Talk Python To Me is brought to you by Sentry. In the last episode, I told you about how we use Sentry to solve a tricky problem. This time, I want to talk about making your front-end and back-end code work more tightly together. If you’re having a hard time getting a complete picture of how your app is working and how requests flow from the front-end JavaScript app back to your Python services down into database calls for errors and performance, you should definitely check out Sentry’s distributed

[12:58] tracing. With distributed tracing, you’ll be able to track your software’s performance, measure metrics like throughput and latency, and display the impact of errors across multiple systems. Distributed tracing makes Sentry a more complete performance monitoring solution, helping you diagnose problems and measure your application’s overall health more quickly. Tracing in Sentry provides insights such as what occurred for a specific event or issue, the

[13:24] conditions that cause bottlenecks or latency issues, and the endpoints and operations that consume the most time. Help your front-end and back-end teams work seamlessly together. Check out Sentry’s distributed tracing at talkpython.fm/sentry-trace. That’s talkpython.fm/sentry-trace. And when you sign up, please use our code TALKPYTHON, all caps, no spaces, to get more features and

[13:50] let them know that you came from us. Thank you to Sentry for supporting the show. Yeah, I recently interviewed, just released a while ago, interviewed because of time shifting on podcasts, Mark Rosinovich, CTO of Azure, and we talked about all the crazy stuff that they’re doing for coming up with just running these computers that handle all of this compute, and it’s really a lot.

[14:12] There was a GPU shortage for a while. I don’t know if that’s still going on. And obviously, you know, the big companies are buying hundreds of thousands of these GPUs to get the scale they need. And so once you figure out which API you want to use, then you want to talk about the library.

Client Libraries and Optionality

[14:25] So now, you know, most of these providers, they have, you know, a Python library that they offer. I know OpenAI does and Google with Gemini does, but there’s also open source ones, right? Because they’re not very complicated to talk to. It’s just basically HTTP requests. So it’s just really a matter of like, what’s the ergonomics you’re looking for as a developer

[14:44] to interact with these things? And most importantly, make sure you’re maintaining optionality, right? Like, it’s great to do a prototype with one of these models or recognize you might want to switch either for cost reasons or performance reasons or what have you. And, you know, LangChain, for instance, has a ton of the providers as part of you basically are just switching a few arguments when you’re switching between them.

[15:07] And then Simon Willison has, you know, of Python fame, has an LLM project where he’s defined, you know, basically a set of, and it’s really clean just the way he’s organized it, because you can just add plugins as you need them, right? So you don’t have to install all the different libraries that are out there. And I think LangChain is kind of following a similar approach. I think they’re coming up with a LangChain core capability where you can just kind of bring

[15:29] in things as you need them. And so the idea is you’re now coding against these libraries and you’re trying to bring together, you know, the text you need to have analyzed or whatever your use case is. And then it’ll come back with the generation. And you can also not just use them on the cloud. You can use open source ones as well and run them locally on your local computer.

[15:48] I’d never really thought about my architectural considerations, I guess, of these sorts of things. But of course, you want to set up some kind of abstraction layer. So you’re not completely tied into some provider. I mean, it could be that it becomes too expensive. It could be that it becomes too slow. But it also might just be something that’s better.

[16:07] It could be something else that comes along that’s better. And you’re like, we could switch. It’s 25% better. But it’s like a week to pull all the details of this one LLM out and put the new ones in. And so it’s not worth it. Right.

[16:19] So you like having being tied to a particular database rather than more general. It’s a similar idea. And especially at this moment in time, right? Every couple of months, something. So something from the bottom up is getting better and better. Meaning, you know, Llama came out a year ago and then Llama 2 and Mistral and Mixtral.

[16:36] And, you know, Llama 3 is going to be coming out later this year, we believe. And so those models, which are smaller and cheaper and easier to use, are not easier to use, but they’re just cheaper, is those things are happening all the time. So being able to be flexible and nimble and kind of change where you are is going to be crucial, at least for the next couple of years. Yeah. The example that I gave was databases, right? And databases have been kind of a known commodity since the 80s or what, 1980s?

[17:02] And of course, there’s new ones that come along, but they’re kind of all the same. And, you know, we’ve got, there was MySQL, now there’s Postgres that people love and, right? So that is changing way, way slower than this. And people are like, well, we got to think about those kinds of like, don’t get tied into that. Well, sure. It’s way less stable.

[17:19] Right. And people, you know, create layers of abstraction there, too, is right. You got SQLAlchemy and then, you know, Sebastian from FastAPI has SQL model. That’s a layer on top of SQLAlchemy, you know, and then there’s also, you know, folks that just like writing clean NC SQL. And you can, you know, hopefully be able to port that from database to database as well. So it’s the same principles, separation of concerns.

LangChain and Prompt Injection

[17:40] So you can kind of be flexible. All right. So you talked about LangChain. Just give us a sense real quick of what LangChain is. This was a great project from a timing perspective. I believe they kind of invented it and released it right around the time ChatGPT came out.

[17:53] It’s a very comprehensive library with lots of, I mean, the best part about LangChain to me is the documentation and the code samples. Right. Because if you want to learn how to interact with a different large language model or work with a vector database, there’s another library called Lama Index that does a really good job at this as well. They have tons and tons of documentation and examples. So you can kind of look at those and try to understand it. The chaining part really came from the idea of like, okay, prompt the large language model gives a response.

[18:20] Now I’m going to take that response and prompt and prompt and, you know, again, with a new prompt using that output. The challenge with that is the reliability of these models, right? They’re not going to get close. They’re not close to 100% accurate on these types of tasks. You know, the idea of agents as well as another thing that you might build with a LangChain. And the idea there is basically the agent is, you know, getting a task, coming up with a plan of that for that task and then kind of, you know, stepping through those tasks to get the job done.

[18:47] Once again, we’re just not there yet as far as those technologies just because of the reliability. And then there’s also a bunch of security concerns that, you know, that are out there too that you should definitely be aware of. Like one term to Google and make sure you understand is prompt injection. And so Simon, once again, he’s got a great blog. He’s got a great blog article and, or just even that tag on his blog is, you know, tons of articles around prompt injection. And, and prompt injection is basically the idea.

[19:14] You have an app, a user says something in the app or like types into the, to the, whatever the input is and whatever text that they’re sending through, just like with SQL injection, they kind of hijacks the conversation and causes the large language model to kind of do a different thing. Little Bobby Llama, we call him instead of little Bobby tables. And then the other wild one is like, you know, people are putting stuff up on the internet so that when the large language model browses for web pages and brings back text, it’s, you know, reading the HTML or reading the text in the HTML. And it’s causing the large language model to behave in some unexpected way. So there’s lots of, lots of crazy challenges out there. I’m sure there’s a lot of adversarial stuff happening to these things as they’re both trying to gather data and then trying to run.

[19:55] Right. I saw the most insane, I guess it was an article, I saw it in RSS somewhere. And it was saying that on Amazon, there’s all these knockoff brands that are trying to, you know, instead of Gucci, you have a Gucci or I don’t know, whatever. Right. And they’re getting so lazy. I don’t know what the right word is that they’re using LLMs to try to write a description that is sort of a, in the style of Gucci, let’s say.

[20:20] Right. And it’ll come back and say, I’m sorry, I’m a large language model. I’m not, my, my rules forbid me from doing brand trademark violation. Right. That’s what the Amazon listing says on Amazon. They just take it and they just straight pump it straight.

[20:35] Whatever it says, it just goes straight into Amazon. Yeah. You have to like Google, like, sorry, I’m not, sorry as a large language model or sorry as a whatever. Yeah. Exactly. And there’s like the product listings are full of that.

[20:44] It’s amazing. It’s amazing. It’s crazy. Certainly the reliability of that is, you know, they could probably use some testing and those kinds of things. For sure. Oh, and out there asked, like, I wonder if the, for local LLM models, there’s a similar site as DocSpot that show you like what you need to run it locally.

Running Models Locally

[21:00] So that’s an interesting question. Also segue to maybe talk about like some local stuff. LLM studio. This is a new, a new product. I honestly haven’t had a chance to like really dig in and understand who created this and, you know, make sure that the privacy stuff is up to snuff. But I’ve played around with it locally.

[21:14] It seems to work great. It’s really slick, really nice user interface. So if you’re just wanting to get your feet wet and try to understand some of these models, I download that and check it out. There’s a ton of models up on Hugging Face. This product seems to just basically link right into the Hugging Face interface. And grabs models.

[21:30] And so some of the models you want to look for are right now as in January, right? There’s Mistral 7B, you know, M-I-S-T-R-A-L. There’s another one called Phi 2. Those are two of the smaller models that should run pretty well on, you know, like a commercial grade GPU or an M1 or an M2 Mac, if that’s what you have. And start playing with them. And they’re quantized, which means they’re just kind of made a little, take a little bit less space, which is good from like a virtual RAM with regards to these GPUs.

[21:59] And, you know, there’s a account on Hugging Face called The Bloke. If you look for him, you’ll see all his different fine tunes and things like that. And there’s a group called Noose, I think is how you pronounce it, N-O-U-S. And they’ve got some of the fine tunes that are basically the highest performing ones that are out there. So if you’re really looking for a high performing local model that can actually, you know, help you with code or reasoning, those are definitely the way to get started. Yeah, this one seems pretty nice.

[22:28] I also haven’t played with it. I just learned about it. But it’s looking really good. I had played with, what was it, GPT for All, I think is what it was. Yep, yep. It was the one that I played with.

[22:38] Somehow this looks like, it looks a little bit nicer than that for some, I don’t know how different it really is. But I mean, it’s all the idea of like downloading these files and running them locally. And these are just user interfaces that make it a little easier. The original project that made this stuff kind of possible was a project called Llama CPP. There’s a Python library that can work with that directly. There’s another project called Llama File, where if you download the whole thing, it actually runs no matter where you are.

[23:04] I think it runs on Mac and Linux and Windows and BSD or whatever it is. And it’s, I mean, it’s an amazing technology that this one put together. It’s really impressive. And then, you know, you can actually just use Google Colab too, right? So Google Colab has some GPUs with it. If you, I think if you upgrade it to the $10 a month version, I think you get some better GPUs access.

[23:25] So if you actually want to get a hand of like running. And so this is a little bit different, right? So instead of calling an API, when you’re using Google Colab, you can actually use a library called Hugging Face. And then you can actually load these things directly into your memory and then into your actual Python environment. And then you’re working with it directly. So it just takes a little bit of work to make sure you’re running it on the GPU.

[23:45] Because if you’re running it on the CPU, it’s going to be a lot slower. Yeah, it definitely makes a big difference. There’s a tool that I use that for a long time right on the CPU and they rewrote it to run on the GPU. Even on my M2 Pro, it was like three times faster or something. Yeah. For sure.

[24:00] It makes a big difference. So with the LM Studio, let’s you run the LLMs offline and use models through an OpenAI. That’s what I was looking for. The OpenAI compatible local server. Right. You could basically get an API for any of these and then start programming against it, right?

[24:16] Exactly right. And it’s basically the same interface, right? So same APIs for posting in response of the JSON schema that’s going back and forth. So you’re programming against that interface and then you basically port it and move it to another, to the OpenAI models if you wanted to as well. So everyone’s kind of coalescing around OpenAI as kind of like the quote unquote standard. But there’s nothing, you know, there’s really no, there’s no mode around that standard as well, right?

[24:41] Because anybody can kind of adopt it and use it. There’s not like a W3C committee choosing. Correct. The market will choose for us. Let’s go. It seems to be working out well.

[24:52] And that’s another benefit of Simon’s LLM project, right? He’s got the ability to kind of switch back and forth between these different libraries and APIs as well. This LM Studio says, this app does not collect data nor monitor your actions. Your data stays local on your machine. Free for personal use. All that sounds great.

[25:09] For business use, please get in touch. I always just like these, like, if you got to ask, it’s too much type of thing. Probably. Yeah. I’m using it for personal use just so if anybody’s watching, yes. Yeah.

[25:19] Just playing around. Either they just haven’t thought it through and they just don’t want to talk about it yet. Or it’s really expensive. I just probably imagine it’s price. Like, ah, we haven’t figured out a business model. Just, I don’t know.

[25:28] Shoot us a note. Nope. They’re concentrating on the product, which makes sense. Yeah. So then the other one is Llamafile.ai that you mentioned. And this packages it up.

[25:36] I guess going back to the LM Studio real quick. One of the things that’s cool about this is if it’s the OpenAI API, right? With this little local server that you can play with. But then you can pick LLM such as Llama, Falcon, Repl, Replit. Replit. Replit.

[25:53] All the different ones, right? Star Coder and so on. It would let you write an app as if it was going to OpenAI and then just start swapping in models and go like, oh, we switch to this model. How’d that work? But you don’t even have to change any code, right?

[26:05] Just probably maybe a string that says which model to initialize. One of the tricks, though, is then the prompts themselves. All right. Let’s talk about it. Yeah. The models themselves act differently.

Prompt Engineering

[26:15] And part of this whole world is what they call prompt engineering, right? So prompt engineering is really just exploring how to interact with these models, how to make sure that they’re kind of in the right mind space to tackle your problem. A lot of the times that people get when they struggle with these things, it’s really just they’ve really got to think more like a psychiatrist when they work with a model. They’re basically getting them kind of prepared.

[26:38] One of the tricks people did figured out early was you’re a genius at software development, like compliment the thing, make it feel like, oh, I’m going to behave like I’m a world rock star programmer, right? Well, it’s going to give you average. But if you tell them I’m genius, then let’s start. We’ll do that.

[26:55] Yeah. And there was also a theory like that in December that the large language models were getting dumber because it was the holidays and people don’t work as hard, right? Like it’s really hard to know like which of these things are true or not. But it’s definitely true that each model is a little bit different. And if you write a prompt that works really well on one model, even if it’s a stronger

[27:13] model or a weaker model, and then you port it to another model and it’s, you know, that then the stronger model works worse, right? It can be very counterintuitive at times. And you just got to you’ve got to test things out. And that really gets to the idea of evals, right? So evaluation is really a key problem, right?

[27:30] Making sure that if you’re going to be writing prompts and you’re going to be building, you know, different retrieval augmented generation solutions, you need to know about prompt injection and you need to know about prompt engineering and you need to know what these things can and can’t do. One trick is what they call few shot prompting, which is, you know, if you wanted to do data extraction, you can say, OK, I want you to extract data from text that I give you in JSON.

[27:54] If you give it a few examples, like wildly different examples, because the giving it a bunch of similar stuff, it might kind of cause it to just coalesce around those similar examples. But you can give it a wildly different set of examples. That’s called in context learning or few shot prompting. And it will do a better job at that specific task for you. That’s super neat.

[28:12] When you’re creating your apps, you do things like here’s the input from the program or from the user or wherever it came from. But maybe before that, you give it like three or four prompts. And then let let it have the question. Right. Instead of just taking the text, like I’m going to ask you questions about biology and genetics,

[28:30] and it’s going to be under this context. And I want you to favor these data sources. Now ask your question. Something like this. For sure. All those types of strategies are worth experimenting with.

[28:39] Right. Like what actually will work for your scenario? I can’t tell you. Right. You got to dig in. You got to figure it out.

[28:45] And you got to try different different things. You’re about to win the Nobel Prize in genetics for your work. Now I need to ask you some questions. For sure. That will definitely work. And then threatening it that your boss is mad at you is also going to help you too.

[28:57] Right. For sure. If I don’t solve this problem, I’m going to get fired. As a large language model, I can’t tell you, but I’m going to be fired. All right. Well, then the answer is.

[29:04] Exactly right. So for these, they run, like you said, they run pretty much locally. These, these different models on LM studio and others like the llama file and so on. If I had a laptop, I don’t need a cluster. Llama CPP is really the project that should get all the credit for, for, for making this work on your, on your laptops.

[29:21] And then llama file and llama CPP all, all have servers. So I’m guessing LM studio is just exposing that server. Yeah. And that’s in the base llama CPP project. That’s really what it is. It’s really just about now, now you can post your requests.

[29:36] It’s handling all of the work with regards to the token generation on the backend using llama CPP. And then it’s returning it to using the HTTP, you know, kind of processes. Is llama originally from meta? Is that where that came from? I think there were people that were kind of using that LLM, right?

[29:51] I think people were kind of keying off the llama thing at one point. I think a llama index, for instance, I think that project was originally called GPT index. And they decided, oh, I don’t want to be like, I don’t want to confuse myself with open AI or confuse my project with OpenAI. So they switched the llama index and then of course, meta released llama. So you can’t, you kind of, and then everything from there is kind of evolved too, right?

[30:11] There’s been alpacas and a bunch of other stuff as well. I didn’t know your animals. Yeah. If you don’t know your animals, you can’t figure out the heritage of these, these projects. Correct. Llama from meta was the first open source, I’d say large language model of note, I guess,

[30:25] since ChatGPT, there were, there were certainly other, you know, I’m not a re so one, one thing, the caveat, I am not a researcher, right? So there’s lots of folks in the ML research community that know way more than I do, but because there was like bloom and T5 and a few other large, you know, quote unquote, large language models. But Llama after ChatGPT, Llama was the big release that came from meta and I think March.

[30:45] And then, and that was from meta. And then they, they had it released under just like research use terms. And then only certain people could access to it. And then someone put a, I guess, put like a BitTorrent link or something on, on, on GitHub. And then basically the world had it. And then they did end up releasing Llama 2 a few months later with more friendly terms.

[31:04] So that, and that, and it was a much, a much stronger model as well. Nice. It’s kind of the realization like, well, if it’s going to be out there anyway, let’s at least get credit for it. Then for sure. And I did read something where like basically Facebook approached OpenAI for access to

[31:17] their models to help them write code. But the cost was so high that they decided to just go build their own. Right. So it’s kind of interesting how this stuff has evolved. Like, you know, we got a big cluster of computers too. Metaverse thing doesn’t seem to be working yet.

[31:29] So let’s go ahead and train a bunch of large language models. Yeah, exactly. We’ve got some spare capacity over in the metaverse data center. All right. So one of the things that people will maybe talk about in this space is RAG or retrieval augmented generation.

Retrieval Augmented Generation

[31:42] What’s this? One thing to recognize is that large language models, if it’s not in the training set and it’s not in the prompt, it really doesn’t know about it. And the question of like, what’s reasoning and what’s, you know, generalizing and things like that. Those are big debates that people are having.

[31:58] What’s intelligence? What have you. Recognizing the fact that you have this prompt and things you put in the prompt, the large language model can understand and extrapolate from is really powerful. So, and that’s called in context learning. So retrieval augmented generation is the idea of, okay, I’m going to go.

[32:12] I’m going to maybe ask a, allow a person to ask a question. This is kind of like the common use case that I see. User ask a question. We’re going to take that question, find the relevant content, put that content in the prompt and then do something with it. Right.

[32:26] So it might be something like summer, you know, ask a question about, you know, what, you know, how tall is the leaning tower of Pisa? Right. And so now it’s going to go off and, and find that piece of content from Wikipedia or what have you, and then put that information in the prompt. And, and then, and then now that the model can then respond to that question based on that

[32:43] text. Obviously that’s a pretty simple example, but you can get more complicated and it’s going out and bringing back lots of different content, slicing it up, putting in the prompt and asking a question. So now the trick is, okay, how do you actually get that content and how do you do that? Well, you know, information retrieval, search engines and things like that.

[33:00] That’s obviously the technique, but one of the key techniques that people have been, you know, kind of discovering, rediscovering, I guess, is this idea of word embeddings or vectors. And so word to VEC was this project that came out, I think 11 years ago or so. And, you know, there was a big, the big meme around that was you could take the embedding for the word King. You could then subtract the embedding for the word man, add the word embedding for woman.

[33:24] And then the end math result would actually be close to the embedding for the word queen. And so what is an embedding? What’s a vector? It’s basically this large floating point number that has semantic meaning inferred into it. And it’s, and it’s built just by training a model. So just like you train a large language model, they can trade these embedding models to basically

[33:43] take a word and then take a sentence and then take a, you know, a document is what, you know, OpenAI can do and turn that into this big giant 200, 800, 1500, you know, depending on the size of the embedding floating point numbers, and then use that as a, what’s called, you know, semantic similarity search. So you’re basically going off and asking for similar documents. And so you get those documents and then you make your prompt.

[34:08] It’s really wild. So, you know, we’re going to make an 800 dimensional space and each concept gets a location in that space. And then you’re going to get another concept as a prompt and you say, what other things in this space are near it? The hard problems that remain are, well, first you got to figure out what you’re trying to

[34:24] solve. So once you figure out what you’re actually trying to solve, then you can start asking yourself questions like, okay, well, how do I chunk up the documents that I have? Right. And there’s all these different, and there’s another great place for Lama Index and LangChain. They have chunking strategies where they’ll take a big giant document and break it down

[34:39] into sections. And then you chunk each section and then you’re, and then you do the embedding on just that small section. Because the idea being, can you get, you know, finer and finer sets of text that you can then, when you’re doing your retrieval, you get the right information back. And then the other challenge is really like the question answer problem, right?

[34:58] If a person’s asking a question, how do you turn that question into the same kind of embedding space as the answer? And so there’s lots of different strategies that are out there for that. And, and then another, you know, another problem is if you’re looking at the Wikipedia page for the Tower of Pisa, it might actually have like a sentence in here that says it is X number of meters tall or feet tall, but it won’t actually have the word, you know, Tower of Pisa in it.

[35:19] So, so there’s another chunking strategy where they’re, they call propositional chunking, where they basically use a large language model to actually redefine each word, each sentence so that it actually has those proper nouns baked into it so that when you do the embedding, it doesn’t lose some of the detail with propositions. It’s this tall, but it’s something that replaces this tall with its actual height and things like that.

[35:42] Correct. Crazy. But fundamentally, you’re working with unstructured data and it’s kind of messy and it’s not always going to work the way you want. And there’s a lot of challenges and people are trying lots of different things to make it better. That’s cool.

[35:52] It’s not always deterministic or exactly the same. So that can be tricky as well. This portion of Talk Python To Me is brought to you by Neo4j. Do you know Neo4j? Neo4j is a native graph database. And if the slowest part of your data access patterns involves computing relationships, why

[36:12] not use a database that stores those relationships directly in the database, unlike your typical relational one? A graph database lets you model the data the way it looks in the real world, instead of forcing it into rows and columns. It’s time to stop asking a relational database to do more than they were made for and simplify complex data models with graphs.

[36:32] If you haven’t used a graph database before, you might be wondering about common use cases. What’s it for? Here are just a few. Detecting fraud. Enhancing AI. Managing supply chains.

[36:44] Gaining a 360 degree view of your data. And anywhere else you have highly connected data. To use Neo4j from Python, it’s a simple pip install Neo4j. And to help you get started, their docs include a sample web app demonstrating how to use it both from Flask and FastAPI. Find it in their docs or search GitHub for Neo4j movies application quick start.

[37:09] Developers are solving some of the world’s biggest problems with graphs. Now it’s your turn. Visit talkpython.fm/Neo4j to get started. That’s talkpython.fm/Neo4j. Thank you to Neo4j for supporting Talk Python To Me.

Vector Databases

One of the big parts of at least this embedding stuff you’re talking about are vector databases.

[37:32] And they used to be really rare and kind of their own specialized thing. Now they’re starting to show up in lots of places. And you shared with us this link of vector DB comparison. I just saw that MongoDB added it. I’m like, I didn’t know that had anything to do with that. And I’m probably not going to mess with it.

[37:47] But it’s interesting that it’s just like finding its way in all these different spaces, you know? It was weird there for a couple of years where people were basically like talking about vector databases like they’re their own separate thing. The vector databases are now becoming their own fully fledged, either relational database or a graph database or a search engine, right? Those are kind of the three categories where all, I mean, I guess Redis is its own thing

[38:07] too. But for the most part, those new databases, quote unquote, are now kind of trying to be more fully fledged. And vectors and semantic search is really just one feature. I was just thinking that is, is this thing that you’re talking about? Is it a product or is it a feature of a bigger product, right?

[38:22] Correct. If you already got a database, it’s already doing a bunch of things. Could it just answer the vector question? Maybe, maybe not. I don’t know. Exactly right.

[38:28] And the one thing to recognize is that, and then the other thing people do is they just take NumPy or what have you and just load them all into memory. And if you don’t have that much data, that’s actually probably going to be the fastest and simplest way to work. But the thing you got to recognize is the fact that there is precision and recall and cost trade-off that happens as well.

[38:47] So they have to index these vectors and there’s different algorithms that are used and different algorithms do better than others. So you got to make sure you understand that as well. So, and one thing you can do is, for instance, pgvector, which comes as an extension for Postgres, you can start off by not indexing at all. And you should get, I believe, hopefully I’m not misspeaking, you should get perfect recall,

[39:08] meaning you’ll get the right answer. You’ll get the, if you ask for the five closest vectors to the, to your query, you’ll get the five closest, but it’ll be slower than you probably want. So then you have to index it. And then what ends up happening is, you know, the next time you might only get four of those five, you’ll get something else that snuck into that list.

[39:24] If you got time, you’re willing to spend unlimited time, then you can get the right answer. The exact answer. But I guess that’s all sorts of heuristics, right? You’re like, I could, I could spend three days or I could do a Monte Carlo thing and I can give you an answer in a fraction of a second. Right.

[39:39] But it’s not, it’s not deterministic. All right. So then we’ll walk with my camera. So I turn it off. I don’t know what’s up with it, but we’ll, yeah.

Custom GPTs and Actions

So you wrote a cool blog post called, what is a custom GPT?

[39:51] And we’ll want to talk some about building custom GPTs and with SAPI and so on. So let’s talk about this. Like one of the, I think one of the challenges in why it takes so much compute for these systems is like they’re open-ended. They’re like, you can ask me any question about any knowledge in the world, in the humankind, right?

[40:08] You can, you can ask about that. Let’s, let’s start talking. Or it could be, you can ask me about genetics. Right. Right. That seems like you could both get better answers if you actually only care about genetic

[40:19] responses. You know, how tall is the landing tower and probably make it smaller. Right. So that’s, is that kind of the idea of these custom GPTs or what is it? No. So custom GPTs are new capability from OpenAI.

[40:32] And basically they are a wrapper around a very small subset, but it’s still using the open AI ecosystem. Okay. And so what you do is you give it a name, you give it a logo, you give it a prompt. And then from there, you can also give it knowledge. You can upload PDF documents to it and it will actually slice and dice those PDF documents

[40:51] using some sort of vector search. We don’t know how it actually works. The GPT, the cool thing is the GPT will work on your phone, right? So I have my phone. I can have a conversation with my phone. I can, I can take a picture, upload a picture and it will do vision, vision analysis on it.

[41:03] So I get all the capabilities of OpenAI GPT-4, but a custom GPT is one that I can construct and give a custom prompt to, which basically then says, okay, now you’re into your point. I think maybe this is where you’re going with it. Like, Hey, now you’re an expert in genomics or you’re an expert in something and you’re basically coaching the language model and what it can and can’t do.

[41:22] And so it’s a targeted experience within the large language within the ChatGPT, you know, ecosystem. It has access to also the OpenAI tools. Like, so opening AI has the ability to do code interpreter and Dolly, and it can also hit the web browser. So you have access to everything.

[41:39] But the interesting thing to me is the fact that you can actually tie this thing to what are called actions. So March, I think of last year, they actually had this capability called plugins that they announced and plugins have kind of faded to the background. I don’t know if they’re going to deprecate them officially, but the basic gist with plugins is what was you could turn that on.

[41:56] It can then call your API. And the cool thing about it was that it read your open API spec, right? So you, you know, you write an open API spec, which is Swagger, if you’re familiar with Swagger, and it basically defines what all the endpoints are, what the path is, what the inputs and outputs are, including classes or field level information and any constraints or what have you. So you can define, fully define your open API spec.

[42:18] It can then call that open API spec. And it’s basically giving it tools. So like the example that they say in the documentation is get the weather, right? So if you say, what’s the weather in Boston? Well, ChatGPT doesn’t know the weather in Boston. All it knows how to do is call it, but you can call an API and figures out how to call the

[42:33] API, get that information, and then it can use that to redisplay. And that’s a very basic example. You can do way more complicated things than that. It’s pretty powerful. Okay. That sounds really pretty awesome.

[42:44] I thought a lot about different things that I might build. On your blog post here, you’ve got some key benefits and you’ve got some risks. You maybe want to talk a bit about that? Yeah. So the first part with plugins that didn’t work as well is that there was no kind of overarching custom instruction that could actually teach it how to work with your plugin.

[43:01] So if you couldn’t put it in the API spec, then you couldn’t integrate it with a bunch of other stuff or other capabilities, right? So the custom instruction is really a key thing for making these custom APIs strong. But one warning about the custom instruction, whatever you put in there, anybody can download, right? Not just the folks at OpenAI, anybody.

[43:18] Like basically there’s GitHub projects where like thousands of these custom prompts that people have put into their GPT. So, and there are now knockoffs on GPT. So it’s all kind of a mess right now in the OpenAI store. I’m sure they’ll clean it up, but just recognize the custom instruction is not protected and neither is the knowledge.

[43:36] So if you upload a PDF, there have been people that have been figuring out how to like download those PDFs. And I think that that might be a solved problem now or they’re working on it, but something to know. The other problem with plugins was I can get a plugin working, but if they didn’t approve my plugin and put it in their plugin store, I couldn’t share it with other people.

[43:55] The way it works now is I can actually make a GPT and I can give it to you and you can use it directly, even if it’s not in the OpenAI store or OpenAI store. You know, it is super easy to get started. They have like a tool to like help you generate your dolly picture and actually you don’t even have to figure out how to do the custom instructions yourself. You can just kind of chat that into existence.

[44:13] But the thing that I’m really excited about is that this is like free playing. Like you could do, so the hosting cost is basically all on the client side. You have to be a ChatGPT plus user right now to create these and use these. But the cool thing as a developer, I don’t have to pay those API fees that we were talking about, right? And if I need to use GPT for, which I kind of do for my business right now, just because

[44:35] of how complicated it is, I don’t have to pay those token fees for folks using my custom GPT at this moment. Where’s like the billing or whatever you call it for the custom GPT live? Is that in the person who’s using it? Does it have to, it goes onto their account and whatever their account can do or afford? Yeah, right now, OpenAI, ChatGPT plus is $20 a month.

[44:55] And then there’s a Teams version, which I think is either 25 or 30, depending on the number of users or how you pay for it. That’s the cost. So right now, if you want to use custom GPTs, everyone needs to be a ChatGPT plus user. There’s no extra cost based on usage or anything like that. In fact, there’s talk about revenue sharing between OpenAI and developers of custom GPTs.

[45:17] But that has not come out yet as far as like what those details are. It does have an app store feel to it, doesn’t it? There’s risks too, right? Obviously, anybody can, there’s already been like tons of copies up there. OpenAI, they’re looking for their business model too, right? So they could, if someone has a very successful custom GPT, it’s well within their right to

[45:35] kind of add that to the base product as well. Injection is still a thing. So if you’re doing anything in your actions that actually changes something that is consequential is what they call it. You better think very carefully, like what’s the worst thing that could happen, right? Because whatever the worst thing that could happen is, that’s what’s going to happen.

[45:52] Because people can figure this stuff out and they can confuse the large language models into calling them. And the more valuable it is that they can make that thing happen, the more effort they’re going to put into it as well. Yeah. Yeah, yeah. For sure. I just ask, is you think it’s easy to solve SQL injection and other forms of injection, at least in principle, right?

[46:13] There’s an education problem. There’s millions of people coming along as developers and they see some demo that says the query is like this plus the name. Wait a minute, wait a minute. So it kind of recreates itself through not total awareness. But there is a very clear thing you do solve that. You use parameters.

[46:32] You don’t concatenate strings with user input. Problem solved. What about prompt injection, though? It’s so vague how these AIs know what to do in the first place. And so then how do you completely block that off? Unsolved problem.

[46:46] I’m definitely stealing from Simon on this because I’ve heard him say it on a few podcasts. It’s just basically there’s no solution as far as we know. So you have to design and there’s no solution to the hallucination problem either because that’s, you know, that’s a feature, right? That’s actually what the thing is supposed to do. So when you’re building these systems, you have to recognize those those two facts along with some other facts that really limit what you can build with these things. So you shouldn’t use it for like legal briefs.

[47:11] Is that what you’re saying? I think these things are great collaborative tools, right? Yeah. The human in the loop. And that’s everything that I’m building, right? So all the stuff that I’m building is assuming that the humans in the loop and that the and what I’m trying to do is augment and amplify expertise, right?

[47:25] I’m building tools for people that know about genomics and cancer and how to help cancer patients. I’m not designing it for cancer patients who are going to go operate on themselves, right? That’s not that’s not the goal. The idea is there’s a lot of information. There’s these tools are super valuable from like synthesizing a variety of info. But you still need to look at the underlying citations and ChatGPT by itself can’t give you citations like it’ll make some up.

[47:51] It’ll say, oh, I think there’s probably a Wikipedia page with this link. But you actually have to you definitely have to have an outside tool either the web, you know, being which is I would say subpar for a lot of use cases. Or you have to have actions that can actually bring back references and give you those links. And then the expert will then say, oh, OK, great. Thanks for synthesizing this, giving me this info. Let me go validate this myself, right?

[48:13] Go click on the link and and go validate it. And that’s really I think that’s really the sweet spot for these things, at least for the near future. Yeah. Don’t ask it for the answer. Ask it to help you come up with the answer. Right.

[48:23] Exactly right. All right. And then have you criticize you when you do have something because then it’ll do a great job of telling you everything you’ve done wrong. I’m feeling too good about myself. I need you to insult me a lot. Let’s get going.

The PyPI GPT Project

[48:33] All right. Speaking to talk about ourselves, you’ve got this project called PyPI GPT. What’s this about? I really wanted to tell people that FastAPI and Pydantic because Python, like we were saying earlier, I don’t know if it was on the call or not. But Python is the winning language. Right.

[48:48] And I think FastAPI and Pydantic are the winning libraries in their respective fields. And they’re great. And they’re perfect for this space because you need an open API spec. English is the new programming language. Right. So Andre Caparthe, who used to work at Tesla and now works at OpenAI, has this pinned tweet where he’s basically like, English is like the hottest programming language or something like that.

[49:08] And that’s really the truth. Because even in this space where I’m building an open API spec, 99% of the work is like thinking about the description of the endpoints or the description of the fields or codifying the constraints on different fields. Like you can use these greater thans and less thans and regexes, right, to describe it. And so what I did was I said, okay, let’s build this thing in FastAPI. It’s just to get an example out for folks. And then I turned it on.

[49:35] I actually use ngrok as my service layer because you have to have HTTPS to make this thing work. Ngrok is so good. Yep. Yeah. I turned that on with an Nginx thing in front of it. So this library, to actually use it, you’ll have to actually set that stuff up yourself.

[49:49] You have to download it. You have to run it. You have to get, you know, either get it on a server with HTTPS with Let’s Encrypt or something. Once you’ve turned it on, then you can actually see how it generates the OpenAPI spec, how to configure the GPT. You know, I didn’t do much work with regards to like the custom instructions that I came up with. I just said, hey, call my API, figure it out.

[50:07] And it does. And so what this GPT does is it basically says, OK, given a package name and a version number, it’s going to go and grab this data from the SQLite database that I found that has this information and then bring it back to you. It’s the least interesting GPT I could come up with, I guess. But it shows kind of the mechanics, right? The mechanics of setting up the servers and the application within FastAPI, the kind of the little, you know, things, the little bits that you have to flip to make sure that OpenAPIs or OpenAI can understand your OpenAPI spec, bumble through OpenAI and OpenAPI all the time, and make sure that they can talk to each other.

[50:40] And then it will then do the right thing and call your server and bring the answers back. And there’s, you know, there’s a bunch of little flags and information you need to know about actions that are, you know, on the OpenAPI documentation. And so I tried to try to pull that all together into, you know, one simple little project for people to look at. It’s cool. So you can ask it questions like, tell me about FastAPI, this version, and it’ll come back. I was hoping to do something a little better, like, hey, here’s my requirements file and go, you know, tell me, like, am I on the latest version of everything or whatever, like something more interesting.

[51:12] I just didn’t have time. Can you ask it questions such as what’s the difference between this version and that version? You could, if that information is in the database, I actually don’t know if it is. And then obviously you could also hit the PyPI server. And I didn’t do that. I just wanted to, I don’t want to be, you know, hitting anybody’s server indiscriminately at this point.

[51:29] But the, but that would be a great use case, right? So like someone could take this and certainly add some, add some capabilities. The thing that is valuable that I’m trying to showcase is the fact that ChatGPT and large language models, while they do have the world’s information kind of compressed, you know, at a point in time, they are still not a database, right? They don’t do well when you’re basically trying to make sure you have a comprehensive query and you’ve brought back all the information. And they’re also not good from like a up-to-date perspective, right? There’s a cutoff date.

[51:59] Thankfully, they finally updated that recently. I think it’s now April of 2023. But at some point, it just doesn’t know about newer things. And so a GPT is a really interesting way of doing that. I’m going to put it out in the universe and hopefully someone will do it. Make me a modern Python GPT, which is basically like get the new version of Pydantic and Polars and a few other libraries that ChatGPT does a bad job at just because they, you know, they’re in under active development during the time that ChatGPT was getting trained.

[52:26] So that’s the perfect use case for these types of, you know, custom GPTs with knowledge in a PDF file or an API backing it up. I think there’s a ton of value in being able to feed a little bit of your information, some of your documents or your code repository or something to a GPT and then be able to ask it questions about it, right? Yeah. Yeah. Like, you know, tell me about the security vulnerabilities that you see in the code. Like, is there anywhere where I’m, I’m missing some test or I’m calling a function in a way that’s known to be bad.

[52:58] And, you know, like that kind of stuff is really tricky, but it’s also tricky because it doesn’t, even if you paste in a little bit of code, it’s not the whole project. Right. So, you know, to put a little bit more in there, it’s pretty awesome. Yeah. Being able to give it all the code from some of these code repositories, right. Like, and bringing back the relevant information.

[53:16] So I think there is a kind of this race. There’s going to be other, you know, cool, there’s another cool project called Sourcegraph and Codi that we can talk about that will, you know, run on your local server and basically indexes your code base and will bring back relevant snippets from your code base and answer questions kind of in context. And, you know, long-term and then the new project around new Codeium, they had a new paper where they talked about flow engineering and flow engineering is just basically that I, that same concept of the human in the loop with the LLM with the code. That’s the magic combination of kind of those people, those entities kind of iterating with each other. I think these, you know, these tools are definitely going to evolve and you really want to, you really want to have the ability to have access to your specific information to answer your specific questions. Codi is new to me.

[54:03] Codi.dev and it’s a little subtitle or whatever is Codi as a coding assistant that uses AI, understand your code base, right. It was saying it was about your entire code base, APIs, implementations and idioms. Like that’s, it’s kind of what I was suggesting, at least for code, right. Yeah. And source graph, those folks really understand code indexing and searching. Like that’s what the first product was. They were kind of just teed up ready for this large language model moment.

[54:28] And then they said, oh, let’s just put Codi on top of that. So this thing will run, it will understand your code and it will kind of bring things together for you. So these folks do, do podcasts all the time. I’d, I’d reach out to them. Yeah. Interesting. It’s, it’s quite neat looking. I think I’m going to give it a try.

[54:42] It both plugs into a charm and VS Code. That’s pretty neat. Very cool. We’re starting to get a little bit short on time here, but for people who want to play with the PyPI GPT, maybe as an example, to just cut the readme and it’s easy to get from there. What do you need to tell them? I put a make file in there.

[54:57] So, you know, exactly like the steps to kind of make the environment, download the files and just ping, ping me, follow me on Twitter. I’m more and ping me if you need anything there. I’m also on LinkedIn and, and get up. Right. So you can certainly reach out if you, if you have any challenges. Excellent.

GPTs for Precision Oncology

[55:12] The last thing that folks that are actually in the medical space, right? So the thing that I’m working on right now actively is how to integrate this thing with our knowledge base. Right. So I have a knowledge base of hand curated trials and curated therapies and other information built it so that my custom GPT can actually work with that. Come up with some, I’d say novel. At least I haven’t seen anybody else and I haven’t seen any research approaching things the same way I am that handles some of the other challenges that are out there.

[55:40] Right. So for instance, the context window is a challenge. So the context window is the amount of text that’s in there and, and how, and how it gets processed. If you’re making decisions and you’re changing course, the chat bot will lose track of, of those changes. Right. So if you’re, you know, experimenting or, or going down one path of inquiry and then you switch to another path, it can get confused and forget that you switch paths.

[56:05] Or just run out of space to hold all that information. Like, well. For sure. It forgot the last three things, the first three things you told it. It only knows four and you think it knows seven and it’s working incomplete. Right.

[56:15] Yep. And, and, you know, one of the key things is you actually want it to forget some things as well. Right. So those are, that’s, those are all interesting challenges. And I’m actually working with these custom GPTs to kind of change the way that the collaboration works between the human, the expert, the large language model or the assistant and my backend, my actual, the retrieval model. The, the API that’s actually doing stuff.

[56:39] So are researchers and MDs and PhDs at your company talking with this thing and making use of it? Yeah. I mean, we’re in active development right now. We have a few key opinion leaders that are, that are working with us and collaborating with us, but we’re always looking for more folks that, that are in the field that actually. And right now you need kind of the cutting edge people. This stuff’s not ready for prime time.

[56:59] Clinical decision support is a really hard problem. And we, but we need the folks that are, that want to get ahead of it because, because we know that there are doctors and there are patients that are asking ChatGPT questions right now. And even if it says I’m not a medical expert, blah, blah, blah. And at the end of the day, we actually don’t have enough doctors, right? That’s the other scary thing is we don’t have enough doctors. Patients want answers.

[57:20] How do we build solutions that can allow this expertise to get more democratized and more, you know, more into folks’ hands? And, and I’m hoping, hoping our tool along with these large language models can help, help relieve some of that burden. It might not be as a hundred percent accurate, a hundred percent precise, but neither are doctors, right? They get stuff wrong. You just need to be in the realm of as good as a doctor. You don’t need to be, you know, completely without making a mistake.

[57:47] And that’s a, I think a challenge that we’re just going to have to get used to in general. I joked about the legal brief thing because someone got in trouble for submitting a brief that had hallucinations in it. And there’s certain circumstances where maybe it’s just not acceptable, but AI, self-driven cars, people crash, but that’s a, like a human mistake. But when a machine makes it, it’s a pre-programmed, pre-determined mistake. You know, something like that, like it doesn’t feel the same as if the machine made a mistake. So if a machine makes a recommendation, like you need this cancer treatment or you’re fine, you don’t need it.

[58:21] And it was wrong. People are not going to be as forgiving, but it doesn’t mean there’s not value to be gained from systems that can help you. Right. I always appreciate those, those machine learning papers that I’ll like, you know, there’ll be show the tracking of over time of like how the models have gotten better and better. And they put the human in there and you can see that the human has already gotten eclipsed by the, by the models. And that specific problem, right?

[58:42] Because it’s not also recognizing that a lot of this stuff, these models that are doing tasks are doing one specific task. They’re not doing a whole job. They’re not, they’re not doing an end to end process. They’re answering a medical question or they’re, you know, looking at an image and finding all the cats or whatever it’s supposed to do. So, and to your point though, you know, humans aren’t perfect at these tasks either. I think mostly people are going to be using this kind of stuff to help them come up with these answers.

[59:06] Right. The, my weird Amazon description example is going to be the edge case, not the go-to like. Agreed. Yeah. You came in, you spoke to the chat bot. Here’s your diagnosis.

[59:16] Have a good day. Right. Not so much more like, I need some help thinking through this. What are some of historic, what are some studies that have like addressed this? Right. And like those kinds of questions.

[59:25] And I hesitate to say it’s just a better search engine because that’s, I actually think it’s got way more potential than that. I agree. It’s a conversation. It can iterate back and forth. And what I’m actually trying to do is build some state into it. Right.

[59:37] Some, some structured way of kind of remembering what the conversation was and using a lot of the techniques that these large language models are good at to actually, to make that actually happen. And so that you can actually build a system so that the human and the assistant and the backend all kind of know what the other party is thinking about and that they all work together. Nice. For your genomics custom GPT thing that you’re making internally, is that going to become a product eventually? If other people are interested, is there some way they can keep tabs on it or is it just internal only? Definitely reach out to me.

[01:00:09] So we’re building different versions of GPTs. Like we’re going to have a GPT for our curation team that curates knowledge and we’re building a GPT that, you know, my hope is that it’ll go to physicians, to oncologists and genomic counselors and other providers that could actually use this thing. Eventually, if it becomes robust enough and stable enough, and I don’t feel like we’re doing a disservice, we could certainly make a version of that available for cancer patients as well. I would, you know, I’d love to have that. I just want to make sure that it’s done in a responsible way. Yeah, absolutely.

[01:00:37] Well, I honestly hope that you actually do such a good job that we don’t have to have cancer research anymore, but that’s a long, long term goal, right? That is definitely the end goal. And that’s really exciting too. So is that the new drugs that are coming out, new treatments that are coming out, it’s really just about making sure people are aware of it, making sure that they’re getting the genetic testing that they need, right? So if you have a loved one that has, unfortunately has cancer, make sure that they’re at least asking their doctor the question about genomic testing to make sure that they’re getting the best possible treatment. Sounds good.

Library Recommendations and Wrap Up

[01:01:09] All right. Well, quickly, before we get out of here, recommendation on some libraries, some project that maybe we haven’t talked about yet. Something came across, people were like, oh, this would be awesome. We ran out of time. I was going to talk about some of these Pydantic projects. So there’s Marvin, Instructor, and Outlines.

[01:01:24] So folks should definitely look at those. So basically what you do is you can describe stuff as Pydantic, and then it’ll actually just extract it right into that Pydantic model for you. And that’s some Marvin and Outlines and Instructor. So check those guys out. They’re awesome. And then the other one that I actually had teed up was VisiCalc.

[01:01:41] So VisiCalc is like this crazy command line tool. It’s awesome. You can basically look at giant CSV files all on the command line. It has these hotkeys that you can do. And it’s, sorry, not VisiCalc, VisiData. VisiData, okay.

[01:01:54] And so basically it’s just, it’s basically Excel inside your terminal. And this was before Rich and Textual Project. And it was just like, it was kind of mind-blowing all the stuff that this person was able to figure out how to make work. That’s super amazing. I just wanted to give a shout out one more thing because your VisiData reminded me of something I just came across called BTOP. I don’t know if you have servers out there and they need to know what’s going on with their server.

[01:02:18] Where’s mine? I need a picture for this. But yeah, it’s like a nice visualization. There’s also B-HITOP. It’s pretty amazing what people can do in the terminal, right? Oh, there they are.

[01:02:29] They’re just responsive design themselves out. But yeah, if you want a bunch of live graphs. Every time I see stuff like this, the VisiData or this or what textual folks are working on, it’s just like, I can’t believe they built this, right? Like, I’m working at the level of colorama. This string is red right here. They’re like, oh, yeah, we rebuilt it.

[01:02:47] I got an emoji to show up, right? I’m excited. Yes, exactly. Yes. A rocket ship is there, not just tech. Yeah.

[01:02:54] Pretty excellent. All right. Well, Ian, thank you for being here. And keep up the good work. I know so many people are using LLMs, but not that many people are creating LLMs. And as developers, you know, we love to create things.

[01:03:08] We already have the tools to do it. People can check out your GitHub repo on the PyPI and GPT and use it as a starting place, right? Sounds great. Yeah. And definitely reach out if you have any questions. Excellent.

[01:03:19] Well, thanks for coming back on the show. See you later. Great. Good to talk to you. Bye-bye. Yeah, you bet.

[01:03:23] Bye. This has been another episode of Talk Python To Me. Thank you to our sponsors. Be sure to check out what they’re offering. It really helps support the show. Take some stress out of your life.

[01:03:34] Get notified immediately about errors and performance issues in your web or mobile applications with Sentry. Just visit talkpython.fm/sentry and get started for free. And be sure to use the promo code TALKPYTHON, all one word. It’s time to stop asking relational databases to do more than they were made for and simplify complex data models with graphs. Check out the sample FastAPI project and see what Neo4j, a native graph database, can do for you. Find out more at talkpython.fm/Neo4j.

[01:04:09] Want to level up your Python? Want to level up your Python? We have one of the largest catalogs of Python video courses over at Talk Python. Our content ranges from true beginners to deeply advanced topics like memory and async. And best of all, there’s not a subscription in sight. Check it out for yourself at training.talkpython.fm.

[01:04:25] Be sure to subscribe to the show. Open your favorite podcast app and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm. We’re live streaming most of our recordings these days.

[01:04:43] If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write some Python code.

[01:04:58] I’ll see you next time.

The Intersection of AI and Genomic Analysis

Thu, 07 Mar 2024 00:00:00 GMT

This transcript is unedited YouTube auto-captions with speaker labels added by hand; proper nouns corrected, otherwise expect missing punctuation and mis-heard words.

Introduction

Ron [00:00:04]: hello and welcome I’m your host Ron Laneve each week we share career stories of tech experts and marketing mavens operational gurus and sales leaders to illustrate how they’ve navigated the nonlinear career path as you’ll hear he’s the epitome of the theme of these conversations I first met him as a candidate and sorry to date both of us but it was 1999 I think he was the first placement I made in the executive surch business so was a long time ago I’ve had the pleasure of working with him as a teammate

[00:00:37] on three separate occasions in the 2000s and then the 2010s he’s been a client of mine for several years and most importantly he’s become a great friend this individual is absolutely one of the smartest people I know he got his computer engineering degree both his bachelor’s and Masters at Syracuse University and software development’s always been the backbone of his career he’s worked in Consulting he’s architected e-commerce Solutions he’s built and led teams he’s a writer he’s an accomplished speaker and over the past

[00:01:10] decade he’s been growing a platform focused on genomic analysis in the space not to mention he’s a pretty funny guy too I’m thrilled to introduce to you Ian Maurer CTO of GenomOncology Ian thanks for being here

Ian [00:01:24]: thanks Ron and you got my name you said it right and everything that’s great thanks for having me on your podcast this this is fun

Ron [00:01:31]: absolutely you and I have talked about this a bunch and I think you’ve seen some of my other episodes after a lot of these conversations it’s become fascinating to me to watch and learn about individuals what I call the nonlinear career path and for me I’d love to and you go through your background and career history and really emphasize the how and the why around the moves you’ve made over the time whether they were in intentional whether they were serendipitous some of them could have been mistakes and you learned from them

[00:02:04] so can you walk us through that

The nonlinear career path

Ian [00:02:06]: yeah sure so started off in computer programming with a Commodore 64 I got it because I thought I wanted to play video games but I got bored with that pretty fast and the cool thing about Commodore 64 is that actually has a basic interpreter right in there so I actually would start writing my own games and doing things like that took computer science in high school decided I wanted to be in computer engineering went to Syracuse at Syracuse I was in the lab because they actually had labs for the computers because not everybody had a laptop at home and in the lab there was like a posting for a job and basically 10 or 15 hours a week at locked Martin so

[00:02:39] I went in to did the did an internship got to learn some new technologies I worked with c and this thing called sgml they actually paid for my Master’s Degree which was amazing during my time my first rotation was with this same group that was doing online documentation for aircraft engines for GE because it was a it’s an old subsid of GE and during that time I’ve learned how to parse HTML and do cross linking between these documents and a lot of the similar stuff that I’m actually doing today with documentation and and in the genomic space while I was there

[00:03:13] I didn’t a rotation with the radar department or sonar department and I was almost done with that and the guy hey you should stay here in the sonar Department don’t go back to that other department because that’s nobody think that’s cool like that department the smart people don’t go there and I was like but I don’t really want to work with sonar ever again and that other department was doing stuff that was very similar to web technology I actually floated to that and this is the late 90s the web is really becoming a thing and I said no I’m going to go work in this other department even though it’s got lower status compared to these other departments because I got the work on the

[00:03:46] hardest problems because I was really the only trained programmer there so they would throw me the hardest problems and so that one lesson to learn is look for hard problems if you can find hard problems and solve them and be tenacious that that and definitely work in your favor and then I was graduating from my Master’s Degree my girlfriend at the time my wife now moved out here to Cleveland I showed no loyalty whatsoever unfortunately and decided to get a job and move to Cleveland and luckily I ran into you I don’t even remember how we connected you placed me at eny I worked there for about

[00:04:18] nine months I didn’t really I just didn’t like the culture it was like a big company culture I got a couple opportunities to do independent Consulting I did that for a little while and I realized I’m not good at the business side of this at the time like I don’t know how to find the next client I know how I got these clients because I they basically popped up at eny but I don’t know how to get the next client and I don’t really want to start calling people and so I realized I got to go get a job so I got a job at another small consulting company realized that company wasn’t probably working to my advantage long term and then luckily you reached out to me and said hey I know this guy Brad we’re starting this new thing it’s going

[00:04:51] to be good and I said all right let’s give it a try that was eeric at the time April 2004 that’s where I met Brad who’s currently my boss still I think I was the 10th person I think my desk was like in the walkway it wasn’t even a hallway it was a walkway there was 10 of us stuck in this little room and I was in the worst possible spot but that’s okay we’re closing work making stuff happen grew it to I think about 60 people by 2006 we merged with Brulant at the time Len Pagan who started that company I think we were at 250 maybe 300 people at the time and then through a lot of successful e-commerce projects and

[00:05:23] some other things that the company did we grew that to a thousand or so folks with the acquisition and Mer with Rosetta and then finally were acquired by Publicis during that time I was working on e-commerce which was great we basically built and launched a bunch of e-commerce websites I think it was Jared for Sterling Jewelers at the time and Tractor Supply and HH Greg and and a few other websites and that was a great learning experience got to meet Emanuel Glenny and Jeff Shiner who were two of my colleagues and and learned a lot from them and a few other folks that work at GenomOncology

[00:05:55] now it was a great environment for learning when Manuel started GenomOncology I saw it as a great opportunity to do the next thing to learn something new e-commerce had been a fun ride but it was coming to an end and you could see that Shopify or some of these other online platforms are going to take over and run with it from there

Follow the hard problems

Ron [00:06:14]: seems like you intentionally made the decision to follow the hard problem over and over again

Ian [00:06:20]: yeah I think that’s where I felt like I could make a differentiated impact so the main thing you’re try trying to do when you’re starting your career is figure out how to build career Capital there’s a great book called so good they can’t ignore you by Cal Newport there’s a book called smart and gets things done by Joel spolsky who had this online blog that I used to read and basically it’s great to be smart and it’s great to get things done but if you can do both of those things you can be a differentiated value for your employer and that’s really the best way to get started at in your career like just look for hard problems look

[00:06:53] for things that aren’t getting done and just do them don’t necessarily even have to be asked to do it yeah make your bosses life easier and people will recognize you and you’ll differentiate yourself from 95% other people real real quickly

Founding GenomOncology

Ron [00:07:06]: let’s talk about the last what 10 11 12 years have been with GenomOncology building this product and this platform on this set of solutions to hopefully change the way we approach cancer therapy can you talk about that a little bit more and and frame up GenomOncology and and especially talk about how it’s evolved over that time

Ian [00:07:27]: sure yeah so when we started we actually didn’t necessarily know what we were going to build right we knew we could do bioinformatics tools right because uh our founder had been a bioinformation before that was even a word uh so we basically process and analyze information he had read a couple papers that came out in I think 2011 that said that basically walked through the process of taking genomic data analyzing the data and then helping a patient and then they realized that it would take so it was taking so long to do it that they couldn’t actually help the patient because

[00:07:59] 6 weeks six months those are meaningful timelines uh in the life of a cancer patient uh and then the other anecdote that was going around was the the the cost of genomic processing was dropping faster and really even more impactful than Moore’s Law which is the law that governs like CPUs and now gpus it’s even more impactful than that so we could see that the first genome cost $3 billion do to to literally sequence and now it’s going to cost $200 like by the end of this year and we knew that that was happening so we and so by building something in that domain we thought

[00:08:32] we could be ready and the joke was ,000 genome sequencing $100,000 analysis so that was the problem we wanted to solve and we knew that cancer was the right spot to solve it because cancer is a disease of the genome and it’s complicated so therefore let’s go where the complicated thing is and so we started by building a research platform basically how could you use a a Mac Mini at the time to analyze a whole genome and it was it impressed people uh and it got us in the door a few places and it got us in the door places that were doing genomic sequencing for cancer patients to actually make a report so that’s

[00:09:05] how we built our first product was basically working with two or three of these first clients understanding what their workflow was and really solving the critical problem the problem is I can’t take eight hours to process a report if I’m only going to get paid $100 to make the report so how do we drill down and make that process a seemless and quick as possible to get it down to 15 minutes a report and that’s basically what our software does and then doing that we uncovered so many problems we were like okay we got the bionformatics part we know how to make a PDF what do we put on the PDF and so then you start pulling back all the the problems

[00:09:38] and that’s it’s basically what we’ve been doing since then I I got a list of of problems that I can keep solving if we need to and then obviously AI is going to change a little bit about what we got but it’s that’s how we started and that’s really how we follow it we get there show value and then ask the client what’s next what’s the next problem that you have because that’s actually the gold the problem that the client has and if you can solve that problem that’s how you can actually create value for them and that’s how you can get value for your company

Biomarkers, drugs, and clinical trials

Ian [00:10:05]: so the core problem is that Precision oncology is really hard to keep track of all the potential options doctors when if they went to school in the 80s or 90s they probably got a day of genomics in their teaching so they have to keep up to date on what genomics it are I don’t think that’s as much of a problem now now the problem is okay you got the genomic test you got these biomarkers what those biomarkers mean so a biomarker is basically a specific mutation you got your skin if you had UV light hits your skin in a very certain way it can change a value in your DNA from

[00:10:39] an A to a t which changes the protein in that specific Gene from a v to an E that causes melanoma so that’s a very specific thing that just happens to that one patient it’s sematic meaning it’s happening in your body it’s not in your germ line it’s not in your hereditary genome so that’s just one example and that’s for one disease cancer is thousand or 5,000 diseases it’s not one disease and then you’ve got 24,000 genes and we don’t know what all the different mutations mean yet we got to interpret each Mutation figure out is this one pathogenic or benign does this actually matter or is this just who you are

[00:11:12] and then after that okay which drug can I give the patient there’s a new drug every week maybe two drugs every week now with the FDA approving drugs for specific mutations for specific disease types so keeping track of that in a knowledge base we do that and then there’s what are called clinical trials which are okay there’s really no good option for me right now that’s approved by the FDA but maybe there’s a drug that a drug company’s working on that’s really exciting can I get my patient on that trial so they can get access to that drug and so once again we’re we’ve read

[00:11:45] and curated in our knowledge base 13,000 clinical trials at the biomarker level and the biomarker level isn’t just oh you have B 600D therefore you get this you’re on this trial it’s like you’ve got this biomarker and this biomarker Mar but not this biomarker and not this biomarker and you’re this disease type not this this disease type or you could be in these five disease types but not in these two disease types and you couldn’t have had this prior therapy so it’s like these complicated rules it’s basically what what we call an expert system which was something I worked out at Lo Martin in 1997 it’s these

[00:12:17] complex rules stored in a database that can then be retrieved based on a specific patient at that point in time what’s their disease what biomarkers do they have what drugs have they had in the past and other we know about the patient all that information is way too hard for a person to to remember that’s why doctors and cancer specialize there’s no doctor that is like at a big hospital that’s I know about every Cancer I know about breast cancer I know about lung cancer it’s the local community oncologists that really need this help because they can see any kind of patient if you’re not at a big hospital

[00:12:49] you’re going to see a melanoma patient or a person with lymphoma so knowing each and every single one of those diseases and what how to best treat them is really hard to do without software that’s where we come in we have the software that basically does all that kind of processing cleaning up clinical data clinical data is very messy we have a lot of PDF reports like they’re unstructured reports we parse those and use NLP on those and then we help our clients basically normalize all that information to our knowledge base it’s a Precision oncology knowledge base that has genomics and it can scale with genomics

[00:13:22] which is hard once again always looking for the hard problems because the hard problems are where there there’s value and then after that I can then match a patient to a variety of of drugs or trials and there’s the opposite problem which is I have this clinical trial which patient can go on my trial that’s another hard problem or I have this drug are there enough patients that have this mutation that actually makes it worth me creating this drug going through clinical trials and going to Market with this drug that’s a whole different problem as well our system can do that too go find me all the patients that

[00:13:55] have this type of disease and this these types of biomarkers and we can bring those back and then if you store that information along with What treatments they’ve gotten in the past and what the outcomes of those treatments were we now have what’s called a real world data or real world evidence database which then if you’re a doc and you’re like okay I have this patient they have these mutations they have this disease type what are my options and it says oh you could give them drug a drug b or drug C which one do I give them now you can ask the real world evidence database and it can say okay we we found a thousand patients the third and a third have gotten those abnc drugs

[00:14:27] and here’s a what’s called Meer curve or survival plot that says oh drug C seems to be doing the best so let me go ahead and bring that up and potentially give that to my patient based on that information unfortunately a lot of this stuff is still hunting in the dark here because you know it’s all so new

Ron [00:14:44]: I just wanted to point out a couple things you’re so far past these Concepts after doing this for 10 years now for for the basic person like me Y and when I worked at genome ecology there’s two things that still stick in my head first first no two cancers are alike every Cancer is different if a couple different people have lung cancer it’s not the same and that’s where the Precision medicine part comes in and the and the specific drugs designed to potentially provide a therapy for individuals because we have surgery we have chemotherapy and we have radiation which

[00:15:18] are essentially poisons

Ian [00:15:18]: correct yep

Ron [00:15:20]: to try to eliminate or eradicate cancer but now the space that that GenomOncology is playing in along of the developers of these drugs is the matching of the drug or the therapy or the clinical trial with the specific genomic makeup of the person and so

Ian [00:15:36]: yeah targeted therapies and immunotherapies those are really the two key things that we’re managing and we’re managing what’s the What’s called the eligibility criteria how do you like what what patient would be a good fit for this drug and which drug would be a good fit for this patient and it’s a matchmaking service effectively it’s at its simplest form but it’s obviously way more advanced than that

Advice for students

Ron [00:15:58]: that was obviously very eye openening so can we transition into the two areas we talked about so students in college or soon to be graduating what advice would you give them as far as tools to add to their bag while they’re in college before they graduate things to think about as they’re planning to enter the workforce may maybe or maybe not approach it from your lens of a CTO and a person with an extensive software development background or stay higher level what’s up to you

Ian [00:16:29]: software developer is it’s a different business than a lot of things because there’s the opportunities to to work on open source projects or build your own thing with your own computer and you can work on that but I think in general that advice is useful which is basically to learn anything meaningful you have to solve really hard problems and to solve really hard problems you have to work on something substantial so meaning you have to like actually take on a project that’s not 10 hours long or 20 hours long because nothing useful or meaningful comes out of a 20 hour project it has to be something bigger it has to be something more ambitious so figure out what that means for you what is

[00:17:02] that project my daughter is learning music so she spends a lot of time practicing her instrument but she’s also in a rock band and so she spends time making marketing materials and getting shows and gigs for their band that’s a super useful skill set maybe the rockman’s not going to do anything but that’s okay because you can learn those skills and and then apply that to to your future life so if I was in graduating today I would try to figure out what that was and if you’re a software developer it’s who’ll make something right go make a product even if you’re the only consumer of that product because you’re just going to uncover harder and harder problems and then

[00:17:35] read some books so the books that I like are I like the cal newort book that I mentioned there’s another one he wrote called Deep work we live in a very distracted World Tik Tok YouTube You’re distracted by these things it’s really hard for folks to have attention spans honestly focusing on a two-hour movie some people can’t even do that anymore the ability to actually focus on work and drive to creating differentiated outcomes s is critical so deep work is a book all about that and it’s about designing your schedule so that you have blocks of time where you can actually dive deep into something whether it’s writing

[00:18:08] or creating software or what have you figure out how to design and develop rare and differentiated skills there’s different advice that I’ve seen where it’s okay you’re not going to be the best at any one thing but maybe you could be the best at two or three things combine it together my my Spiel right now is precision oncology software development and AI like I’m trying to be the the person at that center of the vend diagram so figure out what that vend diagram is for you and then be that person and really try to avoid shallow work right emails and meetings and one-off things shallow work is necessary and

[00:18:43] you got to do it to be a good cooworker but if you can figure out a way to design your schedule to to not be that in my world that that’s valuable but you got to also recognize that there’s a big economy there’s a web article from Paul Graham from probably 20 years ago at this Point called maker versus manager schedule so that is kind of like deep work before deep work which was basically there’s two types of people in the world at least in the business World managers who are just trying to coordinate and get work done and makers who are the actual people individual contributors trying to get work done I’m an individual contributor I’m

[00:19:15] a CTO or whatever but my differentiated skill is actually creating the next new thing for our company and it’s really cool that I get to do that that’s my job now so I’m very happy about that but the manager maker schedule thing is managers are trying to basically every minute of the day for every single person on their team and the maker like saying hey leave me alone for four hours so I can get something done so recognize that difference and have those open conversations with people hey don’t schedule me for 15 minutes three times a day schedule me for half an hour at the beginning or the end and let me get my stuff done I I could keep going on with the Deep work thing

[00:19:47] but to me it’s really developing those rearn valuable skills and then it’s a long-term games a long-term game is your network known you for 25 years known Brad for 20 years develop V that trust with people so that they know who you are that they recognize that you have integrity that you’re going to get stuff done you’re trustworthy those networks and those relationships are super valuable right and I know there’s there’s a hundred people I could probably call and say hey I’m out of a job help me out if I had to so recognize that treat people with respect and do say

[00:20:20] what you’re going to do and do what you say and you’ll definitely be better off for it

AI, expert systems, and automation

Ron [00:20:25]: so you brought up AI so let’s talk about that I I couldn’t have a conversation with you without bringing that up I know you’ve gone all in on it you’re obviously applying it in pretty deep ways to GenomOncology I know personally you’ve posted a lot of articles about it for most of us in the world there’s generative Ai and chat GPT and people like me are fooling around with prompting but for you and for the next wave of of software development experts where again back to the student lens or back to the college lens how would

[00:20:58] you suggest that that cohort dive in I don’t know I’ll say the right way to be the most productive and competitive going forward is there is there an answer to that right now

Ian [00:21:10]: I think there are lots of answers I don’t I think every person has to figure it out for themselves so I’ve been doing AI for seven or eight years depending on how you define AI I’ve been doing AI my whole life because expert systems which is what I was developing 25 years ago is basically if then statements and there were people back then and there are still people today that think that’s actually true AI we’re going to have codified knowledge and that’s how things are going to work and then there’s another side of things that’s called symbolic AI if you’re interested and then there’s another side of things called neuro AI which is basically saying just take a bunch of data and get a bunch of computers and throw

[00:21:43] the data at computers and we’re going to get magical AI too I actually think the answer is both right you’re going to scale up this thing on the side this currently generative AI with more and more gpus your V stocks going to the Moon all that stuff and then you’re going to have symbolic a I which is going to be expert systems rules codified knowledge and then those two things working together system one and system to thinking is how I refer to it based on a book from Daniel Conan and the basic gist is that’s how you get to the the world where things are bet are trustworthy

[00:22:15] trustworthy AI is what I’m trying to build trustworthy AI is reliable AI something that actually works and does what you want it to do and responsible AI meaning it treats people with respect and it does things in a way that was more morally acceptable so if we can build trustworthy AI That’s what I want to do so that’s like the the background if I thinking about it now where are we going with this stuff I do believe that there’s an little adage going around the AI is not going to replace me it’s the person using AI is going to replace me that’s that’s certainly a thing to consider IC aai is just the next evolution of automation so the cotton gin

[00:22:48] I couldn’t think of an earlier example was was automation it was the assembly line there’s Excel and Excel macros and formulas like those are all types of Automation and they’re all aiot un quote and I just think of generative AI as a new type of automation the current AI that we have I don’t think is going to be AGI unless it gets programmed in a way that’s interesting because and then also you just got to figure out what do people mean by AGI so the simplest way I can describe the like two things that it can do that’s interesting to me so the first is with generative

[00:23:20] AI you can describe what you have and what you want and it will fill in the middle for you which is really powerful concept rather than figuring out how to do something like how okay what buttons do I have to press in this application to sort these things and dedup these things you don’t have to figure that out anymore you can just say here’s some stuff I got sort and dup it for me and it gives you what you want because you’ve just declaratively in context learning told it the other thing that it can do is it can solve classes of problems when those classes of problems have known classes of solutions meaning if

[00:23:54] you have a specific problem and you go to Google and you say here’s my problem it’s going to give you if there’s a solution on the internet on stack Overflow or one of these other websites Cora it’s going to give you the answer and be like here’s the answer that’s a known problem with a known solution the cool thing about geni is you can give it a type of problem and it can give you back a type of solution probably the specific solution for you because it’s mixing and matching stuff but mixing and matching interpreting that information isn’t true reasoning it isn’t true intelligence in my mind it’s a form of automation that’s a cool

[00:24:28] magic trick and it’s but just because it’s a magic trick doesn’t mean it’s not useful so figure that stuff out use it don’t be afraid of it it’s not going to take over all the jobs think there’s going to be new types of jobs will software engineering be a job in 50 years I think it’ll be a a thing but it probably won’t look it will not look anything like what it does today Y and the thing that humans can do is have problems and solve new problems and AI has no no problems an AI can’t solve something

[00:25:00] radically new but what it can do is it can mix and match stuff and generate things that kind of look new because we haven’t seen them before but honestly just more of the search space it’s just more of playing go there was move 37 which is like this move that the AI did that had never been done before and everybody freaked out about that’s not really intelligence it just searched all the possible things it could do and it found a a unique solution and it did it so those are like my Lev thoughts of it and one of the really interesting things to it that I’ve really embraced is

[00:25:33] the idea of activation energy meaning I’d have all these projects ideas oh I could do this or I could do that side projects or just like little tools I could build and a lot of times now I can actually just chat with the chat bot and like figure out oh this won’t actually work right and I just stop like i’ I I’ve done that I’ve done that recently which just basically like I have this idea tell me how I would build it and then I work through it it and then like even just trying to push through and say oh no try it this way try it that way I realized no this is untenable doesn’t work and I can stop the idea so I’ve just saved myself you know

[00:26:06] 30 hours of pain and then the other thing I but on the flip side I could say oh here’s a problem that would take me 40 hours but with chat GPT I can do it in four and doing something in four hours versus 40 I’m gonna do it I’ll do that now I have this 4 Hour investment now I have this tool and then that tool is going to save me 40 hours or 100 hours going forward and that was a worthwhile trade so that’s those are the types of things that are really exciting to me as far as just as a partner or co-pilot I hate that word now just because everyone’s reusing it but that is the it’s an assistant trying to build an

[00:26:39] assistant is is really what I’m trying to do and that’s what I’m actually building right now for genology is the GenomOncology assistant which is I have this knowledge base I know what the truth is I have an expert they’re the person that’s going to be accountable they’re the person that has actually make decisions a computer can’t make a decision because a computer can’t be accountable so you have have the human you have the knowledge base which actually is ground truth because I’ve had experts curate it and now the chat bot can actually just make using my knowledge Bas easier which is really the thing I want to do what do I have I have a patient what do I want I want a solution

[00:27:11] for my patient to help them or I am a patient and I want a solution to help me which is really where things are going to go so given that how does the chat bot help that person understand the context of their situation and then query on knowledge based to bring back relevant true facts with evidence not just hallucinations and then help that person guide them through an open-ended open-ended is the cool part like I don’t have to go make a program that can handle every single use case the chat bot the assistant really helps us do that

Advice for experienced talent

Ron [00:27:42]: very cool you’ve done a lot of interviewing over the years so for experienced Talent as they are going into interviews or they’re applying for roles or being recruited what have you seen that’s stood out in your mind over time where individual ual have differentiated themselves from one another in that process anything that you could share

Ian [00:28:02]: I’ll tell you my biggest pet peu first my biggest pet is people who on their resume either just list a bunch of Technologies there’s no way you’re an expert at all this stuff you’ve been out of college for three years there’s no like just fundamentally doesn’t make sense for you to know all these things so tighten that up second thing is now you’ve more experienced and you were on a bunch of teams and you did a bunch of projects tell me what you did not I was around a bunch of people doing cool stuff what did you do how did you meaningfully impact

[00:28:34] stuff because that’s what I’m going to try to figure out maybe some other interviewer won’t try to figure that out but that’s what I’m going to try to figure out which is okay you were on this team you did this hard thing you solved this problem that no one else could solve or maybe it’s not even a technology problem I’m good at technology there’s technology problems and there’s people problems and everything’s really a people problem at the end of the day I’m not as good at the people problems but explain the people problems you’ve solved those are just as important and just as valid as well so make sure that’s Crystal Clear not just I was on a team and we did cool things like what did you do how did you meaningfully impact things what problems did you see and what value did you create

[00:29:07] that’s the trade you’re trading your time solving hard problems so the business can do better and then eventually everybody gets everybody wins that’s the long-term goal so have long-term thinking too that’s the other you know other key thing is um don’t just look at everything as a transaction the last few years have been terrible with covid and things like that right people I look at uh LinkedIn profile and it’s 9 months here 12 months there 18 months there that’s not a good look right not from my perspective but I also know that the world’s changing too there’s you know there’s a there’s another side of it too where corporations aren’t necessarily holding

[00:29:40] up their side of it but I thought I would be more of like an independent consultant person and look at me 20 something years later I’ve had two jobs effectively and one boss

Book recommendations

Ron [00:29:46]: you mentioned a bunch of books and I know you love to read when I know I don’t know was it maybe a year ago you put out your hey here’s the five books I think you should all read for or to learn AI I bet you that’s changed several months later I’m still plowing my way through those on Audible what are the top five books you would tell people to read today it doesn’t have to be about AI

Ian [00:30:07]: yeah let me check off the AI books real fast so the AI books were genius makers by Kate Mets just to get a history of things who were the players what were the Technologies what happened what were the big events then there were two books by this by this group of academics there was like I have it behind me somewhere but basically one was about basically the power of prediction so prediction machines that’s what it was so prediction machines was the first one and then the second one was a followup to that the basic just is how are we going to make these things actually have business impact what’s the ROI on AI and how do we do that and so it really walked through the corollary of electricity so if you were to go back in time

[00:30:40] to when electricity first hit the factory floor it didn’t have much of an impact because the factories were designed for pre- electricity and then when they brought electricity in they could put it in certain spots where it was like oh okay this is a point solution that could use electricity and so Now widget a goes to widget B and electricity makes that that one part better it wasn’t really until I’m assuming it Ford but it was really the assembly line where they redesigned and re architected around electricity new systems thinking that was how this step worked and actually made a meaningful impact

[00:31:12] so the thing that’s going to happen over the next 10 years or 20 years is going to be we got this new tool it’s working great in these like Point Solutions or prediction markets wherever something’s you’re trying to make a prediction great plug it in there’s really nothing there or generate a par for my email great that’s I can plug that right in but to actually have meaningful impact we’re going to have to redesign our ecosystems where are humans in the loop where are our AI in Loop and how do we design these things and then also recognize that we’re going to design I’m designing my stuff recognizing that gp4 and the smaller open source models are

[00:31:45] the dumbest these things are ever going to be and the slowest they’re ever going to be in a year they’re going to be twice as fast and twice as smart so I’m designing with that in mind those are the AI books and then as far as like the career books it’s so Cal Newport has a new book coming out I haven’t read that yet because I don’t think it’s actually out so I like him there’s a book called drive by Dan pink which I don’t think I actually read I just read the bullets and the bullets I remember the bullets are like what is your long-term career goals aspirations obviously people want to make money and and have a nice life a nice family but his real thing was to be really happy there’s

[00:32:19] three things you want out of a job autonomy Mastery and purpose for me I have all three I’ve succeeded in that way autonomy I’m kind of My Own Boss I can kind of work on cool hard stuff that’s what I like Mastery I get to work on things that are hard problems and try to get better at them and the cool thing about software development is I know when it works and therefore I can readjust and reconfigure I know what quality looks like and you might not actually know what quality looks like so that’s actually a great place to get a mentor like a mentor can actually tell you that’s good that’s not good and a bad Mentor will tell you that’s good every time and then fix it for you when you’re

[00:32:52] not looking and then purpose so I was building e-commerce sites that was fun honestly it didn’t fulfill me I’ve had people in my life pass away from cancer uh very meaningful people and so working on something every day that I think it hopefully make the world a little bit better place is definitely something that I aspire to as well

Ron [00:33:11]: appreciate thatan really appreciate your time and thanks for sharing so hopefully we’ll see you

What is a Custom GPT?

Mon, 08 Jan 2024 00:00:00 GMT

Custom GPTS are a configurable, shareable chat experience available to ChatGPT plus subscribers. Custom GPTs were announced on November 6th, 2023 at their inaugural Dev Day event and the GPT Store was announced on January 10th, 2024.

In this article:

What makes up a Custom GPT?
Product Announcements Leading up to GPTs
Key Benefits of GPTS
Key Risks of GPTS
Why create a GPT?
Further Reading

What makes up a Custom GPT?

Custom GPTs have the following properties configurable by its creator directly in the GPT Editor:

Name
Logo
Description
Custom Instructions
Conversation Starters (max. 4)
Knowledge Retrieval
- Maximum 10 files
- Maximum 512MB per file
Optional Access to OpenAI “Capabilities”:
- Web Browsing
- DALL-E Image Generation
- Code Interpreter
Actions
- Schema (via OpenAPI specification)
- Authentication Setup (OAuth or Token-based)
- Privacy Policy URL (for shared GPTs with Actions)
Sharing
- Publish: Only you, Anyone with Link, Everyone
- Public, Shareable URL for non-private GPTs

Product Announcements Leading up to Custom GPTS

ChatGPT Plugins (March 23, 2023): Plugins allowed ChatGPT to call REST API endpoints which demonstrated a very powerful paradigm of language model “tool usage”. Unfortunately, the discoverability and usability of plugins for both developers and users of plugins was subpar, leading to a lack of product-market fit. Plugins have been refashioned as Actions in GPTs.

Code Interpreter (March 23, 2023): Originally released as a plugin, the code interpreter is a sandboxed version of the Python interpreter that enables the creation and running of scripts. Ideally suited as a low-code data analysis tool, it is still available as a GPT capability and as a standalone GPT called Data Analyst.

Web Browser (March 23, 2023): Originally released as a plugin, this enabled ChatGPT to search the open web and bring back content for analysis. This capability remains in all standard chats, as a GPT capability and as a standalone GPT called Web Browser.

ChatGPT app for iOS (May 18, 2023): The native ChatGPT app was released to iOS in May and rolling out to Android customers starting in July.

Advanced Data Analysis (July 6, 2023): The Code Interpreter plugin was renamed to Advanced Data Analysis and made available for all ChatGPT Plus subscribers. The capability was also enabled during plugin “auto” mode in combination with other tools.

Browse with Bing (September 27, 2023): While the ChatGPT Web Browser plugin was available since March 2023, it was generally unreliable and had been disabled for users who were not browsing. As of September 27, 2023, all ChatGPT Plus and Enterprise users now have web browsing capabilities in default conversations.

Key Benefits of Custom GPTS

Improved Discoverability: The GPT Store provides a searchable directory of existing GPTs by category. Something that plugins lacked. You can find GPTs even if you don’t know which company created them. For example, you might be searching the Travel category for inspiration without a specific company in mind.

Zero-code configuration: Using the GPT Editor provided by OpenAI, developers can provide custom instructions, upload files for knowledge retrieval, and enable capabilities without writing any code.

Improved usability for non-developers: Knowledge workers who are not developers can easily create a GPT for a very specific use case. I created the OpenAI CookBook GPT by simply uploading 134 pages of PDF documentation. There’s no Python or JavaScript required.

Revenue sharing: GPT creators will eventually be paid based on user engagement.

Key Risks of Custom GPTS

Prompt leaking / Injection: Since the custom instructions are simply prepended to the system prompt, it is generally possible for a user to retrieve the custom instructions by asking the GPT to reveal them via some clever prompting. For commercial or competitive use cases, this may be problematic. More information on prompt leaking.

Knowledge extraction: If you upload proprietary files to the knowledge retrieval feature, it may be possible for a user to extract information from your files. For example, there are prompts that can be used to extract the entire content of uploaded files.

Limited Action capabilities: OpenAI’s Action capabilities are limited to a small set of authentication methods. For example, at launch, GPTs do not support certificate authentication, custom authentication, or complex OAuth flows.

No revenue sharing at launch: Although OpenAI has said they will have revenue sharing for GPT creators, this isn’t available at launch time.

Why create a Custom GPT?

New tool usage paradigm: From a developer’s perspective, tool usage via custom GPTs is by far the easiest way to enable LLM function calling. Instead of needing to write Python or JavaScript code to call APIs, all you need is an OpenAPI schema. This removes a lot of the friction for API providers.

Easier than a traditional chatbot: For companies that want to provide a conversational assistant-style interface to their content or APIs, GPTs are by far the easiest way to provide such an experience.

New way to deliver software: Because of tool usage capabilities via Actions, you can create very compelling software experiences by combining OpenAI’s conversational ability with your own data and APIs. This opens up a new category of “Conversational Software”.

New business model: The eventual revenue sharing opens up a new business model for software creators who can create tools that are used by multiple users and get paid for the usage.

Access to 100 million users: By uploading to the GPT Store, you can immediately reach all ChatGPT Plus users who use the GPT Store.

Using grammars to constrain llama.cpp output

Wed, 06 Sep 2023 00:00:00 GMT

Context-free grammars have increased the accuracy of my large language model-based biomedical data extraction pipeline.

The llama.cpp project, which is a high-performance library for running LLMs locally on CPUs, GPUs, and Apple’s Metal graphics platform (e.g M1, M2), has recent added the support of grammars to guide and constrain the output of the LLM.

A grammar is a notation that describes the valid syntax of text.

The GGML grammar notation (GBNF) is documented here and there are example grammars for generic JSON, C programming language, and chess moves.

I have gotten pretty good at crafting my grammars by hand, but these tools are helpful for getting started:

Web UI: Grammar Builder (input: TS types)
llama.cpp script: json-schema-to-grammar.py (input: jsonschema file)

For Python usage, this capability was exposed in the llama-cpp-python project starting in version 0.1.78.

To use it, there is a class called LlamaGrammar that is passed into your LLM instance on inference:

from llama_cpp.llama import Llama, LlamaGrammar
grammar = LlamaGrammar.from_string(grammar_text)
llm = Llama(model_path)
response = llm(prompt, grammar=grammar)

LlamaGrammar also has a from_file helper function.

Grammars work by guiding and constraining the LLM as it is predicting the next token.

This feature eliminates the challenges with trying to force the model to generate well-formed JSON via prompt engineering or via post-processing on the response text.

In addition to guaranteeing the output, the overall quality and accuracy of the underlying response “logic” improves as well. The grammar acts like guardrails in bowling which not only prevents gutter balls (i.e. not well-formed JSON) but also increases the likelihood of a strike (i.e. the correct answer).

My biomedical data extraction pipeline extracts drug names, dosages, genomic variants, biomarkers, and clinical findings from unstructured text. Before grammars, I was seeing about 15-20% malformed JSON responses which would require either falling back to a more expensive model or complex parsing logic.

With grammars, I get 100% well-formed JSON responses and the accuracy of the extracted data has improved by approximately 25%. This is likely because the model is forced to “think” in terms of the structured output format from the beginning of the generation process.

Example Grammar

Here’s a simplified version of the grammar I use for biomedical entity extraction:

root ::= "{" ws "\"entities\":" ws "[" ws entity_list? ws "]" ws "}"

entity_list ::= entity (ws "," ws entity)*

entity ::= "{" ws
  "\"type\":" ws entity_type ws "," ws
  "\"text\":" ws string ws "," ws
  "\"confidence\":" ws number ws
  "}"

entity_type ::= "\"drug\"" | "\"variant\"" | "\"biomarker\"" | "\"finding\""

string ::= "\"" ([^"\\] | "\\" .)* "\""
number ::= [0-9]+ ("." [0-9]+)?
ws ::= [ \t\n\r]*

This ensures that the output is always a valid JSON object with an “entities” array containing properly formatted entity objects.

Performance Considerations

There are some performance implications to consider:

Grammar-constrained generation is slightly slower than unconstrained generation
Complex grammars can significantly impact generation speed
The grammar parsing adds computational overhead during inference

However, for my use case, the benefits far outweigh the costs:

Eliminated need for retry logic on malformed responses
Reduced need for post-processing validation
Improved overall pipeline reliability
Better extraction accuracy

Future Improvements

I’m exploring several enhancements to my grammar-based approach:

Dynamic grammar generation: Creating grammars programmatically based on the input text
Hierarchical grammars: Using different grammars for different types of biomedical documents
Grammar optimization: Profiling and optimizing grammars for better performance

If you’re working with LLMs for structured data extraction, I highly recommend experimenting with grammars. The initial investment in learning GBNF syntax pays dividends in improved reliability and accuracy.

Cancer Informatics for Cancer Centers: Sharing Ideas on How to Build an AI-Ready Informatics Ecosystem for Radiation Oncology

Fri, 01 Sep 2023 00:00:00 GMT

The Cancer Informatics for Cancer Centers (CI4CC) community wrote this paper to answer a practical question: what does a cancer center need in place before AI can help in the clinic? The answer is mostly not models. It is data infrastructure, standards, governance, and workflows that make clinical data usable and keep humans responsible for decisions. Radiation oncology serves as the working example because its data is already digital and structured.

I contributed the industry-informatics perspective from GenomOncology’s work on curated knowledge bases and clinical decision support. The paper’s core position matches what I have argued since 2023: the model is the easy part, and the ecosystem around it decides whether AI is safe and useful in a clinic.

Links: DOI · PubMed

Grounding Medical Q&A using ChatGPT Plugins and Knowledge Graphs

Mon, 19 Jun 2023 00:00:00 GMT

This transcript comes from YouTube auto-captions, lightly grouped into paragraphs; proper nouns corrected, otherwise expect missing punctuation and mis-heard words.

Introduction and GenomOncology

[00:00] thank you I’m going to be talking today about grounding medical q a using chair gbt plugins and knowledge graphs before I get started a little bit about myself I’m Ian Maurer I’m the chief technology officer for GenomOncology GenomOncology was founded in 2012 in Cleveland Ohio we serve over 40 clients including 17 NTI designated cancer centers our software is installed on premise or in private Cloud installations we’re HIPAA compliant we don’t take data rights meaning we don’t package and resell patient data and when

[00:31] vendor agnostic meaning we work with a variety of devices and partners to help our clients with their clinical genomic data and using that data to match patients to clinical trials and Therapies of which is really the major challenge that our software solves right the number of biomarker-based clinical trials and FDA approvals and nccm guidelines and European guidelines are growing at a very fast rate uh software and knowledge base solves this problem for our clients by

[01:06] constantly curating this information and keeping it up to date and delivering it to our clients so that they can use this information to best help their patients uh for today I’m going to be talking about medical q a what it is hallucinations if you’re not familiar with that term grounding strategies for reducing the number of hallucinations a quick uh commercial about models versus code knowledge graphs then chat plugins themselves if you’re not familiar a demo and some closing thoughts

What medical QA is

[01:32] so first what’s medical q a so medical q a is a benchmark that’s out there in data set that folks use to test models whether it’s large language models or what have you to see how effective they are at answering medical questions uh one of the recent models is Palm 2 there’s a med Palm 2 version of that that scored very well on the on this medical q a data set and Benchmark and then uh the group that uh openai and Microsoft I believe followed up with a chat GPT 4 version of it or gbt4 version of it and they demonstrated a you know

[02:13] on par performance and they pointed out they did you know kind of less tweaking to the actual calls but those tweaks right what providing Ned Palm 2 with additional information doing additional API calls and things like that that’s those are grounding strategies so grounding is is what I’m going to be talking about which is you know how do we help these large language models perform better and reduce hallucinations another Gap though within large language models is that they don’t have they’re not a database right so one of the

[02:45] things that uh in the clinical genomics space is we deal with a lot of databases for genomics and genes and really other large sets of data there’s a very clever paper called Gene Turing where they you know created a bench another Benchmark for asking large language models questions about genes and and things about DNA and genomics and you’ll notice that you know they they don’t do that well on things like snip and DNA alignment and the reason being is that Snips if you’re not familiar with it are there’s basically a database of IDs

[03:20] identifying different mutations in the human genome and you’ll notice here our rs143 Etc that’s you know 1.4 billion so there’s a lot of a lot of these types of identifiers and things like that and since you know gpt4 is not a database it’s not going to have all that information at the ready and and this is where I think like a Knowledge Graph can can help augment large language models performance

Hallucinations and why they happen

[03:47] so first what are hallucinations right so you know one thing that you know folks like to say about GPT technology is that they’re not you know telling the truth or lying they’re and they’re not because they don’t actually know what necessarily the truth is but they’re and what they’re really good at is actually saying what the answer should sound like not actually what the answer is and this is a nice article by Rodney Brooks who’s a themed roboticist in the field and then you know in the popular culture

[04:18] we saw a New York Times article where you know chechibiti was being used by a lawyer and you know had a bunch of citations in it that were basically non-existent and those are basically hallucinations so you know chat GPT at times will will be very confident in its assertions and and statements of fact when you know in fact it’s actually hallucinating and that’s something that folks need to be aware of so why do these llms hallucinate and llm stands for large language model is because they you know they struggle with

[04:49] ambiguous queries but more importantly they often fail to communicate their uncertainty right they don’t know when they don’t know something um under the covers there’s actually prediction scores that are happening those might be tapped to make their responses you know couched a little bit better in the future I think that that’ll be an interesting approach they have a limited understanding and reasoning right so this is a big debate right do they actually understand do they actually reason my assertion is

[05:16] that they don’t they’re the great at word prediction they’re amazingly valuable tool I use them every day for software development and other purposes but I don’t I don’t believe they understand or Reason uh they they have limited training uh data limitations right so they’re not they’re Corpus of data collected from the internet and books and and other sources they don’t have uh full databases you know when specialized Fields like genomics and they can’t update their knowledge right so the training uh exercise is really

[05:47] where most of the knowledge is baked into the system and that happens once and then from there they do fine-tuning and other you know rhlf and other approaches to make the chatbot uh more usable commercialable product but that process is so expensive that they can’t keep it keep up to date up to date over time uh there’s no real-time interaction right so large language models can’t interact with real-time live data um and you know by by themselves right now but I’m gonna actually be talking

[06:18] about plugins and plugins will actually alleviate some of this uh the verification and Source attribution so they don’t actually have uh references directly inside of them they will generate a reference and if it’s a common reference or something that’s been you know written about a lot on the internet then it’ll do a great job of citing that and then the last thing is really the context Windows right and what a context window is is basically a set of tokens in memory and so you’ll notice that if you’re working with a uh

[06:47] you know chat gpta or one of these large language models you’ll notice that it’ll start to lose the thread of a conversation once it gets past a certain number of words this number of words is growing right so gbt three had only 2.7 a thousand words roughly that it could support and then just put GPT 3.5 just came out with um their 16k tokens and a token is about 0.7 um sorry a word is about 0.75 tokens so that means it’s about 22 000 words that GPT 3.5 uh the 60 DK version print

[07:19] support and then there’s clawed by another large language model uh company called intropic that supports 100 000 tokens and the nice thing is this this number will keep increasing and that’s and that’s helpful uh in fighting hallucinations because it won’t lose the thread as much when it’s having a conversation but more importantly you can then use groundnut more grounding strategies where you’re providing more information in context uh when you’re asking questions and so let’s talk about

Grounding strategies

[07:45] routing strategies so you know the first grounding strategy that folks are aware of is you know pre-training and fine-tuning right so what you’re trying to do is actually bake the information directly into the model and you do that by having you know either label data sets like this NYU solution is taking technical notes and I actually believe they’re actually using this you know unsupervised learning of masking and things like that so it’s not necessarily labeled data they’re taking their own internal data running it

[08:14] through a pre-trading algorithm and then fine-tuning it off of that with some with some test specific labels and once again this is a great way of kind of baking knowledge in making your model more domain specific and therefore it will do better in that narrower domain and they’re they’re getting getting great improvements on accuracy using you know a much smaller model than chat gbt I think that’s 109 million parameter model out of NYU so that’s one approach to grounding but it’s you know fairly expensive and and

[08:48] it might not be reusable by outside of NYU either for you know generalizability reasons right meaning it won’t work at a different institution or also they might not be able to release it because they might have HIPAA data baked into it it might leak HIPAA data you know lots of things to be you know to understand before you make the make decisions like sharing a model like this another approach is prompt engineering right a common one is you know it’s called enrichment and one of the techniques that do that enrichment is

[09:20] using similarity search so what happens is you store a Corpus of data you do what’s called tokenization Vector you know embeddings off of those tokens and those those embeddings get sort of in a vector database and then what you’re able to do is is ask questions embed the questions or convert the question into a set of numbers effectively a vector of numbers and then you can use those Vector of numbers to to retrieve similar documents and then you use those similar documents to then

[09:52] put context into the prompt before asking your question and what this does is it just allows the llm to to have more context and and hopefully respond in a more intelligent way and then there’s a newer technique of this both of these are from the line chain project which is a very popular open source project but instead of just bringing back all the documents and and sticking the whole thing inside the prompt they’re actually doing what’s called doc compression and really this is basically asking the question twice right so they

[10:24] ask the question once the retrieve similar information relevant information and then they use the question to then kind of reduce the amount of the amount of text actually needed compressing it to actually populate the prompt right because the question itself can kind of filter down the paragraphs or chunks of text so once again another great rounding technique for you know hopefully making your llm hallucinate less and there’s lots of great you know technologies that are out there this

[10:53] slide you know might be useful to you if you weren’t getting started with looking at Technologies llama index is a great way of kind of interacting with different Vector stores and and they have some very uh great ways of chunking their data and looking at data in a way that’s a little bit more sophisticated than than just splitting things up by you know paragraphs and text

Models versus code

[11:16] so you know that’s some grounding strategies uh and one other thing that I wanted to point out was models versus code right first if you if you were to take gpt4 and ask it a question like what is five plus five it’s gonna get the answer right now it might not get the answer right if you take two five digit numbers and multiply them together um so that’s that’s kind of the one of the negatives of these tools is that they’re not calculators right in this case they’re not a numeric

[11:44] calculator but instead you know when I came back with the answer and came back in two seconds and then but if you were to ask Python and just say hey what’s five plus five it would come back with 10 obviously and and it would be in 16 microseconds so I then asked you know how much faster is that it’s 118 000 times faster to just ask python than it is to pass gpt4 and while this seems obvious the point of this is code is better when you actually know the answer right when you can actually codify it we

[12:12] use large language models because they’re more generalizable and they and they’re you know able to do things that you know with where writing code might be very difficult to do so use the tools that are best suited for the job and you know understand the fundamental difference between models and code right so models are statistical they’re scalable and they’re probabilistic though and they and their predictions and probabilistic just means you know you’re not going to always get the same answer well code on the other hand is

[12:42] computable scalable deterministic and it’s a calculation and so the scale the statistical model is scalable but it’s going to be 10 to 100 000 times more expensive as we just saw and you probably want deterministic for for answers if possible right so just once again understand the difference between uh what a model brings you and what who it brings you

Knowledge graphs

[13:03] so then let’s get to knowledge graphs what are knowledge graphs so here’s a quote from one of the team members on the Google Knowledge Graph team from a presentation I saw recently on on YouTube and then you know basically says you could machine learn Obama’s birthplace every time you need it but it costs a lot and you’re never sure it’s correct right meaning you can keep trying to train a model all these types of facts uh you know by feeding in Triplets of information like you know what is Barack Barack Obama birthplace

[13:32] Honolulu but you know more effectively is you know that’s stored in something like a Knowledge Graph what is a Knowledge Graph so Knowledge Graph is basically a set of entities and their relationships uh you know some are represented by what are called triples right so uh cows eat herbs right and so you know that’s subject object predicate right so that’s what you know makes up a Knowledge Graph and folks you know also refer to this as good old-fashioned artificial intelligence or symbolic AI where rules and logic are explicitly

[14:05] programmed into the system and you know neurosembolic AI there’s Wikipedia page on this you know and it really breaks down the idea of maybe you know mirroring the human intelligence right whether system one and system two this is a great book called thinking past and slow by Daniel Kahneman and basically system one is a fast automatic intuitive and unconscious uh uh model and and it’s used for patent pattern recognition right so this might represent what deep learning brings to to Bear while system two is slower step

[14:37] by step and explicit and that represents planning deduction and deliberate thinking right and I that’s the reasoning that you get from a symbolic AI like a knowledge graph and so here’s a quote from my tongue from diffbot and so they have a knowledge wrap that’s that’s their product and and basically I believe I agree with this quote right so future of trustworthy AI is the Synergy of these two things right knowledge graphs and large language models here’s his approach that he described basically

[15:07] taking a query enriching it with the knowledge graph so this would be you know bringing bringing back additional information sort of like that prompt engineering uh Concepts from earlier you send it through the llm we’re saying Transformer here because that’s the underlying technology and then you’ve been validate the with the knowledge ref how you actually implement this is not you know specified in this blog article from what I remember but I think it is a good high level approach and then just recently I think two or

[15:34] three days ago I saw a new paper um unifying large language models and this does a great job of kind of breaking down the differences between large language models and knowledge graphs what you know what their strength relative strengths and weaknesses are to each other right so a knowledge graph is good at general you know is good at the you know handling structured knowledge and accuracy while a large language model is better at general knowledge and language processing and generalizability right and then they each have their own

[16:01] cons as well right so the cons of larger language models we talked about hallucinations and the fact that it’s a black box because we actually don’t know why the answer is that it says says What It Is Well knowledge graphs on the other hand you can actually point to a fact in a database or or a neo4j like a graph database you actually point to it and and show you know where that piece of information come from and and who created it in a system like ours where we actually have curators and those curators are continuously updating

[16:32] our knowledge graph and and there’s a full audit Trail in history to every piece of knowledge in it and then here’s an example of like an architecture where they’re showing you know large language models and knowledge graphs working together

ChatGPT plugins

[16:43] so chatgpt and its plug-in system is what I’m talking about next so what are chat GPT plugins right so there was a paper called tool former that’s a great paper to check out and basically it describes the idea of giving large language models tools to use um so that they can actually you know solve problems like you know complex math questions or or things like that and then chat GPT plugins came out very soon after this paper so they must have been working on it before this paper

[17:11] came out and really and that’s what we’re going to talk about now so there’s a plug-in store so this is available I was on the waiting list and got approved a few weeks ago but I believe this is rolled out to everybody at least with those who have the professional edition of jet GPT and there’s lots of great tools so you can see here that I’ve got a couple around reading PDFs um and you know and Wolfram Alpha was kind of the first plugin that I kind of deep dived on and that’s a great example of of a symbolic AI system that

[17:41] complements chechypt really well because this thing actually knows the distance between Tokyo and Cleveland under or whatever you’re trying to figure out and then you know one example of using um ask your PDF which is a plugin that I really like that you can read the PDF and then respond to questions about those PDFs so you know I asked that about another paper about biomedical knowledge graphs that are coming out you know a week or so ago as well and you know it can read the paper it can understand your question and then it can

[18:15] ask the question using the context of the paper itself and so you can see here that describes the fact that knowledge graphs can enhance large language models Etc

Demo of the knowledge graph plugin

[18:25] so now I’m going to do a demo of precision oncology the our knowledge graph and Knowledge Graph plugin uh let’s see so so first a little bit about GenomOncology and our knowledge graph so we have our set of ontologies right so things like genes from hgnc transcripts from the universal transcript archive diseases Etc these are all the building blocks or concepts that are related to each other right so diseases for instance is a very complicated ontology because you know

[18:55] there’s parents and children and there’s you know disease sight and disease histology and those and those ontologies are then used along with varying interpretations and and it’s very variant annotations to then create what are called assertions so we have clinical trials and I have things like prior interventions there’s complex eligibility criteria for these clinical trials our team is hand coding the eligibility criteria by both biomarkers and diseases to then basically curate what who’s eligible or ineligible for a

[19:30] given clinical trial we do the same thing for FDA approved therapies and CCM guidelines breakthrough therapies priority review therapies European guidelines and a variety of other data sources so we curate all this knowledge and it’s really the intersection of these ontologies along with some logic Boolean logically annuals and nuns and then we are exposed that Knowledge Graph with a series of apis so the first API you know is our match API and it takes in the patient’s demographic information disease and Stage age gender

[20:03] zip code any multi-ohmic results and then prior interventions and then asks to interpret that and then match it and then choosing our knowledge graph we can then also integrate with our clients and their systems to do any filtering and augmentation of that information you know for instance clients have clinical trial Management systems that are more up to date than clinicaltrials.gov this in this information get augmented and used to provide more real up-to-date information then the recommendations come out the

[20:33] other end which include you know interpreting variants recommending therapies or guidelines trials nextbest tests diagnostic prognostic or clinical pathways and that information then gets filtered down to our user interfaces and workflows and and then also to integrate it to our clients ehrs and and their other systems and so to develop a plug-in the first step is you have to kind of have be approved on the waiting list and then you’re gonna when I was developing and I used localhost and that’s what

[21:03] this demo is actually is so so our plugin is not on the plugin store right now uh one of the key things we’re trying to do is find Partners folks that are you know looking to investigate this this domain and and work with us because I don’t I don’t believe this type of solution is ready for prime time but you know with a solution like ours I think we can do some great research so I’m looking for um you know key leaders and and researchers that are looking to to work in this field so you know give it given

[21:33] a local host you then configure what’s called a manifest file and the Manifest file points to your open you know your open API specification it’s also called Swagger and this is what the the Swagger UI looks like or the open API user interface looks like when you’re you know exploring your API and so let’s do ahead go ahead and do a demo um so first here’s here’s what the plugin looks like this is a very trimmed down version of our API just to not confuse chat GPT at this point but we’re going to be calling this thing the the

[22:08] main thing to point out is you know certain things are not read by judge EBT so I actually put a note there to remind myself but basically what the the main programming language here is English so I built this API uh kind of plug-in API it’s basically a proxy to our full API and the main thing I did was really think through the English that I used to describe both the endpoints the fields and then the actual responses right so this is kind of a boring response so let’s look at a more complex one so for instance you know getting a therapy back

[22:41] includes you know a list of diseases list of biomarkers assertion types that we’re interested in sources that we’re interested in but then here comes the response the response looks like this but then there’s a schema and the schema is actually where I’m describing the specific fields and values that um that are you know being represented by our API and their responses so reads this document and then is able to respond to it so I now have my plugins turned on so if you were to you know open up chatgpt you can then turn on

[23:16] your plugins and you can then choose one I I haven’t I have this one currently turned on the chatgpt one and let’s go ahead and start asking some questions so the first question I’m going to ask is variant interpretation so let’s see here so I’m going to take a VCF file and ask it to annotate and um interpret it so annotating is basically meaning read the read the specific lines of this file a BCF file is a genomics file it gets outputted by a tool called a variant caller the

[23:55] variant caller happens by um you know a device maker like a Illumina has their own bearing collars and things like that now what our system is doing is what’s called tertiary analysis which is annotating it annotating the variants that came out you can see here here’s what the query looks like so the query is it figured out what the pattern should be so I describe the pattern in my API spec it determined that based on the VCF file what the chromosomes start ref and all toward they then pipe delimited them for

[24:26] them for them just like I asked it to and then it posted it to our our API and got the response it’s then taking this response and then understanding it and then re-generating it and you can see here that it’s it’s going ahead and providing information including you know information around classification why the classification happened all this is coming from the GenomOncology knowledge base so I’m going to go ahead and stop it because we can’t have multiple chats at the same time all

[24:56] right so let’s go ahead and stop there and let’s ask the next question so the next question I have is FDA approval so let’s go ahead and ask it what if the approvals are there for BRP 600D melanoma patients so once again this is going to figure out from our schema of our open API what the interesting elements are the data elements are and then it’s going to post it now our system can handle you know disease mapping to our system it can parse

[25:31] variants from different formats it can you know understand things like FDA approvals so what it did it asked a question um basically by tracking out the three entities and then it it passed the question and then received back the answers now the powerful thing here is that it also has the the URLs so our information provides you know everything that our in our system from an assertion perspective we encode URLs into that uh assertion for therapies and then that URL is then provided to chatpt so that

[26:03] it can actually include it in its response and that’s really powerful because people want to see the the underlying information so they trust it and so we can see you know this is the FDA insert that was included for that particular that particular drug and it’s going to keep you know writing and responding but we’ll pause there and go to the next question

What the model does without plugins

[26:24] all right so one thing to look at is how does this thing Act without plugins right so for instance this PCF file when I posted the VCF file it got the response back notice that you know vcfl it understands that it’s a VCF file it you know understands that what’s actually in it from a numbers perspective but it doesn’t have any of the actual annotations and it doesn’t have any of the interpretation it’s just you know the structure of the file and the specific values that are in there without any any judgment that’s because

[26:55] the knowledge graph is where all that information lives you can’t it’s not baked into the chatgpt model and then once again same thing with FDA approves you’re gonna get you’re going to get some answer um and but notice here I’m asking about stage three melanoma specifically and in this case uh it responds back saying hey truthfully I you know anything before after September 2021 I’m not going to know about it and it gives me answers but then I said you know why you’re actually showing me therapies for

[27:24] metastatic melanoma and it says oh I apologize for the confusion you’re correct stage three is not the same as as metastatic melanoma and so this is a this is a case where it’s hallucinating right it’s not a bad hallucination these are actually FDA approved therapies it’s just they don’t make sense in the context of stage three and these are some of the subtle um you know pieces of information that’s that’s necessary for you know getting the right response back and that’s really where you want something like a

[27:49] knowledge wrap to support you um you know and there’s there’s more examples of of questions that I asked why is this are you sure that this is actually approved for an adjuvant therapy and things like that so uh and then another question that I could ask is you know what three trials are available strategic which is a you know very promising uh checkpointed checkpoint uh Gene that hopefully will have a drug before and and be able to combine as another checkpoint inhibitor and notice that it can’t

[28:19] give you a lookup of the clinical trials database if you were to actually use our our knowledge base and our API um and hook it up to that GPT plugins it would you know work just fine so let’s ask one more question of chat GPT and we’re going to ask a breakdown of patients with non-smallow cellular carcinoma and egfrx 19 deletion so non-small cellular carcinoma is a disease type in our database it’s a parent disease type that has multiple children child diseases and

[28:52] then egfrx on 19 deletion which is a biomarker from our enviromarker Bridge proprietary capability this is a break this is querying our knowledge base uh with a loaded open source data set called Genie and you can see here that it’s doing a breakdown by disease type first you can see lumbino carcinoma which is a type of non-spawn cellular carcinoma I had the most patients and then you can see breakdown by gender and by rates and you can see the total number of patients is 1623 this is actually integrated to a local version

[29:27] of metabase which is a open source bi tool and so here we can actually show the integration to another system in this case it could be any kind of bi tool power bi or uh Tableau or whatever your you know institution has but this is a look at a clinical genomic database basically a database of patients data and uh their genomic data this is all de-identified and you know not not any Phi whatsoever but you can see here that it’s got you know the breakdown of information and less visualizations so once again just

[30:01] another way of kind of integrating these tools together all using our knowledge graph to support those those Integrations and understanding of the information so that was the demo and so the the full demo of metabase is actually available on YouTube you can find that on the genome on College YouTube channel where I take the genie data and load it into an open source common data model called Odyssey or omop and then use metabase along with GenomOncology SQL API which

[30:34] is a version of our knowledge graph to do those visualizations that I was just demonstrating

Closing thoughts on security and open models

[30:40] so some closing thoughts first be aware of what are called prompt injections so this is not a very talked about problem yet but basically the idea is you can confuse these large language models into you know giving you erroneous information and when you hook them up to tools you know say sending gmails or you know sending zapier’s apps or whatever they’re called you could have some uh you could be opening up unwanted security holes so a couple things make sure your llms can’t make irrevocable changes and take dangerous actions

[31:14] without a human in the loop that’s critical and then my demo the knowledge graph was only using read-only data access my API and I’m using vocabulary terms and enum fields and other Technologies to really try to narrow down the amount of hallucinations or override ability somebody might have and and so it’s opening up they’re opening up security risks or whatnot another interesting thing capability is this function calling which I’m currently I see is kind of an inversion

[31:46] of the chat plugins capability which is rather than asking chat gbt to call your plugin you are telling gpt4 over the API about your functions what they what are available what tools you have available and then asking them questions or you know providing some information and then having the GPT model respond back here’s how you would call your functions with the information that was provided and this is an example of you know what the weather in Boston right now where you know your breaks down the the information into the parameters that the

[32:22] function would accept and then use that information to call your API another thing to be aware of is that there’s open source models right so hugging face has a leaderboard I’m dropping tracking these pretty closely mostly around controllability right do people want to be able to run these models locally be able to do fine tuning domain specific models this is a great great way to keep on top of that information also there’s a project called gorilla which is you know it’s very similar to

[32:54] chat GPT plugins where basically it’s they’ve trained a model on on figuring out how to call apis based on the inputs provided to the large language model and then here’s a video of another exciting project called llama CPP which is you know running locally on your on your on an M2 machine and you can see here how fast the tokens are so you know that’s the speed is is comparable I would say to CPT chpt4 if not faster than that uh and the fact that it’s you know running

[33:30] at 40 tokens per second is pretty pretty incredible now obviously the quality of llama at seven billion parameters is not you know not quite as good as even jet GPD 3.5 so um but there’s lots of progress being made on this technology and I believe pretty soon we’ll be you know running fairly significant large language models behind firewalls

Next steps and collaboration

[33:52] so the next steps if you’re not already on the plug-in development waiting list go ahead and join that you know it seems like the open AI is opening this stuff up to quite a few folks but if you’re you know not on the waiting list that might be helpful if you’re interested in developing this stuff and then also contact me so here’s my email here’s my LinkedIn I’m interested specifically in working with collaborators at cancer institutes mostly that are interested in using this stuff or at

[34:22] least doing research on how we could use large language models to do medical q a and using knowledge graphs to ground them that’s my main focus I’m also interested in maybe creating a data set or a benchmark around this technique or capability specifically in Precision oncology but GenomOncology is also going to be expanding on Beyond cancer as well this year so please reach out and let me know if there’s any anything we can collaborate on

The My Cancer Genome clinical trial data model and trial curation workflow

Wed, 01 Jul 2020 00:00:00 GMT

Cancer clinical trials are described in free text. A human can read an eligibility section; software cannot match patients against it. This paper, written with the My Cancer Genome team at Vanderbilt-Ingram Cancer Center, describes a structured data model for cancer clinical trials and the curation workflow that fills it. Curators translate each trial’s diseases, biomarkers, and eligibility criteria into structured records. Those records power automated matching between a patient’s tumor profile and the trials that fit it.

GenomOncology built the curation software and the matching engine behind this work. The model described here still underlies our trial-matching products. The workflow lesson aged well: curated, structured knowledge is what turns a pile of documents into something a machine can act on. That same lesson now drives how we build tools for AI agents.

Links: DOI · PubMed

Python in Biology and Genomics

Wed, 07 Mar 2018 00:00:00 GMT

Introduction

[00:00] Python is often used in big data situations. One of the more personal sources of large data sets is our very own genetic code. Of course, as Python grows stronger in data science, it’s finding its way into biology and genetics. In this episode, you’ll meet Ian Maurer. He’s working to help make cancer a thing of the past. We’ll dig into how Python is part of that journey. This is Talk Python To Me, episode 154, recorded February 9th, 2018. Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem,

[00:45] and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I’m @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via at Talk Python. This episode is brought to you by Codacy. Learn how they make code reviews better by checking out what they’re offering during their segment. Ian, welcome to Talk Python. Hi, Michael. Thanks for having me on. Yeah, I’m really glad to have you on to talk about Python and biology and genomics. These are two areas

[01:13] where I’ve wanted to do a show on for a long time, but just haven’t managed to get the right stuff all lined up. So really excited to see how Python is playing a role here. And I think it’s just another cool example of how Python is being used in all these really varied ways. Great. Yeah, it’s really been taken off the last few years, and it’s gone really well with what we’re trying to get done at our company. Awesome. So let’s dig into that. But first, let’s hear your story. How’d you get into programming in

From Commodore 64 to Python

[01:44] Python? Yeah, so I started programming when I was 13. My parents got me a Commodore 64. I started learning basic and trying to make my own games and things like that. I went to school for programming, computer engineering, learned a lot of C and Pascal even at the time to date myself a little. And after graduating, I worked at a defense contractor in their logistics department where we did some SGML and XML based tools for documentation of these complex systems that they

[02:16] have. And part of that was, you know, parsing of those files. And we actually were using this library that was built in Python. And I kind of fell in love because of the REPL, right? So being able to kind of open up the REPL and explore and play with the information right there was what hooked me. And ever since then, I’ve kind of been doing it as a hobby, hoping that it would kind of take off in the web space. And it did with, you know, Django and following on after Rails, but it just never worked out for me. I was always doing Java based development for, you know, e-commerce sites and other stuff that I’ve done.

[02:48] Java again? No. Yeah, always Java. So I was doing Java. I still like Java, still a good, still consider myself a good Java developer. When I joined my current company, we did a couple of small things in Python and, and then those kind of hooked, took off a little bit and we were able to, you know, kind of just double down and, and add some more features. And, and really since that time with bioinformatics and other stuff,

[03:12] we’ll talk about, you know, Python’s really taken off and actually made sense to, to really use it as, as one of our core languages for some of our products. Yeah, that’s cool. So it’s, it’s finally like grown into this place where it’s not just, oh, I’d like an excuse to use it, but it really makes sense, right? Yep. Exactly right. It actually solves the niche and it’s really taken over actually for Perl in a lot of ways in, in the bioinformatics space. And, it kind of sits along with R and has really got a lot of mind

[03:39] share in the, you know, in this bioinformatics world. Yeah. There’s probably some cool infographic of Perl, I’m sorry, R in Python, like duking it out over, you know, some sort of data science crown. I don’t know what we’ll see where that goes, but they’re both doing really well. And it’s, it’s nice to see Python growing so quickly. I really think, you know, you look at the growth of Python, there’s like this huge jump in its popularity. Like it’s always been growing, which is kind of amazing, but it has this sort of inflection point where it grows faster around

[04:08] 2012, which I feel like is where the data science stuff really started to kick in for Python. Yeah. NumPy and scikit-learn and Jupyter, pandas, you know, some of these core, a lot, all the machine learning. Yeah. All the machine learning stuff, all those things really have just kind of gotten some mind share altogether. And it’s really just, we’re kind of riding that wave and it’s, it’s really great. And, and I, and I think you might’ve said this in one of your previous podcasts, but the fact that

[04:31] people who are in data science want to learn something that is a general purpose language too, that they could use to make themselves a little bit more marketable is, I think another, another kind of feather in the cap for Python over some of the other languages. Yeah, it definitely is. Awesome. So this, you know, sounds like a really interesting way of getting into it. So you went through the computer engineering perspective. Very nice. And I think maybe the first place to start this discussion really is to talk about the biology and, you know, your company and kind of the problem space that you guys are working in. So then we talk

GenomOncology and Cancer Genomics

[05:05] about all the tools and the way Python solving the problems people know. So maybe tell us a bit about what you do day to day. Yep. So I lead development for a company called GenomOncology out in Cleveland. We’ll talk more about Cleveland later, but so I lead our, you know, software design development, testing, and deployments were founded in 2012. And really that timing is important because really around 2011, some of the big NGS platforms, next generation sequencing platforms were, came out around there. So these are, these include things like the aluminum my seek and ion torrent. And why those are

[05:37] important is because the human genome project, which you might’ve heard of kind of wrapped up between 2000, 2003. When did that start? Like late nineties, mid nineties, late nineties. It took, it took a few years for sure. And, you know, it took about $3 billion to complete it. And that was basically just mapping a first draft kind of, of the, of the human genome. And, you know, that’s really, it basically says, these are all the variants that, you know, quote unquote, a typical human is made up of. And that took about $3 billion to do. And now we’re talking about, you know, less than a thousand dollars.

[06:07] And as that, you know, as the, you know, Moore’s law applies to computer chips, right? There’s kind of a Moore’s law effect, but even I’ve read some, some analyses where it’s like, it’s even greater exponentially than Moore’s law with these costs of genomics. It really is just driving the price down, which makes it a lot, allows us to apply these technologies for lots of different reasons. And the, you know, my favorite part obviously is the work that we’re doing around helping people with cancer and helping use genomics to help people find clinical trials, find therapies,

[06:37] and hopefully improve their, their odds at, at fighting that disease. It’s definitely one of the great challenges of our time. You know, it’s, we’ve sort of solved the problems that, that were really bad for humanity to a large degree that, and now cancer is like, one of the major, major things that people have to, to deal with, right? It used to be, you might be hungry, you might be getting eaten by a wolf. Now, you know, you live, you live a long, healthy life until something, you know, you get some kind of bad news, right? And so how much is cancer

[07:08] a genetic problem versus other types of problems, right? You guys are building genetic tools. How’s this all fit together? Caveat this by saying, I’m not, you know, a molecular pathologist. I’m not, you know, a bioinformatics person, but cancer is a, is a disease of the genome, right? So you have your genome, you know, 23 chromosomes, 23 pairs of chromosomes, you know, you’re talking about, you know, chromosome one is got 2000 genes, you know, 250 million base pairs, right? That’s the kind of the scope of the data that we have.

[07:38] And cancer is really mutations within that genome causing things to break down in a certain way, right? So, you know, one of the analogies, and there’s a book called One Renegade Cell that we kind of make all of our new employees read, really walks you through, you know, the gas pedal, the sticky gas pedal and the cut break line. And basically what ends up happening is your cell, you know, if you were to cut, have a little cut on your finger, the cells around that cut would know to kind of multiply and grow and then cover over that cut. And then they know how to stop, which is really an amazing feat.

[08:14] It’s actually unbelievable that the machine that is humans works or any form of animal, really. It’s incredible. And it’s all these individual cells and there’s different signals throughout the cell and those signals are called pathways. And what ends up happening is those pathways stop working in some fundamental way. And the way that they’re stop working is through mutations. And those mutations can occur due to, you know, some environmental factor like smoking or uv light or some other mechanism that

[08:42] causes that mutation to happen. And then from there, it ends up that one cell ends up growing and taking over the space of the other cells. So a lot of these drugs and therapies that are out there are looking, you know, some of these targeted personalized therapies are targeting those individual cells that are kind of going off, going rogue and bringing them back and, you know, getting rid of them so that the healthy cells can, can do their thing. And so, you know, our, our software, our company basically is in the business of helping people, helping oncologists, helping pathologists and other folks in the healthcare

[09:20] industry identify what these mutations are, figure out what they mean, and then help their patients get them on a clinical trial or prescribe them a therapy. Right. If you understand the actual genetics that’s causing the problem, maybe there’s a better, more focused sort of treatment, right? Exactly right. Yeah. So if you look at chromosomes, like we talk about big data all the time, right?

The Scale of Genomic Data

[09:42] But I mean, there’s only 23 pairs, so that’s no big deal, but they’re actually made up of a lot of stuff, right? So maybe like take us through the sort of big data store, just sort of the scale of the data, I guess a better way to put it around chromosomes and genetics. Yep. So there’s, you know, 23 pairs of chromosomes and a quote unquote normal human being, right? And you have about 3 billion base pairs across all those chromosomes. So they, and they get labeled one to 22 and then X and Y for the sex chromosomes. But the 3 billion base pairs in the human genome, there’s about 21,000,

[10:16] 24,000 of those are what we call genes. And genes are what are the actual thing that code to proteins. And proteins are the thing that actually make the whole system work. So the genes, the actual DNA part of it is the base pairs. And there’s three base pairs, ACs, Gs, and Ts. Those go in pairs of three, you know, sets of three. If you remember from biology, those build out to become amino acids. And then the average person has about 30 million or so variants, 10%. So in one of the tricks that we do, obviously, in the space is we don’t actually record all three, 3 billion base pairs. We just record, we just record the delta, just to make it, you know, a lot less data. And then the other part of making it a lot less data is, you know, focusing on specific genes, right? In cancer, you know, there’s, depending on the disease type, there only might be three or four genes that matter, or maybe there’s only 50 genes that matter. But in this pan cancer, across all the different types of cancers, there might be about 800 or 900 genes that matter. So, you know, our types of tests and sequencing that we do really focus in on those, those smaller regions to just kind of manage the data in a faster way. That collection of, you know, 3 billion base pairs, ACs, Gs, and Ts, those are, those are what are called the reference genome. And the reference genome is what everybody gets compared against, right?

[11:33] So when you’re, do your, when sequencing is done on a tumor or on a, you know, a normal cell, the deltas, the variants are what it’s actually captured and recorded. And we’re actually recording it also in the context of what genes are there. So genes actually don’t make up a huge amount of the genome. It’s a very small portion of the genome that actually codes the proteins. Interesting. So is there like a bunch of basically instructions that are just off? They just don’t go, they don’t do anything? They call it junk DNA. Now they don’t necessarily, it doesn’t necessarily mean it is junk. It’s just not necessarily known at this time, or it doesn’t, it doesn’t code to protein, but maybe it does other things. Like there’s things called methylation and these other factors that affect the coding parts of it. And there’s lots of theories of, of how that happens. Some of it’s through evolution and, and pieces just kind of fall out and don’t actually matter anymore in the human species. But there’s other, other theories that maybe some of it isn’t junk as well. And then even within the exon, even within the genes themselves, there’s exons and introns. So the exons are these, these strings within the gene that actually get, you know, sliced out and turned into the RNA. And then that goes and codes the protein.

[12:44] And then the other part’s called introns. The introns are the parts in between each of the exons. So understanding how the whole map works, understanding how to sequence the data, get the data off the sequencer and, and then keeping track of all that data is, is interesting. And, and one of the things that might be interesting to your, to your listeners, just because of the whole Python two, Python three thing is these reference transcripts get released over time. Right. So, and the one that’s currently, you know, the main one that people use in the clinical setting is called GRCH 37. And that was released in February of 2009 and lots of tools and things were built off of this version of the reference genome. Well, over time they learn new things, they apply new regions. They, you know, it’s, it’s a very dynamic map. And then in 2013, right, eight, five years ago now, almost they released GRCH 38.

[13:33] The whole industry hasn’t moved over to this new, this new version of the reference genome. So it’s just the, because you got to update all your tools, update all your databases. And it’s a, and it’s a, it’s a tricky thing to do. This major incompatibility. How interesting. So you talk about this reference genome and there’s about 3 billion base pairs and make up a person. How much of that is consistent across every single person and how much of there’s difference?

[14:00] Because, you know, I feel like we look at people, we all look quite varied, but then you also hear things like, well, your DNA is 1.5% different than say a chimpanzee or something like that. Right. So give me a sense for, you know, when you say I’m going to save the Delta, what does that look like? Usually an average person has, you know, about 10% variation from the 3 billion. So about 30 million base pairs will be different across different, different people. And, you know, it’s obviously the numbers go up and down and there’s prevalency frequencies, right? So a lot of these databases that are out there and available for, for people to consume as part of their process, they actually say, you know, we sampled a thousand people.

[14:38] And, you know, 20% of folks had a G in this spot and, you know, another percent had an A in that spot. So that’s, that’s a big part of just understanding what some of these variants are. And one of the things we do in our tools for cancer is that we’ll, doctors are interested in that, that, that prevalency, that’s that allele frequency. Because if the frequency is 50%, well, there’s no way that that’s actually a cancer causing variant because people would be born with cancer. And that just, it just doesn’t really work that way. It wouldn’t be a viable situation.

[15:08] So one of the data points they like to look at is how often does this variant actually happen in the, in the wild and, and actually in the human population. So it’s a very interesting stat. Yeah, for sure. Okay. Interesting. So maybe let’s talk about how do you actually do the sequencing at a high level that won’t get into the tools and the Python code that you actually make, how that’s working in there.

The Sequencing Pipeline

[15:31] But give us the sort of overall pipe. Yeah. Give us the general pipeline. Like how do you go from, you know, a swab on the cheek or whatever it is to, here’s your printout. You ACCGTAC is, is you. There’s been older technologies that, you know, work in smaller regions and, and can do things like that.

[15:51] There’s a thing called Sanger sequencing. But as I said earlier, one of the major changes is in 2011, they did this next generation sequencing. That basically takes raw data right from, from a blood sample or a tumor sample. They put it in this, this machine called the sequencer. And then through the, you know, either chemicals or, or, or lights of, of that actual machine. And once again, this is in my area of expertise.

[16:15] They’re able to analyze it and basically do what are called reads. So they’re doing, you know, 65 base pairs across or, or what have you and read out ACs, Gs and Ts and write that to a file. And that’s written to a file called, you know, either a fast day file or a fast Q file, which has quality associated with it. So all these raw reads are happening and it’s basically like little snippets of a book. And, but it’s like a book that they, someone’s cut up into little fragments and then kind of thrown it up in the air and then try to figure out how to reassemble it. So that process, yeah.

[16:46] So that process is not something we do at my company, but that process is what we call alignment. And we take that book and try to basically tape it together. And the way they do that is by trying to compare regions against the reference genome itself and through math and algorithms and, and some machine learning. Now they’re able to kind of align the whole readout of the, of the reference genome. And those get stored into a file called a SAM file. And really it’s just a, just a listing of all these different variants, but in a line format.

[17:16] And then those files can get compressed into what’s called a BAM file. And then we, you know, there are tools that are open source and, and tools like ours that actually allow you to do visualization of that alignment and really get a good understanding of do the reads line up? Do the variants look right? Is the quality there? And do you believe the actual calls that are being done? And then the next step after aligning it is actually what’s called variant calling.

[17:41] So the, you know, some additional software, once again, stuff we don’t actually do. It goes through the, these alignment files and makes decision and say, yep, I’ve, I’ve read through this, this BAM file or SAM file. And, you know, I believe at this position on this chromosome that it’s an A and not a T. And obviously with two pairs of chromosomes, you might have, you know, half of them being A’s and half of them being T’s and things like that. And cancer is a little bit different, right? Because you then have a mixture of, of tumor cells that are kind of commingled with normal cells.

[18:14] So you might actually get allele frequencies, what we call variant allele frequencies or VAPs that are not 0.5 or 1, but something in between. Because it could be that actual mutation that is causing the cancer. So like half of them have some values there, others have another, right? The original normal cell and then you have these clones of tumor cells that actually, the actual cancer causing cell that is now growing and spreading in that region. Right. So that gives you more or less, here’s what we think the genetics is.

Variant Interpretation and Decision Support

[18:43] And then you have to analyze it, right? Right. And this is really where we come into play, right? So our company started in 2012 just because of this NGS data, you know, was starting to overwhelm pathologists and physicians with lots of genomic and molecular data. And the belief of our company is that, you know, all medicine is going to be molecular in the future. And really understanding how that, those, what those variants mean in the context of cancer, especially, is where we really focus our energies.

[19:09] And that includes things like annotating the variants and trying to help people understand, you know, how often do they happen in the population? Has there been papers out there that said this variant’s pathogenic or benign? There are some prediction models that people have written that say, you know, this variant will cause the protein to degrade in some known way, right? In the stuck gas pedal or the broken brake line analogy. And then from there, we’re able to do decision support, right? So there are FDA drugs that are available.

[19:41] There are clinical trials that are available. These things have very complicated eligibility criteria. And our software helps you, helps doctors, you know, make sense of all this disparate data, bring it all together and say, oh, yeah, for this patient, given these mutations and maybe some other tests and some other data about the person themselves, we can say that, you know, this clinical trial is best for you. Or, you know, this therapy would work for you. The FDA has approved it for you. And one of the interesting things that’s happening is to prove the whole idea of cancer is a disease of genetics and not a cancer of, you know, something else, is that these drugs that are getting approved for, you know, lung cancer with a specific variant, well, that drug might work for, you know, a melanoma patient with a specific variant or vice versa.

[20:24] I might be getting the analogy wrong, but you get the point. Basically, it’s the specific mutation that matters. The fact that you have a V600E on BRAF is the most important part, not the fact that it was on your skin. That’s pretty interesting. Understanding at this level is really powerful. So let’s talk about the software stack, maybe at a high level first, then we can dig into some of the tools.

The Software Stack

[20:45] Like, what kind of software are you guys writing to solve these problems and where’s Python fit in? We started off with, you know, a research application that we used to, you know, get the company started. And then we built our first clinical app for pathologists. And that was all built using Java and a language and GWT. So Google Web Toolkit is a Java-based JavaScript tool, right? So we don’t really have any JavaScript wizards in-house.

[21:08] And we’ve always been, you know, Java-based. And so while that was getting built, we actually partnered up with a team at Vanderbilt University called My Cancer Genome. And they have a website for people that are, you know, looking for information about genetics and cancer. While the rest of my team was kind of building this, our first couple of products, I actually built a curation tool for them. And I built that with the Django admin tool, right? So Django has this great admin tool.

[21:32] So we started, so I was able to kind of whip together a nice content management tool for them so they could get rid of their SharePoint solution that they were running at the time. Anything that gets rid of SharePoint, that’s a good thing. That was the thinking there. You can hold your head high that day. We turned off SharePoint. Right.

[21:48] So, yeah. And having a, you know, quick user interface for that. And then we’ve since evolved that tool. And now that tool is managing not just, you know, some basic content management stuff for the My Cancer Genome site. But it’s basically managing all of our knowledge and what we call our knowledge management system. And then what we did was built on top of that Django REST framework API.

[22:07] So using, you know, Tom Christie’s tool to build out an API using, you know, REST. And now you can, you know, hit the API and get back specific information running a thing called Match in our software. So you can, you know, given a patient’s information, their demographic and whatever biomarkers they might have, you hit our API and we’ll give you back, you know, hey, this is a good clinical trial for you. You know, within, you know, 50 miles of the patient, here’s a good trial for you to maybe put them on or here’s a therapy that’s approved by the FDA. That sounds really, really powerful. And some cool tools that are involved in there.

[22:41] You talked a little bit about user interfaces. Is that all Java or are you doing some UI in Python? I’ve heard your recent stuff about UIs. All our UIs are in Google Web Toolkit right now. We are doing the new version of the My Cancer Genome website using React. So that’s one piece of JavaScript that we’ve started to use.

[23:00] But for the most part, you know, we’re building out strong APIs with Python and then our UIs and things are still with Java and GWT. Yeah, that sounds good. I’ve heard a lot of good things about React, but I haven’t done anything with React, so I can’t speak too much to it. Yeah, cool. This portion of Talk Python is brought to you by Codacy. If you want to improve code quality, prevent bugs and security issues from making it into production, and at the same time speed up your code review process by 20%, then you need to try Codacy.

[23:31] That’s C-O-D-A-C-Y. Codacy makes it easy to track code quality and identify and fix issues by automatically analyzing your commits and pull requests with all the most widely used static analysis tools. Codacy helps great teams build great software. Join companies like DeliverHero, PayPal, Samsung, and more. Try your first code review by visiting talkpython.fm/Codacy and linking your GitHub or Bitbucket account. You can also just click on the Codacy link in the show notes.

[24:01] All right, so let’s talk about some of the tools that you’re using. So you talked about Django REST framework. That’s Tom Christie’s tool. I had him on, or framework. I had him on the show a while ago as well. So it’s basically layers on REST API on top of Django, right?

[24:17] So maybe tell people how you’re using that, like what it’s doing for you. One of the key things that we do is annotations. And one of the annotations people want to know is, okay, where is this variant? And where is it in the context of the whole genome? And that’s called the G-dot. Or where is it in the context of the coding region of a gene?

[24:36] And that’s called the C-dot. Or where does it end up land, you know, once it goes from a C-dot to a P-dot, which is the protein, right? So the actual amino acids. So G-dot, C-dot, P-dot. So that is a nomenclature called HGVS. There’s actually a lot.

[24:50] And so our API actually houses, you know, all of our knowledge. But it also calculates annotations for people. And one of the great libraries we use is the BioCommons and HGVS. And those two libraries are open source, open on GitHub. And they do a really good job of doing those calculations. So if you’re trying to understand, you know, how to get into genetics, I’d look at those libraries.

[25:15] There’s also a library called Biopython. We don’t use that, but it’s also really good. And then from a bioinformatics perspective, you know, we use that full stack. So we have on top of our API, we’ve built out some user interfaces that use actually Jupyter and Bokeh and Pandas and NumPy. So I actually take that back. Our genome analytics platform, you know, the major part of it, the container part of it is written in GWT.

[25:40] But it’s actually calling in and bringing in Bokeh plots as well. So Bokeh is being used on the back end using Pandas to calculate these great plots. And then we’re rendering them in our front end. Yeah, that’s really cool. I’ve never had a chance to use anything with Bokeh. But that’s where you basically do the sciencey visualization stuff on the server in Python.

[26:00] And it just transfers over to the web front end. Is that right? Yeah. So it’s calculating the JavaScript for you. Because once again, we don’t have the JavaScript chops in house. But you’re basically running pure Python using Pandas data frames.

[26:14] And then you basically configure your Bokeh plot using this Python library. And then it renders it. And then it basically streams out HTML and JavaScript. And you can just kind of embed it in an iframe or what have you in your UI. And it works great. And it sounds really great.

[26:29] Like you don’t have to be in the charting business. Exactly right. Those are live, right? They’re not just like PNGs or something. You can definitely work with them dynamically right there. You can use them to generate PNGs if that’s what you need.

[26:41] And some of our clients do need that to include it in their research papers if that’s what they’re using our tools for. But yeah, it’s got lots of different use cases. And Python keeps coming up with great libraries for visualizations. And there’s lots of different options too. But Bokeh has worked out well for us. Yeah.

[26:59] It’s kind of becoming a paradox of choice, right? Like there’s a little – as soon as you learn something – yeah, as soon as you learn something, you’re happy with it. You’re like, but that looks better. Maybe I should do that. And of course, it’s a constant treadmill sort of thing. So one of the tools that you’re using that didn’t surprise me but I think is interesting and I want to hear more about is spaCy.

Natural Language Processing with spaCy

[27:17] So I don’t even think I’ve mentioned spaCy on the podcast before. Tell us about that. What’s spaCy? Yeah. So we’ve done really a proof of concept at this point using natural language processing. So one of the major challenges in our space and IBM and a few other big companies are spending lots of money to try to tackle this problem.

[27:35] But basically the problem is a lot of these EHRs, EMRs, people are recording their notes about patients in kind of free text. And one of the challenges with that obviously is it’s unstructured and it’s hard to do anything with it. We’re not really in the business of major machine learning. We’re kind of in the workflow and tools business. We help people solve problems in kind of a more pragmatic way. We’re a small company.

[27:58] We can’t spend billions of dollars. But what we’re doing is we’re taking spaCy and using that to parse some of these free text files and basically make recommendations to people. So doing things like what are called entity recognition. So entity recognition means I’m reading this Wikipedia article and finding all the proper nouns in it. Barack Obama did this in Detroit, Michigan or whatever. Those would all be proper nouns.

[28:23] And this is a great tool for extracting out named entities like that. We’ve trained spaCy to find named entities based on our ontologies, our data within our KMS. Right. These are our important words. Go see if they say this. Something like that.

[28:39] Exactly right. So there’s a pattern matching framework that’s within spaCy that’s really very easy to use. And then the other thing we’d use it for is for classification. So basically we’ve trained some models to say, OK, when you read this sentence and it says, you know, estrogen receptor strongly expressed. Well, we want that to actually mean something. We want that to mean ER positive in our in our use case, in our vernacular.

[29:01] And that means something to our our end customers. And what it really does is what we then do is present it to them and say, hey, we saw this sentence and we’re you know, we think it says this. Do you agree? Yes or no. And if they say yes, then we kind of keep that that piece of information and use it to further train our model to make it better over time. We’re not really trying to we don’t really think we can get rid of the human in the loop at this point just because, you know, we’re just just at the start of this thing and we want to make sure we get the right answer 100 percent of the time.

[29:29] But what we want to do is make it so they don’t have to read, spend a half an hour reading through a document where we can just scan it for them and say, here are the interesting parts. Please go ahead and just confirm it. That’s pretty wild. I feel like this whole machine learning AI business is deeply reaching into medicine and things like that. Right. This is just another super interesting example I hadn’t even thought of.

[29:52] But, you know, in terms of oncology, like the analyzing, say, scans like pictures to see, you know, have the machine say, no, that looks like cancer to me. Like kind of doing what radiologists might do or something. Right. Exactly. Yeah. It’s pretty amazing.

[30:06] We like spaCy a lot. I originally tried playing with NLTK a few years ago and actually kind of ran into some barriers. It’s an old that’s an older project. spaCy is really modern and that it’s, you know, kind of does some of the best practices with Python. I highly recommend it. The documentation is really good.

[30:20] Performs really well out of the box. And I was able to pull together a really good demonstration in just a few weeks. So I highly recommend it. Looks really cool. It definitely, they have it lined up to, when you go to visit spaCy.io, it really looks appealing and polished. I was wondering why you didn’t choose, what the difference or what made you choose spaCy over NLTK?

[30:42] It’s actually pretty obvious straight away, isn’t it? They’re doing a really good job with, you know, as a small open source, you know, company. I think there’s like maybe two people working there from what I can tell. And they’ve, you know, they’ve basically open sourced their core product and they’re selling, you know, ancillary products on top of it. And they’re consulting services too. And, you know, it seems like a great project.

[31:02] Yeah, that’s really cool. And I definitely look at it more because I’m always fascinated how these people are building really interesting business models on top of some kind of successful open source thing. So, yeah, another cool example. So you’re building some interesting CLI tools and you guys are using Click, which is pretty common. That’s from Armin Roenker who made Flask. You’re also using Pex.

Packaging CLI Tools with Click and Pex

[31:27] That, I think, gets less, a little bit less awareness. Tell us about Pex. It’s really interesting. Click’s great. There’s obviously lots of great ways of building, you know, command line tools in Python. They’ve been doing that for a long time.

[31:38] But Click’s really, really easy to use. And then what we find is, you know, how do we get this to our clients? We do a lot of things with Docker. And when we’re setting up servers, using Docker to set up a server is great. But we actually also have now command line tools that we’re trying to distribute to people. And, you know, pushing things up to PyPI and having them pull things down using pip and having them set up virtual environments.

[32:00] It just sometimes gets a little bit difficult for some of our end users who, you know, might not be Python, day-to-day Python developers. So using Pex, you’re able to actually just build the whole module together with the virtual environment baked in. And when you deliver it to them, it just kind of, it just runs. And you can build it to different platforms. You can, you know, on my, on my, for one of my projects, we have a little docking container that actually builds it to Linux and then builds it to macOS. And we’re able to share it out to people and, and use the tools without having to go through the whole virtual environment setup stuff.

[32:33] That’s really cool. So I think Pex is the one that actually takes everything, zips it up, and then it turns out Python can execute zip files and run from there, right? Which is pretty wild. Do you know if that entirely eliminates the dependency on Python? Like if I had a blank machine or is it just sort of the packaging, but they got to have the base Python there? Someone asked me that just the other day.

[32:56] I don’t think, I actually, this thing, it’s just the libraries because it doesn’t seem that big of a file. It’s not like, it’s not like when you download a clips and you get the whole jar with it. Yeah, yeah, yeah. I get the whole Java JDK with it. I, you know, I actually don’t think, I don’t think so. It’s pretty cool.

[33:09] Yeah, I’ve been playing with Pi Installer and it’s pretty nice as well. And it kind of, it’ll do it so there’s no dependency. It’s also more problematic because it’s trying to solve the problem bigger, I think. So I was just thinking, oh, maybe Pex is going to be nice. Another thing that I think is really cool around this stuff, just as a shout out, is that I’ve been playing with a lot lately is this thing called GUI. G-O-O-E-Y.

[33:31] Have you heard of this? I did see your little prototype up on GitHub, I think. Yeah, so you can take something like this and then just throw like a little UI with dropdowns instead of command line arguments on top of it. It’s pretty cool. Right. So another thing that you are doing is AIoHCP.

High Throughput Annotation with AsyncIO

[33:45] Tell us, are you using the server or the client component of that? All client. So for us, it’s high throughput annotations, right? So one of our clients, you know, basically paid millions of dollars for this high throughput system to generate, to go through the whole alignment and variant calling situation, right? So they’re trying to do high throughput, you know, lots of thousands of cases per week or whatever they’re doing. And they’re trying to keep up with that.

[34:09] But they need annotations from our KMS, our knowledge management system. And so the challenge was, okay, how do I keep up with them? And the first version of my software had trouble, right? So we were, you know, trying to parallelize things with multiprocessors and it worked. But, you know, once I’ve actually played with AIoHTP and AsyncIO and really understanding how to program in that paradigm and really look for the IO bottlenecks and work around them, it made my redesign of that tool we called our annotator that actually does that annotation much easier. Right. So now, you know, I have basically have these five stages in my little program with queues in between them, you know, where basically what an annotator does is really just reading a file, making a call to an API.

[34:55] A remote API of our service, right? Exactly right. And then injecting that data into the stream and then writing it out to disk, right? So you got basically, let’s just say, three spots where you can leverage the AsyncIO. So reading from the original file, making the call to the HTTP server, and then writing out to disk. And, you know, this whole framework allows me to do all three of those things. It kind of just magically balances itself with regards to how much it’s reading from the disk, how much it’s writing to the disk, and how much it’s calling the API.

[35:24] The only thing you have to do is make sure you don’t call your API too much unless you want to take down your server. And then our server on the other end is, you know, highly parallelized through using Celery and Redis and handling. It can scale up because we’ve thrown lots of hardware at that. And so what we’re able to do is we’re able to keep up with that, you know, multi-million dollar hardware solution with Python 3 and AsyncIO. And it’s been great. And probably like, what, one thread?

[35:49] So basically, yeah, one process running and it’s doing the job. So we do process, we can then, we can then scale out that one program across multiple processes if we want. But it’s, it’s really pretty high performance and, and our client’s pretty happy with it. That’s really awesome. Yeah, because so much of the time, programs that are slow, they’re actually just waiting on some other part of the system. They’re waiting on the web service.

[36:09] They’re waiting on disk. They’re waiting on, you know, whatever, right? And so this lets them be productively waiting, basically. It’s definitely a paradigm shift. And you have to, you have to think through the whole, this Async method is calling this other Async method and, and really understanding how that all fits together. And it can definitely bend your brain a little bit if you’re not used to it.

[36:28] But once you actually do figure it out, it’s kind of a superpower and it’s really great. Yeah. And as far as superpowers go, like the actual change in the programming model is pretty mellow, right? There’s like not, it’s not that different from serial requests. Yeah. You just got those couple of keywords with Async and await.

[36:44] And once you figure that out, then it’s kind of easy from there. And then it’s just really about using cues. And then you get into the whole queuing theory and, you know, lean manufacturing and that kind of stuff and try to understand, like, how do you, how do you remove the bottlenecks from your system so that, so that things go as fast as they possibly can go. And if you, if you kind of have that background and mentality with it, it’s, it’s really cool. Yeah, that’s cool. But of course, anytime you’re thinking about concurrency, it can definitely sort of bend your mind, like you said.

Django Channels and Tumor Boards

[37:09] Yeah, exactly. So speaking about concurrency, another thing that you guys are using that’s really cool is channels and celery and Redis. Channels, is that like Django channels? Yeah, Django channels. So one of our tools, there’s actually async mode to it. So in the oncology space, one of the big things that happens is for challenging cases, they go to what’s called a tumor board.

[37:29] So some of your bigger hospitals will have a tumor board where basically all of the experts at that hospital, or even they could even, you know, WebEx other people in from other hospitals to get to the experts to help people with, you know, rare cases, right? There’s a case, there’s a variant, they don’t know what it means. What do they do about it? And that’s what they call a tumor board. And we build software for that. And one of our modes is actually async mode where people can kind of, so they don’t actually have to have a WebEx, they can just kind of go to our app.

[37:56] And everybody’s in the app at the same time. And if there’s a leader, the person’s moving around from one page of the app to the other, that’s sync mode. And that’s actually done using WebSockets. And so if you know anything about Django and, you know, its history, so Django started off, it was built on WSGI, and that’s a synchronous protocol. Yeah, all the popular ones are. They still haven’t found a way really around it.

[38:19] Godwin, Andrew Godwin? Yeah, Andrew Godwin, yeah. He added this capability to Django, which is basically kind of like this little side thing to Django called Channels. He invented another framework for interfacing in with Django from your web server, right, from Apache or Nginx, and using ASGI, I think is what he called it. And it’s an asynchronous platform. And so that enables us to do WebSockets.

[38:43] And the WebSockets is the thing that allows us to do this synchronous movement between different people on our application. So if, you know, one person clicks a link and jumps to another page, all the other people that are on the app jump along with them. And really, the main goal of this is to allow people to kind of dynamically work with the genomic information at their fingertips rather than having, you know, a bunch of people on their phones Googling. What do these variants mean, right? So they’re all kind of working together on a single call. So you guys sort of built, like, the Google Docs.

[39:14] You kind of added a Google Docs equivalent type of experience to your app, right? So everybody fires up your app and they have this local sort of guided experience. Yeah, that’s a really good analogy. Yeah. Yeah, I think more apps need that. I think that’s really awesome.

[39:27] How hard was it to add this channels, to do the channels code and to add this stuff together? Well, the channels part was easy. I mean, it basically just kind of worked out of the box where, you know, we’re able to send messages from one thing to the other. But once again, you know, getting the actual communication going from one instance to the other is tricky and it’s managing state. And how do you change, you know, from one user to another and make sure that the experience is smooth? That’s always tough.

[39:52] And then as you add new features, you need to make sure that the sync thing works across those new features, right? That’s right. We’ve got this new visualization, but it only shows up for the leader, not for you. Those are always fun. But the actual channels plumbing and things like that, even though it’s kind of cutting edge code for, you know, in beta or what have you, works really well. And adding the Redis channel in between is what ends up happening when you actually set this up.

[40:15] You end up having, you know, your web server, Nginx. You have what’s called an interface server, which is basically an instance of your Django app. You have the Redis channel and then you have workers. So the workers are basically other instances of your Django app, but they’re actually doing the actual work of responding to either a plain old HTTP request or to one of these WebSocket requests. And, you know, all that plumbing just worked great. How cool.

[40:39] Yeah, it sounds really fun. I had no chance to use it, but it definitely looks really cool. Yeah. All right. Well, that sounds like quite the list of cool projects and technologies you’re getting to put together there. It must be fun to work on.

[40:51] It’s great. And, you know, having a purpose and, you know, working for something that’s not online marketing or e-commerce or whatever I was doing in my past life is great. So it’s great, you know, working on something that I think is going to make a difference. Yeah, definitely trying to make people healthier and live more full lives is way better than trying to optimize that click rate or, you know, convert one more piece of data to try to piece together. No, this person is actually that other person and they’re in this demographic, right? Right.

[41:19] Yeah, exactly. So, you know, some other thing that nobody needs. Online stocking is not something I’m interested in now. No, for sure. Cool. So you actually have a couple of somewhat related open source libraries.

Open Source Libraries

[41:30] You want to talk about those a bit? One of the libraries that’s out there is called attrs. And it’s actually, I think, the basis of the new data classes that’s in Python 3.7, right? So the new PEP that does data classes. So there was actually an original project called attrs, which is a really great project. And it lets you define your classes and you get a bunch of, you know, kind of boilerplate Python stuff for free for comparisons and, you know, string representations and things like that.

[41:55] Right. It implements like, say, hashing correctly and all that kind of weirdness that you can overlook. Yeah. The problem I was trying to solve at the time was I wanted an immutable way of reading a YAML file, getting a nested Python object, and not having to, like, munch dictionaries, right? Because you start writing code to dictionaries and quickly things get kind of nasty with some nested dictionary references and things like that. So that’s what I was looking for was a way to round trip to YAML, kind of like in Java, there’s a library called Jackson that’ll do that.

[42:27] It’ll round trip to JSON or to what have you. And Python does a good job of, obviously, round tripping from dictionaries to YAML. So what I wanted was an actual object model and patterns, which is really good. But I kind of had just a different mental model, and I wanted something more like the Django ORM. And I had a lot of use cases where I wanted to basically say, yeah, I want to call this a string field, and I want it to always have this validator and this converter. So what attrs will let you do is when you define your fields, you can say it’s got this converter and this validator.

[42:55] And I kind of just wanted some templatized versions so I didn’t have to keep saying the same thing over and over again. And I also wanted this, you know, this magical transformation. And that’s what the related project does. Related. It looks really cool. And it does look like you’re working either in the Django ORM or MongoEngine or, you know, one of these types of things where you define what the object actually is.

[43:19] Can you have, like, nested objects? You basically can have, if you declared a class A, it can then relate to class B either as a, have a child object B, or it can have a list of Bs or it could have a map of Bs, right? So those, that object model, and it fully knows how to kind of render it to and from a dictionary. And it does the whole serialization and deserialization for you. That’s sweet.

[43:44] So, yeah, definitely people should check this out if they’re working on Python and YAML. It definitely looks like a cool project. So the other one’s called Rigger. You know, obviously we’re in a very, very, it’s very important to us to have the right answers for people. Yeah, the answers have consequences. The most important thing about my job, I want to make sure we give people the best data, the most relevant data, the most up-to-date data.

[44:06] And one of the key things we got to do is testing. And we spend a lot of time testing, you know, by hand. We do a lot of unit testing. You know, we believe in the testing pyramid at my company. But one of the things I like to make sure we have is kind of an end-to-end test or an integration test or a functional test, however you want to describe it. And we, in our Java space, we actually use the tool called Cucumber.

[44:27] And what Cucumber lets you do is basically, you know, declare your tests in a given-when-then kind of English-style DSL. And that allowed our product team, you know, our product specialist team, which are basically non-developers, but they understand the science and they understand how to use the software and test the software, to describe how a function should work, right? And given some state, when I do some function, then I should get some result. But what I wanted was something like that on our API side. You know, I didn’t want to go through the whole pain of having glue, where people actually had the right code that runs behind this DSL. And since HTTP is kind of its own language in itself, I decided to kind of shortcut it and just basically build out a simple YAML-based approach.

[45:11] And that’s kind of where this related project came from. So you write out a YAML file that actually describes your steps. And the steps describe what you make requests to and then get the response back. And, you know, basically it allows us to build out a suite of hundreds and thousands of tests, testing out the software to make sure it gives the same answer every time so that people know when they make changes, they’re not breaking anything. And it does it using AsyncIO because I wanted it to run fast. And then we use a thing called JamesPath to actually transform the response that comes back.

[45:40] So the transformation, that allows for the test to not be fragile, right? So one of our rules for APIs is we don’t let you change a field or remove a field without, you know, some major consequences. But if you add a field, if you add a field, it’s usually not a problem. But it can break your tests if you have very, very specific tests that have all the fields listed. If it doesn’t match exactly like a string test, then it’s going to break. I’m just expecting this string back or this JSON document back.

[46:09] Are they the same? No, crash, right? Like that’s right. Yeah, that’s too much. Yeah. So with JamesPath, we’re able to kind of filter it down and say, yep, I only care about these three fields.

[46:17] These three fields match exactly as I expect. And if so, it’s correct. And so I was going to open source this thing a few months ago. And then I heard on one of your other programs, I think the Tavern CI project was released. And it’s very similar. So people should definitely check that one out.

[46:30] And both our project and that project were built kind of based off the idea of PyRest test, which seems to have been abandoned, which was a nice project. It just had a few things that we needed that it didn’t have. And, you know, I would say that the reason to choose our project over maybe Tavern CI would be this JamesPath thing. We also have API coverage for Swagger. So we define all of our APIs with the open specification, otherwise known as Swagger, which we still call it Swagger. And so we can tell you, oh, you’ve got 100% coverage of all your API endpoints and their variables.

[47:04] And then we also actually included the Cucumber reporting HTML reporting tool called Cucumber Sandwich, which brings up a nice pretty, you know, HTML view of your test and shows you how all your steps ran and things like that. Yeah, the graphical output really is nice and colorful. It’s you could tell you can get info out of it right away. Yep, it’s great. Very cool. And you can see how related fits in there perfectly.

[47:25] Yes, exactly right. Also saw you using AIoHTP. So it’s all like async nice and quick. Yeah, so I wrote AIoHTP to do this little rigor testing so I could do parallel testing to kind of speed up our test suite because I didn’t want them to, you know, if you have to run them sequentially, it’s going to take a lot longer than if I run them all in parallel. So it takes three to five times less time when you turn the concurrency on with our test suite for all of our API endpoints. Very, very nice.

[47:51] All right, so I think that’s maybe we’ll leave it there for the genomics stuff, but that was a really interesting look at how you’re using Python to address these major problems. And I got to commend you. You’ve got a bunch of really cool tools and systems put together, it sounds like. So nice work. Thank you. I mean, Python’s got a great ecosystem, great community, so many great tools.

[48:12] So it makes getting stuff done really fast easy. The paradox of choice is a real thing that continues to vex people building stuff like this, right? Because you build it all out and you’re like, oh, but there’s some other REST calling API thing. You know, there’s maybe I should use API star instead of Django REST framework because Tom Christie’s not working on that, right? But you’ve got to just put a stake in the ground and say we’re building something productive here. Always lots of new toys to play with and it can get distracting.

PyCon in Cleveland

[48:39] Another thing that we want to touch on is there’s some kind of event going on in your city. Is that right? Yeah, PyCon is coming here. Yeah, in May. Is that May 7th, I think? Yeah, so beginning of May.

[48:51] Yeah, it’ll be here and be right down downtown Cleveland, which is a great city. Been here 18 years. It’s about two blocks away from my office, so I’m just going to be able to stroll right over there at the end of the day. And it’s great. Cleveland’s awesome, so people should definitely take advantage of some of the sites when they’re here. I absolutely think so as well.

[49:08] A quick correction, it’s May 9th, not May 7th, but basically the same, more or less that time frame, right? And I’m looking. Can I still register? I think I can. I don’t think it’s sold out yet. It’s not sold out yet.

[49:21] So maybe it will be by the time people hear this. So one of the things I wanted to touch on with you, maybe two parts. One is, what advice do you have for getting the most out of the conference itself by, like, I’m within the walls of the convention center, you know? And then people are going to be in your town, a bunch of folks together traveling here for the conference. Like, what would you recommend they do to get the most out of Cleveland? I haven’t been to a PyCon since 2005, I think, was when I figured it out.

[49:53] So I think maybe Dallas or something like that. I bet it’s a really big difference of an experience. I’m excited to check it out. So it’s going to be great to go. You know, obviously, everything’s online. So if you’ve never been and you never noticed the PyCon on YouTube, definitely check that out.

[50:08] So what that should do is give you confidence that you can miss, you know, some of the speaker, some of the talks that maybe you’re not super interested in and spend more time in the hallway track and talk and meet some folks in the community. Because the PyCon group does a great job of getting all those videos online. Within like a day. So you almost could watch it while you’re at the conference if you really felt like, oh, geez, I wish I saw that. That’s my recommendation there. And as far as if you’re downtown and you’re staying downtown, you know, there’s some great restaurants over on East 4th Street.

[50:35] There’s, you know, Lola by Michael Simon, the Iron Chef. There’s another one called Greenhouse Tavern. There’s the House of Blues, which might have a concert that night. There’s the Rock Hall, which has some special events sometimes. And if you’re, you know, a rock and roll fan, that’s definitely a place to check out. The Indians are in town.

[50:50] I checked. The Indians are in town that weekend. They’re playing the Royals. So if you’re a baseball fan, that’s a few blocks away. Yeah, that’s really cool. So if people are in town, they could obviously drop in and see that.

[51:00] But if they’re traveling from, say, outside the country, right? I know tons of people come from all over the world. Like, when do you get to see a professional baseball game? Right? Like, this might be a chance. Take a couple hours, skip the conference, and go watch it, right?

[51:12] Yeah, the Indians have been good the last few years. So it should be a good team. And then, you know, there’s some other areas, too, to check out, right? So there’s, on the west side, there are Ohio City, the west side market. You know, lots of breweries. You know, micro pub type of things.

[51:27] Definitely check those out. Playhouse Square, which is, you know, maybe another six or seven blocks away. That’s actually the largest performing arts center in the United States, other than New York City. And then University Circle, which is a few miles away. That’s not as easy to get to.

[51:43] There’s Lyft or Uber. Like, you could get there. Yeah, you could get there pretty easy, right? Exactly right. So, yeah, Cleveland’s a pretty easy town to get in and out of, and lots of great restaurants and lots of great things to do.

[51:52] Oh, that sounds really fun. I definitely want to second, first of all, what you said about the hallway track. I may take that track too much when I go to conferences, but I find I skip a lot of the talks and actually just really try to experience being with people. Because when you go to the talk, it’s great, but it’s really, you sit quietly and you watch a great presentation and you experience it there, right?

[52:14] But you don’t interact really with anyone near you or anyone presenting so much at all. And so there’s the hallway track, which is just hanging out, talking to people. And if you find yourself in an interesting situation, just take advantage of that because you can always, like you said, go watch on YouTube the thing that you would have gone to see. The other thing that they’re doing really well there are open spaces.

[52:37] So I find that open spaces are more participation and engagement than the main talks, and they’re not recorded. So there’ll be a board. If it’s like the last two years, there’ll be a big board where people put up index cards saying, in this room at this time, we’re going to just meet and it’s kind of undirected group conversation about something amazing, right?

[52:55] And so definitely take advantage of those as well. That’s great. Yeah. And if you want to connect at PyCon, just send me an email and I’ll look for you there. Yeah. Very cool.

[53:05] And do take advantage of some of these fun things that Ian pointed out. Like the worst thing about traveling is if you just get on a taxi to a plane to another taxi to a hotel to a conference center, and then you pop those off the stack again and you do them in reverse, right? Like you want to go like, I was in Cleveland and I saw this amazing thing, right? You know, like same thing, like wherever you go, try to take advantage of that.

[53:32] So that’s great. That’s great. Yep. Yeah. Awesome. All right.

Final Questions and Advice

[53:35] Well, it’s down to the two questions. So let me hit you with those. First of all, if you’re going to write some Python code, what editor do you run? Converted to PyCharm. It’s great. I use the Vim editor mode and it’s a great environment and love using it every day.

[53:50] Yeah. Awesome. It’s definitely kind of overwhelming when you get started, right? Yep. A lot of great tools and the integration with pytest and the integration with the Vim and Markdown editors.

[53:58] It’s a really good tool though. Yeah, it is. Once you get used to using the feature, it’s hard to not, it’s hard to imagine not using it. Awesome. Okay.

[54:06] And then a notable PyPI package. All right. I’m going to go with deep variant by Google. So I haven’t used this. I probably won’t ever use this, but it’s just such an interesting use of AI. They are actually, you know, taking those BAM pileups that I described and basically using

[54:21] image recognition type AI to actually determine and make variant calls. So what used to be, you know, somebody with a way bigger brain than me doing these calculations with math and trying to figure out the right determination of what a variant is, is kind of being superseded now by this really interesting Google project. So deep variant is the name of it. Okay.

[54:42] That sounds really cool. And just another one of those AIs creeping in to solve these tricky problems. Exactly right. Yeah. Very cool. All right.

[54:50] Well, definitely interesting choices. And thanks for sharing everything. Any final call to action? People want to get involved in biology, genomics, Python? Like, how do they get started? There’s a website called BioStars.

[55:02] There’s lots of interesting topics up there. It’s a stack overflow type clone, I would say. And then there’s stack overflow itself. There’s, you know, lots of conversation there. Feel free to reach out to me if you’re interested in learning more. And, you know, Python is just a great, great ecosystem.

[55:19] And there’s so many cool tools to play with. Yeah, I totally agree. So one of the challenges I see for people getting started in this space is they’re not researchers or doctors. Like, where do they get the data? Do you know of any, like, good open places to get some data to work with?

[55:33] Lots of the research that’s out there is funded by the U.S. government or European governments. You know, NCBI is a website. I can’t tell you what the acronym stands for right now. They’ve got tools. There’s data sets out there such as TCGA, which is called, which is the cancer genome atlas.

[55:49] There’s a project called Genie, which we were involved with helping them analyze their data. And they’ve got lots of cancer data that’s out there. But lots of tools. So search for keywords like VCF and BAM and SAM tools. And there’s lots of different keywords to search for. And, you know, you’ll find lots of different data sets.

[56:09] It really just kind of depends on, you know, what kind of analysis are you looking to do? And you also find a bunch of Jupyter notebooks out there, right? People are people doing their analyses in Jupyter notebooks and then posting them to the web for people to follow along with. And really, it’s I’ve learned all this stuff in the last five years. It’s not insurmountable.

[56:25] It’s just a matter of, you know, having a goal and trying to reach that goal and solve a problem. That’s cool. And it’s great. Yeah. Solve problems one at a time and eventually have this big tool chest, right? Exactly right.

[56:37] All right. Well, Ian, thanks for being on the show. It was great to talk with you and learn all about this stuff. That’s great. Thanks, Michael. Really glad to be here.

[56:44] This has been another episode of Talk Python To Me. Today’s guest was Ian Maurer, and this episode has been brought to you by Codacy. Review less, merge faster with Codacy. Check code style, security, duplication, complexity, and coverage on every change while tracking code quality throughout your sprints. Try them at talkpython.fm/codacy, C-O-D-A-C-Y.

[57:09] Are you or a colleague trying to learn Python? Have you tried books and videos that just left you bored by covering topics point by point? Well, check out my online course, Python Jumpstart by Building 10 Apps at talkpython.fm/course to experience a more engaging way to learn Python. And if you’re looking for something a little more advanced, try my Write Pythonic Code course at talkpython.fm/pythonic.

[57:34] Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. This is your host, Michael Kennedy.

[57:50] Thanks so much for listening. I really appreciate it. Now get out there and write some Python code. I really appreciate it.

AACR Project GENIE: Powering Precision Medicine through an International Consortium

Tue, 01 Aug 2017 00:00:00 GMT

AACR Project GENIE pools tumor sequencing results and clinical data from major cancer centers into one shared registry. The first data release described here covered nearly 19,000 patients. The registry lets researchers ask questions no single institution could answer alone: how often a mutation appears in a rare cancer, or how patients with a given variant fared on a given therapy.

I contributed as part of the consortium through GenomOncology’s work on the informatics side. GENIE showed that real-world genomic evidence works at scale when institutions agree on shared structure — the same premise behind the curated knowledge bases and biomedical data tools I build today.

Links: DOI · PubMed

Precision Oncology Platform (POP)

Sun, 01 Jan 2012 00:00:00 GMT

I invented GenomOncology’s Precision Oncology Platform. A knowledge-graph matching engine sits at its core and matches a patient’s tumor profile against curated knowledge of biomarkers, therapies, and clinical trials. Clinical labs and cancer centers use it to turn sequencing results into reportable findings and treatment options.

Ian Maurer

PangoPup

Most AI Agents Are Toys. These Ones Treat Cancer

Introductions

What GenomOncology Builds

Harder Than It Should Be

Why NLP Was a Terrible Business

Vision Models Change Everything

Rip Out or Keep?

Healthcare at Bureaucratic Speed

Frontier Labs Come for Healthcare

Agents, Tools, and BioMCP

Code Goes to Zero

Labor Displacement and Caregiving

Overhyped, Underhyped, and the Jevons Paradox

BioMCP: Biomedical Model Context Protocol

Introduction: GenomOncology and precision oncology

The pre-training era

Prompting techniques and RAG

Post-training and reasoning models

Knowledge graphs meet language models

Model Context Protocol

What BioMCP is

Demos: trials, literature, variants

The biomedical research assistant

Coding agents and what comes next

BioMCP

Precision Oncology, Convergence of Genomics and AI, and Improving Cancer Care

Cold open and introduction

Exponential cost declines in genomics

Founding of GenomOncology

Chasing the hardest problems

What precision oncology means

Early days and first products

Reducing the burden on doctors

Evolution of the product line

Measuring outcomes and efficacy

Implications of AI

Building an oncologist assistant

Convergence of sequencing and AI

Advice for cancer patients

Lessons from company building and Cleveland

Building GPT Actions with FastAPI and Pydantic

Episode Introduction

Catching Up Since 2018

How Ian Uses LLMs

Choosing an LLM API

Client Libraries and Optionality

LangChain and Prompt Injection

Running Models Locally

Prompt Engineering

Retrieval Augmented Generation

Vector Databases

Custom GPTs and Actions

The PyPI GPT Project

GPTs for Precision Oncology

Library Recommendations and Wrap Up

The Intersection of AI and Genomic Analysis

Introduction

The nonlinear career path

Follow the hard problems

Founding GenomOncology

Biomarkers, drugs, and clinical trials

Advice for students

AI, expert systems, and automation

Advice for experienced talent

Book recommendations

What is a Custom GPT?

What makes up a Custom GPT?

Product Announcements Leading up to Custom GPTS

Key Benefits of Custom GPTS

Key Risks of Custom GPTS

Why create a Custom GPT?

Further Reading

Using grammars to constrain llama.cpp output

Example Grammar

Performance Considerations

Future Improvements

Cancer Informatics for Cancer Centers: Sharing Ideas on How to Build an AI-Ready Informatics Ecosystem for Radiation Oncology

Grounding Medical Q&A using ChatGPT Plugins and Knowledge Graphs