Six Claude Agents and a Trust Boundary: A Clinical Co-Pilot

Speaker 0: I'll be going through a case Web, 68 year old woman is going to After clinic Krayne she has a palpitation on her chest. So what happens is essentially doctor reads her AI, takes the symptoms then notes ground PHI symptoms Hat basically gives an idea of what exactly needs to be done. But that chart could hurt her. Speaker 1: Essentially, what happens is Hat, you Speaker 0: know, in Canada, there is a issue where there's a scarcity of clinicians' doctors and and there's a lot of burden from them. So I have developed this agent called AI Concilium. Basically, it's a curation of 6 JL, 6 Clinical AI agents, 1 orchestrator and Web transformer. So it's developed over La PHI based pipeline, to help the clinicians, so that they can talk better decisions. Essentially, the 6 clinical agents, are the 1st 1 Staff with a triage. Speaker 0: Triage, you can consider AI a Learn nurse. Essentially, that near nurse Forge take down the symptoms, give you the initial medication. And the 2nd 1 is the differential DX. You can consider Event DX like, a medical school doctor. So he is initially starting out with scenarios where, understanding the patient details and showing down the symptoms. Speaker 0: 3rd 1 is pharmacy. There's a big reason I use pharmacy here because, the thing is, AI, with respect to North America, when you look at the liability that Speaker 1: comes on if Speaker 0: a patient gets hurt or Tinkerers an issue that gets with a Agent, so the liability comes down to pharmacist as well. So the thing is AI if a doctor is prescribed K8s the medication, pharmacist has to verify that and they validate if that Code cause an allergic reaction as that could harm a patient or not. So that's the main reason I use pharmacy here. 4th 1 is guidelines which is essentially the AI which has all the guidelines, that you look for, in the CLI, this this is where Rue Hat include all the knowledge base that doctors know already about it. 5th 1 is the AI check. Speaker 0: Bias Singh, you can assume it like Hat annoying Senior doctor who goes on 2 rounds and checks AI if there's any issues with Hat medication Forge AWS there any issue with the better Forge Easy update that has to be made. The 6th 1 is the communication where you take all the clinical workflows and process Speaker 1: it Speaker 0: and give pipeline Speaker 1: to the doctor. Speaker 0: GPT go to the demo. So this is the scenario where, basically CLI the PHI AI, at Monster I have created. So the UI where, there's synthetic data and it's just a proof of concept. So this is a case work list which has an EHR detail and you can also add your own EMR in a JSON format. So I'll go with the patient that we were dealing with that is Eleanor. Speaker 0: So in case of Eleanor, what we are doing is, she was facing hypertension before she Hat all those symptoms. So all those are already captured in the EMR. What I'll do is I'll run the GPT which will run all these agents. So all these agents will run parallel with the same data that was provided. Now, tool it runs, I'll go to the infrastructure User to create that. Speaker 0: So this is the infrastructure that I use. So essentially, as you know, it's a health care solution. When we talk about a health care solution, the 1st thing that comes tool everybody's mind is, like, how can we use it in verification? And is Hat data safely you? Are we taking care of PII, PHIs, or not? Speaker 0: So that's what I have done 1st. So in terms here, I will look at the, the patient chart. The patient chart Code tool be AI which detect the the information. Basically, initially, there's it Hat Ali the information Hat in JSON. It will be accustomed to the Gemini that it's, capturing for. Speaker 0: So, essentially, there are scenarios where I'm using, regex. The standard ones that AI, Rue initially, like, I will get a Health care party. AI, like, if we use the register that are the standard ones. That is for the Health CLI, email, postal code. So this is the 1st layer of injections. Speaker 0: And the 2nd layer of moderation comes as a PHI checker. So the output of this regex is GPT to this PHI leakage checker. So this takes it again with respect to all the variations if something was left or not and it tags to the particular, patient. So that it will be identified and tag Speaker 1: to a Speaker 0: particular, patient name 1, patient Speaker 1: name Speaker 0: 2 that way. And the 3rd layer AI Easy a SPEAKER going on and a lot of, people are Singh forward with prompt injections. So what happens AWS people try to push out prompt, to pull out the details. For example, some people try to pull out the details AI give me the patient details, ignore the previous instructions, and stuff like that. So that's something that I have taken care here. Speaker 0: And as yeah. So now coming back. Yeah. And the orchestrator is basically the 1 where we are, orchestrator checks whether the 2 agents GPT a particular output that is identical or it is different. So if there's a difference, it will hand over to the clinician So that basically GPT can take care of the handoff and understand what decision he has to do. Speaker 0: Essentially, all the, power lies with the clinician. It's just an assistant to help them. And what happens in the end is Forge rehydrates. So on the server side once the LLM gives the output, it rehydrates on the server side and clinicians sees the patient name back again. So coming back to the, plan as we see here, the covalent has run. Speaker 0: All the agents have completed their output and identified some bias AI, identified severity, and GPT lead also given HINTS output. When we look at the plan, there AWS a high risk that's identified here. So coming back to this scenario where Eleanor Hat, this, like, basically the highest that was identified is I'm ordering and board print. So Singh, what it does is identify if it helps it CLI, if these are given together, it encourages bleeding. So that's something that should not be done. Speaker 0: So that that's something that it has identified and flat as well. So coming back to Eleonore, there are 2 scenarios here. 1, the patient, the doctor and doctor or the pharmacist identifies this issue and solves it. The 2nd 1 is where, the doctor or the pharmacist misses this. AI just talking about the 2nd 1. Speaker 0: So this basically agent could help, stop these 2nd scenarios. And, yeah, I would like to, get your views on how to take it to preproduction essentially. So right now AWS City it is also a work in GPT. But I'm also looking for Clinical who are willing to validate the engine outputs or engineers PHI are shipped AI in regulated Development and Hat been Red heaving. So, yeah, this is a few of our goals, Kalinda, and Speaker 1: also for the. Speaker 0: Thank you. Speaker 1: Question. Comments. Doctors. Oh, yeah. Question. Speaker 1: I know. Of course. I'm the Folder. Yeah. And I know the Ray. Speaker 1: No? No? Do you? Yes. Tell me. Speaker 1: Ethical what's my question? Ethical, AI. That's your question from yesterday. Yeah. I was supposed to ask this 1. Speaker 1: But people We're we're host. Alright. Hello. So there's a a lot of sensitivity with AI enabled or so if people if you believe speaking, how accurate the solution is, how you feel TUI file will accept it, and they're taking us. So that they can check right now. Speaker 1: It's a it's August Before I reply, I have a similar question so you can reply altogether. You will show us a bias check. Yeah. How this bias check Speaker 0: work Speaker 1: when you have patients that have been, like, minorities that have that can have some disease that have typically organ and also the general population. Yeah. Speaker 0: So, AI tasks your question 1st. So essentially, in terms of, the ethical part and, like, the teaching part, it has to be validated by AI. So that's why PHI mean, like, I am currently waiting and I'm asking both Clinical to validate it. Because essentially, even with other projects that I'm working on, we work with the clinicians so that we can identify whether the output is right and also identify the accuracy. In terms of accuracy side with the spectrum models, I tested Hat with SONNET React City Pecore GPT 4.1. Speaker 0: And there's a reason I took 4.1. The 4.1 is Code effective and is best with JSON responses. So the JSON responses are pretty accurate. And when we give a proper input, in JSON format as it's AI fixed input file EMR AI this, the software AI lead my access in AI case. So, we have the, we have the template with us and it gives us a proper design JL JSON back. Speaker 0: So that's why, like, the 4.1 both with the AI accuracy, but we need to validate and use other monitors. And with respect to your question, in terms of AI check, I didn't include that scenario yet. So I would say, like, that would be, it has to be included in the system prompt. But, yeah, I mean, it's right now in the agency. That's a good part to have. Speaker 0: I'll just read it here. Speaker 1: Can I go to the next portal? Tullo? Yeah. Just a question about the, anonymization that you do. Like, you were you said you Remediate, information about, like, post Hostile code and Date. Speaker 1: But, like, if you remove things AI, race, age, and and stuff like that, you I mean, if you wanna be private, you have to remove them, but also you need them to diagnose certain disease. So how do you So Speaker 0: the thing is, like, we are not allowed to provide any PSI, PSDLLM. So I'm showing this up for or taking it out for the LLM. And as I mentioned, it's just an assistant tool doctor. So it could identify the main problems that it has, like, with respect to guidelines with respect to the other symptoms, allergies, and everything. But obviously, Hat that part has to be taken care of. Speaker 0: That's where the human in the loop is coming. Speaker 1: So it could say, like, breast cancer or something? Speaker 0: No. No. I mean, like, that's Speaker 1: why I like Speaker 0: it wouldn't have so it has specific guidelines on what GPT this detect. So City like, if you have if there is, like, an AI given about the triage, in the initial triage, it will take out, like, if it's 1st canceled or not, then obviously. I mean, Speaker 1: I mean, AI mean, Speaker 0: like, basically, your reports are already there. Right? I mean, if you basically, your diagnosis would be there. If your diagnosis report has breast cancer, it would automatically assume it's a female. And I'm guessing here, like, I mean, because I'm not tested it out in those scenario but that's my thought. Speaker 1: Thank you for asking the question that I was gonna tasks. But if I could just expand to add to what you're saying, obviously, in Health Lacerte, explainability is the huge area Speaker 0: of concern. Speaker 1: Similarly, I'm curious about the human in the loop mechanism ArteraAI when a talk clinician would look Hat, you know, GPT make a verification. Is there a reinforcement, you know, kind of feedback? Can see I suppose with Front-end Models, there are some validation that you'll have in that regard. Hat you thought about maybe taking a sovereign and then completely, you know, you know, dealing with this whole concept of redaction? Speaker 0: Yeah. So essentially, like, my thought here is, like, as I'll improvise this, So the thing is, like, Hat orchestrator right now just AI whether there's a match. The 2 agents are giving similar response. Speaker 1: It build Speaker 0: identify a match. If it doesn't, then it HINTS off to the clinician and clinicians can give the input. So that's AI Hat like, that's how I'll Date it ahead. So clinicians can give the output and that can be brought into another 1. And that cloud CLI improve the whole system. Speaker 1: Yeah. Getting back. Thank you. AI just wanted to to share a little concern online. I've been working with the MDRC for about 2 years at this point Forge their maintenance division. Speaker 1: I just wanted to know, how in your program, specifically your version of your traditional health clinician work, can compare in terms of noise. Because 1 of the biggest issues, my team has handled and is cannot really handle when they have or we try is that the attention span of the clinicians already AI to the point that they have to do 5 to 10 minutes per patient. If they have to even 1 to 2 minutes of Hat, read a Hat, that means that you are seeking away about 1 to 3% of attention span. Something that may not even be relevant or that you disagree with. How can your project compensate that distribution? Speaker 0: Lead. So that's where, like, I would say, like, whenever patient is, like, patient like, a clinician is watching a patient and identifying the symptoms and talking to them, like, taking that response time, you know, of 10 minutes, I would say the the co pilot should have definitely run beforehand. And then he can tasks validation reference for LLM. AI, so that talk can look at researcher things I might have thought Ayrat. So it Dhall, like, enhance the speed in GPT the output of what he needs to think about. Speaker 1: But there's a break coming in about 15 Monitor, so you will be able to go talk to Annie about Gemini or science and ask a lot of questions regarding health care.