How do you know when a tweak broke your AI agent?

<!-- SC_OFF --><div class="md"><p>Say you&#39;re building a customer support bot. Its supposed to read messages, decide if a refund is warranted, and respond to the customer. </p> <p>You tweak the system prompt to make the responses more friendly.. but suddenly the &quot;empathetic&quot; agent starts approving more refunds. Or maybe it omits policy information in responses. How do you catch behavioral regression before an update ships?</p> <p>I would appreciate insight into best practices in CI when building assistants or agents:</p> <ol> <li><p>What tests do you run when changing prompt or agent logic?</p></li> <li><p>Do you use hard rules or another LLM as judge (or both?)</p></li> </ol> <p>3 Do you quantitatively compare model performance to baseline?</p> <ol> <li><p>Do you use tools like LangSmith, BrainTrust, PromptFoo? Or does your team use customized internal tools?</p></li> <li><p>What situations warrant manual code inspection to avoid prod disasters? (What kind of prod disasters are hardest to catch?)</p></li> </ol> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/Tissuetearer"> /u/Tissuetearer </a> <br/> <span><a href="https://www.reddit.com/r/ChatGPTCoding/comments/1rovgo5/how_do_you_know_when_a_tweak_broke_your_ai_agent/">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/ChatGPTCoding/comments/1rovgo5/how_do_you_know_when_a_tweak_broke_your_ai_agent/">[comments]</a></span>