Technically Legal - A Legal Technology and Innovation Podcast

• Benchmarking Legal AI: Measuring the Delta Between Man and Machine (Anna Guo Legalbenchmarks.ai)

  Is artificial intelligence custom-made for legal tasks better than general AI tools like Google Gemini and ChatGPT? That is the topic of this episode featuring Legalbenchmarks.ai Founder Anna Guo. Anna is a former BigLaw lawyer who left the practice to become an entrepreneur and now focuses her energies on quantifying the utility of AI in the legal industry. Anna's initial anecdotal research for colleagues quickly revealed a strong community interest in a systematic approach to evaluating legal AI tools. This led to the creation of Legalbenchmarks.AI, dedicated to finding out where the promise of humans plus AI is truly better than humans alone or AI alone. The core of the research involves measuring the "delta," or the extent to which AI can elevate human performance. To date, Legalbenchmarks.ai conducted two major studies: one on information extraction from legal sources and a second on contract review and redlining. Key Findings from the Studies: Accuracy vs. Qualitative Usefulness: The highest-performing general-purpose AI tools (like Gemini) were often found to be more accurate and consistent. However, the legal-specific AI tools often received higher marks in qualitative usefulness and helpfulness, as they align more closely with existing legal workflows. Methodology: The testing goes beyond simple accuracy. It includes a three-part assessment: Reliability (objective accuracy and legal adequacy), Usability (qualitative metrics like helpfulness and coherence for tasks such as brainstorming), and Platform Workflow Support (integration, citation checks, and other features). Human-AI Performance: In the contract analysis study, AI tools matched or exceeded the human baseline for reliability in producing first drafts. Crucially, the data demonstrated that the common belief that "human plus AI will always outperform AI alone" was false; the top-performing AI tool alone still had a higher accuracy rate than the human-plus-AI combo. Risk Analysis: A significant finding was that legal AI tools were better at flagging material risks, such as compliance or unenforceability issues in high-risk scenarios, that human lawyers missed entirely. This suggests AI can act as a crucial safety net. Strengths Comparison: AI excels at brainstorming, challenging human bias, and performing mass-scale routine tasks (e.g., mass contract review for simple terms). Humans retain a significant edge in ingesting nuanced context and making commercially reasonable decisions that AI's instruction-following can sometimes lack. Discussion Highlights: [0:00] – Introduction and background of Anna Guo and Legal Benchmarks AI. [4:30] – The impetus for starting systematic AI benchmarking. [6:00] – Explaining the concept of measuring the "delta" in performance. [9:00] – Detailed breakdown of the three-part AI assessment methodology. [15:00] – Discussion of the contrasting results: general LLM accuracy vs. legal AI qualitative value. [19:00] – Results on AI performance matching human reliability in contract drafting. [21:00] – Debunking the myth about Human + AI always outperforming AI alone. [23:00] – The finding that legal AI excels at surface material risks that lawyers miss. [27:00] – A SWOT analysis of when to use humans and when to use AI. [30:00] – Future roadmap for Legal Benchmarks AI research.  

  --------  

  26:40

  --------

  26:40
• Want to be a Crypto Lawyer? Rule # 1: Use the Technology. Rule #2: Beware of Hyper-Specialization (Justin Wales-Head of Legal, Crypto.com & Author of Crypto Legal Handbook)

  Justin Wales, Head of Legal for the Americas at Crypto.com, and author of The Crypto Legal Handbook visits the show to provide his unique perspective on pivoting from a career in Constitutional Law, including work on high-profile appellate cases like the Obergefell gay marriage decision, to becoming a trailblazer in crypto law and blockchain technology.  He shares his serendipitous journey stemming from a law school article that launched his legal career and his subsequent deep dive into crypto, sparked by WikiLeaks accepting Bitcoin donations. The discussion covers the evolution of his practice to one of the first successful crypto legal groups at a large law firm. Justin emphasizes the necessity for any lawyer in the space to use the technology and become a generalist to navigate the multi-jurisdictional and rapidly evolving industry.  Finally, the conversation touches on the critical distinction between centralized finance (CeFi) and decentralized finance (DeFi), and his outlook that crypto's infrastructure will ultimately serve as the underpinning for future advancements like Artificial Intelligence (AI) and the Internet of Things (IoT). Episode Highlights:  * Justin's start in Constitutional Law, beginning with a law school paper on unconscionable cruise ship employment contracts that led to Supreme Court brief writing.  * The shift to crypto law: WikiLeaks and Bitcoin's role in it introduced Justin to blockchain technology.  * Building a pioneering crypto legal practice at a large firm in the early days (2013-2015).  * Why the best in-house lawyers, especially in a cutting-edge field like crypto, need to be generalists, not hyper-specialized.  * The inspiration and philosophy behind writing The Crypto Legal Handbook—creating an affordable, regularly updated, and candid resource for students and practitioners.  * Advice for aspiring crypto lawyers today: the field is more mature and requires blending traditional financial regulatory expertise with an industry-wide approach.  * The fundamental distinction between Centralized Finance (CeFi) and the legally complex, more innovative world of Decentralized Finance (DeFi).  * Rule #1 for Crypto Lawyers: Why you must use the technology and the risks of lawyers who are frozen in time with their technical understanding.  * High-level overview of US policy efforts to regulate crypto, including the GENIUS Act (Stablecoins) and the Clarity Act (Securities vs. Commodities regulation).  * Justin's crystal ball: The long-term view that crypto infrastructure will eventually become an unseen layer beneath the rise of AI agents and IoT.   Things We Talk About in this Episode    * Book: The Crypto Legal Handbook by Justin Wales  * Book: Read Write Own by Chris Dixon  * JustinWales.com (for more information on the book and author)  

  --------  

  38:14

  --------

  38:14
• The Business of Law: From IP Lawyer to Investor to CEO to Legal Tech Ecosystem Builder (Avaneesh Marwaha, Litera CEO)

  Avaneesh Marwaha, the CEO of Litera, visits the show to discuss his journey from IP lawyer to becoming a legal tech CEO and investor. He discusses the motivations behind his career pivot, including his desire to be a decision-maker and his passion for the business side of law. The conversation delves into the evolution of Litera, from its origins as a document-focused migration software company to its current role as a comprehensive legal tech ecosystem. Avaneesh highlights the company's strategic shift from acquiring to building new technologies. He also emphasizes the importance of Litera’s strong partnership with Microsoft and the company's focus on enhancing law firms' productivity and client service. Key Takeaways: Pivoting from Law to Business: Avaneesh's move from practicing IP law to an in-house role and eventually becoming a CEO was driven by his desire for a more active role in business decision-making. He saw lawyers as reactive to business decisions, and he wanted to be at the forefront of the action. The Value of a Law Degree in Business: Avaneesh shares his perspective on the value of a law degree for business professionals, highlighting the critical thinking and risk tolerance skills it provides. He notes that the corporate transaction course he took in law school was particularly beneficial, teaching him about corporate governance and fiduciary duties. Litera's Evolution: Avaneesh discusses how Litera has grown through strategic acquisitions and a recent focus on internal development. He explains how the company integrates various tools, like Kira and Foundation, to create a seamless ecosystem for lawyers within their daily workflows, primarily in Outlook and Word. The Impact of Generative AI: The discussion touches on the transformative impact of AI on the legal tech industry. Avaneesh explains how AI has enabled Litera to accelerate its product development and introduce features that proactively assist lawyers with tasks like document comparison and client relationship management. Build vs. Buy Strategy: Avaneesh outlines Litera's shift from a buy-heavy strategy to a more balanced build-and-invest approach. This change is largely due to the increased efficiency and speed of development enabled by AI tools. Future of Legal Tech: Avaneesh shares his vision for the future of Litera, which includes a focus on "maniacal client service" and helping law firms proactively grow their business. He believes that as automation handles routine tasks, lawyers will have more capacity for business development and client-centric work. Things We Talk About Kira Systems - A company acquired by Litera specializing in AI-powered contract analysis. Foundation Software - A business acquired by Litera focused on knowledge management. Doxly - A document formatting and repair tool.  

  --------  

  39:27

  --------

  39:27
• Ben Chiriboga (Nexl) on Legal Tech, Innovation, and the Future of Law

  A conversation with Ben Chiriboga, Chief Growth Officer at legal CRM company Nexl and host of the This Legal Life podcast. Ben shares his journey from a litigator to a legal tech innovator. He recounts how an early e-discovery tool using natural language processing completed in six hours what had taken him six months, leading to an epiphany about the power of legal technology. The conversation delves into the legal industry's historical resistance to technology, prioritizing billable hours over efficiency, and how Ben leveraged his legal experience to transition into a new role in tech. Ben discusses the challenges of building a legal tech company and the critical distinction between a general CRM and one designed specifically for law firms. The episode concludes with a thought-provoking discussion on the impact of AI on the legal profession, focusing on what aspects of law are truly "fundamental" and what new roles and jobs AI will create. Key Takeaways The Power of Tech: Legal tech tools can drastically improve efficiency, as demonstrated by an e-discovery tool that performed six months of work in just six hours. Transitioning Careers: A background in law can be a valuable asset in the legal tech sector, but success requires adapting to new skills like sales, marketing, and growth strategy. The "Why" Behind a Legal CRM: General CRMs are built for sales teams with built-in incentive structures, whereas a legal-specific CRM, like Nexle, is designed to passively collect and enrich data without requiring lawyers to manually input information. The Fundamentals of Law: The human element—relationship building, counseling, and strategic advisory—is the core function of a lawyer that AI cannot replicate. The Future of Legal Jobs: AI's true impact will not be in better-drafted motions, but in the creation of new "legal adjacent" roles and business models that merge legal expertise with technology and data.

  --------  

  34:50

  --------

  34:50
• Do What the Bots Cannot: How AI Is Reshaping Legal Marketing, Data Analysis and Biz Dev, But Not the Human Connection (Gyi Tsakalakis, AttorneySync)

  We welcome back legal marketing expert Gyi Tsakalakis, President of AttorneySync and host of the Lunch Hour Legal Marketing podcast. Five years after Gyi’s first appearance on Technically Legal, he discusses the dramatic shifts in digital marketing for law firms, driven largely by the rise of artificial intelligence. Gyi highlights how AI is changing the landscape for lawyers, from automating back-office functions and data analysis to influencing content creation and search engine optimization (SEO). He explains why law firms must diversify their marketing channels beyond just Google to adapt to the new era of conversational search and AI overviews. Despite these technological advances, Gyi emphasizes that the human connection, including a simple handwritten note, remains the most powerful tool in an attorney's arsenal. He argues that as technology automates the mundane, it forces lawyers to focus on the relationships and empathy that truly differentiate their practice. Key Takeaways: Diversify Marketing Channels: Law firms must move beyond a "one-trick pony" approach with platforms like Google and diversify their client acquisition strategies to include social media, email, and local events. The Problem with Client Intake: Gyi discusses the persistent issue of lawyers not responding to inquiries and how AI can help automate certain parts of the intake process, though the human touch is still crucial for building trust. Do What the Bots Cannot: Attorneys should leverage AI for tasks that can be automated (e.g., scheduling, data analysis) to free up time to focus on complex legal work and building meaningful client relationships. AI for Data Analysis: A powerful, overlooked use case for AI in a law firm is analyzing internal data, such as call transcripts and financial records, to identify blind spots and inform marketing strategies. The Future of SEO and Search: Gyi predicts that search will become more conversational and influenced by AI Overviews and that ad-based models will be integrated into all major AI platforms.  

  --------  

  36:54

  --------

  36:54

Mais podcasts de Notícias

Podcasts em tendência em Notícias

Sobre Technically Legal - A Legal Technology and Innovation Podcast