Thoughts on ai

I haven’t posted anything in a while.  This is just a random thought, not really a full post.  

It’s not quite “new,” but ai has recently become ubiquitous.  The technology isn’t going anywhere, has amazing capabilities, and does things astoundingly quickly.  In the law, the big firms are all using it, and even with carefully planned guardrails and implementation there’s been highly publicized errors (fabricated cites, etc) and probably many that aren’t so public. 

Everyone has it, or some variation, and there’s downsides.  For seemingly routine tasks, it gives you a polished seeming product, that’s usually unnecessarily wordy and complicated.  We’re entering an era where discovery disputes - which already become easily protracted - are becoming automated.  Tons of firms, particularly in volume practice areas, already used formulaic demands and objections, but now those form docs have a life of their own.  It’s not all bad, the ai does more thinking on those types of tasks than some lawyers or paralegals, but it’s also a lot of automated churning that we humans have to wade through. 

There’s many use cases for it, and it’s not going away.

I was really impressed for a while with westlaw’s ai function, and google gemini can also be very good for deep internet research, but they have limits.  I always loved doing deep dives.  You research a concept, and as you read and learn you refine what you’re searching for.  

I recently thought I had a research task that was perfect for ai: search every case nationwide for how a statute has been applied to variations on a fact pattern.  I knew the cases were out there, but it was the sort of thing that doesn’t generate a lot of published decisions and you had to actually read the facts because judges wouldn’t always use identical phrasing.   

So westlaw ai initially gave me junk.  Decent junk, the right legal principals, but not the fact patterns I needed.  Gemini, on the other hand, gave me hallucinations.  It gave me one or two legitimate leads on cases to look into, but probably 80% was just fake, and then I’d call it out and get a “you caught me” response.  I knew that was a possibility, but I hadn’t seen it like that before, and it’s hard to understand why it happens, particularly when I was very clear in the prompts as to what I wanted. It’s also really surprising because google scholar is great, and and they already have an organized database of every modern state and federal court decision.  I got the ai to agree that it would only give me cites from google scholar, then it proceeded to make up more stuff anyway.  Something’s off, maybe even intentionally, because Google’s ai should be able to do that.  

So back to Westlaw, with old fashion terms and connectors, but now that the search wants to think for you it’s a battle to make it just a tool that does exactly what you ask and nothing else.  I did, eventually, find exactly what I was looking for.  

I’ve noticed a similar thing with google just with personal things lately.  It will give you an ai response, based on ai slop that’s flooding the internet, and struggles to assess the reliability of its sources.  

What prompted this post, although it was percolating in my head from research i did earlier in the week, was trying to figure out if a nonsense Facebook post about a decorative park bench was real or fake.  It was such a close call that I googled it, but google sees the same ai generated story and pictures gave it back at me as if there were multiple sources.  Looking a little deeper, I’m 95% sure it’s fake, but there’s a feedback loop of generated content being the source for more generated content.  

It’s the same with using ai for legal research.  It can be done and done well, at least if used as a background memo where you double check everything, particularly with westlaw or a similarly constrained database, but there’s is an avalanche of ai generated material out there (including legal blogs and websites) that is now going to be looked at as a source to answer new queries.  We all know or should know that just because it says it somewhere on the internet doesn’t make it true, but humans’s already struggle with it, and LLM’s may increase rather than reduce that problem.