The Association Intelligence Podcast by Betty:AI

0:00 20:48

[Thomas Altman] (0:05 - 0:10)

Today's big question, how did we make the Association Intelligence Podcast and why?

[Rob Barnes] (0:10 - 1:06)

We are Thomas Altman and Rob Barnes and this is episode zero of the Association Intelligence Podcast. Thomas, I'm so glad that you came up with the idea of doing an episode zero because it reminds me of the conversations we were having before we started writing the Association Knowledge Cycle book, which is now nearly three years ago. But also because I think that, and I've said this to everyone that's bought the book or taken a copy of the book, I said, please just go to the epilogue.

My favorite part of the book is the epilogue that you put together, which is about how we wrote the book. Because it was such an interesting experiment, seeing that we wanted to experiment with the interplay between our human endeavor, the thoughts, the ideas that we wanted to share and how we used AI tools to help us ultimately write the book.

[Rob Barnes] (1:06 - 1:26)

And over the 12 months it took us to write the book, how dramatically the AI that we were using to do this changed in capability and intelligence and insights and pushback, like that whole experiment. And I feel like this is exactly the same experience again, but in a completely unique way.

[Thomas Altman] (1:26 - 2:50)

Yeah, I think it's really, really cool. So the idea here is we've now recorded all the episodes for this podcast. So kind of set the table a little bit because hopefully this is the first one you're listening to.

What this season of the Association Intelligence Podcast has is a series of 12 questions, right? 12 questions that we either are asked a lot and deserve really, good questions, or that we think aren't asked enough, but are really, really important. And kind of, we went through this whole process of creating this really cool, awesome podcast for you guys, and then took a step back and my thought went to the same place to the epilogue of our book, the Association Knowledge Cycle.

And wanted to give you guys a little peek behind the scenes, like how did this all come together? Why is this episode just humans, but the rest are going to be humans and AI kind of back and forth? And how can, I think more importantly, how can you use sort of this experience to inform your own ideas around AI and how you can employ it at your association, right?

I think that that's really, really important. I think all of it is grounded, for those of you who have not kind of, are not familiar with the Association Knowledge Cycle concept that we've been playing with for a few years now. The idea here is that, one, associations are the organizations best situated to benefit from generative AI.

We fully believe that, we've believed that since before ChatGPT launched.

[Thomas Altman] (2:51 - 3:26)

And what we, the way we see that happening is kind of through this interplay around intellectual property, which associations are the owners of, the creators of, the guardians of for your industry, right, or your profession. And the interaction between members and the public with that intellectual property, with that knowledge. And what generative AI gets us when done correctly and deployed correctly, is the ability to see those interactions at scale, across all your members, across time, and then learn from those interactions in a way that allows you to create the next wave of knowledge.

[Thomas Altman] (3:27 - 4:10)

And as we've been building Betty and deploying Betty across kind of almost 200 different associations now, we've been good about that first part, right? We've gotten to the point where we've got, we've scaled the conversations and people can see them, but we haven't quite executed on that last, that last mile of taking those and creating something useful. So this podcast is kind of us putting our own words into action, looking at how people interact with our version of Betty, and turning that into something that we believe is kind of more broadly useful across the entire association ecosystem to help understand sort of good and bad and sort of beyond use cases for AI.

So that's kind of where this, this all came together.

[Rob Barnes] (4:10 - 7:00)

I remember when we first started scoping out, like, how did we want this to kind of take shape? And I was reflecting on how the nature of the conversations I'm having about Betty and AI, the application, the insights, those conversations have markedly changed over the last 12 months because of where the maturity around conversations, but also where things like AI fatigue are really starting to, to hit association executives. And so we wanted to go back a little bit and listen.

And so the first part of the first step was, well, let's build a repository of knowledge around what the conversations are in the association sector about AI. And that repository of knowledge would then become, you know, part of the source, you know, the major part of the source of how we would produce, you know, the episodes. And that's how we arrived at the, you know, the big questions of the day, even that experiment alone, and how long and how purposeful we stepped through what the listening tool that we built, how it was following that process, right?

You know, we wanted, like, there's amazing consultants in the association industry. There's amazing other podcasts, some of the, you know, 14, 15 best podcasts already around the association industry. There is books, there's newsletters, there is, you know, ASAE's Association Now, there's associations for associations.

You know, we had the AI tools listening to all of this over a period of time. It wasn't just a one and done. We didn't just accept the first outputs that came.

We were like, okay, that's super interesting. That doesn't sound right. Like, tell us where you got that from and where's that?

Like, we really kind of interrogated that knowledge base pretty critically. And even that in itself is an experiment that, you know, any association can think about what is being said about our industry or our profession out there broadly on the internet from reputable sources, and what do we do with that? And so, yeah, this view of that broad kind of listening, but really narrowing in on some key questions that are still being asked, that are being debated.

Then, okay, we're going to turn this into something that is usable, was always where this was going to go. We did it with the book. We had a thought.

We had a methodology. We had research. We produced a book.

Now, what are we going to produce that's also showing that interplay, that interoperability between us as humans and AI can produce something that's really meaningful? So, I think the hosts are synthetic, but the questions aren't. That's a really great way to frame where we ended up, right?

[Thomas Atlman] (7:01 - 10:49)

So, yeah, I think that's the point. So, what we want to do in this episode zero is kind of pull back the curtain a little bit, tell you how we came to this, how this was made. We want to inject that transparency, that human element back into it, keep this pretty unpolished for that purpose.

So, you know, it is more human. And the way we did this, the way we came across this is we started kind of with the conversations, right? We did this listening.

And then we had this repository of kind of the conversations that are being had, our own external research. And we said, what do we, we didn't even know we wanted to do a podcast at this point. We kind of went to the AI and said, what are some ideas here, right?

And one of the ones that popped out was, hey, what about a podcast, a serial style podcast where we ask and answer the most important questions per episode and the light bulb and all our heads went off. Like, that's a great idea. I don't, I don't know that any of us would have thought about that on our own.

And the way we made this, I think is important. As I think if you're not careful, if you guys are thinking about trying to do something like this yourself, if you're not intentional about how you employ these tools, you will end up with AI slop, right? We do not want, this is not slop.

Hopefully it's not slop. That is not our goal here. And the reason we're confident it's not is because we didn't just throw stuff at AI and copy and paste it and send it on.

We were very active in the way that we interacted with the AI on the front end, right? We were very intentional. Our own ideas were what initiated this process when we started kind of creating the podcast themselves.

We listened to the podcast, right? When we used a notebook LM to create these, it's got a very distinct style. We listened to the podcast.

We read through the transcripts. We were very critical, right? Most of these episodes went through six, seven, eight iterations where a couple interesting things happened.

On the first end, sometimes it was just not what we wanted. Most of the time, the first episode was just not it. Maybe it was mostly it, but there was stuff about it that was like, oh, no, I don't like that.

So we'd go back and kind of rewrite kind of our instructions back to the AI and have this dialogue back and forth. But the other thing, and I think maybe the more important thing that happened, so one, we were governed here. We advocate governed AI.

We absolutely governed what we did here. The other thing that happened, though, that I thought maybe was more interesting was that dialogue itself, like when I was getting frustrated that it wasn't doing what I wanted it to do, it forced me to do some self-reflection, right? It forced me to think about what actually I wanted, right?

It actually forced me, because I wasn't just accepting what AI did and pasting it, as being critical of the output. I was being critical of my own ideas, and it made me crystallize kind of my own thoughts about this in a way that I don't think I ever would have if I wasn't using AI to challenge myself. I frame it like that on purpose.

It's not AI was challenging me. AI was a tool that I used to challenge myself to think more deeply about the concept. And what we wanted to do here was kind of, one, open up that, you know, pull back that curtain and show you guys, you know, how we did this.

But then, two, as we did it, you'll find the structure of all our episodes starts with myself and Rob having a little bit of a back and forth, throwing it over to the Notebook LM produced podcast, and then an outro where we talk about it, right? We want to cap in both of those with the human experience and the way we interact with it. And what we're hoping is you can kind of see how that interaction happens, and then you yourself can contribute that too.

Hopefully, this forces you to be a little bit more critical, more thoughtful, disagree with, you know, both the AI and with us, right? But do so in a way that has kind of concrete, crystallized actions behind it. And that's kind of our goal here, and that's why we did this and how we did it.

[Rob Barnes] (10:50 - 14:54)

Yeah, the point that you made there, like most people who know me know that I'm relatively self-aware, but self-reflection hasn't necessarily been the strong suit. And you're quite right. At this point in the Betty journey, you know, we're two and a half years into this company, and we've done almost 200, you know, AI projects with different associations around the world.

And I didn't want to use that as a, I didn't want to make any assumptions about what would be meaningful or purposeful in this exercise, just because of that experience. And so you're right, on a far more rudimentary level than you, but my frustrations with what the AI was outputting was making me stop and think about, is it just assuming a bunch of things because the only thing that I've given it is my learned experience, I'm not asking it to explore outside of that. And so you're quite right, the process of, you know, interrogating it or sharing my frustrations with it or asking it to come back and challenge me to answer questions that I wouldn't have thought about, in and of itself created, again, some knowledge assets that became super useful in working out, well, what am I learning from these episodes each time?

You know, so actually going and reading a transcript and listening to an AI-produced podcast about stuff that we've researched is, I mean, it's a lot of fun, if nothing else, and particularly in the Notebook LM format. But it's actually insightful to look back and go, I would never have gleaned that insight from the same research just by myself, I just wouldn't have. And so being able to go and say, yeah, our format that we wanted is intentional, the human experience up front, us talking about what we, a preview about the episode, what we liked about it, where we had some challenges with it perhaps, then the outro, you know, the end of it, like where we're recapping it, but also saying, you know what, this was one of those episodes where we didn't necessarily agree with exactly what came out, nor do we have to agree, it just became the point of conversation and the point of learning, which every listener is going to interpret for themselves.

And that in itself is another important point, that you can go through this exercise to produce an asset, if you like, and in this case for us, it's 12 podcast episodes on the big questions that are being talked about in associations right now. But the learning is in the learning, you know, it's like it's learning how to learn from these things, the process by which we produced it, how we then fell into this really nice routine of previewing it and critiquing it afterwards, the number of times we went through that, but in really quick time, again, because the AI allowed us to do the fun stuff, which was to dig into the things that we learned and liked or didn't like.

And then also, you know, just making sure there's teachable takeaways, you know, from each of this, right, we want to make sure that association executives who do invest the time to listen to us and listen to what we've produced, understand the very clear distinction between what we're talking about, what the AI has produced, but also the companion asset, which again was a great idea. Let's have something that they can just take away and use, even if they don't ever listen to the podcast ever again, there is a document here that they can download and go, I can actually use this today in my job, then kind of our work is done, you know, and the amount of time we've invested to pull this together for everybody is well worth it if just one person says, well, I used this checklist or I got this, you know, I asked these questions in my staff meeting, it's like, okay, great, you know, that's exactly what we wanted to do it for. It's still the most fun thing to talk about how we did it and why we chose some of the tools as it is actually listening to the quality of the content that was produced from the research that we did.

[Thomas Atlman] (14:55 - 16:07)

I want to lean in that the companion asset, which I think is a really important part of this. So each episode has a companion asset, a workbook that allows you to kind of engage in that dialogue, you know, take it offline and think deeply about what you may have learned in each episode. And I want to call that out, one, it is a tangible benefit to you as you go through this that you can kind of hold on and reflect on and, you know, share around as you need to.

And I think you'll get a lot of value from that. I want to call that out too, because at Betty, we are all about this idea of governed AI, right? This is where we see the value for associations in that the AI is governed, it's traceable, it's auditable, it's coachable, right?

And we talk about that throughout the 12 questions. So the podcast itself is about that, but I want to call this out too, is that it's also an example of it. So the way that those companion assets were produced was through a process of governed AI, right?

It wasn't just like this idea of AI swap. We didn't just create stuff. We know where the sources came from.

We traceable to it. We understand how it was made. It's auditable.

When it was wrong, we coached it so that the next time it was right.

[Thomas Altman] (16:07 - 16:16)

And we've got a legacy, a repository of that coaching that then will inform kind of future versions of this experiment, whatever that might be, right?

[Thomas Altman] (16:16 - 16:36)

We're can kind of now expand out our knowledge asset creation. So yes, we are all about governed AI, but those companion assets are examples of how that is put into action. So kind of leading by example in this situation or just showing kind of eating our own, no, drinking our own champagne, almost said eating our own dog food, but we drink our own champagne at Betty.

[Rob Barnes] (16:36 - 18:39)

That's right. Yeah. And I think there's a great, you know, for us, it was a great experiment with some of the different tools that are available to us right now and, you know, how we produce those assets and, you know, even how we produce the name of the podcast in the first place and kind of iterate it on all of these sorts of things.

It can be a lot of fun with that kind of disciplined approach to the application of the applications as well. So the episodes as well are interchangeable. Like you don't have to follow it from one to 12.

You grab a topic, grab a question that's of interest to you right now, dig into that particular one. So in any order, I think it's going to be, you know, useful for people. And I think while voice has become very topical, I don't necessarily think it's that important, but it's very topical at the moment, like giving AI a voice.

The podcast was just a natural extension of that exploration because, you know, NotebookLM has had this audio capability for a good 12 months now and it's pretty cool. And so like, okay, what does our research sound like if we gave it to that? And I think that's the part that's like, you know, the sources are the important point.

It wasn't the hosts. It wasn't what the hosts sounded like. It had much nicer voices for podcasting than we do probably anyway, Thomas.

But because we governed the sources and we governed the way we wanted it to be produced, the hosts followed that. And by the time we got to the publishable versions of the episodes, we are stand behind. We are very confident that this is not AI slop, that this is rooted in genuine research and genuine questions and then capped off with our own experiences, you know, at the start and the end.

[Thomas Altman] (18:41 - 19:15)

So what I think we want you to do is, yeah, jump in wherever you need to. This is not, you know, you don't have to go from one through 12. Find the topic that's interest to you and then engage in the dialogue, whatever that means to you, whether that's through the companion asset, whether that's just sort of chatting with your colleagues at work or wherever you may be.

Engage with the dialogue yourself and kind of try to crystallize your own thoughts about AI. And if you disagree with us, that's great. That's what we, that's what you want.

You can disagree with us and the AI and have your own idea, but we ask that you share it. So find a way to share that back with us and we will love to learn from you guys.

[Rob Barnes] (19:15 - 20:42)

Yeah, I feel like that's actually a little, that is a little permission slip for everybody that's listening to the podcast at any point, is if you want to take the companion asset and load it into your AI tool of choice, if you're a co-work, if you're on a chat GPT, if you like perplexity, load it in there and have it do some critical analysis for you and see what it comes back with. And if you start getting some advice as to, like, this is a crackpot idea, this one, whatever, we would love to, like, book time with us at any point. If you want to have a conversation with us about the things that we learn from these big questions, just book time.

We can talk, you know, we can talk about this stuff under wet cement with a mouth full of marbles, I think, as my mother used to say. So we're open to those conversations, truly, because every time we have one of these conversations, we learn so much more about how these issues are impacting associations. And that's our wheelhouse.

That's where we, that's where we work. That's the work that we love every day. And, you know, producing the podcast is just another contribution, I think, to the body of knowledge for the associations industry that we know and love, that we hope people will have a bit of fun with and hopefully do some of their own experimenting after this.

Thanks, Thomas. Thank you, everyone, for listening to the Association Intelligence Podcast. This was episode zero. Join us on the first episode where we discuss this question: "Are we ready for AI?"

What this helps your team answer

How to use AI to research a topic without ending up with "AI slop" - what a rigorous, governed process for turning AI research into a real deliverable looks like.

An example model for pairing human judgment with AI output.

How to make AI-assisted work feel human and trustworthy instead of generic or robotic.

0:00 31:28

[Thomas Altman] (0:10 - 0:13)

Today's big question, are we ready for AI?

[Rob Barnes] (0:14 - 1:04)

We are Thomas Altman and Thomas Altman, and this is Episode 1 of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the AI Content Governance Readiness Checklist. Yes, that's a lot of words, but it's a very important asset.

It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work. Today, we are digging into the first big question that we discovered online, and it's really one of the things that we hear a lot in conversation with associations, this idea of are we ready for AI and what that even means. Thomas, when you were doing the research for pulling together these big questions for associations, how did this resonate with you as one of the big things that we needed to start with?

[Thomas Altman] (1:05 - 1:51)

Yeah, so this was, to me, the reason why we're starting with this one is it's the biggest one, right? I think it's the one where there's the most hesitation. And what we've been finding is that this hesitation a lot of times is unfounded or turned on its head, but then, too, it's the most value-destroying hesitation you can have when approaching this question.

So one of the things we wanted to do through this episode was to kind of flip the script a little bit, kind of show you how you can get started with AI in a way that's going to be super valuable to you. And it all starts with this idea, the two dreadful words I know I hear a lot, Rob, I think you hear it every day, is digital transformation, and digital transformation specifically being a stumbling block. Rob, do you ever hear that?

Do you ever hear people say, oh, we're in the middle of a digital transformation?

[Rob Barnes] (1:52 - 3:17)

And it's funny how quickly in the last year or two that the meaning of that has changed dramatically. And yet it's still being treated the same. You know, like depending on what level of association executive we're talking to, yeah, digital transformation is either a really cool strategic excuse not to deal with a software vendor, a particular vendor, or literally that's the funded mandate from the board, is to undertake this digital transformation, and it needs to cut across user experience online, the tech stack itself is going to be completely either consolidated or expanded depending on which administration is in at the time. And I also hear that we need to get our house in order first.

You know, this concept that association's knowledge is so disparately housed that until you get it all consolidated or collaborated into one kind of place, you can't move forward with AI. And that's really interesting to me, this idea that it's fragmented, but it's the right content. You know, it might be incomplete, but only incomplete from what we know about our own body of knowledge.

You're right. It's such a challenging environment, and it is holding a lot of associations back from moving forward.

[Thomas Altman] (3:18 - 5:56)

So I think flipping the script a little bit, it's not get our house in order, a digital transformation so that we can start AI. What we've seen, and we've done nearly 200 successful implementations of AI projects for associations now, it's the other way around. Get AI in order so that you can get your house in order.

The process of getting AI set up is quicker time to value, and it informs a better strategy, a more future-looking strategy around kind of how you can do that digital transformation. And the way we want you to think about this, the thing that I think leads to people getting the most of value out of AI first, and then a digital transformation later, is when you think about what's the value that one, an association serves as related to its intellectual property, right? How can AI enable kind of those gaps, right, in getting that value from the intellectual property?

And then how can you inform that through whatever your strategy is going forward? So what we recommend is instead of thinking about, like, how do I do these crazy taxonomies, or how do I kind of reorganize my website, or I've got to get another new AMS, it's been five years since our last AMS, so we've got to get a new one, right? All that other stuff, I'd say, not so important.

More focused in, what's that end value? If what you're doing as an association is creating valuable intellectual property that will be consumed by your members, its general public, or whoever your constituencies are, and the problem is people are either not able to find it, not able to use it, not able to incorporate it in their day-to-day workflow so that they get value from this content. You can put AI first, but you need to do, the way you get your house in order is not digitally.

It's not through technology. It's through understanding your members, understanding what value they get, understanding where that content already lives. You don't have to reorganize or put it all in a single sort of data warehouse.

It can live in disparate places, in different kind of platforms, but as long as you know where it is and you can point the AI to that with the goal of helping people consume it, getting that process up and running, one, it shortens your time to value. You get more value in a shorter time frame by going that route first. Two, it informs the digital strategy going forward.

You're going to learn so much by doing that process first that once you finally get to kind of reorganize your website, you'll know what you want to do there. That's a lesson we've learned over and over again, and it's the biggest hurdle we've seen organizations like yours have trouble in overcoming before they get started. There's this fear of movement, and by taking that barrier down, we see people have a lot of success, both with AI and beyond.

[Rob Barnes] (5:57 - 6:36)

What I liked about this episode and the challenge that lives there is this idea that the biggest hidden ROI isn't really about member satisfaction. It's about an organization agreeing what the official answers are, what are the responses that the organization should be giving to a request to an AI, and is that really the silver lining? I think that's where this episode goes, and I really enjoy it is what is the silver lining that comes from all of this work, and then the interface with the members.

So I'm really looking forward to digging into it.

[Thomas Altman] (6:37 - 7:03)

Before we hand this over to our AI podcast host, a quick reminder that the following is generated using Notebook LM, but with heavy involvement from the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach, we recorded a full episode where we discuss how and why we made this. If you're interested in understanding how this all came together, we recommend you listen to episode zero.

And with that, over to our AI hosts.

[AI Host 1] (7:07 - 7:14)

Right now, thousands of organizations are deliberately delaying artificial intelligence.

[AI Host 2] (7:14 - 7:15)

Yeah, they really are.

[AI Host 1] (7:15 - 7:30)

And they're doing it for what sounds like the most responsible, mature reason in the world. If you spend any time in leadership meetings, you've probably heard someone sigh and say, we can't deploy AI yet because we're in the middle of a massive digital transformation.

[AI Host 2] (7:31 - 7:32)

Right, the classic stall.

[AI Host 1] (7:33 - 7:47)

Exactly. It's this looming cloud. The assumption is that you simply must wait for the new website to be built or the new content management system to launch or that endless taxonomy overhaul to be perfectly finalized before you can even think about AI.

[AI Host 2] (7:47 - 7:54)

I mean, it sounds incredibly logical on the surface, right? You don't put new furniture into a house while you're still pouring the foundation.

[AI Host 1] (7:54 - 7:54)

Makes sense.

[AI Host 2] (7:54 - 8:05)

But based on the sources we're diving into today, that assumption is actually strategically backward. Waiting until your digital transformation is quote-unquote done is the wrong play.

[AI Host 1] (8:05 - 8:08)

Because the AI changes the transformation itself.

[AI Host 2] (8:08 - 8:26)

Exactly. Governed AI fundamentally changes the requirements of the very transformation you're trying to finish. See, that completely flips the standard boardroom logic.

Because the ultimate question everyone asks is, are we ready for AI? But what you're saying is that the premise of that question is just flawed from the start.

[AI Host 1] (8:26 - 8:49)

It is flawed. If governed AI is going to become a primary interface to your organization's knowledge, bolting it on at the very end of a website rebuild makes no sense. It alters how you need to think about natural language access, how you connect different sources, and how you establish access rules.

It really needs to come along for the ride.

[AI Host 2] (8:49 - 8:50)

Not just an afterthought.

[AI Host 1] (8:50 - 8:59)

Exactly. And we should ground this right away for you listening, this isn't just theory. These insights are drawn from patterns across nearly 200 association AI implementations.

[AI Host 2] (8:59 - 9:00)

Wow, 200.

[AI Host 1] (9:00 - 9:08)

Yeah. We're looking at the recurring hurdles and the actual successful pivots leaders make when dealing with vast amounts of member or staff knowledge.

[AI Host 2] (9:09 - 9:23)

That context is vital. And as you think about your own organization's digital ecosystem, which I'm guessing has a few messy corners, we actually have a highly practical tool linked in the show notes for you. Yeah.

That's the AI content and governance readiness checklist.

[AI Host 1] (9:23 - 9:38)

Exactly. Keep that handy because we're going to cover exactly how to apply it during this deep dive. But let's start with the paralysis itself.

Why do leaders feel they have to wait for the perfect website or system before bringing in AI?

[AI Host 2] (9:39 - 9:48)

Well, it stems from what we can call multi-platform fear. Most leaders know their institutional knowledge isn't sitting in one pristine database. It's totally scattered.

[AI Host 1] (9:48 - 9:49)

Oh, yeah. Everywhere.

[AI Host 2] (9:49 - 10:08)

You have data in your membership database, your AMS. You have articles in the website's CMS, courses in the learning platform, the LMS. Plus, like, decades of trapped knowledge in old PDFs and recorded committee meetings. Right. So the assumption becomes we have to clean all this up and move it into one centralized master system before an AI can read it.

[AI Host 1] (10:08 - 10:10)

Which sounds like a five-year project, honestly.

[AI Host 2] (10:10 - 10:22)

Minimum. And it creates a false binary. Leaders think the only options are either to pause everything and spend years moving files into one repository or to just recklessly connect an AI to everything and see what happens.

[AI Host 1] (10:22 - 10:24)

Upload everything and pray.

[AI Host 2] (10:24 - 10:34)

Yeah. But neither is true. Governed AI doesn't need you to build a single new filing cabinet.

Technologically, it functions more like a highly secure smart librarian.

[AI Host 1] (10:35 - 10:36)

OK, a smart librarian. I like that.

[AI Host 2] (10:36 - 10:46)

Yeah. So through API connections and secure indexing, it maps out where the information lives across your existing platforms without physically moving the files.

[AI Host 1] (10:46 - 10:48)

Oh, so the files stay where they are.

[AI Host 2] (10:48 - 10:55)

Exactly. It just needs to be told where the approved sources are and what the access rules are for those specific sources.

[AI Host 1] (10:55 - 11:07)

I want to make sure I understand the mechanism here, though. If the files are in completely different formats, say an old Word document on a shared drive, a modern Web page and, I don't know, a transcribed video file.

[AI Host 2] (11:07 - 11:07)

Yeah.

[AI Host 1] (11:07 - 11:11)

The AI doesn't need us to standardize all of those into one file type first.

[AI Host 2] (11:11 - 11:19)

Not at all. Modern AI parsing and indexing systems can ingest text from wildly different formats and convert them into mathematical representations. They're called vectors.

[AI Host 1] (11:20 - 11:20)

Vectors.

[AI Host 2] (11:20 - 11:34)

Yeah. So the AI isn't reading a PDF and a Web page differently. It's reading the underlying concepts mapped out in those vectors.

As long as it has a secure key to the room where that PDF lives, it can retrieve the answer.

[AI Host 1] (11:34 - 11:42)

So waiting to use AI because your files are scattered is basically like refusing to hire a brilliant researcher just because your books are in three different rooms.

[AI Host 2] (11:43 - 11:44)

That's a great way to put it.

[AI Host 1] (11:44 - 11:58)

The researcher doesn't care as long as they have the keys to the rooms. But wait, if the AI is just indexing concepts across all these different systems, what happens to the, like, hundreds of hours organizations spend building website navigation?

[AI Host 2] (11:58 - 11:59)

The taxonomy projects?

[AI Host 1] (11:59 - 12:03)

Yeah. The endless debates over taxonomy and drop-down menus.

[AI Host 2] (12:03 - 12:23)

Well, this is where Govern AI changes the priority of your digital transformation. It drastically reduces your dependence on perfect site navigation. Really?

Yeah. Think about the mental load we currently put on a user to find, let's say, a 2022 policy update. They have to reverse engineer the mental model of the person who designed the website.

[AI Host 1] (12:23 - 12:27)

They have to guess if it's under resources or advocacy or maybe about us.

[AI Host 2] (12:28 - 12:29)

Right. And if they guess wrong.

[AI Host 1] (12:30 - 12:31)

They assume the document doesn't exist.

[AI Host 2] (12:31 - 12:37)

Precisely. But AI doesn't navigate by clicking folders. It operates on natural language intent.

[AI Host 1] (12:37 - 12:39)

Meaning they just ask for what they want.

[AI Host 2] (12:39 - 12:55)

Yes. A member can just type, what changed in the safety policy a few years ago regarding site inspections? The AI understands the semantic intent of that question, the relationship between safety, inspections, and recent history, and fetches the relevant approved source directly.

[AI Host 1] (12:55 - 12:56)

That's huge.

[AI Host 2] (12:56 - 13:02)

It is. And even better, deploying Govern AI early actually accelerates your taxonomy project.

[AI Host 1] (13:02 - 13:06)

Wait, how so? By telling you what people are actually looking for.

[AI Host 2] (13:06 - 13:19)

Exactly. It produces real-time signal. Instead of guessing how members navigate, you see exactly what natural language shorthand they use, how they group topics in their minds, and where your current structure is failing to deliver.

[AI Host 1] (13:19 - 13:36)

So you use the AI's data to build a better website map. Let me push back on that, though. Because if the AI can perfectly read intent and deliver the right document, a web developer might hear this and think, great, we don't need a structured website anymore.

Just put a search bar on a blank page.

[AI Host 2] (13:36 - 13:46)

No, no, no. Structure still matters immensely. Members still want to browse, they still want curated experiences, and you definitely still need an organized digital presence.

[AI Host 1] (13:46 - 13:47)

Okay, good to clarify.

[AI Host 2] (13:47 - 14:02)

What AI does is remove the urgency of that structure being absolutely perfect before you can deliver value. You don't have to freeze your AI rollout while committees debate a sitemap for six months. The AI bridges the gap.

[AI Host 1] (14:02 - 14:16)

That shifts the pressure in a really healthy way. So if we accept that we don't need perfect navigation or a single master database to begin, we have to look at the content itself. Right.

Because the fear isn't just that the content is scattered, it's that the content is messy.

[AI Host 2] (14:16 - 14:32)

And let's be honest, everyone's content is messy. The key is understanding what kind of messy you are dealing with. The implementation patterns give us a very practical framework for this.

Instead of trying to clean everything at once, you triage your existing content into three distinct buckets.

[AI Host 1] (14:33 - 14:44)

Okay, let's unpack this. It's like deciding what to pack when you're moving houses, right? So if the goal is to figure out what's ready for AI without a massive cleanup, I imagine the first bucket has to be the content we actually trust.

[AI Host 2] (14:44 - 14:48)

That's it. The first bucket is content that is fragmented but accurate.

[AI Host 1] (14:49 - 14:50)

Fragmented but accurate.

[AI Host 2] (14:50 - 15:01)

Yeah. The knowledge might be spread out across recent journal articles, formal policy PDFs, and official conference transcripts. The storage is fragmented, but the source truth is highly trusted by your organization.

[AI Host 1] (15:02 - 15:03)

So what's the action for that bucket?

[AI Host 2] (15:04 - 15:11)

For this bucket, the action is simple. It is ready for launch or a strong pilot. You don't need to reorganize it.

You just let the governed AI index it.

[AI Host 1] (15:11 - 15:28)

That makes sense. It's like your good silverware scattered in different drawers, just pack it. But what happens when we have a topic where the documentation isn't entirely comprehensive, where maybe the official handbook has the basics, but the real nuance lives in the heads of your senior subject matter experts?

[AI Host 2] (15:28 - 15:45)

Ah, that brings us to bucket two. Incomplete or uneven. You have valuable knowledge, but the coverage is spotty.

If you put an AI purely on this content, it might give surface-level answers because it lacks the deeper context your staff usually provides in person.

[AI Host 1] (15:45 - 15:49)

So do we hold off on AI for that content until the staff writes everything down?

[AI Host 2] (15:49 - 15:55)

No. You run a narrow controlled pilot. You use the AI system as a diagnostic tool.

[AI Host 1] (15:55 - 15:56)

Oh, to see what it can't answer.

[AI Host 2] (15:56 - 16:09)

Exactly. When the AI struggles to answer a complex question from a test group, the system's data highlights the exact gap. You expect to coach the system here, and you use the AI to figure out exactly which documents your experts actually need to write.

[AI Host 1] (16:09 - 16:23)

Rather than having them just guess what's missing. So it's like a puzzle missing a few pieces. Pack it, but note it.

Which brings us to the third bucket. If bucket one is ready to go, and bucket two needs a pilot, bucket three has to be the liabilities.

[AI Host 2] (16:23 - 16:43)

Yes. Bucket three is contradictory, obsolete, permission confused, or high risk. Oh, bad stuff.

The bad stuff. We are talking about stale technical standards that were superseded five years ago, but never deleted. Or like conflicting compliance policies that could create legal risk if interpreted incorrectly.

[AI Host 1] (16:43 - 16:44)

So what do we do with this?

[AI Host 2] (16:44 - 16:56)

The action here is a hard stop. Do not scale this broadly to your members. You must clean this content, establish authoritative ownership, and set strict guardrails before AI is ever allowed to surface it.

[AI Host 1] (16:56 - 17:01)

So it's not about feeding everything into the machine. Bucket three is actual garbage don't pack the garbage.

[AI Host 2] (17:01 - 17:03)

Exactly. Don't pack the garbage.

[AI Host 1] (17:03 - 17:13)

And that garbage, the high risk, obsolete content, really naturally leads to the most critical part of governed AI, right? Making sure the system understands what it's not allowed to share.

[AI Host 2] (17:13 - 17:15)

Yes, the governance guardrails.

[AI Host 1] (17:15 - 17:29)

Let's talk about permissions, because a lot of organizations have deep archives on shared network drives or, you know, internal SharePoint sites. If an organization just plugs a generic AI search tool into their internal servers to index everything, what actually happens?

[AI Host 2] (17:30 - 17:34)

It creates a massive exposure problem. And it usually involves stale permissions.

[AI Host 1] (17:34 - 17:36)

Stale permissions, like what?

[AI Host 2] (17:36 - 17:48)

Well, think about how organizations typically work. Six years ago, an HR team drafts a sensitive internal policy. They put it in a shared folder.

Over time, people move on, folder permissions get inherited or changed.

[AI Host 1] (17:48 - 17:48)

Right.

[AI Host 2] (17:48 - 17:56)

And suddenly that draft is technically accessible to a wider group of staff, but no one realizes it because it's buried 30 folders deep.

[AI Host 1] (17:57 - 18:01)

So it's protected by obscurity. It was so hard to find that the broken permissions didn't matter.

[AI Host 2] (18:01 - 18:06)

Exactly. But an AI doesn't experience obscurity. It just reads the permissions.

[AI Host 1] (18:06 - 18:06)

Oh, wow.

[AI Host 2] (18:06 - 18:21)

If you ask a generic AI a question about that HR policy, it will instantly retrieve that half-baked sensitive draft because it perfectly matches the semantic intent of your prompt. And the system technically has access to it. The AI is just doing its job.

[AI Host 1] (18:21 - 18:22)

That is terrifying.

[AI Host 2] (18:22 - 18:33)

It is. Which is why the safest first move is to draw a hard boundary. When you start, you point the governed AI only at approved, external, or clearly member-facing content.

[AI Host 1] (18:33 - 18:35)

You just leave the internal stuff out.

[AI Host 2] (18:35 - 18:43)

You leave the internal drafts, and the messy staff network drives completely out of scope until you have audited who actually owns what.

[AI Host 1] (18:43 - 18:52)

Okay, that covers the internal permissions. But here's where it gets really interesting for me. What about public-facing content that just changes over time?

[AI Host 2] (18:53 - 18:53)

Versioning.

[AI Host 1] (18:53 - 19:05)

Yeah, versioning. Let's say we have an official safety standard from 2015 that is fully approved and public. But then we published a totally updated safety standard in 2025.

Both are public. Both are technically approved.

[AI Host 2] (19:06 - 19:06)

Yeah.

[AI Host 1] (19:06 - 19:11)

How does the AI know how to handle a question about safety standards? Doesn't it just look at the date?

[AI Host 2] (19:11 - 19:32)

You would hope so, but a generic generative model doesn't inherently understand version control unless it's explicitly governed. Yeah. Generative AI is designed to synthesize information.

Left to its own devices, it might pull a foundational paragraph from the 2015 standard and combine it with a new bullet point from the 2025 standard because both documents conceptually answer the prompt.

[AI Host 1] (19:32 - 19:36)

So it synthesizes them into a Franken-standard that doesn't actually exist in the real world.

[AI Host 2] (19:36 - 19:58)

Precisely. And in technical or compliance-driven fields, a synthesized Franken-standard is incredibly dangerous. We can imagine.

To prevent that, the system requires explicit current source rules. You have to apply source labels. Now, the 2015 standard is still valuable, right?

Members often need to compare the new regulation against the old one so you don't delete the 2015 document.

[AI Host 1] (19:58 - 19:59)

Use governance.

[AI Host 2] (19:59 - 20:13)

Exactly. The governance layer explicitly instructs the AI, treat the 2025 document as the authoritative truth for all current queries. Only reference the 2015 document if the user explicitly asks for historical context.

[AI Host 1] (20:14 - 20:17)

The AI cannot be left to infer that nuance on its own.

[AI Host 2] (20:17 - 20:18)

No, it can't.

[AI Host 1] (20:18 - 20:32)

That completely demystifies the idea that AI is just a plug-and-play magic wand. It requires human guidance to set those rules, which implies that rolling out AI isn't just an IT project. It's an ongoing operational practice for the staff.

[AI Host 2] (20:33 - 20:46)

It requires an active process called coaching. An AI model, out of the box, does not know the unique culture or the specific dialect of your organization. Your staff and your subject matter experts have to teach it over time.

[AI Host 1] (20:46 - 20:49)

What does coaching actually look like in practice, though?

[AI Host 2] (20:49 - 20:58)

It's about teaching the AI your organizational shorthand, abbreviations, retrieval cues. For instance, your members might constantly ask about the big show.

[AI Host 1] (20:58 - 21:00)

Oh, like a nickname for an event.

[AI Host 2] (21:00 - 21:15)

Right. The AI doesn't know what that means. A staff member coaches the system by mapping that phrase to the official annual global convention.

You are taking the tacit knowledge that usually only lives in the heads of your senior staff and embedding it into the AI's understanding.

[AI Host 1] (21:15 - 21:24)

That makes the system increasingly valuable over time because it's learning the specific language of the community. But, you know, coaching requires knowing what the members are actually asking. How do we see that?

[AI Host 2] (21:25 - 21:32)

Through usage data, which is often referred to as insights. This is where Governed AI becomes a strategic engine for the whole organization.

[AI Host 1] (21:33 - 21:33)

How so?

[AI Host 2] (21:33 - 21:49)

Well, every time a member asks a natural language question, the system attempts to answer it using the approved sources. When you review the insights data, you aren't just looking at search volume. You are looking for the queries where the AI struggled to generate a high confidence answer.

[AI Host 1] (21:50 - 21:56)

Because if the AI couldn't answer it based on our approved documents, it means our approved documents are missing the information.

[AI Host 2] (21:56 - 22:09)

Exactly. It turns real member questions into a literal roadmap for your content team. Unanswered questions show you exactly what standards you need to update, what blog posts you need to write, or what training modules you need to develop next.

[AI Host 1] (22:09 - 22:11)

That's incredibly powerful.

[AI Host 2] (22:11 - 22:21)

It is. We see this dynamic play out quietly with tools like Betty, which is a really practical example of this kind of governed, source-bound, multi-tier AI.

[AI Host 1] (22:21 - 22:21)

Right.

[AI Host 2] (22:21 - 22:29)

A system like Betty is coachable, and it's designed to activate the knowledge you have approved and then use insights to reveal the gaps you didn't know existed.

[AI Host 1] (22:30 - 22:33)

But importantly, a system like that doesn't write the new standard for you.

[AI Host 2] (22:33 - 22:41)

No, it doesn't. It doesn't magically fix your bad guidance, it doesn't repair your broken SharePoint permissions, and it absolutely does not replace human governance.

[AI Host 1] (22:41 - 22:43)

It's just activating what you have.

[AI Host 2] (22:43 - 22:51)

Yes. It relies entirely on the parameters you set. Which brings us to what we should call the honest beat of this entire readiness conversation.

[AI Host 1] (22:51 - 22:53)

The honest beat. I like that. The reality check.

[AI Host 2] (22:53 - 23:14)

Yes. Here's the candid truth. If an organization has a high-stakes, highly sensitive topic, but no one on staff can confidently identify what the actual authoritative source document is, or who owns the access rights to it, that topic is simply not ready for broad, member-facing AI.

[AI Host 1] (23:14 - 23:25)

Wow. So what does this all mean for you, the listener? It means AI isn't a magic wand that cleans your messy room.

It is simply a very bright flashlight that shows you exactly where the mess is.

[AI Host 2] (23:25 - 23:26)

That's exactly it.

[AI Host 1] (23:26 - 23:31)

And if the mess is a liability, you don't shine a spotlight on it for your members. You fix the source first.

[AI Host 2] (23:32 - 23:40)

It illuminates the gaps. That is the core of readiness. It's not about waiting for a pristine digital ecosystem.

It's about knowing your sources.

[AI Host 1] (23:40 - 23:58)

We've covered a lot of ground here in this deep dive. We started with the stall, that fear of the incomplete digital transformation. We walked through how AI actually acts like a smart librarian, indexing across platforms without moving data, and how it navigates by intent rather than rigid website structures.

[AI Host 2] (23:59 - 23:59)

Right.

[AI Host 1] (23:59 - 24:11)

We looked at sorting content into those three actionable buckets, the dangers of stale permissions, Frankenstandards, and the necessity of coaching. So how does a listener actually start this tomorrow without feeling overwhelmed?

[AI Host 2] (24:11 - 24:13)

You do a readiness pass, and you start small.

[AI Host 1] (24:13 - 24:14)

Okay.

[AI Host 2] (24:14 - 24:23)

Pick one single use case for your organization. Don't try to solve the whole website, just one use case. First, list the authoritative sources for that specific topic.

[AI Host 1] (24:23 - 24:24)

Just the trusted ones. Right.

[AI Host 2] (24:24 - 24:28)

Second, name the actual human content owners who are responsible for those sources.

[AI Host 1] (24:28 - 24:28)

Right.

[AI Host 2] (24:28 - 24:35)

Third, set the access tier. Clearly define exactly who gets to see this information. Is it public?

Is it for members only?

[AI Host 1] (24:35 - 24:39)

And then flag any of the versioning or permission risks we discussed.

[AI Host 2] (24:39 - 24:56)

Yes. And once you have that use case mapped out, look at the content and assign it to a bucket. Is it bucket one, fragmented but accurate and ready to launch?

Is it bucket two, incomplete and ready for a pilot? Or is it bucket three, and you need to clean it before moving forward?

[AI Host 1] (24:56 - 25:04)

It takes something that feels like an impossible multi-year IT overhaul and turns it into a targeted content strategy.

[AI Host 2] (25:04 - 25:05)

It makes it manageable.

[AI Host 1] (25:05 - 25:18)

And to make that strategy tangible, you can go to the show notes right now and grab the AI content and governance readiness checklist. It walks you through that exact sorting and planning process so you can take the safest next step for your association.

[AI Host 2] (25:18 - 25:48)

It gives you the map. And, you know, I want to leave you with one final thought to mull over. We spend so much time talking about the ROI of artificial intelligence in terms of member satisfaction or search efficiency.

But what if the greatest immediate ROI of AI isn't actually about giving your members perfect answers on day one? What if the greatest value is that implementing governed AI finally forces your organization to sit down, look at your scattered files, and agree on what your official answers actually are?

[AI Host 1] (25:48 - 26:00)

That is something to chew on. Forcing organizational alignment as the hidden ROI of AI. Love that.

We will leave it there for today. Grab the checklist in the show notes, start sorting those buckets, and thanks for joining us on this deep dive. We'll see you next time.

[Rob Barnes] (26:07 - 26:41)

So the big question today was, are we ready for AI? And I thought it was really interesting, Thomas, that the research pulled out this kind of risk use case around a document that was buried deep in, you know, and I think it's called it buried by obscurity. But all of a sudden it was resurfaced again, just because of the capability of generative AI and some of the challenges, you know, getting ready.

Do we even have to worry about being more cautious because of things like that potentially happening?

[Thomas Altman] (26:42 - 29:06)

Yeah, I think to me, that was super interesting, too. It wasn't something I was expecting, but it made me think. And one of the things I liked about it was this idea of understanding sort of the buckets of content that you can get started with first, right?

So this idea of if it's in SharePoint, a lot of times if you've got a legacy sort of Microsoft system that was set up in the 90s, and you've gone through like 10 different IT directors in the interim, who knows who has permissions or access to what? And really what's preventing people from accessing stuff they shouldn't access is the fact that they don't even know that they can. But AI would be able to, right?

So that's a genuine, real concern. And it's something that I really caution people around when they're thinking about adopting a solution like Copilot, right? Not that Copilot's bad, but like, it's something you need to think about.

And to me, that is definitely like a risk. It's something you don't want to think too much about. So then flipping that around, where is there content that is de-risked?

So I liked that the AI duo kind of dialed in on, it was around like the seven, seven and a half minute mark, where they started talking about like these different buckets, right? And the idea that you've got some content that I think you already are pretty confident in, right? So that might be your, you know, if you've got journal articles that you publish that went through peer review, if you've got stuff that's published on your website, you know, around upcoming events or things that just need to be correct and have gone through some process where you trust it.

Or if you've got standards or a handbook or a textbook, right? These are kind of sources of truth that have been vetted, right? So understanding what those vetted pieces of content are, they're more trustworthy, and you probably know who the audience is, you know, who's going to use it, you know how they want to use it, right?

So we're starting to build these sort of defendable use cases here, you know, the content, it's trustworthy, you know, who should have access to it and how they are intended to use it. And then you internally probably already have a set of rules that govern not just access, but you know what it can be used for. And once you have those three things, an audience, the sources, and the governing rules, that is a kind of a clean bucket that you then can use AI for.

So kind of carving those out and starting with those while you clean up your SharePoint, right? Those kind of permissions.

[Rob Barnes] (29:06 - 31:28)

And we hear this quite often, that sense of, what I love about associations is even through the natural onboarding of new teams over time, people who work there now know what the trusted sources of information is almost immediately. They know that the policy documentation that exists is the current truth, the journal articles, you know, the white papers, these sorts of things. They might not know the full extent of the library of all of that, but they know that that's the stuff that everybody can trust and that the members, the profession trusts in them sort of straight away.

So, you know, when they started talking about what AI isn't going to do, you know, in terms of fixing bad guidance or repairing, you know, the broken permissions of who's got access to what to do what, that human oversight, and I think, you know, we've always talked about the human in the loop, uh, is still an incredibly important part and a much more focused function.

And that’s what’s useful about today’s, um, companion asset that’s available for everyone to download, it’s really look at what is the purpose for implementing AI. You know, instead of just taking the board’s recommendation, which is “We need to be doing AI,” whatever that really means, look for the purpose first. And don’t be so concerned about the full, kind of, audit. You know, it’s not about creating a single clearing house necessarily. AI is going to help you get to that sort of path, but form follows function has always been a fun sort of thing for me. If you decide on the function that you need to solve, the form that the solution takes, in some cases, becomes kind of self-evident. I think the readiness checklist that we’ve prepared for everyone today is something that everyone is going to get a benefit from following this.

So, I would say, one more time before you go, everybody, download the companion asset today. It’s the fastest way to actually put today’s episode to work for you.

Thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question: “What do we tell the board?”

What this helps your team answer

Why you should get AI in order first, and let it inform your digital transformation strategy.
How to triage content into readiness buckets before launch: fragmented but accurate, incomplete/uneven, and contradictory/obsolete, or permission-confused.
Two big governance risks to guard against: stale permissions and outdated versioning. Both require explicit source labeling, access rules, and ongoing human coaching.

Companion Asset

AI Content Governance Readiness Checklist

0:00 30:41

[Thomas Altman] (0:05 – 0:09)

Today's big question, what do we tell the board?

[Rob Barnes] (0:09 – 1:27)

We are Thomas Altman and Rob Barnes, and this is episode two of the Association Intelligence Podcast. Before we get into today's episode, this episode's companion asset is the board brief, Governed AI for Association.

It's linked in the show notes, grab it now if you want to follow along, or download it afterwards and put it to work. Thomas, this is an excellent question, and I'm glad it came up in the research that we did for Association Intelligence Podcast, because oftentimes, and I think even what do we tell the board? I'm hearing a lot is like what the board has told us, you know, from association executives.

It's like, as we talked about in the last episode, it's like, we have to do AI, but without any kind of real framework for any of that, and, you know, bringing a board a solution without any of the, I guess the preambles, like, I think about it from an alignment perspective, again, purpose, the purpose of the organization is not to become a software company and start developing its own AI. You know, do we see organizations tie themselves in knots now at a board level because of this question around, we have to be doing AI?

[Thomas Altman] (1:27 - 3:31)

Who doesn't love, who doesn't love the board meetings, right? Who doesn't love getting up and having to, you know, stand in front of a group or do your presentation, right? And to me, anytime I'm in front of a board, and we've done several of these, right?

It's the ultimate herding cats experience, right? You've got a bunch of different people from a bunch of different backgrounds, all who do care about the well-being of the association, but they bring their own biases or their own experience, right? So, you've got sort of the person who's obsessed with AI, and they want to be the AI guy, and they kind of get too into the weeds, and they lose sight, or you have kind of the, the person's a bit maybe more fearful of AI, and right, they're like, well, we don't want to do this, or we don't want to do that.

And how do you get all of these people on the same page, so that you're delivering value back to your members at the end of the day, right, that you're executing on the mission of your association, the best way you can with the technology that we have. And that's kind of the goal of this episode. And then sort of where people ask, like, I got to prep for my board meeting, what I need to do.

The thing that I thought was interesting is calling out that it's not really the AI that you want to focus on. It's the mission of the association, right? You don't want to bring a demo, because demos, or you don't want to bring just a shiny demo, and then that becomes a tax service, like people just start picking apart, like, oh, should this button be blue or red, right?

You don't want to get to that level. Or you don't want to start going in the weeds around sort of technical jargon, right, because, or like your favorite model, this model or that model, like, that doesn't matter. At the end of the day, what I think the thread that we were able to pull on with this, and kind of it's been at least my experience, I think Rob, yours as well, you want to be able to deliver a kind of a finely crafted, this is advancing our mission in this way.

It's protecting us from these particular concerns that are fairly common throughout any board meeting and they're valid, right? And this is the way we're going to do it. So focus on what you're delivering, how you're protecting your downside, and the value you're creating at the end of the day.

And Rob, I know you've been in many board meetings, I don't know what your take is.

[Rob Barnes] (3:31 - 5:53)

Yeah, the board meeting that has the self-anointed AI expert on the board, because they happen to do it in their job or for their company or wherever, it's that I've seen boards be derailed by that level of enthusiasm, because that individual has forgotten their role as a steward of the organization. And got into the exciting kind of new tool. And that also has an impact on the staff that are trying to navigate alignment.

And this is what this episode, I think, really brings to the fore quickly is this concept of alignment. When you are thinking about AI that's to solve a problem and create value, value needs to be aligned to strategic purpose. And this is where we want to keep our board is on the strategy.

They've agreed to three to five years strategic plan. They're still very common. So when we're looking at value, align it to the things that the board has already agreed to and remind them at this opportunity to say, hey, this is the stewardship that you've already agreed to.

This is the strategy going forward. This is an opportunity for us to, and you're right, it's less, hopefully at that point, it's kind of less about the tool itself. The boards that we talk to, they've got, they understand their fiduciary risks.

They're looking to protect the organization from any potential threats that seem to be well publicized around AI these days. But if we can bring that back to, again, the blueprint, this is the opportunity. This is the value.

This is the alignment to strategy. Then it does get us a little bit closer to that. And there are some great guardrails.

You know, I think in the episode it talks about some really interesting guardrails that everybody can kind of learn from. And, you know, that's part of the companion asset as well. You know, we see some of this stuff go sideways pretty quickly with boards and with an association, but we're also seeing ones that have got a little bit of courage mixed in with a real understanding of their stewardship, start to take great strides very quickly and get great value and realize the impact.

And ultimately that's what we want, right? Is we want the association to have impact, right?

[Thomas Altman] (5:52 - 6:18)

And with that, over to our AI hosts.

[AI Host 2] (6:24 - 6:25)

What do we tell the board?

[AI Host 1] (6:25 - 6:32)

Oh, man, that is, I think that's the single most critical question hanging over executive leadership right now.

[AI Host 2] (6:32 - 6:40)

Yeah, absolutely. I mean, if you're listening to this deep dive, chances are you are an association CEO or, you know, a COO, maybe a senior leader.

[AI Host 1] (6:41 - 6:42)

Staring down an upcoming board meeting.

[AI Host 2] (6:42 - 6:50)

Exactly. You know, artificial intelligence is a priority. Your members are asking about it.

Your staff is like already experimenting with it on their phones.

[AI Host 1] (6:50 - 6:52)

Yeah, on their personal devices, for sure.

[AI Host 2] (6:52 - 7:09)

And you are expected to present this coherent path forward to your governing body. But there's this massive gap between knowing you need a strategy and actually standing in front of a fiduciary board without the whole meeting just, you know, turning into a complete derailment.

[AI Host 1] (7:09 - 7:14)

Which happens a lot. And the approach we're outlining today really relies on what we called the demo versus blueprint dynamic.

[AI Host 2] (7:15 - 7:16)

Right. Break that down for us.

[AI Host 1] (7:16 - 7:30)

Well, the absolute worst maneuver an executive can make is walking into that boardroom, pulling up a screen and just asking the board to, like, approve AI based on some flashy generic technology demonstration.

[AI Host 2] (7:30 - 7:32)

Because that just screams risk to them.

[AI Host 1] (7:32 - 7:41)

Exactly. So we're going to map out the alternative. We're going to detail why you need to bring your board a one page blueprint instead of a pitch.

[AI Host 2] (7:41 - 7:55)

A blueprint that details what concrete value, specific risk controls, internal ownership and really strict rollout boundaries. Got it. Now, before we get into the actual mechanics of that, I do want to give you all a tool right out of the gate.

[AI Host 1] (7:56 - 7:57)

Oh, yeah. The companion asset.

[AI Host 2] (7:57 - 8:08)

Right. We have a companion asset for this deep dive called the board brief. Consider it like your show notes tool, specifically designed to help you structure this exact internal board conversation.

[AI Host 1] (8:08 - 8:09)

It's super helpful.

[AI Host 2] (8:09 - 8:19)

It really is. We'll circle back to it later. But just know that you don't need to furiously take notes or memorize the framework we're about to build.

That board brief is your practical next step.

[AI Host 1] (8:19 - 8:27)

Perfect. So before we can discuss how to present the right blueprint, we really have to look at what the board is actually apprehensive about.

[AI Host 2] (8:27 - 8:28)

You know, what's keeping them up at night.

[AI Host 1] (8:28 - 8:39)

Right. Looking at patterns from nearly 200 association AI implementations, the mapping shows us that board anxiety generally clusters around a few very specific areas.

[AI Host 2] (8:39 - 8:40)

Like what?

[AI Host 1] (8:40 - 8:45)

Content scope, intellectual property leakage, multi-tier access, traceability.

[AI Host 2] (8:46 - 8:47)

Traceability is a big one.

[AI Host 1] (8:47 - 8:55)

Huge. Plus answer quality, member trust, reputation risk. And honestly, who actually owns the review process?

[AI Host 2] (8:55 - 9:04)

OK, let me play devil's advocate here for a second, because if I'm an executive who's really pushing for modernization, I might find those fears a bit overblown.

[AI Host 1] (9:04 - 9:05)

Really? How so?

[AI Host 2] (9:05 - 9:18)

Well, I mean, if an association's PDFs and articles are already sitting on their public facing website, hasn't a system like ChatGPT already scraped and read them anyway? Like, why is the board acting like this is some sudden new threat to our intellectual property?

[AI Host 1] (9:18 - 9:31)

OK, I hear that a lot, but there is a profound mechanical difference between a public search engine scraping your public marketing copy and an association taking its proprietary multi-million dollar member research.

[AI Host 2] (9:31 - 9:33)

Oh, you mean the stuff hidden behind the paywall.

[AI Host 1] (9:33 - 9:38)

Exactly. Taking those paid assets and just plugging them into an unmanaged third party AI model.

[AI Host 2] (9:38 - 9:39)

Right. That makes sense.

[AI Host 1] (9:39 - 9:53)

Board members have a fiduciary duty to protect the organization's assets. So when they ask if a new AI tool is going to absorb their peer review journals and just feed them to the open Internet, I mean, they're asking a fundamental governance question.

[AI Host 2] (9:54 - 9:55)

They want to know if the pipe flows both ways.

[AI Host 1] (9:55 - 10:07)

Exactly. They need to know the pipe is sealed. But the common mistake we see leaders make is misinterpreting this apprehension as like a lack of technical understanding.

[AI Host 2] (10:08 - 10:09)

So they try to overcompensate?

[AI Host 1] (10:09 - 10:19)

Yes. The leader walks into the boardroom and tries to overcompensate by explaining the plumbing. They start throwing around terms like large language models, our RAG, vector databases.

[AI Host 2] (10:19 - 10:20)

Just trying to sound cutting edge.

[AI Host 1] (10:20 - 10:22)

Right. And it completely backfires.

[AI Host 2] (10:23 - 10:32)

Wait, before we move past that, for the executive listening who keeps hearing AAG from their IT department, what actually is that? Let's just define it quickly so we know what concept we're supposed to be avoiding in the boardroom.

[AI Host 1] (10:32 - 10:50)

Sure. So RAG stands for retrieval augmented generation. Instead of asking an AI to answer a question from its general memory, which is, by the way, how you get hallucinations and totally made up facts, RAG forces the AI to run a search through your specific documents first.

[AI Host 2] (10:50 - 10:52)

Oh, OK. So it has to look it up.

[AI Host 1] (10:52 - 11:02)

Yeah. It retrieves your document, reads it, and then generates an answer based strictly on what it just read. Think of it as changing the AI's task from a closed book exam to an open book exam.

[AI Host 2] (11:02 - 11:22)

Where the only book allowed is your official archives. That makes total sense. But to your point, pitching the mechanics of retrieval augmented generation or showing some flashy demo to a board without a governance model.

I mean, it's like trying to sell someone a sports car by only showing them the top speed right when they're asking to see the safety crash ratings.

[AI Host 1] (11:22 - 11:31)

That is a perfect analogy. And treating the phrase we have an AI policy as if it is the same thing as having a governed system. That's another major pitfall.

[AI Host 2] (11:31 - 11:34)

Yeah. A policy on a piece of paper isn't a system.

[AI Host 1] (11:34 - 11:45)

Right. Leading with hype or competitor fear, like saying we have to do this because rival associations are instead of leading with architecture. Well, that guarantees the board will pump the brakes.

[AI Host 2] (11:45 - 11:50)

But safety ratings alone don't sell the car either. Nobody buys a Volvo just to leave it parked in the driveway.

[AI Host 1] (11:50 - 11:51)

Fair point.

[AI Host 2] (11:51 - 11:59)

We need governance. Sure. But governance is simply the condition that makes it safe to unlock the actual value sitting in the association's archives.

[AI Host 1] (11:59 - 12:10)

Right. The board conversation cannot be a referendum on AI enthusiasm. If the boardroom devolves into a philosophical debate about whether AI is good or bad for humanity, the initiative is dead.

[AI Host 2] (12:11 - 12:11)

Oh, totally dead.

[AI Host 1] (12:11 - 12:22)

The conversation must be entirely about value activation. Because associations are sitting on a staggering amount of intellectual property that they have already paid to create.

[AI Host 2] (12:22 - 12:37)

Yeah. We are talking about decades of conference recordings, webinars, certification guides, industry standards, peer reviewed research, FAQs, not to mention the subject matter expertise that's just trapped in the heads and like the email outboxes of senior staff.

[AI Host 1] (12:37 - 12:49)

Exactly. And much of that value is completely inert right now. It's trapped in a PDF on some clunky legacy portal or buried in an MP4 video file from an annual conference three years ago.

[AI Host 2] (12:49 - 12:59)

Right. If a member needs to know the association's stance on a highly specific regulatory change, they are definitely not going to sit down and watch a two hour conference video from 2022.

[AI Host 1] (12:59 - 13:04)

No way. Not just to find the three minute segment where a panelist happened to discuss it.

[AI Host 2] (13:04 - 13:15)

Right. A recorded annual conference session shouldn't be a one time asset. A member shouldn't have had to be sitting in a specific hotel ballroom on a specific Tuesday two years ago to get the value of that insight.

[AI Host 1] (13:15 - 13:23)

Beautifully said. And what a governed source bound knowledge assistant does mechanically is multiply that existing value by making it findable and evergreen.

[AI Host 2] (13:23 - 13:25)

It brings it back to life.

[AI Host 1] (13:25 - 13:39)

Yes. It transforms an association's static dormant archives into an active on demand mentor. The AI itself is just a translation layer.

The actual product is your historical knowledge delivered at the exact moment a member experiences a problem.

[AI Host 2] (13:39 - 13:50)

OK. So if the goal is to safely unlock that trapped value without, you know, triggering the board's risk aversion, we really need to talk about how to construct that one page blueprint.

[AI Host 1] (13:50 - 13:51)

The blueprint is everything.

[AI Host 2] (13:52 - 14:02)

You mentioned five specific boundaries that define the exact shape of the box we are asking the board to approve. Let's look at the mechanisms behind those. The first one is the source boundary.

Right.

[AI Host 1] (14:02 - 14:18)

The source boundary dictates exactly what content the system is mechanically permitted to read. You present a hard line to the board. This system will only pull from our approved 2023 and 2024 conference transcripts, our peer reviewed journals and our official policy handbook.

[AI Host 2] (14:18 - 14:21)

And it is completely disconnected from the open Internet.

[AI Host 1] (14:21 - 14:28)

Exactly. That immediately lowers the temperature regarding intellectual property leakage. You are defining the universe of knowledge.

[AI Host 2] (14:28 - 14:47)

But what about the access boundary? Because how does a system actually differentiate between what a non-member is allowed to see versus a premium paying member? Like if I'm on the board, I am terrified that an AI is going to accidentally summarize our $500 certification study guide for a free website visitor.

[AI Host 1] (14:47 - 14:55)

Oh, absolutely. And that's why the access boundary requires the AI to integrate with your existing association management system or AMS. OK.

[AI Host 2] (14:56 - 14:56)

How does that work?

[AI Host 1] (14:56 - 15:10)

Well, when the user logs in, the AMS passes a token to the AI saying, hey, this person is a tier one premium member. The AI then searches the vector database, but it only retrieves documents tagged for tier one.

[AI Host 2] (15:10 - 15:11)

Oh, wow.

[AI Host 1] (15:11 - 15:18)

So if a non-member asks the exact same question, the system physically cannot see the premium study guide. It respects the entitlement structures you have already built.

[AI Host 2] (15:18 - 15:24)

That is super smart. OK, let's push on the third boundary, because I think this is where implementations actually die.

[AI Host 1] (15:24 - 15:25)

The answer and traceability boundary.

[AI Host 2] (15:25 - 15:33)

Right. Because even if you restrict the sources and lock down the access tiers, large language models are just notorious for trying to be overly helpful.

[AI Host 1] (15:33 - 15:34)

They want to please you.

[AI Host 2] (15:34 - 15:51)

Right. They combine two correct facts into one massive, embarrassing hallucination. If I am a board member, my nightmare is our official AI giving a member legally disastrous advice.

How do you prevent them from shutting the project down the very first time it makes a mistake?

[AI Host 1] (15:51 - 16:02)

You architect the system so it is fundamentally incapable of returning an answer without providing a literal clickable footnote back the specific paragraph it pulled from.

[AI Host 2] (16:02 - 16:11)

So instead of thinking of the AI as a college graduate who read all your files and is trying to summarize them from memory, it's more like an ultra fast intern with a highlighter.

[AI Host 1] (16:11 - 16:22)

I love that. Yes, an intern with a highlighter. It isn't generating original thoughts.

It is only allowed to fetch and highlight the exact paragraph in your archives. It has to show its work.

[AI Host 2] (16:22 - 16:24)

So mechanically, how does it do that?

[AI Host 1] (16:24 - 16:33)

The system maps the user's query to an exact digital coordinate in your database, a vector point. It finds the paragraph sitting at that exact coordinate and retrieves it.

[AI Host 2] (16:33 - 16:34)

And if it can't find a match?

[AI Host 1] (16:34 - 16:49)

If it cannot find a match within the approved source boundary, it is programmed to say, I don't know, but here is a human you can contact. Traceability gives the board the accountability they require. Proving the system is only fetching your approved truth.

[AI Host 2] (16:49 - 17:02)

OK, but what happens when the intern highlights the wrong paragraph? Or, you know, what if the source document it fetches is technically correct based on a 2019 policy, but functionally wrong because the industry moved on?

[AI Host 1] (17:03 - 17:08)

And that brings us to the human coaching boundary. AI is not a set it and forget it technology.

[AI Host 2] (17:08 - 17:09)

No, definitely not.

[AI Host 1] (17:10 - 17:17)

The board needs to know whose desk the buck stops at, which specific staff members or subject matter experts own the coaching and review process.

[AI Host 2] (17:17 - 17:19)

So there has to be a way to course correct.

[AI Host 1] (17:19 - 17:31)

Exactly. If an answer needs to be adjusted, there must be a dashboard. Where human staff member can look at the query, see the document the AI fetched and manually correct the pathway for the future.

You are showing the board who holds the steering wheel.

[AI Host 2] (17:31 - 17:43)

Which naturally leads to the final piece of the blueprint, the rollout and success boundary. I imagine you never walk into a board meeting and announce a plan to launch an AI to 50,000 members globally on a Tuesday.

[AI Host 1] (17:44 - 17:53)

Oh, please do not do that. A phased pilot is universally more credible than a sweeping, untethered transformation plan. You contain the blast radius.

[AI Host 2] (17:53 - 17:54)

Start small.

[AI Host 1] (17:54 - 18:06)

Right. You propose rolling the system out exclusively to the internal membership team for the first 30 days to help them answer phone queries faster. Or you run a beta test strictly with your top tier committee members.

[AI Host 2] (18:06 - 18:25)

You tell the board exactly who gets it first and what metrics will determine if it expands. Exactly. So we have built this highly governed, deeply accountable box.

But how do we measure if it is actually worth the investment without promising the board guaranteed financial outcomes? Because if we over promise ROI in year one, we are setting ourselves up for failure.

[AI Host 1] (18:26 - 18:42)

We definitely frame the rollout as a series of measurable hypotheses. You are bringing the board pads to test. And looking at the implementation patterns across those 200 associations, we see three primary hypotheses that organizations test during a pilot phase.

[AI Host 2] (18:42 - 18:43)

Let's hear them.

[AI Host 1] (18:43 - 18:45)

First is a member return behavior hypothesis.

[AI Host 2] (18:45 - 18:53)

OK, so how does the AI mechanically shift a member from visiting the website once a month to, say, once a week?

[AI Host 1] (18:53 - 18:57)

Think about it. Traditional association search bars are often incredibly frustrating.

[AI Host 2] (18:57 - 18:58)

Oh, the worst.

[AI Host 1] (18:58 - 19:07)

A member types in a query about a complex compliance issue and the search bar returns 40 unhelpful links to massive 100 page PDF documents.

[AI Host 2] (19:07 - 19:12)

And then the member has to download them and use CTRL plus F just to find their answer. Right.

[AI Host 1] (19:12 - 19:25)

But if the new system provides an instant plain language summary with a direct citation to the exact page they need, all that friction vanishes. So the hypothesis is, if we remove the friction of finding knowledge, does the frequency of member engagement increase?

[AI Host 2] (19:26 - 19:27)

I'd bet it does. What's the second hypothesis?

[AI Host 1] (19:28 - 19:36)

The second is the non-member preview or the join path hypothesis. This seems like it could fundamentally alter how marketing departments operate.

[AI Host 2] (19:37 - 19:38)

Oh, interesting. How so?

[AI Host 1] (19:38 - 20:02)

Consider the psychological trigger of the access boundary we discussed earlier. A non-member lands on your site and asks a highly technical industry question. The system provides a high level two sentence summary.

But the deep dive citations, the actual mechanics of the solution. Are blurred out. Yes, they're blurred out or locked behind a prompt that says log in or join to read the full standard.

[AI Host 2] (20:02 - 20:16)

Oh, that is the ultimate show don't tell for membership value. You aren't just sending them some generic marketing email claiming you have the best resources. You are proving you have the exact answer to the problem that you're trying to solve right now and offering them the key to unlock it.

[AI Host 1] (20:16 - 20:25)

Exactly. It's incredibly powerful. And the third hypothesis might actually be the most operationally transformative.

The intelligence and content gap hypothesis.

[AI Host 2] (20:26 - 20:29)

Moving from search bar zero to actual query intelligence.

[AI Host 1] (20:30 - 20:45)

Exactly. When a member uses a traditional search bar and finds nothing, they just leave. The association never knows what they were even looking for.

But with a conversational system, you capture the exact natural language questions your industry is asking.

[AI Host 2] (20:45 - 21:04)

So if hundreds of members suddenly query the system about a new international supply chain regulation and your system returns empty handed because your association hasn't written a policy on it yet, you now have real time intelligence. You are literally crowdsourcing your editorial calendar based on immediate member needs rather than just guessing what topics to cover at the next annual conference.

[AI Host 1] (21:05 - 21:19)

Bingo. A practical example of this structural approach in the real world is a system like Betty. It is designed around these exact patterns, a governed, source bound, multi-tier coached and traceable association AI.

[AI Host 2] (21:19 - 21:20)

Right.

[AI Host 1] (21:20 - 21:30)

And when we look at implementations like the one at ISA, the International Society of Automation, what is really notable is how they built their deployment team. They didn't just treat it as an isolated IT project.

[AI Host 2] (21:30 - 21:30)

Which is so common.

[AI Host 1] (21:30 - 21:37)

Right. Instead, they assembled a cross-functional group involving governance, membership, sponsorship and IT.

[AI Host 2] (21:38 - 21:53)

And having a board liaison involved directly in the pilot group is just an incredibly strategic move because then the board is no longer sitting in judgment of some black box technology at the end of the quarter. They are active participants in tuning the governance model.

[AI Host 1] (21:54 - 22:03)

It demonstrates profound organizational maturity. However, any executive presenting this blueprint must deliver an honest caveat to the board.

[AI Host 2] (22:03 - 22:04)

There's always a catch.

[AI Host 1] (22:04 - 22:15)

Always. Even with the perfect blueprint, the most robust governance model and a brilliant cross-functional team, we have to set realistic expectations about what source bound AI absolutely cannot do.

[AI Host 2] (22:15 - 22:26)

Right. Because if the association's source material is an unorganized mess full of contradictory policies from different decades, the AI is not going to magically organize it into a cohesive strategy.

[AI Host 1] (22:26 - 22:52)

Exactly. Source bound AI is a mirror. It does not fix bad source content.

If your PDS from 2018 directly contradicts your web copy from 2024, the AI will simply fetch both and expose that conflict with incredible speed. Yeah. It will not resolve internal policy disputes.

It does not remove bias that is already present in your historical source material. And it absolutely does not solve existing organizational governance problems.

[AI Host 2] (22:53 - 22:54)

Garbage in, garbage fetched.

[AI Host 1] (22:55 - 22:55)

Precisely.

[AI Host 2] (22:56 - 23:13)

But honestly, I would position that as another point in favor of the blueprint approach. An executive can stand in front of the board and say, look, this system is going to shine a harsh light on where our archives are messy and where our content is weak. And we welcome that.

We want to know where the gaps are so we can clean them up.

[AI Host 1] (23:14 - 23:29)

That is a great spin. It turns a potential operational negative into a strategic positive. It manages expectations beautifully.

The board won't expect a flawless omniscient oracle. They will expect a tool that retrieves what the association has built exactly as it was built.

[AI Host 2] (23:29 - 23:44)

We have covered a tremendous amount of ground today regarding the mechanics of governance, boundaries and hypotheses. The core takeaway for the listener preparing for that upcoming meeting is clear. Do not walk into the boardroom and ask your governing body to approve AI.

[AI Host 1] (23:44 - 23:44)

We don't do.

[AI Host 2] (23:45 - 23:54)

If you do, you are asking them to underwrite an unlimited, unbounded concept and their fiduciary risk aversion will shut the initiative down immediately.

[AI Host 1] (23:55 - 23:57)

Bring them a one page architecture and value brief.

[AI Host 2] (23:57 - 23:57)

Yes.

[AI Host 1] (23:57 - 24:06)

Show them the bounded box. Show them exactly what historical knowledge assets will be activated, what the system can access and what it physically cannot access.

[AI Host 2] (24:06 - 24:12)

Show them the traceability mechanism and define where the initial rollout is intentionally restricted.

[AI Host 1] (24:12 - 24:12)

Yes.

[AI Host 2] (24:13 - 24:25)

And to help you translate the strategy into a physical document, you can actually slide across the boardroom table. I want to remind you to grab the board brief companion asset we mentioned at the start of this deep dive.

[AI Host 1] (24:25 - 24:26)

It's basically a cheat sheet.

[AI Host 2] (24:27 - 24:33)

It really is. It's structured specifically to help you build out this exact one page conversation. It is your immediate practical next step.

[AI Host 1] (24:34 - 24:56)

And as you prepare that brief and define the value of unlocking your archives, consider the long term mechanical shift this creates. Once a member can instantly query your association's entire history of standards, webinars and peer reviewed research at the exact moment they face a crisis at their desk. How does that fundamentally shift the core value proposition of an association?

[AI Host 2] (24:56 - 24:57)

That's a huge question.

[AI Host 1] (24:57 - 25:05)

It is moving away from just being about networking and annual events to actually be an indispensable daily operational partner. That's the real transformation.

[Rob Barnes] (25:11 - 25:19)

Super interesting. The big question today, what do we tell the board? Thomas, what did you take away from that?

[Thomas Atlman] (25:20 - 27:23)

So I like this one and I like the way it kind of it drove to what I think was a pretty conclusive ending, right? That the idea that you want to approach the board with sort of this value brief, this this way to unlock kind of your archives, consider the long term mechanical shift. I think that was the the idea here.

How does your value proposition as an association change and have a brief ready for the board in that matter? So the idea to me is thinking along those terms of. Kind of what are what are we actually asking the board to approve?

Like you don't you're not just blindly saying we're asking the board to let us use AI. We're asking the board to let us use AI so that we can fill in the blank. Right.

What is that so that we can and then thinking in those terms like who are who are the people you're using it for? Like frame it more as value towards your member, advancing your association's ambition, your three to five year plan concretely using this tool. Right.

And it's not you want to shift the conversation away from the tool and more about the value and thinking in those terms, thinking the value breeding thing, how it contributes to the larger strategic mission that the board is there to oversee and then thinking about the guardrails, sort of how you want to protect your your downsides. Right. So if questions are.

Does this protect our IP or are we giving away our IP to be trained on some other model and we actually lose control? Right. Understanding what that is, that's a very common concern that I think gets addressed.

The other one is who has access to it by letting people use AI? Are we kind of undercutting our value proposition? Right.

That's another concern that I think you need to have ready to address there. Like do non-members get access to paid materials and how do we govern that? So once you identify those guardrails, you can create a governing strategy around that and including that as part of your brief, I think that makes it much more powerful and it makes it something the board is excited to approve instead of worried about.

[Rob Barnes] (27:23 - 29:10)

It's super interesting to me that the research that picked out International Society for Automation part of this, because I felt like, you know, when we look at the five boundaries that are shown in the companion asset for today's episode, I reflect back on the conversations I was having with the team at ISA and they had a very good sense of these boundaries like right from the get-go and which probably speaks to the success of MIMO, award-winning MIMO, I might add, now because they really did have a good sense of what was the, you know, what's the source, what's the approved content that we wanted to make accessible in ways that wasn't possible before, you know, who can get access to it, how they had learned that, you know, that automation engineers, professionals around the world were wanting, you know, the responses that they needed from this content source, you know, these assets themselves, and always this really strong sense of the human in the loop, you know, they have this cross-functional team that own MIMO and the way MIMO performs, it's not just an IT thing, it's not just a membership or governance or marketing thing, it's very much some cross-ownership, and so that human in the loop element was there right from the get-go, and I think that the, well, I know that the way they have sort of reported upstream and downstream, you know, the way the board have taken ownership of MIMO's value and alignment to strategy has really helped keep them not just, that's thriving, like it's growing in its capability and its use because of that strategic alignment, the board owns that now.

[Thomas Atlman] (29:11 – 30:09)

And I think, like it ended on this note, it really stuck with me, this podcast, where it says you want to move away from being sort of about networking and annual events and more into an indispensable daily operational partner, right? So the idea here is that because you've invested this time in identifying the value and helping people unlock kind of value that's already there, like get access to stuff, you aren't this punctuated, like at our annual conference, people are engaged, but then we don't see them again until the next annual conference, or people stay alongside with you as an operational partner, the association then gets embedded in the workflows and the daily lives of its members. And kind of reframing the value around that, which I think is something ISA did really really really well. And kind of presenting that as the end goal back to your board, I think makes it much more of an exciting prospect and let’s you get to yes much more easily than kind of getting dragged into the weeds around what model do you want to use.

[Rob Barnes] (30:10 – 30:41)

So one more time, before you go, thanks everyone for listening today. Download the Board Brief: Governed AI for Associations from the show notes. It’s the fastest way to actually put today’s episode to work. Thank you for listening to the Association Intelligence Podcast. Join us next week to answer this question: “Why not just use ChatGPT or Copilot?”

What this helps your team answer

How to frame AI adoption without triggering board risk aversion.
Which outcomes matter to governance-minded leaders, and what boundaries (source, access, traceability, human ownership, rollout scope) need to be defined to earn board confidence and approval.
How to set realistic, measurable expectations for a pilot instead of overpromising ROI, and what you should track to prove its working.

Companion Asset

Board AI Briefing One-Pager

Tool Selection

Why Not Just Use ChatGPT or Copilot?

A practical breakdown of when to use ChatGPT, Copilot, or governed AI — separating personal productivity, Microsoft ecosystem work, and the official answers your association has to stand behind.

36:07

0:00 36:07

00:00.000 --> 00:08.140
Today's big question, why not just use ChatGPT or Copilot?

00:08.140 --> 00:13.380
We are Thomas Altman and Rob Barnes and this is Episode 3 of the Association Intelligence

00:13.380 --> 00:16.020
Podcast.

00:16.020 --> 00:22.320
Before we get into it, this episode's companion asset is the ChatGPT, Copilot or Governed

00:22.320 --> 00:25.160
AI Use Case Decision Guide.

00:25.160 --> 00:29.640
It's linked in the show notes, so grab it now if you want to follow along or download

00:29.640 --> 00:32.120
it after and put it to work.

00:32.120 --> 00:40.440
Thomas, one of the most fun episodes I think, simply because the analogies that came out

00:40.440 --> 00:48.960
of the research and the interaction with Betty and the AI tools, it's just incredibly fun.

00:48.960 --> 00:56.740
Talk us through the custom GPT as a treehouse bit because this really resonates and I want

00:56.740 --> 01:02.820
to prep everybody that's about to listen to the call to stay on track with this.

01:02.820 --> 01:06.300
This is one of those things where this whole project of creating this podcast has been

01:06.300 --> 01:13.300
so eye-opening for me because it's this interaction with human talking to AI, the dialogue, us

01:13.300 --> 01:15.660
kind of debating and going back and forth.

01:15.660 --> 01:19.420
And then something comes out of that that's not just from the AI or just from us, it's

01:19.420 --> 01:20.580
something more.

01:20.580 --> 01:26.420
And the anecdotes or the metaphors it comes up with are really helpful.

01:26.420 --> 01:29.300
They help me sort of crystallize it in this idea.

01:29.300 --> 01:32.580
So the topic today is like, why not use chat GPT or copilot?

01:32.580 --> 01:36.380
Why do you need sort of a governed solution for associations?

01:36.380 --> 01:41.560
The idea of a treehouse being a custom GPT is so perfect.

01:41.560 --> 01:46.220
The idea that you can build a treehouse and you can go to the backyard and sort of do

01:46.220 --> 01:49.980
it over the weekend and it's going to be cool and you'll learn a lot by doing it.

01:49.980 --> 01:52.100
You'll kind of learn how to build stuff.

01:52.100 --> 01:54.900
But are you going to move your corporate headquarters into that treehouse?

01:54.900 --> 01:55.900
There's no plumbing.

01:55.900 --> 01:56.900
There's no AC.

01:56.900 --> 01:57.900
You're not going to live there.

01:57.900 --> 01:59.660
You can't like put a bed in it.

01:59.660 --> 02:06.860
There's so much more that goes from a cool project that is cool and is fun and is learning.

02:06.860 --> 02:11.100
And we're not saying don't build the treehouse, but there's a difference between that and

02:11.100 --> 02:15.820
something kind of built for being a corporate headquarters.

02:15.820 --> 02:18.820
And just when it came up with that, when I heard it's like a treehouse, it's like, how

02:18.820 --> 02:21.020
did that even come into play?

02:21.020 --> 02:23.300
It's so smart.

02:23.300 --> 02:27.620
I think the perspective that I'm hearing a lot in the conversations I'm having every

02:27.620 --> 02:35.980
week is this, again, this idea of yes, and right.

02:35.980 --> 02:40.700
The Jobs to be Done framework that we use at Betty, it really plays into this nicely.

02:40.700 --> 02:43.820
It's like chat GPT, yes.

02:43.820 --> 02:44.820
Individual productivity.

02:44.820 --> 02:49.980
Obviously, we use called co-work instead of chat GPT, but we use codecs for a bunch of

02:49.980 --> 02:50.980
things as well.

02:50.980 --> 02:53.140
Like that stuff, yes.

02:53.980 --> 02:54.980
And co-pilot.

02:54.980 --> 02:59.020
If you're a Microsoft office shop and you came into the Microsoft office in the 80s

02:59.020 --> 03:01.300
and 90s and you're still there, great.

03:01.300 --> 03:07.780
It's an ecosystem specific kind of environment across applications.

03:07.780 --> 03:13.300
Governed AI is around what do you do when you need to scale capability and skill and

03:13.300 --> 03:18.260
your knowledge assets to external parties.

03:18.260 --> 03:21.980
So I'm often having conversations with Association staff.

03:21.980 --> 03:25.580
And when they ask this question about why can't we just use an enterprise license of

03:25.580 --> 03:31.060
GPT, it's like you're thinking about it like a staff member instead of thinking about it

03:31.060 --> 03:41.580
in terms of your whole Association's mandate and the audiences that it serves.

03:41.580 --> 03:48.020
And then there are some real risks with playing the internal productivity or my internal ecosystem.

03:48.020 --> 03:51.700
When you start to try and think that you can scale that outside and into the, I'll call

03:51.740 --> 03:56.740
it the public domain, but external to the staffing environment, there's some risks still

03:56.740 --> 03:59.420
associated with that which come up in this episode, right?

03:59.420 --> 04:00.900
Yeah, that's exactly the point.

04:00.900 --> 04:04.260
So yeah, co-pilot's great for its purpose.

04:04.260 --> 04:09.340
Chat GPT or Claude co-work, great for their purpose.

04:09.340 --> 04:13.300
Not all those purposes line up perfectly with the Association use case, right?

04:13.300 --> 04:20.080
So as you're going through this episode, kind of listen for those, listen for this fun imagery

04:20.080 --> 04:21.520
that it comes up with, right?

04:21.520 --> 04:26.920
So on the co-pilot side, think about how you're inheriting the permission structure that was

04:26.920 --> 04:32.480
set up maybe by three or four IT directors ago, where the reason that the intern isn't

04:32.480 --> 04:38.800
accessing sort of the CEO's compensation package might not be because they're not allowed to.

04:38.800 --> 04:41.720
It might just be because they don't know they can, right?

04:41.720 --> 04:45.600
And the second you install co-pilot or you use co-pilot, all those permissions get carried

04:45.600 --> 04:49.320
forward and the intern could just say, hey, what's the CEO's compensation package, right?

04:49.320 --> 04:52.800
Those kinds of things happen all the time, right?

04:52.800 --> 04:57.320
And the imagery there was turning on the fluorescent lights in the dive bar at 2 AM and you get

04:57.320 --> 05:01.160
to see all the screw stuff you just didn't know you were sitting in, right?

05:01.160 --> 05:05.920
So those ideas or kind of when is a custom GPT appropriate?

05:05.920 --> 05:07.880
Because it can be in some cases, right?

05:07.880 --> 05:08.880
And when is it not?

05:08.880 --> 05:14.080
Like how is a custom GPT trained on, you know, three or four documents different from scaling

05:14.080 --> 05:18.960
that to over a hundred thousand documents, which many associations do.

05:18.960 --> 05:23.400
And what happens when you ask the question and it doesn't find the right document?

05:23.400 --> 05:24.600
Can you coach GPT on that?

05:24.600 --> 05:25.600
No, you can't.

05:25.600 --> 05:27.440
A governed AI solution, you can, right?

05:27.440 --> 05:31.480
These are the differences that I want people to keep in mind, not so that you always choose

05:31.480 --> 05:37.000
the governed solution, but so that these can co-exist for the use case that they're appropriate

05:37.000 --> 05:38.000
for.

05:38.000 --> 05:39.800
Enjoy this episode, everybody.

05:39.800 --> 05:41.520
It's super interesting.

05:41.520 --> 05:44.720
And we hand over to the AI podcasts host.

05:44.720 --> 05:50.040
A quick reminder that the following is generated using notebook LM with heavy involvement from

05:50.040 --> 05:54.560
the Betty team to ensure that the content generated is valuable.

05:54.560 --> 05:59.400
For a deeper dive on our approach, please go back and listen to episode zero.

05:59.400 --> 06:08.080
And now over to the AI hosts.

06:08.080 --> 06:14.080
Listen a brand new hire asking, you know, your organization's AI to summarize a recent

06:14.080 --> 06:16.080
departmental meeting.

06:16.080 --> 06:21.520
And instead of just giving them the minutes, the tool spits out the CEOs unredacted compensation

06:21.520 --> 06:22.520
package.

06:22.520 --> 06:23.520
Oh, wow.

06:23.520 --> 06:24.520
Right.

06:24.520 --> 06:27.620
And like maybe a list of internal restructuring plans from five years ago.

06:27.620 --> 06:30.880
And here is the wildest part of that whole scenario.

06:30.880 --> 06:32.520
The AI didn't malfunction.

06:32.520 --> 06:33.520
Right.

06:33.520 --> 06:34.520
It didn't go rogue.

06:34.520 --> 06:35.520
Exactly.

06:35.520 --> 06:36.520
It didn't break.

06:36.520 --> 06:40.720
It was actually Microsoft co-pilot working flawlessly doing exactly what it was designed

06:40.720 --> 06:41.720
to do.

06:41.720 --> 06:47.560
It is a it's a pretty sobering reality check, honestly, because it forces us to look at

06:47.560 --> 06:50.840
the tools we already have from a completely different angle.

06:50.840 --> 06:54.840
Which is exactly what we are doing today, because if you're an association leader or

06:54.840 --> 06:58.240
say a tech director, you are looking at your budget right now.

06:58.240 --> 06:59.240
Yeah, you're seeing it everywhere.

06:59.240 --> 07:00.240
Right.

07:00.240 --> 07:03.260
You're seeing AI integrated into literally every piece of software you own.

07:03.260 --> 07:05.960
So you are inevitably asking a very, very practical question.

07:05.960 --> 07:08.800
Why not just use chat, GPT or co-pilot for everything?

07:08.800 --> 07:10.600
It's the most natural question to ask.

07:10.600 --> 07:11.640
It really is.

07:11.640 --> 07:16.440
So today in this deep dive, we are answering that question by looking at the hard data.

07:16.440 --> 07:22.960
We're pulling from documentation that analyzes patterns from nearly 200 association AI implementations.

07:22.960 --> 07:26.280
Which is a huge sample size for this specific sector.

07:26.280 --> 07:27.280
Exactly.

07:27.280 --> 07:28.600
So we aren't talking about theories here.

07:28.600 --> 07:32.560
We are looking at how this actually plays out in the wild.

07:32.560 --> 07:36.700
But okay, let's unpack this, because I think we should establish a ground rule for this

07:36.700 --> 07:37.860
deep dive right up front.

07:37.860 --> 07:39.220
Yeah, I think that's crucial.

07:39.220 --> 07:43.380
Our approach today is firmly rooted in, well, what we call a yes and philosophy.

07:43.380 --> 07:44.380
A yes and.

07:44.380 --> 07:45.380
Okay, explain that.

07:45.380 --> 07:51.100
So the takeaway here is absolutely not that chat GPT or Microsoft co-pilot are bad tools

07:51.100 --> 07:52.420
or that you shouldn't use them.

07:52.420 --> 07:55.060
I mean, they're remarkable engineering achievement.

07:55.060 --> 07:56.060
Truly incredible.

07:56.060 --> 07:57.060
Right.

07:57.060 --> 08:01.380
So the mission is really to understand how to use generic AI where it fits perfectly

08:01.380 --> 08:06.640
and to avoid the trap of confusing, you know, a personal productivity tool with a system

08:06.640 --> 08:10.780
capable of delivering an official governed answer on behalf of your association.

08:10.780 --> 08:15.320
That distinction is so important because it's just so tempting to consolidate everything.

08:15.320 --> 08:16.320
Right.

08:16.320 --> 08:23.800
I mean, if you were already paying for Microsoft E3 or E5 licenses, co-pilot is likely sitting

08:23.800 --> 08:25.240
right there in your tech stack.

08:25.240 --> 08:27.000
Yeah, you're already paying for it.

08:27.000 --> 08:28.000
Exactly.

08:28.000 --> 08:32.100
So staff are definitely already figuring out how to prompt chat GPT to write their emails

08:32.100 --> 08:33.480
faster.

08:33.480 --> 08:37.720
So the natural instinct for any leader is to maximize the investment you've already

08:37.720 --> 08:40.640
made rather than going out and buying something brand new.

08:40.640 --> 08:41.640
Right.

08:41.640 --> 08:46.120
I mean, sunk costs and existing vendor relationships are incredibly powerful drivers.

08:46.120 --> 08:52.640
But expecting one single tool to manage every AI workflow in an organization is, well, it's

08:52.640 --> 08:54.880
like expecting a Swiss army knife to build a house.

08:54.880 --> 08:56.040
Oh, I like that analogy.

08:56.040 --> 08:57.040
Right.

08:57.080 --> 08:58.840
You can have a screw, you can saw a really small branch.

08:58.840 --> 08:59.840
Yeah.

08:59.840 --> 09:01.720
You definitely wouldn't use it to pour a concrete foundation.

09:01.720 --> 09:07.660
And when we look at the patterns from nearly 200 association AI implementations, the organizations

09:07.660 --> 09:10.600
that succeed are the ones that separate the jobs to be done.

09:10.600 --> 09:11.760
The jobs to be done.

09:11.760 --> 09:12.760
Exactly.

09:12.760 --> 09:15.120
They separate them into three very distinct categories.

09:15.120 --> 09:19.360
Okay, let's lay those out because I think having a mental map of these three jobs is

09:19.360 --> 09:21.840
really going to anchor the rest of our conversation today.

09:21.840 --> 09:22.840
Sure.

09:22.840 --> 09:25.720
So the first category is individual productivity.

09:25.720 --> 09:27.800
And this is really where chat GPT shines.

09:27.800 --> 09:29.760
Right, like writing a draft or something.

09:29.760 --> 09:30.760
Exactly.

09:30.760 --> 09:32.640
It is your ultimate brainstorming partner.

09:32.640 --> 09:35.000
It's your drafter, your summarizer.

09:35.000 --> 09:38.520
Then the second category is ecosystem specific work.

09:38.520 --> 09:40.080
And that is co-pilot.

09:40.080 --> 09:41.720
Because it lives inside Microsoft.

09:41.720 --> 09:42.720
You got it.

09:42.720 --> 09:47.120
It is highly specialized for your Microsoft 365 environment.

09:47.120 --> 09:52.640
So synthesizing your emails, drafting Word documents, summarizing Teams meetings.

09:52.640 --> 09:53.640
That's its lane.

09:53.640 --> 09:54.640
Okay, so that's two.

09:54.640 --> 09:55.640
What's the third?

09:55.640 --> 09:58.840
The third category is entirely separate and that is governed AI.

09:58.840 --> 10:03.240
This is a layer designed exclusively for official approved association knowledge access.

10:03.240 --> 10:05.040
Okay, meaning what exactly?

10:05.040 --> 10:09.120
Meaning this is the space where accuracy and authorization are just non-negotiable.

10:09.120 --> 10:10.120
It's for official answers.

10:10.120 --> 10:14.400
Okay, so we have the individual drafter, the Microsoft assistant, and then the official

10:14.400 --> 10:16.080
spokesperson.

10:16.080 --> 10:19.280
And before we tear into the mechanics of why these are so fundamentally different, I do

10:19.280 --> 10:22.200
want to point you to a highly practical asset.

10:22.200 --> 10:24.000
We've actually linked in the show notes for you.

10:24.000 --> 10:25.600
Yeah, this is a great resource.

10:25.600 --> 10:26.600
It really is.

10:26.600 --> 10:30.440
It's called the chat GPT co-pilot or governed AI.

10:30.440 --> 10:32.000
Use case decision guide.

10:32.000 --> 10:37.440
It maps out these exact boundaries we're talking about and helps you evaluate your own internal

10:37.440 --> 10:38.440
projects against them.

10:38.440 --> 10:43.320
So you can open that up right now actually and use it to follow along with what we're

10:43.320 --> 10:44.760
about to unpack.

10:44.760 --> 10:49.760
And having that guide as a visual companion is so helpful, especially as we dive into

10:49.760 --> 10:52.680
probably the most complicated tool on that list, which is co-pilot.

10:52.680 --> 10:53.820
Oh, absolutely.

10:53.820 --> 10:58.460
Because co-pilot is native to the tools most associations use every single day.

10:58.460 --> 11:03.100
So it is almost always the first place leaders look when they want to deploy AI organization

11:03.100 --> 11:04.100
wide.

11:04.100 --> 11:05.100
Naturally.

11:05.100 --> 11:09.140
And Microsoft pushes it heavily as this seamless, secure integration.

11:09.140 --> 11:10.140
Right.

11:10.140 --> 11:14.780
But let's go back to that scenario I opened with, the intern pulling the CEO's compensation

11:14.780 --> 11:15.780
package.

11:15.780 --> 11:21.020
Like, how does a supposedly obedient tool create such a massive security headache?

11:21.460 --> 11:25.060
Well to understand that we really have to look at the underlying mechanics of it.

11:25.060 --> 11:28.600
So co-pilot relies on the Microsoft Graph API.

11:28.600 --> 11:30.820
It does not ingest the public internet.

11:30.820 --> 11:33.340
It ingest your organization's internal data.

11:33.340 --> 11:34.340
Okay.

11:34.340 --> 11:35.660
So it only knows what's inside the walls.

11:35.660 --> 11:36.660
Exactly.

11:36.660 --> 11:39.980
But the critical feature of co-pilot is that it acts as a perfect mirror.

11:39.980 --> 11:41.700
Oh, perfect mirror.

11:41.700 --> 11:42.700
Yes.

11:42.700 --> 11:47.100
It flawlessly honors the permissions that are currently set within your SharePoint,

11:47.100 --> 11:49.500
your OneDrive folders, and your team's channels.

11:49.500 --> 11:50.500
Right.

11:50.500 --> 11:54.280
And if it is properly restricted to just the HR department, co-pilot won't show it to

11:54.280 --> 11:55.620
a marketing intern.

11:55.620 --> 11:56.620
In theory, yes.

11:56.620 --> 11:58.260
Ah, in theory.

11:58.260 --> 11:59.260
Exactly.

11:59.260 --> 12:03.260
Because here's where the operational reality completely collides with the technical design.

12:03.260 --> 12:04.260
Think about it.

12:04.260 --> 12:09.340
How many associations actually have pristine permission hygiene going back a decade?

12:09.340 --> 12:11.220
Oh, none.

12:11.220 --> 12:12.220
Literally none.

12:12.220 --> 12:13.220
Right.

12:13.220 --> 12:17.900
Think about, you know, a sensitive HR policy that was maybe accidentally saved to a company-wide

12:17.900 --> 12:20.460
SharePoint drive back in like 2018.

12:20.460 --> 12:21.780
It's just been sitting there.

12:21.780 --> 12:22.780
For years.

12:22.780 --> 12:23.780
Yeah.

12:23.780 --> 12:25.900
Because that document was protected by obscurity.

12:25.900 --> 12:30.660
Like no human was ever going to spend a random Tuesday afternoon clicking through eight layers

12:30.660 --> 12:33.500
of badly named subfolders to go find it.

12:33.500 --> 12:34.500
Right.

12:34.500 --> 12:35.940
So wait, here's where it gets really interesting.

12:35.940 --> 12:40.660
You're saying that co-pilot doing its job safely and correctly is exactly what creates

12:40.660 --> 12:45.680
a discoverability risk if our own internal digital hygiene is a mess.

12:45.680 --> 12:47.020
That is exactly what I'm saying.

12:47.020 --> 12:49.220
AI essentially eliminates obscurity.

12:49.220 --> 12:50.220
Wow.

12:50.460 --> 12:51.940
Co-pilot doesn't get tired.

12:51.940 --> 12:55.500
Doesn't get frustrated by your terrible folder naming conventions.

12:55.500 --> 12:59.680
So if an employee asks a broad question and co-pilot has the technical permission to see

12:59.680 --> 13:05.340
that misplaced 2018 HR document, it just grabs it, it instantly retrieves it and synthesizes

13:05.340 --> 13:06.820
it right into the answer.

13:06.820 --> 13:09.500
The discoverability risk just skyrockets.

13:09.500 --> 13:11.740
So co-pilot isn't broken in this scenario.

13:11.740 --> 13:15.260
It is holding up a mirror to the reality of your data governance.

13:15.260 --> 13:19.220
It's like it's like turning on the fluorescent overhead lights in a dive bar at two point

13:19.220 --> 13:20.220
a.m.

13:20.220 --> 13:21.220
Yeah.

13:21.220 --> 13:24.820
Suddenly you see all the dirt and the grime that you were perfectly happy ignoring in

13:24.820 --> 13:25.820
the dark.

13:25.820 --> 13:29.900
That is a very apt if slightly unsettling way to put it.

13:29.900 --> 13:30.900
But yeah.

13:30.900 --> 13:35.500
And the implication for association leaders here is that permission hygiene is an absolute

13:35.500 --> 13:36.500
prerequisite.

13:36.500 --> 13:37.620
You have to clean house first.

13:37.620 --> 13:38.620
You have to.

13:38.620 --> 13:42.340
You cannot simply turn co-pilot on for all staff and just let it run if you haven't

13:42.340 --> 13:47.020
done the unglamorous manual work of cleaning up your legacy access controls.

13:47.100 --> 13:49.700
OK. But let's follow the chain reaction here.

13:49.700 --> 13:53.820
Let's say an I.T. director realizes their SharePoint is an absolute nightmare.

13:53.820 --> 13:55.620
They do the responsible thing, right?

13:55.620 --> 13:59.020
They lock co-pilot down until they can audit all those permissions.

13:59.020 --> 14:00.020
Which could take months.

14:00.020 --> 14:01.020
Exactly.

14:01.020 --> 14:05.120
But the staff still have jobs to do and they know AI can help them do those jobs twice

14:05.120 --> 14:06.240
as fast.

14:06.240 --> 14:11.100
So if the official internal tool is locked behind a massive I.T. audit, staff aren't

14:11.100 --> 14:12.540
going to just sit on their hands.

14:12.540 --> 14:13.540
Definitely not.

14:13.540 --> 14:19.740
They're going to open a new browser tab and use a generic external tool like ChatGPT.

14:19.740 --> 14:23.480
And that introduces the massive challenge of shadow AI.

14:23.480 --> 14:26.940
The source material highlights this as a very recurring pattern.

14:26.940 --> 14:28.380
Shadow AI.

14:28.380 --> 14:29.780
Meaning unofficial use.

14:29.780 --> 14:30.780
Right.

14:30.780 --> 14:33.700
When official tools are too slow or they're too locked down or they just simply don't

14:33.700 --> 14:37.560
exist yet, people will find a workaround to be productive.

14:37.560 --> 14:40.540
And the thing is, the intent is almost always good.

14:40.540 --> 14:42.900
They are literally just trying to be efficient.

14:42.900 --> 14:43.900
Right.

14:43.900 --> 14:46.940
Nobody's maliciously trying to leak company secrets on purpose.

14:46.940 --> 14:51.460
They just want to write a member newsletter without staring at a blank page for two hours.

14:51.460 --> 14:52.460
Exactly.

14:52.460 --> 14:57.340
But from an organizational standpoint, this shadow AI use creates a cascading series of

14:57.340 --> 14:58.340
liabilities.

14:58.340 --> 15:02.740
You know, you have staff feeding proprietary membership strategies or intellectual property

15:02.740 --> 15:04.140
directly into public models.

15:04.140 --> 15:05.260
Oh, yeah.

15:05.260 --> 15:06.260
Which is bad.

15:06.260 --> 15:07.260
Very bad.

15:07.260 --> 15:11.620
And you have potential privacy breaches if member data is included in those prompts.

15:11.620 --> 15:16.300
But perhaps most critically for an association, you completely lose your single source of

15:16.300 --> 15:17.300
truth.

15:17.300 --> 15:18.300
Okay.

15:18.300 --> 15:20.140
I want to drill into that source of truth problem for a second.

15:20.140 --> 15:24.660
Because if I have, say, three different staff members in the membership department and they

15:24.660 --> 15:30.220
all ask a generic AI model, a complicated question about a lap stem or grace period.

15:30.220 --> 15:31.220
Right.

15:31.220 --> 15:32.220
A specific policy question.

15:32.220 --> 15:33.220
Yeah.

15:33.220 --> 15:37.580
They might get three slightly different, completely unapproved answers just based on how they word

15:37.580 --> 15:38.580
at the prompt.

15:38.580 --> 15:39.580
Right.

15:39.580 --> 15:40.580
Exactly.

15:40.580 --> 15:44.700
And the staff members, you know, copies that hallucinated answer and emails it to a dues

15:44.700 --> 15:45.700
paying member.

15:45.700 --> 15:46.700
Oh, boy.

15:46.700 --> 15:47.700
Yeah.

15:47.700 --> 15:51.380
The association is now legally and reputationally on the hook for a policy that doesn't actually

15:51.380 --> 15:53.180
exist.

15:53.180 --> 15:58.140
You've essentially outsourced your institutional voice to an algorithm that you don't control.

15:58.140 --> 16:02.020
So I can already hear an objection forming from some of our more technically inclined

16:02.020 --> 16:03.180
listeners right now.

16:03.180 --> 16:04.180
Let's hear it.

16:04.180 --> 16:07.580
They might say, well, we know the public chat GPT model is too broad.

16:07.580 --> 16:09.620
So we paid for the enterprise tier.

16:09.620 --> 16:16.340
We created a custom GPT and we literally uploaded the PDFs of our 2024 bylaws and our

16:16.340 --> 16:18.140
certification requirements.

16:18.140 --> 16:19.940
The walled garden approach.

16:19.940 --> 16:20.940
Exactly.

16:20.940 --> 16:22.420
We built a walled garden.

16:22.420 --> 16:24.020
Doesn't that solve the problem?

16:24.020 --> 16:25.020
Doesn't that make it governed?

16:25.020 --> 16:28.300
It is the most common pivot we see in the implementations.

16:28.300 --> 16:29.400
It really is.

16:29.400 --> 16:32.660
But the documentation is incredibly clear on this.

16:32.660 --> 16:38.000
Building a custom GPT or a quick prototype is a phenomenal way for a team to learn.

16:38.000 --> 16:39.000
To learn how it works.

16:39.000 --> 16:40.000
Right.

16:40.000 --> 16:45.360
It teaches staff how models interact with documents and how prompting works in general.

16:45.360 --> 16:49.060
But a custom GPT is fundamentally an illusion of governance.

16:49.060 --> 16:53.160
It is simply not sufficient for a governed member facing official use case.

16:53.160 --> 16:55.000
See, I'm going to push back on that a little bit.

16:55.000 --> 16:57.400
If I explicitly upload the PDF, right?

16:57.400 --> 17:02.100
And I literally tell the system only use this PDF, why isn't it governed?

17:02.100 --> 17:03.960
Because of the underlying architecture.

17:03.960 --> 17:08.880
When you upload a PDF to a custom GPT, the system essentially reads it as flat text.

17:08.900 --> 17:14.060
It lacks the metadata and really the architectural rigor required to manage complex access.

17:14.060 --> 17:15.060
Okay, I'm following.

17:15.060 --> 17:16.060
Give me an example.

17:16.060 --> 17:17.060
Let's use your bylaws example.

17:17.060 --> 17:22.520
Suppose a board member and a student member both ask that custom GPT a question.

17:22.520 --> 17:26.240
The custom GPT does not know the difference between those two users.

17:26.240 --> 17:28.160
It cannot enforce audience tiers.

17:28.160 --> 17:29.160
Ah, okay.

17:29.160 --> 17:31.780
So it can't say, oh, you only have student level clearance.

17:31.780 --> 17:35.960
So I'm going to redact this specific financial clause from my answer.

17:35.960 --> 17:36.960
Exactly.

17:36.960 --> 17:38.140
It just gives everything to everyone.

17:38.140 --> 17:44.080
And furthermore, a custom GPT rarely has built in auditable source traceability down

17:44.080 --> 17:47.140
to the specific paragraph level.

17:47.140 --> 17:52.320
Nor does it have an interface for human subject matter experts to intercept and correct weak

17:52.320 --> 17:53.880
answers over time.

17:53.880 --> 17:55.120
So no review loops.

17:55.120 --> 17:56.120
Right.

17:56.120 --> 18:00.940
Unless you are intentionally designing and building robust source scoping, access controls,

18:00.940 --> 18:05.480
traceability layers, review loops, and ownership protocols, which is a ton of work.

18:05.480 --> 18:06.480
Yeah.

18:06.480 --> 18:09.260
GPT is really just a slightly more specialized shadow tool.

18:09.260 --> 18:10.260
Wow.

18:10.260 --> 18:12.740
It's kind of like building a really impressive tree house in your backyard.

18:12.740 --> 18:13.740
A tree house.

18:13.740 --> 18:14.740
Yeah.

18:14.740 --> 18:15.740
Like it was super fun to build.

18:15.740 --> 18:18.760
It works perfectly for a weekend hangout, but you would never move your association's

18:18.760 --> 18:20.500
actual corporate headquarters into it.

18:20.500 --> 18:21.740
I mean, it doesn't have plumbing.

18:21.740 --> 18:23.960
It doesn't meet commercial fire codes.

18:23.960 --> 18:28.700
And the structural integrity basically relies on whoever happened to nail the boards together.

18:28.700 --> 18:30.380
That is a brilliant way to look at it.

18:30.380 --> 18:35.540
The infrastructure of governance is just entirely missing in the tree house, which really brings

18:35.540 --> 18:36.960
us to the alternative.

18:36.960 --> 18:37.960
Right.

18:37.960 --> 18:43.120
If Copilot is a mirror reflecting our internal messes and custom GBTs are just tree houses

18:43.120 --> 18:47.920
lacking infrastructure, what does the actual corporate headquarters look like?

18:47.920 --> 18:53.360
Like what does a tool built specifically for official answers actually entail?

18:53.360 --> 18:56.360
So now we need to define governed association AI.

18:56.360 --> 18:59.320
What are the actual mechanics that separated from generic models?

18:59.320 --> 19:00.320
Yeah.

19:00.320 --> 19:01.320
Break that down for us.

19:01.320 --> 19:05.320
A governed AI layer is architecturally different from the ground up.

19:05.340 --> 19:10.980
Instead of relying on a broad internet scrape or even flat file uploads, it relies exclusively

19:10.980 --> 19:13.500
on a controlled database of approved sources.

19:13.500 --> 19:14.500
Only the good stuff.

19:14.500 --> 19:15.500
Right.

19:15.500 --> 19:19.860
And it vectorizes that data, which basically means it maps the relationships between concepts.

19:19.860 --> 19:24.060
But critically, it tags every single piece of data with metadata.

19:24.060 --> 19:25.060
Okay.

19:25.060 --> 19:26.260
So it's not just reading the text.

19:26.260 --> 19:28.660
It's reading the rules attached to the text.

19:28.660 --> 19:29.660
Exactly.

19:29.660 --> 19:33.060
And the sources point to a specific implementation pattern for this.

19:33.060 --> 19:38.880
They often refer to a system named Betty, but, and this is important, it is vital to

19:38.880 --> 19:41.400
frame tools like Betty correctly.

19:41.400 --> 19:42.400
Meaning what?

19:42.400 --> 19:46.760
Meaning governed AI is not a magic bullet that magically cleans all your messy data

19:46.760 --> 19:47.760
for you.

19:47.760 --> 19:48.760
Oh.

19:48.760 --> 19:51.080
And it absolutely does not replace your core infrastructure.

19:51.080 --> 19:52.360
Oh, that's huge.

19:52.360 --> 19:53.360
Yeah.

19:53.360 --> 19:57.040
It does not replace your association management system, your AMS, where all your transaction

19:57.040 --> 19:58.040
data lives.

19:58.040 --> 20:01.640
It does not replace your learning management system, your LMS.

20:01.640 --> 20:04.300
It certainly doesn't replace your actual human staff.

20:04.300 --> 20:05.300
Okay.

20:05.300 --> 20:09.100
So if it isn't replacing the AMS or the LMS, where does it actually sit?

20:09.100 --> 20:12.920
It sits as a governed knowledge layer on top of your approved content.

20:12.920 --> 20:15.920
So mechanically, it features strict audience tiers.

20:15.920 --> 20:20.140
It cross-references the user asking the question with the permissions attached to the data.

20:20.140 --> 20:21.140
Okay.

20:21.140 --> 20:22.140
So it knows who is asking.

20:22.140 --> 20:23.140
Yes.

20:23.140 --> 20:25.540
Ensuring the board member gets the full financial breakdown and the student member just gets

20:25.540 --> 20:26.980
the public summary.

20:26.980 --> 20:29.540
And it provides complete unbreakable source traceability.

20:29.540 --> 20:30.540
Right.

20:30.540 --> 20:31.540
The show your work test.

20:31.540 --> 20:32.540
Exactly.

20:32.540 --> 20:37.440
Every single claim it generates is directly linked back to the approved association document.

20:37.440 --> 20:43.360
It has to literally say, I am making this claim and here is the exact paragraph from

20:43.360 --> 20:46.480
the 2024 policy handbook to prove it.

20:46.480 --> 20:47.480
Yes.

20:47.480 --> 20:51.200
And if it somehow gets it wrong, governed AI has coaching loops.

20:51.200 --> 20:55.880
So a staff member can review the logs, see a weak answer, and correct the system's logic

20:55.880 --> 20:57.800
so it never makes that mistake again.

20:57.800 --> 20:59.880
Which is so different from a generic tool.

20:59.880 --> 21:00.880
Completely.

21:00.880 --> 21:06.340
And cannot just call up OpenAI and demand they fix how chat GPT interprets your specific

21:06.340 --> 21:08.860
continuing education requirements.

21:08.860 --> 21:13.540
But with governed AI, your staff actually owns the logic.

21:13.540 --> 21:16.360
You know, I want to take a breath here for a second and just deliver what I think is

21:16.360 --> 21:19.020
a really necessary honest beat for you listening.

21:19.020 --> 21:20.020
Yeah, let's do that.

21:20.020 --> 21:24.100
Because when we start talking about, you know, vector databases and metadata and governance

21:24.100 --> 21:27.260
architectures, it is very easy to feel overwhelmed.

21:27.260 --> 21:28.260
Very easy.

21:28.260 --> 21:30.760
It can start to sound like a lot of vendor hype, frankly, where you feel pressured to

21:30.760 --> 21:35.320
buy this massive enterprise system just to like draft a meeting agenda.

21:35.320 --> 21:36.580
And I completely agree.

21:36.580 --> 21:40.760
We need to ground this in the everyday reality of running an organization.

21:40.760 --> 21:46.440
Because the absolute truth is for a vast majority of your daily friction points, chat GPT or

21:46.440 --> 21:48.440
popilot is perfectly enough.

21:48.440 --> 21:49.440
Right.

21:49.440 --> 21:55.040
If my task is just writing a really sloppy first draft of a marketing email, or say taking

21:55.040 --> 21:59.720
a messy transcript of an hour long internal brainstorming session and summarizing the

21:59.720 --> 22:02.960
key takeaway so I don't have to read the whole thing.

22:02.960 --> 22:05.280
Generic AI is a literal miracle.

22:05.280 --> 22:06.280
It really is.

22:06.280 --> 22:08.480
I don't need governance for a marketing brainstorm.

22:08.480 --> 22:10.720
I just need speed and creativity.

22:10.720 --> 22:11.720
Exactly.

22:11.720 --> 22:16.540
Those tools excel at low risk creative or internal productivity tasks.

22:16.540 --> 22:18.280
The pivot really only happens.

22:18.280 --> 22:22.820
I mean, the transition to governed AI only becomes necessary when the association needs

22:22.820 --> 22:27.940
to provide an official answer that the organization must legally or reputationally stand behind.

22:27.940 --> 22:28.940
That's the line.

22:29.180 --> 22:31.700
If someone asks for a summary of a podcast, use chat GPT.

22:31.700 --> 22:36.260
But if a member asks about a compliance standard that could affect their professional license,

22:36.260 --> 22:38.460
you simply cannot afford a hallucination.

22:38.460 --> 22:43.780
So the million dollar question for a leader is how to actually draw that line operationally.

22:43.780 --> 22:49.380
Like when a team brings a new AI project or a new workflow to the table, how do we decide

22:49.380 --> 22:51.140
which tool they should be using?

22:51.140 --> 22:55.300
Well the source material provides this brilliant straightforward decision test.

22:55.300 --> 23:01.460
Rather than arguing about which AI is smarter, you just evaluate the specific task using

23:01.460 --> 23:02.960
a set of criteria.

23:02.960 --> 23:04.980
We should actually apply it to a real scenario.

23:04.980 --> 23:05.980
Oh I love that.

23:05.980 --> 23:06.980
Let's do it.

23:06.980 --> 23:08.180
Let's use a complicated member question.

23:08.180 --> 23:13.700
Say a long time member emails in and asks, hey does the online ethics course I took back

23:13.700 --> 23:18.620
in 2022 still count toward the new 2026 recertification requirements?

23:18.620 --> 23:19.900
Okay perfect test case.

23:19.900 --> 23:23.540
So the first question in the decision framework is who is asking?

23:23.540 --> 23:27.460
Is it a staff member looking for general information or is it a dues paying member,

23:27.460 --> 23:28.460
a partner or the public?

23:28.460 --> 23:32.980
Well in this case it's a member asking a question directly tied to their professional credentials.

23:32.980 --> 23:35.460
So the risk level is immediately very high.

23:35.460 --> 23:36.460
Exactly.

23:36.460 --> 23:38.380
Second question, what sources are allowed?

23:38.380 --> 23:43.780
Should the AI be allowed to search the broader internet to sort of guess the answer or must

23:43.780 --> 23:48.060
it be restricted strictly to your official 2026 recertification handbook?

23:48.060 --> 23:52.600
Oh it absolutely cannot guess based on what other associations are doing.

23:52.600 --> 23:55.720
It has to be restricted to our handbook period.

23:55.720 --> 23:56.720
Right.

23:56.720 --> 24:00.160
Third question, who is allowed to see the answer?

24:00.160 --> 24:02.640
This touches on those audience tiers we talked about.

24:02.640 --> 24:03.640
Right.

24:03.640 --> 24:07.520
And the answer to this specific member needs to be based on their specific member history

24:07.520 --> 24:09.920
and not just generalized public data.

24:09.920 --> 24:10.920
Exactly.

24:10.920 --> 24:14.640
Fourth, can the answer trace back to approved sources?

24:14.640 --> 24:16.200
The show your work test again?

24:16.200 --> 24:20.960
If the AI says yes your 2022 course counts it better provide a link to the grandfather

24:20.960 --> 24:23.280
clause and out policy that actually proves it.

24:23.280 --> 24:24.280
Correct.

24:24.280 --> 24:26.440
Fifth question, who corrects the system when it's weak?

24:26.440 --> 24:30.880
If the AI misinterprets that grandfather clause and tells the member their course doesn't

24:30.880 --> 24:32.440
count, who catches that?

24:32.440 --> 24:33.440
Ah.

24:33.440 --> 24:34.440
Right.

24:34.440 --> 24:39.360
In a governed system your credentialing staff actually reviews the logs, spots the error,

24:39.360 --> 24:43.080
and adjusts the semantic weight of the rules so the AI learns the exact nuance.

24:43.080 --> 24:45.920
Which you simply cannot do in a custom GPT.

24:45.920 --> 24:51.040
Like you can tweak the prompt, sure, but you can't surgically fix the logic architecture

24:51.040 --> 24:52.040
behind it.

24:52.040 --> 24:53.040
Exactly.

24:53.040 --> 24:57.760
And finally the sixth question, which is perhaps the most strategic of all, what signal or

24:57.760 --> 25:00.880
insight comes back to the association?

25:00.880 --> 25:02.760
Oh, meaning the data.

25:02.760 --> 25:03.760
Yes.

25:03.760 --> 25:08.760
Are these complex member queries just disappearing into the black box of a public chatbot?

25:08.760 --> 25:12.680
Or are you capturing the exact wording of what your members are actually confused about?

25:12.680 --> 25:13.680
Wow.

25:13.680 --> 25:15.680
That framework is incredibly clarifying.

25:15.680 --> 25:18.080
Who is asking what sources are allowed?

25:18.080 --> 25:19.080
Who sees it?

25:19.080 --> 25:20.080
Can it be traced?

25:20.080 --> 25:21.080
Who corrects it?

25:21.080 --> 25:22.440
And what signal comes back?

25:22.440 --> 25:23.600
It's extremely practical.

25:23.600 --> 25:27.960
It really strips away all the philosophical debates about AI and makes it a pure operational

25:27.960 --> 25:29.000
risk assessment.

25:29.000 --> 25:31.160
It forces you to match the tool to the risk.

25:31.160 --> 25:36.760
It allows you to confidently say yes to chat GPT for drafting an agenda and no to chat GPT

25:36.760 --> 25:38.600
for interpreting your bylaws.

25:38.600 --> 25:42.360
So to pull all these threads together, the core thesis of our deep dive today is really

25:42.360 --> 25:43.480
this.

25:43.480 --> 25:46.400
Please use the lightest tool that fits the risk of the task.

25:46.400 --> 25:52.880
Do not over complicate your internal productivity, but reserve governed AI strictly for the official

25:52.880 --> 25:57.600
answers your organization must stand behind in front of your members and the public.

25:57.600 --> 26:02.600
It is all about deploying these tools with intentionality rather than just using whatever

26:02.600 --> 26:04.960
happens to be open in your browser tab at the moment.

26:04.960 --> 26:10.520
And to help you execute on that exact intentionality, please go to the show notes right now and

26:10.520 --> 26:16.680
grab the chat GPT, co-pilot or governed AI use case decision guide.

26:16.680 --> 26:21.120
We mentioned it earlier, and it really is the definitive next step for mapping out these

26:21.120 --> 26:23.280
boundaries in your own organization.

26:23.280 --> 26:24.280
It's going to be so helpful.

26:24.280 --> 26:29.320
It will help your team separate the low risk tasks generic AI can handle from the high

26:29.320 --> 26:33.280
stakes knowledge that your operations team still firmly needs to govern.

26:33.280 --> 26:36.520
Honestly, it will save you a tremendous amount of internal debate.

26:36.520 --> 26:42.240
And as we wrap up our analysis of these implementations today, there is one final provocative insight

26:42.240 --> 26:46.040
from the sources that I really think every leader needs to sit with.

26:46.040 --> 26:50.640
We talked earlier about co-pilot being a mirror for your internal permissions, right?

26:50.640 --> 26:51.640
Yeah.

26:51.640 --> 26:53.360
Reflecting back the messiness of your internal legacy folders.

26:53.360 --> 26:54.360
Right.

26:54.360 --> 26:57.200
The 2.0 AM fluorescent lights in the dive bar.

26:57.200 --> 26:59.240
Yes, exactly.

26:59.240 --> 27:03.160
But think about the insights that governed AI brings back to you when it is facing your

27:03.160 --> 27:04.160
members.

27:04.160 --> 27:08.680
If your governed AI is working perfectly, strictly bounded by your official sources,

27:08.680 --> 27:13.960
and it is meticulously tracking exactly what your members are asking every single day.

27:13.960 --> 27:14.960
Yeah.

27:14.960 --> 27:16.080
Are you prepared for the day?

27:16.080 --> 27:20.280
It mirrors back the gaps in your own institutional knowledge.

27:20.280 --> 27:21.640
Oh, wow.

27:21.640 --> 27:25.120
Because if the AI can only answer based on what we've officially published.

27:25.120 --> 27:26.120
Exactly.

27:26.120 --> 27:28.880
What happens when you finally have the data showing all the crucial pressing questions

27:28.880 --> 27:33.800
your members have, and you realize your association has never actually written the answers down?

27:33.800 --> 27:34.800
It's terrifying.

27:34.800 --> 27:36.320
Governed AI doesn't just answer questions.

27:36.320 --> 27:39.400
It reveals the questions you didn't even know were missing.

27:39.400 --> 27:41.720
And solving that isn't a technology problem anymore.

27:41.720 --> 27:45.640
It is a fundamental content strategy problem staring you right in the face.

27:45.640 --> 27:47.520
That is a phenomenal place to leave it.

27:47.520 --> 27:51.160
You came here to figure out which software to use, and you're leaving with a mandate

27:51.160 --> 27:52.920
to fix your content.

27:52.920 --> 27:53.920
That's the real deep dive.

27:53.920 --> 27:58.920
Get the guide in the show notes, map your use cases, and we'll catch you on the next

27:58.920 --> 27:59.440
one.

28:04.800 --> 28:06.760
Let's recap today's big question.

28:06.760 --> 28:11.240
Why not just use chat TPT or copilot?

28:11.240 --> 28:18.720
And I can't stop laughing, Thomas, because again, that line that comes up about the fluorescent

28:18.720 --> 28:27.880
lights and a dive bar at 2 a.m. as a metaphor for understanding that there is genuine, there

28:27.880 --> 28:32.680
can be genuine risk with a mess that you don't even know is there until the lights are kind

28:32.680 --> 28:33.680
of shining.

28:33.680 --> 28:39.720
That is something that we hear and something that we talk to reasonably regularly, right?

28:39.720 --> 28:43.800
Because people, there is a perception that if you've been a Microsoft shop, particularly

28:43.800 --> 28:50.000
for a long time, and you simply turn copilot on, that that is somehow safer.

28:50.000 --> 28:54.000
And yet in our experience, there is still risk associated with that.

28:54.000 --> 28:58.560
So this was really like this episode was particularly interesting to me because I learned something,

28:58.560 --> 28:59.080
right?

28:59.080 --> 29:01.320
This is actually a problem.

29:01.320 --> 29:03.360
There's a secondary market right now.

29:03.360 --> 29:06.760
I did some research kind of post doing this.

29:06.760 --> 29:13.640
There's a secondary market of people who will sell you a service for pretty for pretty penny

29:13.640 --> 29:17.480
to go in and make sure your permissions are okay for adopting copilot.

29:17.480 --> 29:18.560
Well, right.

29:18.560 --> 29:22.400
It is such a big problem that there's now this kind of other layer of kind of consultants

29:22.400 --> 29:24.120
that will fix the problem.

29:24.120 --> 29:24.720
Right.

29:24.720 --> 29:28.120
And I didn't know this was a problem until I started reading about it.

29:28.320 --> 29:32.600
I hope this sort of like helps people think about it because it does like it feels like

29:32.600 --> 29:33.880
Microsoft's a giant name.

29:33.880 --> 29:40.440
They've been around since, you know, what the late 70s, early 80s, they've got it locked down.

29:40.440 --> 29:45.400
But what it doesn't talk about is actually you're inheriting a lot of assumptions, right?

29:45.400 --> 29:51.400
Not necessarily from Microsoft, but maybe even before you kind of joined the association, right?

29:51.400 --> 29:54.080
You've probably been on the Microsoft framework for a really, really long time.

29:54.080 --> 29:54.640
And that's good.

29:54.640 --> 29:58.480
I'm not trying to say anything bad about Microsoft and copilot does have its place.

29:58.480 --> 30:02.840
But if you're not aware of these problems, you don't know what might come out of it.

30:02.840 --> 30:04.480
And that was really eye opening for me.

30:04.480 --> 30:09.640
It really is like you flip on the fluorescent lights and the cockroach is scurrying away.

30:09.640 --> 30:12.080
It's like, oh my God, I was sitting in this this whole time, right?

30:12.080 --> 30:13.840
Like you just don't know what's there.

30:13.840 --> 30:14.320
That's right.

30:14.320 --> 30:19.160
And we've talked about this in terms of like, does Microsoft really care about the business models

30:19.160 --> 30:21.000
and the governance rules of an association?

30:21.000 --> 30:26.000
No, they've turned an application on inside their own applications and just said, hey,

30:26.000 --> 30:26.800
this is pretty cool.

30:26.800 --> 30:27.680
Go ahead and use it.

30:27.680 --> 30:28.200
And you're right.

30:28.200 --> 30:34.560
That secondary market now of consultants offering 30 to 50 thousand dollar engagements just to

30:34.560 --> 30:39.120
come in, teach your copilot to use your content.

30:40.120 --> 30:43.600
All of a sudden, you know, and again, this is the platform player, right?

30:43.600 --> 30:45.560
It's platform and services kind of costs.

30:46.000 --> 30:51.640
That kind of time and money can be spent, you know, essentially working with a governed AI

30:51.640 --> 30:56.040
system that's designed specifically for associations, pardon the plug.

30:56.280 --> 31:02.960
And I think the risk there is really is a really important one as well that's come up through all

31:02.960 --> 31:11.920
of this, that the governance around an AI is not just about governing the responses that it gives

31:11.920 --> 31:12.680
to the queries.

31:12.880 --> 31:17.800
You know, it's it's understanding that that there is a mandate, that there is a purpose that the

31:17.800 --> 31:19.240
AI needs to abide by.

31:19.240 --> 31:24.560
It's putting that kind of there's the intelligence layer, but it is putting that insights layer

31:24.560 --> 31:31.280
around it so that it is driving the purpose of the organization forward as well, but doing it in a

31:31.280 --> 31:32.720
really sort of critical way.

31:33.800 --> 31:39.800
It's it's it's a shame to constantly be talking about this almost from a risk profile, because

31:39.800 --> 31:44.920
again, when we opened the podcast, it was a little bit about it's not just about should we build

31:44.920 --> 31:54.520
this ourselves and should we use a chat TBT or a co pilot or a Claude agent to do that?

31:54.920 --> 31:59.320
It's more about form follows function.

31:59.680 --> 32:01.480
You know, what's the for that?

32:01.480 --> 32:03.600
What's the purpose of this tool?

32:04.000 --> 32:07.960
Does it have an internal productivity, you know, primary angle to it?

32:07.960 --> 32:11.640
Does it have across a whole ecosystem?

32:11.640 --> 32:17.680
And what happens when you have multiple ecosystems of content or knowledge interacting that's never

32:17.680 --> 32:22.520
interacted before and is being used by a co pilot that's not trained to do the job of your

32:22.520 --> 32:23.200
association?

32:23.560 --> 32:30.280
What happens if so again, a lot of experimentation, probably they can coexist.

32:31.880 --> 32:34.560
But there's also some risks that we still need to be mindful of.

32:35.120 --> 32:37.240
Yeah, and I think it's just awareness of it, right?

32:37.400 --> 32:38.600
These are good tools, right?

32:38.600 --> 32:44.320
We use co pilot, we use Claude code, we use, you know, Chad GPT, and we use, you know, our own

32:44.320 --> 32:45.480
Betty pollution, right?

32:45.680 --> 32:46.920
And we use them for different reasons.

32:47.000 --> 32:53.680
And it's more understanding what has what risks, what's what function does the tool serve for?

32:53.720 --> 32:53.920
Right?

32:53.920 --> 32:57.280
Like you're not using a solve where you should have been using a hammer, right?

32:57.280 --> 32:58.000
It'll never work.

32:58.400 --> 32:58.600
Right.

32:58.600 --> 33:01.760
So it's understanding sort of those applications and how you can manage them over.

33:01.760 --> 33:05.000
And I think that's why the companion asset today is really important.

33:05.000 --> 33:10.360
If you can kind of understand where each plays a role, and you can actually understand, kind of

33:10.360 --> 33:14.920
take a step back, understand your ecosystem of different AI tools and what they're being used for

33:14.920 --> 33:16.760
and how to coordinate amongst them.

33:16.760 --> 33:20.680
I think that's a that's a really important part of this process is just a deeper understanding.

33:20.760 --> 33:26.200
So not again, not co pilots bad co pilot has risks, but it's also got benefits, like the eyes with Chad

33:26.200 --> 33:26.720
GPT.

33:27.240 --> 33:28.920
And dare I say even with Betty, right?

33:28.920 --> 33:33.200
There's there's a purpose for Betty that won't apply in the case that a co pilot would apply.

33:33.560 --> 33:38.880
There was a couple of great challenges to the listeners coming from the podcast today is like if your IT

33:38.880 --> 33:43.960
director all of a sudden decided to restrict co pilot tomorrow, but you were used to having this

33:43.960 --> 33:48.120
productivity tool or whatever, it's like, where's everybody all of a sudden go?

33:48.160 --> 33:53.520
You know, does it create an underground internal, you know, everyone's going and using their own GPT or

33:53.520 --> 33:56.760
whatever it is that they've built, because co pilots been restricted.

33:57.680 --> 34:02.960
And I think that one of the things I like to talk to executives about when they're thinking

34:02.960 --> 34:10.440
about this question, it's we've turned on some of these tools without a hypothesis about the impact.

34:10.840 --> 34:16.200
We haven't kind of said we believe that if we make this available to our staff to our volunteers to our

34:16.200 --> 34:21.600
members, hypothesize we believe this is what the impact is going to be.

34:23.000 --> 34:24.600
And there's plenty of data points there.

34:24.640 --> 34:30.840
But some reason with AI experiments are not being approached with the rigor that I think that they

34:30.840 --> 34:34.920
should or could. And in fact, you can get the AI to help you with the rigor of the experiment in the

34:34.920 --> 34:40.760
first place. But what happens like set yourself a 90 day plan and say we believe that 30 days, 60 days

34:40.760 --> 34:46.160
and 90 days, we think by doing this, it's going to get us there. And then just reviewing what that looks

34:46.160 --> 34:51.120
like, understanding and look at some of the qualitative information that comes back from that,

34:51.480 --> 34:55.880
that's going to help people navigate this question far more rigorously.

34:56.520 --> 35:01.680
And then I think that when we get to the point of saying, you know, which is leading into the next

35:01.680 --> 35:06.200
episode, it's like, should we build the details ourselves? But I'd like to see people experimenting with a

35:06.200 --> 35:13.000
little bit more rigor around a hypothesis and the impact that using these tools and

35:13.000 --> 35:18.360
experimenting with these tools is going to have. And I'm glad that they came up in the challenges from today.

35:20.720 --> 35:25.080
Yeah, absolutely. I think that's really important takeaway is sort of understanding the tool on what it's used

35:25.080 --> 35:29.760
for and what the risk for each one is and what the benefits are. And once you understand that, I think

35:29.760 --> 35:36.480
you'll be in a much better position to move forward with AI in a really effective way that adds value to the

35:36.480 --> 35:38.400
association and minimizes risk.

35:39.720 --> 35:45.920
So one more time before you go today, download the Use Case Decision Guide from the show notes. That's the

35:45.920 --> 35:51.760
fastest way to actually put today's episode to work. Thanks for listening to the Association

35:51.760 --> 35:57.800
Intelligence Podcast. Join us on the next episode to answer this question. Should we build this ourselves?

What this helps your team answer

How to tell the difference between individual productivity tools, ecosystem-specific tools like Copilot, and governed AI — and which job each one is actually built for.
Why turning on Copilot can surface risky, forgotten permissions in your SharePoint/OneDrive — and why that's a data-hygiene issue, not a Copilot failure.
The six-question decision framework for any AI use case: who's asking, what sources are allowed, who can see the answer, can it trace back to a source, who corrects it when it's wrong, and what signal comes back to the org.

Companion Asset

ChatGPT, Copilot or Governed AI: Use Case Decision Guide

Build vs. Buy

Should We Build This Ourselves?

An honest look at the hidden costs of building governed AI in-house — why prototypes are easy but operating a real member-facing system is a different game entirely, plus real association case studies on when to build, buy, or partner.

32:37

0:00 32:37

[Thomas Altman] (0:04 - 0:07)
Today's big question, should we build this ourselves?

[Rob Barnes] (0:08 - 1:22)
We are Thomas Altman and Rob Burns, and this is episode four of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the build versus buy scorecard. It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work.

Thomas, I think we have been talking about or working through this question, if you like, build versus buy or build buy partner, if you like, for forever, particularly on the association software side of the table. I remember even when I worked in my associations, it was a constant question that would come up strategically about, hey, we know our business the best, we should just build it ourselves. Do we have the technical expertise to do it?

And it comes up more and more. The rapidity of, or the pace of development in the generative AI era though, is creating a whole new paradigm when it comes to this question. One of the key things that associations should be aware of, in my view, tell me, should they be really thinking about the hidden costs of doing this themselves?

[Thomas Altman] (1:23 - 3:36)
Absolutely. And the costs are surprising. It's this hidden surface area of what it takes to get governed AI up and running.

And it is one of those things that seems on the surface really simple. You can find a million tutorials online on how to create a retrieval augmented generation system, a RAG based system. And people want to.

It's cool. It's the hot new tech and everybody wants to work on it. But one of the things we've seen is that there is a stark difference between kind of the prototype, right?

Like I'm going to load in, you know, 20 documents and sort of have it work and it'll work. It'll work great, right? Versus the reality.
And the reality is many associations, if you want something beyond a simple FAQ bot, right? If you want governed AI that interacts as the interface to your knowledge base and your knowledge base, many times it's very expensive, right? If it's the case that you have thousands, hundreds of thousands, we have several clients on Betty right now with millions of documents, right?

What happens? Like, how does that scale, right? How do you scale from 12 to a million?
And what are the pitfalls you're going to find along the way when 10,000 members are kind of coming and asking questions per month, right? What kind of load does that put on your system? Where kind of the hidden areas, right?
And that's something that I think people feel overconfident a lot of times that they can build because they haven't seen sort of the pitfalls, right? And you can't see them. You have to experience them one after another.

So being aware of what those pitfalls are are very important when you decide to kind of take on a project to build yourself, when you decide to partner with, you know, someone, or if you want to just sort of buy off-the-shelf solution, you want confidence that either you know what they are and you've got a plan for them, right? Or that the person you're partnering with has dealt with them before. And I think that's a really important key takeaway for this, you know, and where does that apply?

Because if at the end of the day, all you need is an FAQ bot, yeah, that's something you can build yourself. But if you want true governed AI, there's a lot more to that story than it seems at first.

[Rob Barnes] (3:37 - 6:26)
We've had some customers over the years now, some good examples. One, you know, looked at a platform, Azure, and what was possible in Azure, and then looked at the services fees they were going to be up for in order to have someone build on that platform, that kind of the bolt-on technology, and they ended up choosing to buy. You know, we had another that's got developers on staff, and I think this is a pretty common one, particularly with sort of, I guess, larger organizations that have got an IT team that even has developers, you know, software engineers on staff now.

They don't have as much to do because the traditional, like, I mean, unless they're infrastructure engineers, but when it comes to application development and things like that, they've got more time on their hands. And I know the organizations that I've spoken to with developers on staff are like, we need to give them something to do, so we're just going to let them build this themselves. But they get 80 percent of the way through the process and can't close the gap on the other 20, because it's really, really hard to do when you're thinking about it the way we thought about it in, you know, 2022 and 2023.

So they've come back to us as well. Others have built GPTs, you know, a custom GPT or a wrapped, you know, and then hit that wall at scale like you talked about, like how do you go from 12, you know, content items to a million content items? We're still hearing that from technical buyers, you know, we still end up talking to directors of IT because they have, you know, they still are the controllers of the keys.

One of the interesting things that I've noted was, and I look back on some of these conversations, whether they're customers that we've won and partner with now or customers that have said, no, we choose not to buy Betty, we're going to go and do our own thing. It's almost always because of one individual person who wants to be the champion, who wants to be the person that did this thing, as opposed to it being a strategic objective. It's one person going, no, no, I can build this myself because I know.

And the two risks that I tend to call out is, one is why are you asking your association to become a software development company? And if you do such a good job at doing this work for that association, someone's going to come, a bigger association is going to come and poach you and say, come and do that for us. And when you leave, what happens to the technology you built for your former employer?

Is there anybody there that can do what you did? Unlikely, it's going to go with you. And then that association is left hanging.
That's a risk that we've seen actually happen in practice. When it's tied into one person. And I don't think that's talked about enough when we talk about build versus buy.

Would that be a fair assessment?

[Thomas Altman] (6:26 - 6:55)
I think that's probably the critical asset. So there are people that are overconfident. That's true.

There are people that are totally capable, right? And the capable people might not be there. Do you know what I mean?

So it's like making sure that your investment is forward-looking into that. And then the ability to either backfill or to maintain, right? A lot of times the people who are really good at building at the first time are not the same people that enjoy maintenance, right?

The people that are kind of, they like the fresh hit of excitement.

[Rob Barnes] (6:55 - 6:56)
That's a good point.

[Thomas Altman] (6:57 - 7:20)
And then maintaining it and keeping it fresh. And there's this idea of context rot within a lot of these types of models, meaning the content that gets fed into this and making sure maybe it works today. Does it work a year from now?
Right? Is it still finding the right stuff a year from now? All of those things take a lot of work, right?

And I may be very passionate about building something from scratch. I might not be so good at keeping it running.

[Rob Barnes] (7:21 - 8:10)
Durability is the word that comes up a lot for us. We tend to use it. And I think associations appreciate it.

They enjoy certainty. They enjoy understanding that whatever is created, whether it's been built, purchased, partnered on, has a level of durability that certainly extends beyond the 10 years of the current board or staff. And durability of internal ownership is something that is not recognised enough when individuals start talking about, you know what, we can just go and build that.

So the number of associations that have come back to us six, nine, 12 months later and said, yeah, we went and did that and we couldn't make it last. So they come back to us is more common than I would like it to be, to be fair.

[Thomas Altman] (8:11 - 9:19)
Which I think can be important too. So one of the things this episode really taught me was that that process itself is not invaluable, right? So kind of going through it intentionally, kind of building something, not so that you have something at the end, but so that you understand what you want better, right?

It kind of crystallises and clarifies the needs that you actually have and the gaps that you're going to find, right? So I liked the nuance of this episode because it wasn't like always buy, right? And that's not what we're saying either.

It's say know where to build, like if it's a simple thing, yeah, build yourself. Or kind of intentionally build the prototype, not so that it gets launched necessarily, but so that you understand better the ask of the eventual partner, right? You'll be much more clear on what you want, what you don't want.

Everybody benefits from that scenario too. So I do encourage people to try and to try and build on their own, but do so in a targeted way. And I think this, both the episode and the companion asset do a really good job of pointing you to like how to do that well, like kind of getting in and learning by doing and then passing off what you don't want to maintain and carry going forward.

[Rob Barnes] (9:20 - 9:42)
Before we hand this over to the AI podcast hosts, quick reminder that the following is generated using notebook LM with heavy involvement from the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach to the association intelligence podcast, please go back and listen to episode zero. And now over to the AI hosts.

[AI Host 2] (9:47 - 9:55)
Um, there is this running joke in software development. Give a good engineer a weekend and a pot of coffee and they'll, you know, build you a miracle.

[AI Host 1] (9:56 - 9:57)
Yeah. Classic weekend miracle.

[AI Host 2] (9:57 - 10:07)
Right. They string together some code, they connect an API. And by Monday morning, they're showing off this working prototype that just feels like absolute magic.

[AI Host 1] (10:07 - 10:18)
It does. And it always looks flawless, right? Because it's in a controlled demo environment.
The questions are preselected. The dataset is small and well, the stakes are basically non-existent.

[AI Host 2] (10:18 - 10:26)
Exactly. But the second you try rolling that magic out to, I don't know, 50,000 association members across all these different membership tiers.

[AI Host 1] (10:26 - 10:27)
The magic crashes.

[AI Host 2] (10:27 - 10:39)
It usually crashes hard. So welcome to our deep dive today. We are answering the single most pressing question, keeping association executives and senior staff awake right now.

[AI Host 1] (10:39 - 10:41)
Should we build this AI system ourselves?

[AI Host 2] (10:41 - 10:50)
That's the one. And for this deep dive, we're drawing on implementation patterns from nearly 200 associations that have actually navigated deploying governed AI.

[AI Host 1] (10:50 - 10:53)
It's a massive sample size of real world operators.

[AI Host 2] (10:53 - 11:07)
It is. And our mission for you today is to look right past the vendor help. We really want to help you decipher the difference, like the actual structural difference between building a chatbot demo and running an operated knowledge system.

[AI Host 1] (11:07 - 11:10)
Which is frankly a massive jumping complexity.

[AI Host 2] (11:10 - 11:18)
Before we get too deep into the mechanics of all that though, I want to drop a highly practical tool for you right at the top. It's called the build versus buy scorecard.

[AI Host 1] (11:19 - 11:20)
Such a great resource.

[AI Host 2] (11:20 - 11:29)
Yeah, we'll link it in the show notes and I highly recommend you grab it. It is the perfect companion resource for the internal team conversations you are inevitably going to have after listening to this.

[AI Host 1] (11:29 - 11:39)
Because the stakes feel entirely new right now. The underlying tools, things like chat GPT, Microsoft Copilot, Azure, they're suddenly highly accessible.

[AI Host 2] (11:39 - 11:39)
So accessible.

[AI Host 1] (11:40 - 11:50)
Right. So it's entirely valid that leaders are asking this build versus buy question. Because of this accessibility, leaders are rightly worried about, you know, redundant spending.

[AI Host 2] (11:50 - 11:52)
Draining their internal staff capacity.

[AI Host 1] (11:52 - 11:58)
Exactly. Risking member trust, establishing proper governance, and just the sheer weight of long-term maintenance.
[AI Host 2] (11:59 - 12:12)
Okay, let's unpack this core dilemma then. Are we about to pay a vendor for something our internal IT team could just whip up over the weekend? I mean, is there a massive hidden cost we just aren't seeing?

[AI Host 1] (12:12 - 12:32)
There is a significant hidden operating cost. Let's look at the foundational mistake association leaders make when evaluating this. They tend to treat build versus buy purely as a standard software procurement question.
Like, they look at a vendor subscription cost and compare it to a presumed zero internal cost.

[AI Host 2] (12:32 - 12:43)
Ah, the myth of the free internal project. Right. The logic usually goes, well, we already pay the salaries of our IT staff, therefore assigning them a new AI project doesn't cost the organization any extra money.

[AI Host 1] (12:43 - 12:47)
And that assumption naturally leads to mistaking a prototype for an operated product.

[AI Host 2] (12:47 - 12:57)
It's like building a concept car versus running a municipal transit system. I mean, one looks great on a test track, but the other has to run safely every single day for thousands of commuters.

[AI Host 1] (12:57 - 13:08)
That is a perfect analogy. Because it is undeniably easy today to make a chatbot answer questions from a small handful of documents. A junior developer can do that in an afternoon.

[AI Host 2] (13:08 - 13:19)
So if the model mechanics, like the actual AI part of generating text, if that's largely solved by companies like OpenAI, what exactly is the association implementation part?

[AI Host 1] (13:19 - 13:22)
Well, they hit what we call the hidden operating surface.

[AI Host 2] (13:22 - 13:23)
The hidden operating surface.

[AI Host 1] (13:23 - 13:32)
Think of the prototype as just the tip of the iceberg. The hidden surface is everything that has to happen after the model can technically retrieve those documents.

[AI Host 2] (13:32 - 13:34)
OK, let's list them out. What's below the water?

[AI Host 1] (13:35 - 13:44)
Let's say your team builds the prototype. Now, you have to manage content ingestion. And I don't mean uploading a PDF once.

[AI Host 2] (13:44 - 13:51)
Wait, why not? Can't I just have my team upload a massive zip file of all our PDFs and call it a day? Why is dynamic ingestion actually hard?

[AI Host 1] (13:51 - 14:18)
Because association knowledge isn't static. It breathes, you know. When you feed documents into an AI system, you aren't just dropping files into a folder.

You are breaking those documents down into mathematical chunks and storing them in a specialized database. So if your board of directors updates a vital policy... The system has to know.

Exactly. Your internal system has to automatically find the chunks of the old policy, delete them, and index the new ones instantly.

[AI Host 2] (14:18 - 14:26)
Oh, wow. So if that pipeline breaks, your AI starts giving members answers based on obsolete, maybe even harmful bylaws.

[AI Host 1] (14:26 - 14:30)
Yes. Building and maintaining that dynamic sync is a heavy engineering lift.

[AI Host 2] (14:30 - 14:31)
Yay.
[AI Host 1] (14:31 - 14:34)
And it doesn't stop at ingestion. You also have to build permission boundaries.

[AI Host 2] (14:34 - 14:39)
Meaning, does the AI know not to show a board-only financial document to a student member?

[AI Host 1] (14:39 - 14:46)
Right. You need source traceability. You need a system for SME subject matter expert coaching.
You need answer review processes.

[AI Host 2] (14:46 - 14:47)
Deep analytics.

[AI Host 1] (14:47 - 14:56)
Deep analytics, hardened security posture, a seamless member UX, launch support, and, well, a framework for continuous improvement.

[AI Host 2] (14:56 - 15:06)
I want to push back on the enterprise tools for a second, though. If we're paying Microsoft for Azure, shouldn't those platforms already know how to handle indexing and security? Why is the burden falling on my internal IT guy?

[AI Host 1] (15:07 - 15:14)
Because raw cloud infrastructure gives you the building blocks, not the finished house. Azure provides incredible raw compute power, sure.

[AI Host 2] (15:15 - 15:16)
But it doesn't know our business.

[AI Host 1] (15:16 - 15:26)
Right. It doesn't know your association's membership tiers or governance rules. The burden falls on your IT team to write the custom code connecting Azure to your association management system.

[AI Host 2] (15:27 - 15:28)
Which means they own it.

[AI Host 1] (15:28 - 15:35)
They own it. Furthermore, the underlying technology is incredibly dynamic. Model and API changes require maintenance.

[AI Host 2] (15:36 - 15:37)
So you can't just build it once and walk away?

[AI Host 1] (15:38 - 15:49)
Absolutely not. If OpenAI updates their API architecture, your internal team has to drop whatever else they were working on, dive into the code, and fix the integration so the system doesn't go offline.

[AI Host 2] (15:49 - 15:59)
Let's talk about the biggest fear most executives have, which is the AI just making things up. Let's say I allocate the budget. Can't we just mandate that our internal build will be hallucination-free?

[AI Host 1] (16:00 - 16:05)
No. As an organization, you must do not rely on blanket hallucination-free promises.

[AI Host 2] (16:05 - 16:07)
Because it's mathematically impossible.

[AI Host 1] (16:07 - 16:17)
It's just not how generative AI models fundamentally work. They are probabilistic engines. Instead, the operational goal is to ground answers in approved sources.

[AI Host 2] (16:18 - 16:25)
Got it. Meaning the system is mechanically locked to only pull answers from the sandbox of data the association explicitly provides.

[AI Host 1] (16:26 - 16:43)
But even that is not enough. To maintain member trust, the system has to show source documents or passages to the user. If a member asks a complex regulatory question, the AI must provide a hyperlinked citation.
They need to click that citation and immediately see the exact white paper that generated the response.

[AI Host 2] (16:43 - 16:48)
And building that traceability, like actually connecting the text back to the database, that takes ongoing engineering effort.

[AI Host 1] (16:49 - 16:49)
Massive effort.

[AI Host 2] (16:50 - 17:03)
Well, here's where it gets really interesting, right? Building that traceability for a handful of documents sounds like a fun weekend project. But common pitfalls happen when an association scales beyond a simple dataset to tens if not hundreds of thousands of documents.

[AI Host 1] (17:03 - 17:12)
Yes. That is the exponential curve that catches teens entirely by surprise. Moving from five documents to 100,000 documents breaks simple prototypes.

[AI Host 2] (17:13 - 17:13)
I can imagine.

[AI Host 1] (17:14 - 17:26)
When you have a massive library spanning decades of publications, the operational questions get very tough. Like, what do you do when your chatbot simply can't find the right document buried in that mountain of data?

[AI Host 2] (17:26 - 17:39)
Right. If I search for a highly specific industry term and the internally built AI just shrugs, spinning up a custom prototype is basically like hiring a genius librarian who has never been inside your building.

[AI Host 1] (17:39 - 17:39)
Exactly.

[AI Host 2] (17:40 - 17:49)
They're incredibly smart, but if they don't have a perfectly maintained system tailored to your association's weird acronyms, they're going to spend hours wandering the stacks.

[AI Host 1] (17:49 - 17:56)
That is the exact mechanical problem. Can your internally built system actually learn organization-specific search techniques over time?

[AI Host 2] (17:56 - 17:58)
Because every association has its own dialect.

[AI Host 1] (17:59 - 18:03)
Right. And if the AI lacks nuance, how does it learn those crunchy details at scale?

[AI Host 2] (18:03 - 18:12)
Ah, those crunchy details. The highly specific, nuanced pieces of industry knowledge that only your SMEs truly understand. How does a machine learn that?

[AI Host 1] (18:12 - 18:23)
It requires a dedicated feedback loop. Teaching a system those crunchy details across 100,000 documents requires an SME coaching interface.

[AI Host 2] (18:23 - 18:30)
It requires deep analytics to see exactly where the AI is failing and a mechanism to correct it without writing new code.

[AI Host 1] (18:30 - 18:37)
Yes. And questions like that make the last mile exponentially difficult to manage over time for an internal team.

[AI Host 2] (18:37 - 18:52)
To ground this theory in reality, let's look at the proof points in the wild. We have actual implementation patterns from associations in the sources who hit this exact crossroads. Let's start with the National Fire Sprinkler Association, or NFSA.

[AI Host 1] (18:52 - 18:57)
NFSA is a great example. They actively considered building this themselves using Microsoft Azure.

[AI Host 2] (18:57 - 18:59)
So they had access to the raw infrastructure.

[AI Host 1] (18:59 - 19:07)
They did. But they realized the technical skills required to build and maintain the operating layer on top of Azure were just significant.

[AI Host 2] (19:07 - 19:10)
They realized they didn't want to build the coaching interfaces and the dynamic pipelines.

[AI Host 1] (19:10 - 19:19)
Yeah. So they opted to bring in Betty, which is an already operated AI platform. Doing that removed those heavy technical needs from their internal team's plate.

[AI Host 2] (19:19 - 19:29)
And it added customer service, which is huge because when you build it yourself, your IT team literally becomes the customer support desk for thousands of members.

[AI Host 1] (19:29 - 19:32)
Which speaks directly to that internal capacity worry we mentioned earlier.

[AI Host 2] (19:32 - 19:41)
But what if you actually have a robust engineering team? I mean, the sources mentioned Cornet. They are a large organization and they had developers on staff.

[AI Host 1] (19:41 - 19:54)
Cornet is a fascinating case of opportunity cost. They had the internal capability to build a custom stack from the ground up. But their leadership took a step back and asked a foundational question.

[AI Host 2] (19:54 - 20:01)
Why should we expend our valuable engineering resources to build something that is already available to be operated for us?

[AI Host 1] (20:01 - 20:07)
Right. Just because you have the tools to build a water treatment plant in your backyard doesn't mean you shouldn't just hook up to the city grid.

[AI Host 2] (20:07 - 20:15)
Exactly. They recognize their developers could be building unique value for members rather than maintaining underlying infrastructure.

[AI Host 1] (20:15 - 20:20)
And then we see the experience of CareerXRoads, which highlights a totally different technical approach.

[AI Host 2] (20:20 - 20:30)
Yeah, I've seen organizations spin up custom GPTs through OpenAI in about 10 minutes. I know CareerXRoads went down a similar path. Why doesn't that work as a permanent solution?

[AI Host 1] (20:31 - 20:42)
Because their experience proved that custom GPTs can get you partway there very quickly. You get that prototype fast. But they found that the exponentially difficult last mile we just discussed is the real problem.

[AI Host 2] (20:42 - 20:43)
It doesn't scale.

[AI Host 1] (20:43 - 20:52)
Exactly. A custom GPT struggles immensely when you try to scale it to 100,000 documents while simultaneously enforcing complex multi-tier permissions.

[AI Host 2] (20:52 - 20:58)
Trying to take a generic custom GPT and integrate it with your single sign-on is a massive leap.

[AI Host 1] (20:58 - 20:58)
It is.

[AI Host 2] (20:58 - 21:07)
So what does this all mean? It sounds like the realization across these patterns is that having developers on staff doesn't mean you should use them for this specific task.

[AI Host 1] (21:08 - 21:11)
Yes, which brings us to a highly practical decision framework.

[AI Host 2] (21:11 - 21:19)
Perfect. If you are an association executive listening right now, and your board is demanding an AI strategy, how do you actually decide when to build and when to buy?

[AI Host 1] (21:20 - 21:32)
The framework is quite clear once you understand that hidden operating surface. You should build when the association has durable internal ownership for every single layer of the stack.

[AI Host 2] (21:33 - 21:41)
Define durable. Because I've seen organizations build incredible tools, and then the lead developer takes a job somewhere else six months later, and the whole system collapses.

[AI Host 1] (21:42 - 21:55)
Oh, absolutely. Durable means you have dedicated staff. Not just a portion of someone's time, but actual headcount for AI engineering, content operations, security, UX, analytics, monitoring, and ongoing governance.

[AI Host 2] (21:55 - 22:01)
Okay. So if you are a massive organization with a dedicated AI engineering department, building makes sense.

[AI Host 1] (22:01 - 22:11)
Sure. And on the flip side, you buy when the association wants the business outcome governed, access to your knowledge, member adoption, getting staff time back without consuming the rest of your internal roadmap.

[AI Host 2] (22:12 - 22:18)
We've mentioned Betty a few times through the lens of these case studies. I want to use it as a practical example of this buy side of the framework.

[AI Host 1] (22:18 - 22:26)
Sure. Betty demonstrates what an already operated system is. Instead of giving you raw compute power, it comes out of the box offering source bound retrieval.

[AI Host 2] (22:26 - 22:29)
It handles the multi-tier access.

[AI Host 1] (22:29 - 22:39)
Right. Respecting who is allowed to see what. Crucially, it allows for that coachability by your SMEs, so they can refine those crunchy details without needing to write code.

[AI Host 2] (22:39 - 22:49)
It also provides analytics. The source material specifically highlights a feature called insights. Mechanically, why is a dedicated analytics dashboard so critical?

[AI Host 1] (22:49 - 22:58)
Because without it, you are flying blind. Insights allows the association to see exactly what members are asking the AI in real time.

[AI Host 2] (22:58 - 23:04)
Oh, so if 50 members ask about a new regulatory change and the AI can't find the answer, that's an immediate signal.

[AI Host 1] (23:04 - 23:08)
Yes. You now know exactly what article to write or what webinar to host next week.

[AI Host 2] (23:08 - 23:16)
It turns the AI from a simple search bar into a listening device for your entire content strategy. That is a tangible revenue pathway right there.

[AI Host 1] (23:16 - 23:23)
Exactly. It's a layer that activates your existing knowledge. But to maintain absolute credibility here, we need to offer an honest caveat.

[AI Host 2] (23:24 - 23:29)
Right. The sources don't say that every single association should ban their internal teams from touching AI.

[AI Host 1] (23:29 - 23:37)
Absolutely not. Some associations can, and frankly should, build narrow internal pilots as learning tools.

[AI Host 2] (23:37 - 23:40)
Like staff facing workflows. Yeah.

[AI Host 1] (23:40 - 23:49)
If your team wants to build an internal tool to help staff summarize meeting notes or quickly search an HR manual, that is a fantastic way to build AI literacy.

[AI Host 2] (23:50 - 23:57)
Because the stakes are low. If the internal tool hallucinates, your staff is inconvenienced, but the organization survives.

[AI Host 1] (23:58 - 24:07)
Exactly. The high-risk move isn't building a pilot. The high-risk move is trying to take that prototype and scale it into member-facing infrastructure without an operating model.

[AI Host 2] (24:07 - 24:15)
Because if your member-facing system hallucinates the wrong answer on a certification question or leaks board materials, you lose member trust entirely.

[AI Host 1] (24:15 - 24:20)
And remember, governance features are hard and expensive to retrofit if you don't build them in from day one.

[AI Host 2] (24:20 - 24:26)
Yeah. The stakes change entirely when it crosses the threshold to a core member benefit. We have covered a lot of ground today.

[AI Host 1] (24:26 - 24:26)
We really have.
[AI Host 2] (24:26 - 24:42)
So, synthesizing all this, the core thesis comes down to this. Your internal team can probably build a demo. The tools are there.
Your engineers are smart. But the real decision is whether your organization wants to own the entire operating model.

[AI Host 1] (24:42 - 24:54)
Yeah. If you are hearing yourself in this question, do not start by asking whether your team can build a chatbot. Start by listing everything the system has to do after the demo works, then decide whether you want to own that operating model.

[AI Host 2] (24:55 - 25:08)
And to help you make that list, I want to send you back to the build versus buy scorecard we mentioned earlier. It will help you methodically list out those hidden requirements and have a structured, honest conversation internally. You can find it linked in our show notes.

[AI Host 1] (25:08 - 25:09)
It's the best next step you can take.

[AI Host 2] (25:09 - 25:24)
Absolutely. I want to leave you with a final thought to mull over. Think about the history of technology.

Decades ago, companies built their own email servers in a back closet. Today, that would be an absurd waste of resources.

[AI Host 1] (25:24 - 25:24)
It really would.

[AI Host 2] (25:24 - 25:42)
So, if AI infrastructure inevitably becomes a standard utility like having electricity or internet access, what unique proprietary association specific value should your internal tech team be creating today instead of maintaining the plumbing? Thank you so much for joining us on this deep dive. We hope it brought some clarity.
Until next time, keep exploring.

[Rob Barnes] (25:45 - 27:35)
Recapping today's big question, should we build this ourselves? And Thomas, this just reminded me of a couple of real experiences that I've had with a couple of reasonably large and international associations in the last few years, one of which we have spoken to every year for the last three years about what they're doing, what they would like to have achieved, and they still don't have anything to show for it. There is nothing in production, despite their best efforts to try and test, despite my best efforts to have them buy Betty and just do some work, they still don't have anything in production after three years, and because technology has changed rapidly.

Another organisation, a society, I learned, has spent over a seven-figure sum of money trying to get a purpose-built tool for that society working at scale, and again, still does not have anything to show for it, and it kind of pains, honestly, it pains me to know that these things are happening, because, yeah, the demos were easy to show, it's easy to show some capability or some, but are we able, you know, it's a bit like the last episode, you know, they built a tree house, but we've got to move into this thing, when are we ever going to be able to move in, and they haven't been able to, and this episode just reminded me of some of those pitfalls, and again, without putting some real guardrails around what does a build even look like and feel like, beyond the excitement of building something new, that's been lost, and it's very costly, and this episode reminded us of that, right?

[Thomas Altman] (27:35 - 29:02)
Yeah, and I think it's not, again, it's not that you shouldn't build, right, and I think there are a lot of cases where you should, right, it makes a lot of sense, either kind of small, quick wins, right, low-hanging fruit, easy to get out the door and get value out of that, I think, obviously, you should build there, but the other thing that I think this organization could have done is sort of build to learn, right, not build to deploy, but build to learn, like, let's see how far we can go, and it's a very targeted, you know, small budget attempt, right, so spending the seven figures was not, I think, where they should have gone, right, they should have spent four figures, three figures, trying to build something small, learning how it works, and then going back and saying, okay, now does it make sense to partner here, we hit a lot of walls, and we would spend seven figures to overcome those walls, let's go talk to some of these other people in our network and say, like, have you overcome, we found this specific problem, have you faced it before, they say no, red flag, they say yes, maybe keep talking, right, a little bit more, so it gives you a more informed position, right, if you do build first, so we're not anti-build, I just want to be clear about that, there's times you can build and deploy, there's times it makes sense, but the workbook, I think, in this episode, I think, will hopefully have helped you clarify how to build in a way that makes positive ROI far, far, far more likely than it becoming a sunk cost that you just keep sinking more and more and more into.

[Rob Barnes] (29:02 - 30:28)
You've made a good point that I'm going to learn from with this, even with our own conversations, is we do lean into the build versus buy conversation much too heavily instead of build and then buy, oh and partner, because you're right, it's not don't build, I mean, ultimately, that's what you and Dre did, you know, originally with Betty all those years ago, it's more like understanding what the guardrails should and shouldn't be, particularly when it comes to investment, I think the difference there would be, we're talking about associations, we were building a software development company to bring Betty to market, that's our purpose, so it makes sense, for an association, it doesn't always make sense, but you're right, there is great ways to do narrow, internal pilots, low stakes, keep it staff facing, so the control group is there, rather than having this kind of absolutist approach about don't build, I need to remember that when we're having some of these conversations. The scorecard that we've shared with the show notes here, Thomas, is there any other factors, any questions that people should be asking themselves when they are thinking about what first initial steps to take when this is something that's been offered, it might even be by the board, why don't we just do this ourselves?
[Thomas Altman] (30:29 - 32:03)
Yeah, I think look for those weekend projects, right, it talks about sort of a miracle happening over the weekend at the beginning of this episode, and you don't have to do them over the weekend, right, the weekend being a short, like, oh, it's a passion project, I would say it would be useful if, and we did this internally, right, like a one-week sprint, right, where everybody on the team, or maybe you get a cohort of people, and you say, yeah, keep the lights on, you know, for a week, but your job for the next week is to just build something cool, and not that you're going to deploy that, or that it's going to become some critical part of your infrastructure, but that on Friday, if you started on Monday, on Friday, maybe you throw the whole projects away, maybe you don't, but maybe you do, and the guaranteed outcome of this is that you're going to have a discussion around what is actually possible based off of real experience, because AI is such a new, it's still a new technology, I know it feels like it's been forever, but it's still fairly new, like there's no clear best practices, and the reason for that is it's also fast-moving technology, there's new stuff every day, so being able to kind of get people's hands dirty trying to build something, even if they're not technical, like have them work with codex, or quad code, or whatever, to sort of build something, and to see how far they get, and then talk about it, right, and then once you talk about it, you'll be making more informed decision around like, okay, this is a good pilot project that we can actually do, or this is a much bigger hornet's nest that we don't want to shake, and we're going to find people who've encountered and solved these problems previously, and it'll be an informed decision at that point that'll cost three digits instead of seven.

[Rob Barnes] (32:04 - 32:29)
Thanks Thomas. So everyone listening, one more time before you go, download the Build Versus Buy scorecard, I'm not going to change the title of it just yet, but download that from the show notes, that is the fastest way to actually put today's episode to work. Thank you for listening to the Association Intelligence Podcast, join us on the next episode where we dig into this big question, we don't have a tech team, can we still do this?

What this helps your team answer

Why a working prototype (a few documents, a small demo) and an operated, member-facing knowledge system are two completely different things — and where the "hidden operating surface" (content ingestion, permissions, traceability, SME coaching, analytics, security) actually lives.
Real association examples — NFSA, Cornet, and CareerXRoads — showing what happened when they weighed Azure builds, in-house developers, and custom GPTs against buying an already-operated platform.
A practical framework for deciding when to build (you have durable, dedicated headcount for every layer of the stack) versus buy (you want the outcome without consuming your internal roadmap) — plus how to run a low-stakes "build to learn" pilot instead of a costly seven-figure attempt.

Companion Asset

Build vs. Buy Scorecard

Getting Started

We Don't Have a Tech Team — Can We Still Do This?

Why AI success for associations is far less about engineering talent and more about organizational clarity — and how to right-size a single governable use case instead of trying to build an org-wide governance framework before you start.

34:01

0:00 34:01

[Thomas Altman] (0:07 - 0:11)
Today's big question, we don't have a tech team. Can we even still do this?

[Rob Barnes] (0:12 - 1:29)
We are Thomas Altman and Rob Barnes and this is episode 5 of the Association Intelligence Podcast. Before we get into it today, this episode's companion asset is the Implementation Readiness Worksheet. It's linked now in the show notes.

Grab it now if you want to follow along or download it after and put it straight to work. This episode is really talking about organizational clarity, not so much a technical discussion. I think that's what I took away from this is I hear often Thomas now in conversations about well, we have to do this big AI policy, AI governance framework, AI assessment first before we can think about it at an application level.

And like all strategic planning and all strategic work in an organization, it's a little bit like herding the cats. It's costly, it can be time consuming, and it doesn't always arrive at the problem that's trying to be solved when it comes to an AI conversation anyway. Are we understating how much governance work is actually technical work in disguise?

[Thomas Altman] (1:29 - 3:09)
I think that's the interesting thing here. And I think it might even be the other way around. So one thing we've seen is we've had plenty of non-technical or groups without an IT staff at all kind of adopt successfully Betty, right?

And I think there's this kind of concern around we don't have a technical team because AI feels like it should be technical, right? And the fact is governed AI a lot of times isn't, right? Like the technology is there, right?

Or the hurdle isn't the technology, the hurdle is the organizational understanding of what is governed, what is not governed, what are the cases that we want to apply AI, right? It's more of an organizational hurdle than a technological one. So we've had teams with strong technical backgrounds struggle, right?

And sometimes because they're strong technical teams, right, they get too in the weeds with it, and they lose sight of kind of what's the value of the project. And we've had teams with strong technical backgrounds also succeed, right? But we've had teams with no technical team be wildly successful.

And the thing that I liked about this episode is it teases out sort of how that happens, right? Where does that actually live? Why is this actually not really a technical problem?
And why is it that strong organizational organizations, organizational organizations, why are they successful regardless of whether or not they have a strong IT team, right? And I think this gets into that a little bit well in a way that I think is super useful and hopefully applicable.

[Rob Barnes] (3:09 - 4:56)
Yeah, for sure. There is some very strong customer success stories where, you know, I know two of our smallest customers are like two full-time staff, and neither of them are technical. But the level of ownership that they've taken over the application and its purpose and its alignment to their strategic intent is where a lot of that success kind of comes from.

Like, I think some people may come into this episode and go, oh, it's another AI readiness checklist. And that's, and I think what's great about this is that it is a checklist, take it or leave it. Look at the eight steps, you know, from a practical perspective.

But ask yourself the question, like, which of these do we skip when we talk about these sorts of things? What breaks if we do skip them? And so use it as an exception, almost as an exception list first rather than a hard and fast rule.

And look at it from, look at it from that perspective because we know how to govern. We know that there are governance frameworks in place for just about every other program of work that an association does. Advocacy is tightly controlled.

You know, event delivery is tightly controlled. Even our marketing management is tightly controlled against brand and persona and those sorts of things. What I really enjoyed about this, again, is taking away the shrink the scope advice, like bring it down to a tangible, more practical, something that we know we can govern almost immediately.

However, I wonder, Thomas, is that too small of a scope in order for a board to approve ROI or to achieve an ROI in these conversations, which can also thwart the whole project?

[Thomas Altman] (4:57 - 6:10)
Right, that's the debate. So I think it is smart to not boil the ocean, right? Like it can be overwhelming to have to have kind of organizational governance over everything, right?

If you're thinking in terms of the whole organization, that can be so much work that you'll actually never do anything at all. And if you never do anything at all, you're going to be left behind. But at the same point, if you start too small, is that not exciting enough to get buy-in from the group?

And I think kind of navigating that tension is what this episode is really all about. How do you find the right size use case that is governable, right? So you don't have to boil the ocean.
You don't have to govern your whole organization. You can govern a specific use case. How do you identify what that is so that it's not too small, right?

You don't lose enthusiasm or momentum because you started too small and too timidly, but it's not so big that you never get started at all. So kind of right-sizing that and listening to kind of how our favorite AI hosts kind of navigate that on this, I think is going to be really hallucinating. And then once you apply that through the companion asset for this episode, I think you'll come away with a really clear, like, this is where we can start.

[Rob Barnes] (6:11 - 6:33)
And before we hand this over to the AI podcast's hosts, quick reminder that the following is generated using Notebook LM. Heavy involvement from the Betty team to ensure that the content generated is valuable to you today. For a deeper dive on our approach to the Association Intelligence Podcast, please go back and listen to episode zero.

And now, over to the AI hosts.

[AI Host 1] (6:36 - 6:40)
Imagine buying, well, like a brand new self-driving car, right?

[AI Host 2] (6:40 - 6:40)
Yeah.

[AI Host 1] (6:40 - 6:45)
You get strapped into the passenger seat, and then you suddenly realize you have absolutely no idea where you're going.

[AI Host 2] (6:45 - 6:46)
Oh, that's terrifying.

[AI Host 1] (6:47 - 6:51)
Right. Or, or worse, you realize you don't even know how to tell the car where to go.

[AI Host 2] (6:51 - 6:53)
Yeah, that sudden rush of panic.

[AI Host 1] (6:53 - 7:00)
Exactly. And that is exactly how most association leaders feel when they, you know, when they look at their organizational chart.

[AI Host 2] (7:00 - 7:02)
They see zero software engineers.

[AI Host 1] (7:02 - 7:11)
Zero. And they realize they're expected to implement artificial intelligence anyway. It's, it's basically the number one concern we hear from you guys.

[AI Host 2] (7:12 - 7:13)
It really is.

[AI Host 1] (7:13 - 7:19)
It always starts with this deep sigh, and then the exact same question. We don't have a tech team. Can we still do this?

[AI Host 2] (7:19 - 7:24)
Which is such a common refrain. And frankly, I mean, it's a completely rational response.
[AI Host 1] (7:24 - 7:24)
Totally.

[AI Host 2] (7:24 - 7:37)
If you're looking at this fundamentally new, really complex technology, and trying to map it onto an organization that specializes in member services, not machine learning. Right. The math just doesn't seem to work out in your head.

[AI Host 1] (7:38 - 7:47)
Yeah, let's validate this right out of the gate for you, the listener. Because if you are the person lying awake at night, worrying about the maintenance, or, you know, the vendor management.

[AI Host 2] (7:47 - 7:48)
And system permissions.
[AI Host 1] (7:48 - 7:55)
Exactly, the permissions, or just the sheer support load this might dump on your already stretched staff, you are not being a Luddite.

[AI Host 2] (7:55 - 7:58)
No, not at all. You're being a responsible leader.

[AI Host 1] (7:58 - 8:06)
You are. Those are the exact operational realities that break bad technology implementations.

[AI Host 2] (8:06 - 8:07)
Yeah, they really do.

[AI Host 1] (8:08 - 8:19)
So today's Deep Dive is based on a stack of implementation case studies. And specifically, the Implementation Readiness Framework. Which is designed for association leaders facing this exact dilemma.

[AI Host 2] (8:20 - 8:25)
And our goal today is to basically fundamentally reframe the whole concept of readiness.

[AI Host 1] (8:26 - 8:28)
Yes, the mission of this Deep Dive.

[AI Host 2] (8:28 - 8:35)
Right, we need to shift the perspective here because the barrier to entry, it isn't about code anymore. It's about organizational clarity.

[AI Host 1] (8:35 - 8:36)
Okay, say more about that.

[AI Host 2] (8:36 - 8:44)
Well, we just need to stop asking, do we have the engineers to build this? And start asking, can we own the decisions that only our association can own?

[AI Host 1] (8:44 - 8:47)
Which brings us right back to that self-driving car analogy.

[AI Host 2] (8:47 - 8:47)
Yeah.
[AI Host 1] (8:48 - 9:06)
Because, I mean, you do not need a team of mechanical engineers living in your garage to safely drive a vehicle. Thank goodness. Right, you don't need to know how to mill an engine block, or program a fuel injection system.
But you absolutely must know your destination.

[AI Host 2] (9:06 - 9:08)
Yes, you have to know the rules of the road.
[AI Host 1] (9:08 - 9:12)
And critically, you need to know exactly who you are allowing to take the wheel.

[AI Host 2] (9:12 - 9:14)
That is the key right there.

[AI Host 1] (9:14 - 9:27)
So to understand why a lack of an internal tech team isn't actually a deal breaker, we first have to draw a very hard line. Like a massive divide between the technical mechanics of AI and the organizational governance of it.

[AI Host 2] (9:27 - 9:36)
The great divide. Let's talk about the heavy lifting first. Because in a modern AI deployment, your vendor, your implementation partner, they carry the technical burden.

[AI Host 1] (9:37 - 9:38)
What does that actually look like?
[AI Host 2] (9:38 - 9:53)
So they handle the data ingestion, which means they do the complex work of actually connecting to your databases and parsing your files. They manage the hosting, they build the underlying retrieval systems, and the security architecture that keeps everything totally safe.

[AI Host 1] (9:53 - 9:54)
So they're building the engine block.

[AI Host 2] (9:55 - 10:00) Exactly. They mill it, they tune it, and they maintain it. They also handle the launch support mechanics.

[AI Host 1] (10:01 - 10:25)
Okay, I am going to push back on this right away, though. Sure. Because if I'm playing the skeptic, I'm listening to this and I'm thinking, well, if the vendor is handling all the technical architecture, all the ingestion and the retrieval, aren't we essentially just handing our organizational brain over to a third party?

Like, how do we not lose control of our own intelligence if we aren't the ones writing the code?

[AI Host 2] (10:25 - 10:32)
Well, it's a really vital question, but it kind of misunderstands how governed AI actually works in practice.

[AI Host 1] (10:32 - 10:32)
How so?

[AI Host 2] (10:33 - 10:46)
Because control isn't in the code anymore. The control is entirely in the content. When we talk about governed AI for associations, we are absolutely not talking about, you know, unleashing an open-ended chatbot that just hallucinates answers from the public internet.

[AI Host 1] (10:46 - 10:47)
The nightmare scenario.

[AI Host 2] (10:48 - 10:57)
Right, the nightmare. We are talking about a closed source-bound system. The AI is strictly constrained by the documents you explicitly approve.

[AI Host 1] (10:57 - 11:04)
Let's pause on that mechanism for a second, actually. Because I think a lot of people assume the AI is just trying to, like, remember things it read once.

[AI Host 2] (11:04 - 11:06)
Yeah, that's a common misconception.

[AI Host 1] (11:06 - 11:10)
So how does a governed system actually stop those famous AI hallucinations?

[AI Host 2] (11:11 - 11:19)
Well, think of it less like a traditional human brain and more like a highly restricted search engine paired with a really brilliant summarizer.

[AI Host 1] (11:19 - 11:21)
Okay, a restricted search engine.

[AI Host 2] (11:21 - 11:35)
Yeah, so when a member asks a question, the underlying code acts like a search engine first. It searches only your approved databases, retrieves the specific PDF or webpage that has the answer, and then it hands that exact text to the language model.

[AI Host 1] (11:36 - 11:36) Oh, I see.

[AI Host 2] (11:36 - 11:42)
The system essentially tells the AI, read this specific document, and only summarize exactly what is in this text.

[AI Host 1] (11:43 - 11:43)
Wow.

[AI Host 2] (11:43 - 11:52)
And if the answer isn't in the document, the AI is programmed to say, it just doesn't know. That is how you kill the hallucination. The code just follows the map you draw.

[AI Host 1] (11:52 - 11:56)
So the authority sits with the association, not the algorithm.

[AI Host 2] (11:56 - 11:58)
100%. You are drawing the boundaries.

[AI Host 1] (11:58 - 12:06)
And for everyone listening who is wondering how they actually draw that map, we have a really crucial companion asset for this deep dive.

[AI Host 2] (12:06 - 12:07)
We do.

[AI Host 1] (12:07 - 12:15)
It's called the Implementation Readiness Worksheet, and it's designed specifically to help you map out these exact ownership decisions.

[AI Host 2] (12:15 - 12:17)
It's such a helpful tool.

[AI Host 1] (12:17 - 12:26)
I highly recommend pulling it up as we talk through this, honestly, because it translates this entire conversation into actionable steps for your team. You can find it right in the show notes.

[AI Host 2] (12:27 - 12:42)
And that worksheet, it really bridges the gap between the theory of governance and the daily practice of it. And understanding that division of labor where the vendor takes the technical burden and you take the governance burden, it actually leads to this really interesting paradox.

[AI Host 1] (12:42 - 12:43)
Which is what?

[AI Host 2] (12:43 - 12:49)
Lacking a massive IT team is actually a compelling argument for partnering right now rather than waiting.

[AI Host 1] (12:49 - 12:50)
Wait, that feels super counterintuitive.

[AI Host 2] (12:51 - 12:51)
I know.

[AI Host 1] (12:51 - 12:56)
Because a lack of a tech team usually means pump the brakes, not move faster.

[AI Host 2] (12:57 - 13:05)
Well, it means move faster through partnership rather than falling into the waiting trap. And we see associations do this, like, all the time.

[AI Host 1] (13:05 - 13:06)
The waiting trap.

[AI Host 2] (13:06 - 13:11)
Yeah. Small teams often think they need to wait for some massive future platform modernization project.

[AI Host 1] (13:12 - 13:12)
Oh, right.

[AI Host 2] (13:13 - 13:17)
You hear the old, well, we'll look at AI when we overhaul our AMS in three years. Yes.

[AI Host 1] (13:17 - 13:27)
The legendary association management system upgrade. The eternal excuse for delaying innovation because, you know, the new database will magically fix everything.

[AI Host 2] (13:28 - 13:32)
Precisely. Or they think they need to hire a bunch of developers before they can even pilot a tool.

[AI Host 1] (13:33 - 13:33)
Right.

[AI Host 2] (13:33 - 13:41)
But trying to build sweeping AI transformations internally, it creates an unmanaged, just overwhelming burden for a small team.

[AI Host 1] (13:41 - 13:41)
It's too much.

[AI Host 2] (13:42 - 13:48)
It is. Partnering with a specialized vendor activates your existing knowledge without that massive technical overhead.

[AI Host 1] (13:48 - 13:55)
So what does that kind of partnership actually look like in the real world? Like, how does a non-technical staff member interact with a system they didn't build?

[AI Host 2] (13:56 - 13:58)
A great practical example of this is Betty.

[AI Host 1] (13:58 - 13:59)
Okay, tell me about Betty.

[AI Host 2] (13:59 - 14:19)
So Betty is a governed implementation pattern designed specifically so that associations can own their content decisions without needing to staff a software team. Got it. Practically, it might look like a chat widget on your certification landing page or maybe a search tool in your member portal.

[AI Host 1] (14:19 - 14:19)
Sure.

[AI Host 2] (14:19 - 14:27)
But behind the scenes, systems like Betty are traceable back to your original source material. And more importantly, they are coached by your staff and your subject matter experts.

[AI Host 1] (14:28 - 14:30)
Coached is a really interesting word there. Walk me through that.

[AI Host 2] (14:30 - 14:30)
Yeah.

[AI Host 1] (14:30 - 14:36)
Because if I'm the director of membership and I don't know how to code in Python, how am I coaching an AI?

[AI Host 2] (14:36 - 14:40)
Well, coaching in this context is really just a structured feedback loop.

[AI Host 1] (14:40 - 14:40)
Okay.

[AI Host 2] (14:40 - 14:55)
It means a human on your staff periodically reviews the chat logs. They might see that, say, a member asked a really nuanced question about membership renewal. Right.

And the AI gave an answer that was technically correct, but maybe it lacked some helpful context.

[AI Host 1] (14:55 - 14:57)
So what does the staff member do?

[AI Host 2] (14:57 - 15:11)
They use a dashboard to flag that interaction, essentially leaving a note that says, hey, next time someone asks this, prioritize the information in paragraph three of the renewal policy, not paragraph one.

[AI Host 1] (15:11 - 15:11)
Oh, wow.

[AI Host 2] (15:11 - 15:15)
You are applying human judgment to guide the system.

[AI Host 1] (15:15 - 15:25)
I'm really glad you brought up the human judgment aspect, actually, because we had to be crystal clear for our listener here. We are definitely not saying this is hands-off.

[AI Host 2] (15:25 - 15:26)
Oh, absolutely not.

[AI Host 1] (15:26 - 15:29)
You don't just buy the software, flip a switch, and then go take a nap?

[AI Host 2] (15:29 - 15:34)
No, there is no such thing as no technical work required, translating to no work required at all.

[AI Host 1] (15:35 - 15:35)
Right.

[AI Host 2] (15:35 - 15:41)
The nature of the work simply shifts. You are replacing coding work with governance and judgment work.

[AI Host 1] (15:41 - 15:47)
It really feels like the difference between building a library and being the head librarian.

[AI Host 2] (15:47 - 15:48)
Yes, that's a great way to put it.

[AI Host 1] (15:48 - 15:56)
Because you don't have to pour the concrete for the foundation or wire the electricity, but you absolutely have to decide which books go in which section.

[AI Host 2] (15:56 - 15:57)
Exactly.

[AI Host 1] (15:57 - 15:58)
And how the Dewey Decimal System is applied.

[AI Host 2] (15:59 - 15:59)
Yeah.

[AI Host 1] (15:59 - 16:01)
And who is actually allowed to check out the rare manuscripts.

[AI Host 2] (16:02 - 16:12)
That is a perfect analogy. And being the head librarian, that is a significant organizational commitment. It is.

It requires deep institutional knowledge and real operational discipline.

[AI Host 1] (16:12 - 16:22)
Which brings us to the core of the readiness framework. How does an organization actually replace that coding work with governance work?

[AI Host 2] (16:22 - 16:22)
Right.

[AI Host 1] (16:22 - 16:30)
How do you define those boundaries so the system doesn't just run wild? This is where the minimum viable implementation checklist comes in.

[AI Host 2] (16:30 - 16:38)
Yeah, this is where the rubber meets the road. Because if you want to know if you are ready, without a tech team, there are eight criteria you have to be able to define.

[AI Host 1] (16:38 - 16:42)
And these map directly to the implementation readiness worksheet we mentioned earlier, right?

[AI Host 2] (16:42 - 16:43)
Yes, they do. Exactly.

[AI Host 1] (16:43 - 16:51)
Okay, so if we think about this checklist, it is really like hiring a brand new, incredibly smart, but completely naive intern.

[AI Host 2] (16:51 - 16:52)
Very naive.

[AI Host 1] (16:52 - 17:00)
Right. First day on the job, you have to define the use case. That is step one.
You would never look at a new intern and just say, hey, go fix the association.

[AI Host 2] (17:00 - 17:02)
That would be chaos. A catastrophe.

[AI Host 1] (17:03 - 17:03)
A complete disaster.

[AI Host 2] (17:04 - 17:19)
You have to be specific. You don't say, use AI to help members. You define the job.
You say, we want to use the system to help members instantly find answers about our 2025 continuing education certification requirement.

[AI Host 1] (17:19 - 17:34)
Right, which naturally leads to the second boundary, which is the source set. Yep. Because if I give that intern their first project, I'm not going to tell them to just, you know, go read every single document ever produced by the organization since 1985.

Please don't. I'm going to hand them a specific stack of folders.

[AI Host 2] (17:34 - 17:46)
Right. You draw a fence around the official documents. You have to ask which specific webpages, videos, standards, FAQs, or policies are deemed authoritative for that specific use case.

[AI Host 1] (17:46 - 17:46)
Okay.

[AI Host 2] (17:46 - 17:55)
If the job is answering questions about certification requirements, the source set is only the official certification handbook and the official FAQ page. Nothing else.

[AI Host 1] (17:56 - 18:09)
Okay, so the intern knows what to read. Next, we have to define who they are actually allowed to talk to, what they are allowed to say. So that covers the next two criteria, which are the audience and the access tiers.

It's like giving the intern the right key card to the building.

[AI Host 2] (18:09 - 18:24)
Exactly. You have to define who gets to ask the questions. Is this tool facing the general public?
Is it locked behind a member wall? Right. Is it only for internal staff or the board of directors?
And then, what exactly can each audience see?

[AI Host 1] (18:24 - 18:26)
Give me an example of that.

[AI Host 2] (18:26 - 18:32)
Well, if a member asks a question about benefits, they should get an answer drawn from the full member packet.

[AI Host 1] (18:32 - 18:32)
Sure.

[AI Host 2] (18:33 - 18:40)
But if a non-member asks the exact same question, they should probably get a totally different answer drawn from, like, public marketing materials.

[AI Host 1] (18:41 - 18:41)
Yeah.

[AI Host 2] (18:41 - 18:43)
You have to govern those permissions.

[AI Host 1] (18:43 - 18:49)
Now we get to my favorite part of the checklist and arguably the one that causes the most internal friction.

[AI Host 2] (18:49 - 18:50)
Oh, yeah.

[AI Host 1] (18:50 - 18:52)
The owner. Every intern needs a boss.

[AI Host 2] (18:52 - 18:53)
Ah.

[AI Host 1] (18:53 - 18:58)
But in an association, ownership of information can be incredibly messy.

[AI Host 2] (18:58 - 19:11)
Oh, it can be a battlefield. You need a named human being who actually approves the sources and the boundaries of the answers. When there is a dispute about whether a document is out of date, whose desk does that actually land on?

[AI Host 1] (19:11 - 19:13)
Let's play that out, actually, because I know that happens all the time.

[AI Host 2] (19:13 - 19:14)
All the time.

[AI Host 1] (19:14 - 19:19)
Say the marketing department has a glossy brochure from last year that says membership costs $500.

[AI Host 2] (19:20 - 19:20)
Hmm.

[AI Host 1] (19:21 - 19:27)
But the legal and finance team just updated the official bylaws last month, and membership is now $550.

[AI Host 2] (19:28 - 19:28)
Yep.

[AI Host 1] (19:29 - 19:34)
If the system is looking at both, the intern is confused. Who breaks that tie?

[AI Host 2] (19:34 - 19:41)
That is exactly why you need an owner. The AI cannot resolve a political dispute between your marketing and finance team.

[AI Host 1] (19:41 - 19:41)
It just can't.

[AI Host 2] (19:42 - 19:53)
No. The owner is the person with the authority to say the bylaws are the ground truth. Remove the marketing brochure from the source set.
You must have someone who owns the business logic of that specific use case.

[AI Host 1] (19:53 - 19:57)
Which flows perfectly into the next item on our checklist, the review path.

[AI Host 2] (19:57 - 19:58)
Yes.

[AI Host 1] (19:58 - 20:00)
Our intern needs a Friday performance review.

[AI Host 2] (20:00 - 20:23)
Because AI is an iterative process. Users are going to ask questions in ways you completely didn't anticipate. They'll use weird phrasing or obscure acronyms.

So who corrects those issues and how often does that happen? Is the owner looking at the chat logs every Friday afternoon to see if the system gave an incomplete answer? Are they coaching the system to do better next time?
You need a cadence for review.

[AI Host 1] (20:24 - 20:32)
Okay, then we have the launch channel. Where does our intern actually sit in the office? Do we hide them in the basement?
Or do we put them at the front desk?

[AI Host 2] (20:32 - 20:36)
You have to place the tool where users will naturally encounter it. Don't make them hunt for it.

[AI Host 1] (20:37 - 20:37)
Okay.

[AI Host 2] (20:37 - 20:45)
If it's an internal staff tool to help your membership team answer phone calls faster, maybe it lives right inside Microsoft Teams or Slack.

[AI Host 1] (20:45 - 20:45)
That makes sense.

[AI Host 2] (20:45 - 20:52)
But if it's a member-facing tool for certification, it needs to be a highly visible widget right there on the certification landing page.

[AI Host 1] (20:53 - 21:02)
And finally, the last item on the checklist. The success signal. How do we know the intern is actually doing a good job and isn't about to get fired?

[AI Host 2] (21:03 - 21:08)
Right. What metric would prove this first narrowly defined scope is actually worth expanding?

[AI Host 1] (21:09 - 21:11)
Give me some examples of what that looks like.

[AI Host 2] (21:11 - 21:23)
Well, are we looking for a 20% reduction in basic support emails to the membership team? Or maybe we're looking to cut the time it takes staff to research an answer from 15 minutes down to two minutes.

[AI Host 1] (21:24 - 21:25)
Okay, tangible metrics.

[AI Host 2] (21:25 - 21:33)
Exactly. You have to define what a win looks like before you start playing. Otherwise, you'll never know if the governance effort was actually worth it.

[AI Host 1] (21:33 - 21:43)
So if we step back and look at that whole checklist. Use case, source set, audience, access tiers, owner, review path, launch channel, and success signal.

[AI Host 2] (21:43 - 21:44)
That's the list.

[AI Host 1] (21:44 - 21:50)
That is the anatomy of governed AI. It proves why small teams need clearer boundaries, not more complexity.

[AI Host 2] (21:51 - 21:51)
Absolutely.

[AI Host 1] (21:52 - 21:57)
You are creating a controlled environment where your existing staff can safely apply their expertise.

[AI Host 2] (21:57 - 22:03)
Exactly. And when you define those boundaries, you prevent the system from becoming an unmanaged, just sprawling liability.

[AI Host 1] (22:04 - 22:07)
Okay, but we have to do the honest beat here. We have to look at the dark side of this.

[AI Host 2] (22:07 - 22:07)
Let's do it.

[AI Host 1] (22:08 - 22:20)
Having boundaries is great in theory, but what happens if an association leadership team sits down, they pull up that implementation readiness worksheet, they look at that eight-point checklist, and they just draw a complete blank?

[AI Host 2] (22:20 - 22:25)
Yeah. This is the reality check, and it's a sobering one for a lot of organizations.

[AI Host 1] (22:25 - 22:26)
I can imagine.

[AI Host 2] (22:26 - 22:48)
Because if leadership sits down and realizes that no one can actually agree on what the authoritative sources are, if they realize that no one on staff actually has the capacity or the willingness to own corrections and review the logs, and no one can define where the system should escalate issues, then they are fundamentally not ready to launch an AI system broadly.

[AI Host 1] (22:49 - 22:59)
Hold on, though. If they can't even agree on their sources, aren't they basically admitting their internal house is entirely out of order? Yes.
That feels like a much bigger operational problem than simply not having an AI tool.

[AI Host 2] (22:59 - 23:00)
You've hit the nail on the head.

[AI Host 1] (23:01 - 23:01)
Yeah.

[AI Host 2] (23:01 - 23:22)
That is an organizational crisis masquerading as a technology problem. If the humans in your building don't know what the single source of truth is, the AI certainly isn't going to magically figure it out for you. Right.
If you put messy, contradictory, unmanaged knowledge into an AI retrieval system, you are just going to get very fast, very confident wrong answers.

[AI Host 1] (23:22 - 23:33)
So if an organization hits this wall, if they look at the checklist and say, wow, our data is a mess, we can't possibly do this for the whole association, is the dream dead?

[AI Host 2] (23:33 - 23:34)
No.

[AI Host 1] (23:34 - 23:39)
Do they just walk away from AI entirely until they spend three years cleaning up their servers?

[AI Host 2] (23:39 - 23:44)
No. The solution isn't to walk away. The solution is to radically shrink the scope.

[AI Host 1] (23:45 - 23:46)
Okay. Shrink the scope.

[AI Host 2] (23:46 - 23:54)
You shrink it until it becomes operationally realistic for the capacity you actually have today. If you cannot govern the whole organization's data, do not try.

[AI Host 1] (23:54 - 23:55)
Right.

[AI Host 2] (23:55 - 23:56)
Govern just one department's data.

[AI Host 1] (23:56 - 23:59)
So cut it down from the whole library to just one section.

[AI Host 2] (23:59 - 24:02)
Exactly. And if one department is still too much, shrink it again.

[AI Host 1] (24:02 - 24:02)
Keep going.

[AI Host 2] (24:03 - 24:08)
Govern just the knowledge for one specific upcoming annual conference. Just the event FAQ.

[AI Host 1] (24:08 - 24:09)
Oh, that's smart.

[AI Host 2] (24:10 - 24:18)
Right. Can you find one owner, a handful of clear sources, and one success metric for just that one event? Usually the answer is yes.

[AI Host 1] (24:19 - 24:25)
The golden rule here seems to be that implementation must reduce your staff load, not add to it.

[AI Host 2] (24:25 - 24:33)
That is the absolute rule. If the scope is so big that governing it feels like a full-time job you don't have time for, the scope is simply too big.

[AI Host 1] (24:33 - 24:34)
Yeah.

[AI Host 2] (24:34 - 24:40)
You have to find the smallest, most contained area of friction in your association right now and start exactly there.

[AI Host 1] (24:41 - 24:46)
Shrinking the scope until it fits your capacity, I mean, that's incredibly empowering. It really takes the pressure off.

[AI Host 2] (24:46 - 24:47)
It does.

[AI Host 1] (24:47 - 24:52)
It means that any association, regardless of size or technical footprint, can find their starting line.

[AI Host 2] (24:52 - 24:59)
Which is exactly why the original question, we don't have a tech team, can we still do this? is really asking the wrong thing.

[AI Host 1] (24:59 - 25:09)
Let's synthesize this for the listener then. What is the core takeaway for the association executive who tuned in today with that creeping anxiety about their lack of software engineers?

[AI Host 2] (25:09 - 25:13)
The main takeaway is that a large internal tech team is simply not the test of your readiness.

[AI Host 1] (25:14 - 25:14)
It's just not.

[AI Host 2] (25:14 - 25:40)
The true test is organizational clarity. The real test is whether your association can define a bounded use case, point to authoritative sources, establish access tiers, assign an owner, commit to a review process, and define a metric for success. If you can do those things, you replace the need for complex custom coding with the power of clear strategic governance.

[AI Host 1] (25:40 - 25:41)
You trade the wrench for the steering wheel.

[AI Host 2] (25:42 - 25:42)
Exactly.

[AI Host 1] (25:42 - 25:51)
And that brings us right back to our companion asset for this deep dive. If you want to put this into practice, please go to the show notes and grab the implementation readiness worksheet.

[AI Host 2] (25:51 - 25:52)
Highly recommend it.

[AI Host 1] (25:52 - 26:01)
It takes everything we just talked about, the use case, the sources, the ownership, the metrics, and puts it into a clear, actionable format for you and your team to fill out.

[AI Host 2] (26:01 - 26:02)
It really makes it tangible.
[AI Host 1] (26:02 - 26:14)
Because here is the real secret. If you can sit down and fill out that worksheet for just one specific use case, you likely have exactly what you need to start a focused, responsible pilot today.

[AI Host 2] (26:14 - 26:15)
Today.

[AI Host 1] (26:15 - 26:18)
You do not need to wait for a tech team to magically appear.

[AI Host 2] (26:18 - 26:28)
It really is just about taking that first bounded step. Once you see it work in a small, governed environment, the path forward becomes so much clearer.

[AI Host 1] (26:29 - 26:46)
Absolutely. So as we wrap up, I want to leave you with a final thought to mull over. Think back to that self-driving car analogy from the beginning.

You do not need to build the engine. You just need to know how to navigate. Now think about the most valuable buried knowledge sitting inside your organization right now.

[AI Host 2] (26:46 - 26:47)
All those hidden gems.

[AI Host 1] (26:48 - 27:14)
The insights, the standards, the rich institutional history that your members desperately need but can never seem to find on your website. Right. What if the only thing keeping that incredibly valuable knowledge from your members isn't a lack of complex code, but simply a lack of clearly defined boundaries?

It completely changes how you look at the problem. It changes everything. The tech isn't the barrier anymore.

You are the barrier. So grab the keys, define your boundaries, and start driving.

[Rob Barnes] (27:17 - 28:28)
In today's episode, we dug into the big question. We don't have a tech team. Can we still do this?

And it's super interesting to hear our AI hosts really make some, I'm going to say, interesting comments. You know, the first thing, Thomas, that I picked up on is this genuine issue that often arises in an association is about if the association leadership, which we know is a mix of staff and volunteer, can't agree on what is the authoritative source, who owns the corrections, the coaching. Our AI hosts were like, that's an organizational crisis masquerading as a technology problem and challenged us to think about it in terms of our own team.

But perhaps that was a little harsh. I'm not sure that it's crisis mode, but it is a challenge. We know that ownership of this work leads to success.

When someone is the steward of this work, it leads to more success. I don't think it's the organizational crisis necessarily our hosts talked about, right?

[Thomas Altman] (28:29 - 30:02)
Yeah, I think they were a little harsh on this one. I agree. There was a few times I'm like, I don't know if I would have said it like that.

It's a good point though, right? Like the idea that if it's not a technology problem, if you or your organization can't agree on what the use case is and what the authoritative kind of assets to support that use case will be. That is a true problem.

Or the kind of lying around like it's not a tech barrier anymore, it's a you barrier. That's a little harsh. I don't know if I would have gone that far.

But I do think it is kind of illuminating a real problem, which is understanding what rules apply, how to apply those rules. The human sort of in the loop and the sort of complex human relationships need to be incorporated into the governing framework, right? And sometimes that's hard, right?

Sometimes that's too much, right? Sometimes you can debate that for months and months and months and never build anything. And that's not going to help anyone either, right?
So the idea that I like, the thing that the tension that seemed to be working or how we want to approach resolving this and the ways we've seen this be successful is in rightsizing a scope, right? You don't have to do everything all at once. You don't have to have all of your ducks in a row in place for everything.

When you can concretely and specifically govern a specific use case. So being able to find out what that is and the thing that actually makes the cats heritable, right? That's where I liked this, how this conversation kind of led us.

[Rob Barnes] (30:03 - 31:35)
There are so many examples of organizations that have real clarity over, you know, the ownership of the steps that even the readiness worksheet, the checklist that we published in concert with this, where there is an owner or a clear owner when you define that scope and perhaps apply that checklist to a particular limited scope exercise. A lot of organizations have an absolute clear, this is the leader of that. This is who is the custodian of that governance framework.

And seeing that being able to be applied to the kind of technology now that's accelerating the purpose of that work, it's very clear. And so I actually thought that the homework that the podcast gave us today about, you know, filling out the worksheet with a single use case and seeing if we can identify a very clear owner. But being okay with the ambiguity that may come from that, depending on who is working within the organization at the time, because a lot of this is going to be ambiguous.

You know, if we don't always know what the decision-making pathway is and who owns the authority around knowledge and distribution, then I think we just have to be okay with that ambiguity because there's a lot of the work in this area that is going to be ambiguous for a lot of organizations for a little bit longer, for a little bit further to come, I imagine, right?

[Thomas Altman] (31:36 - 33:16)
Yeah, I think so. And I think there's sort of, again, it wasn't the best analogy this time, but the self-driving car analogy that kind of came up at the beginning of the end, right? But the idea that, like, you don't need someone to build the car.

You need someone to drive the car, right? Or you need someone to be able to navigate the car. I think that's the important aspect here.
And I get at that point it's not a self-driving car. That's not sold on their analogy this time, right? But the general idea is, like, you don't need the technical expertise to make a car work.
I have no idea how a car works. I'm not a mechanical person at all. If something breaks, I'm out of luck.

But I can drive the car where I need to go because I've been trained in that. And I think the same idea does sort of apply here where it's not complex code that delivers the value. It's an understanding of what the use case is and how it's governed and the value that it creates for your members and being able to steer that.
So being able to have a human being sort of as an owner participating as part of the system to coach it, right? That idea of, okay, when this question comes up, this is the kind of way we want to respond. Or these are the source documents that are available to this use case, right?

Someone who owns that and someone who is actively engaging, at least on the front end of this, with the AI to coach it up so that the system learns and repeats those lessons going forward. That's how the governability is enforced. It's enforced by having an owner who's clearly understanding the use case, who can clearly explain it in natural language, not code, to the AI, and then maintaining it so that the AI enforces those rules that it learns through the interaction over time.

[Rob Barnes] (33:16 - 33:56)
That's a really important point and a lens to put over, even just the use of, the very basic use of the readiness worksheet as part of today is to kind of go into it with that lens. I think that's a layer that I'm really glad that you kind of wrapped us up today with that. So one more time before you go today, listeners, download the implementation readiness worksheet from the show notes.

It's the fastest way to put today's episode to work. Thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question.

How do we do this without burning out our staff?

What this helps your team answer

Why the real barrier for most associations isn't technical skill but organizational agreement — knowing your use case, your approved sources, and who owns corrections and coaching.
How to right-size scope so a pilot is governable and concrete without being so small it loses momentum or so broad it never launches.
Why the owner of an AI use case doesn't need to code — they need to clearly explain the use case, the source material, and the rules in plain language, then coach the system as it learns.

Companion Asset

Implementation Readiness Worksheet

Team Capacity

How Do We Do This Without Burning Out Our Staff?

A candid look at why "we're at capacity" is often a vibes claim, not a data point — and a framework for spotting the hidden ongoing cost of managing AI so it reduces work instead of quietly creating a second job for your team.

30:36

0:00 30:36

[Thomas Altman] (0:07 - 0:11)

Today's big question, how do we do this without burning out our staff?

[Rob Barnes] (0:12 - 1:01)

We are Thomas Altman and Rob Barnes, and this is episode six of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the Staff Capacity Impact Worksheet. It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work.

Thomas, one of the things that I picked up from this episode and from our AI hosts was this notion that if anyone is talking about introducing AI into their organization and someone is saying that this is not going to create new work for the team, it's not being realistic with what's coming. But there's some layers to this, right? There's some nuance here that the episode explores that we really want to make sure that listeners are picking up on, right?

[Thomas Altman] (1:02 - 3:50)

Exactly, and I think the important caveat here is new work doesn't have to be extra work, right? And I think the way that organizations navigate that central theme is what will lead to a successful implementation of AI, whatever AI you're implementing, versus one that burns out your staff, right? And the way that this episode, I think, really does a good job of unpacking is there's basically, there's more than this, but there's basically three buckets of work kind of when you're adopting an AI project.

And it's something we've seen kind of happen over again, and the people that sort of manage that systematically and thoughtfully on the front end do a good job of reducing stressful workloads associated with adopting a new system. So what do we mean by that? Where does this episode go that I think is really important to kind of key into as you're listening?

I think first, this idea of kind of your current jobs to be done, like what are your existing workflows that are underlying from what you do, right? What are the things that are currently done, right? And where does AI intervene?

Like what of that can, I think they call it the robot take on, right? What can the AI assume and replace, right? What are the menial sort of tasks that were kind of, if you've listened to the episodes up to this point, add in governance and trust that the AI can do that and manage this part of the process pretty well, right?

So you're going to identify what the work is, where the robots can step in. And then what you're going to find, and the thing that's often hidden is not only are you kind of doing a lot of the setup work, but there is an ongoing cost to managing those robots, right? If you're replacing some of your work, you need to be, you know, active around making sure that's being done.

But the thing that you get on the other side is there is also work that should never be touched by humans, I mean, by robots, that should be only for humans. Anything that requires relationship building, that requires human judgment, which requires, you know, interpretation, interpretation. Yeah, exactly.

And then that what we see that people do well, and we'll kind of get into this a little bit more, but like, moving people whose time has been freed up from the menial into the value add, right into the fulfilling into the stuff that actually has a meaningful impact on the lives of others and the mission of the organization as a whole. And they can see that they played a part in that. And it wasn't just keeping the lights on by filling out a spreadsheet.

Like, once you can kind of start to replace those, yes, you have new work, you hopefully don't have extra work, and you're able to claw off people from doing menial stuff that's sort of soul-killing into stuff that's a little bit more fulfilling.

[Rob Barnes] (3:50 - 4:52)

One of the things I've learned in the conversations I'm having with the executives, and this brings it up again, is there is a primary, it's almost, the internal use case is the one that comes to the fore more. My staff are already at capacity, how is this going to help them gain some capacity back? But we're not quite sure what the measurement of that is, because we don't, typically associations, and even when I ran the associations that I was running, back in the day now, the idea of trying to measure time and impact of your team, it's just not part of the culture of running an association typically.

It's not like running a software services business where you're accounting for every hour so you can bill it out. That typically doesn't happen, and so it's challenging to put an ROI calculator together without some high level of hypothesis around these things. When we talk about it from a Betty perspective now, I also see it's the value creation around the tool that is designed specifically for members.

[Thomas Altman] (4:54 - 5:20)

And with that, over to our AI hosts.

[AI Host 2] (5:24 - 5:37)

So here is a pretty brutal truth about that quote-unquote magical tech tool your leadership just bought to save you time. Oh boy, here we go. Right.

It is probably going to create a second job for you.

[AI Host 1] (5:37 - 5:45)

It's the great irony of modern technology. I mean, we buy these tools to buy back our time, and we just end up spending all our time managing the tools.

[AI Host 2] (5:45 - 6:00)

Exactly. And you know, if you sit in on enough association board meetings or like those big executive leadership huddles, there's always this one question that's very specific, really heavy concern that just sucks the air right out of the room when a new initiative is proposed.

[AI Host 1] (6:00 - 6:02)

Yep. Someone always leans forward.

[AI Host 2] (6:02 - 6:09)

Always. And it's usually the person who, you know, actually manages the day-to-day operations. And they ask, how do we do this without burning out our staff?

[AI Host 1] (6:10 - 6:17)

It is the ultimate showstopper question. Because the people asking it, well, they have the scars to prove why it matters. They really do.

[AI Host 2] (6:17 - 6:29)

Honestly, that is our mission for today's deep dive. We are unpacking a whole stack of operational frameworks and implementation analyses that are entirely focused on governed AI and associations.

[AI Host 1] (6:29 - 6:39)

Right. Because the goal here is to figure out how to actually responsibly implement AI so it reduces work instead of, you know, just creating that second job for everyone.

[AI Host 2] (6:39 - 6:44)

Because let's validate what you, the listener, are probably feeling right now. You've been burned before.

[AI Host 1] (6:45 - 6:45)

Oh, absolutely.

[AI Host 2] (6:45 - 6:58)

You've seen new tech tools that were promised as the ultimate time savers. And what actually happened? They just created more administrative tasks, endless weekly alignment meetings, and this highly ambiguous ownership structure.

[AI Host 1] (6:59 - 7:05)

Suddenly, a tool that was meant to save time requires a whole committee of like three people just to keep it from breaking.

[AI Host 2] (7:05 - 7:18)

Right. So when you hear your team say they're at capacity, I mean, it's real. But here is the problem with how we typically handle that phrase.

Saying your staff is at capacity, it often functions as a vibes claim.

[AI Host 1] (7:18 - 7:19)

A vibes claim.

[AI Host 2] (7:19 - 7:20)

Yeah.

[AI Host 1] (7:20 - 7:21)

That is painfully accurate.

[AI Host 2] (7:21 - 7:30)

It's just this feeling in the room. It's like saying, you know, everyone is feeling really stressed out lately. It might be 100 percent true, but vibes don't fix workflows.

[AI Host 1] (7:30 - 7:31)

No, they don't.

[AI Host 2] (7:31 - 7:39)

You cannot operationalize a vibe. If a pipe is bursting in your house, you don't talk about the vibe of the water pressure. Right.

You find the shutoff valve.

[AI Host 1] (7:39 - 7:51)

What's fascinating here is that to actually solve this, to find that shutoff valve, we have to reframe the entire conversation scientifically. Like, staff capacity cannot remain a vibes claim.

[AI Host 2] (7:51 - 7:52)

It needs real metrics.

[AI Host 1] (7:52 - 8:05)

Exactly. It requires a quantifiable baseline. It requires a bounded workflow, clear owners, and a strict measurement plan.

So when leaders approach this, the responsible question isn't, how do we make our staff adopt this new AI?

[AI Host 2] (8:06 - 8:07)

Which is what everyone asks.

[AI Host 1] (8:07 - 8:12)

I know, but that is entirely the wrong starting point because it just assumes the AI itself is the goal.

[AI Host 2] (8:12 - 8:14)

The tool becomes the master.

[AI Host 1] (8:14 - 8:29)

Right. The right question, the truly responsible question is, which specific repetitive load are we removing and what human work are we fiercely preserving? And honestly, if you cannot articulate both halves of that sentence with crystal clarity, you're not ready to implement anything.

[AI Host 2] (8:29 - 8:40)

That's such a good point. So if the goal is preserving human energy, why do we keep getting it so wrong? Because, you know, leadership almost always falls for the classic bait and switch.

[AI Host 1] (8:40 - 8:40)

Yeah, they do.

[AI Host 2] (8:40 - 8:55)

Let's talk about how implementations usually fail the actual humans running them. The organization sells this new AI tool to the team as this ultimate staff relief. Like, good news, everyone, the robots are here to do your busy work.

[AI Host 1] (8:55 - 8:56)

We've all heard that speech.

[AI Host 2] (8:57 - 9:04)

But in reality, what they've just done is assign the staff a brand new, completely unmanaged system to babysit after hours.

[AI Host 1] (9:04 - 9:15)

And that babysitting is what the sources call the hidden implementation load. When leadership just throws a tool at a team without changing the underlying workflow, they haven't actually removed any work.

[AI Host 2] (9:15 - 9:16)

They just shifted it around.

[AI Host 1] (9:16 - 9:31)

Worse than that, they've mutated the type of work the staff is doing and usually made it significantly more frustrating. So to avoid this, our sources point to a very specific, grounded framework. We have to separate the staff workload into three distinct categories.

[AI Host 2] (9:31 - 9:40)

Okay, let's unpack this. Paint a picture for us. What does this actually look like for, say, an association professional on a random Tuesday morning?

[AI Host 1] (9:41 - 9:53)

All right, let's imagine a senior membership director. They're currently on a Zoom call trying to coach a longtime member through a really difficult, nuanced mid-career transition.

[AI Host 2] (9:53 - 9:53)

Okay.

[AI Host 1] (9:54 - 10:08)

Now that is highly valuable, deeply human work. But while they're on that call, their inbox is pinging every four minutes. Where is the link to the hotel block for the annual gala?

How many CEU credits do I need to renew my credential? I can't remember my password.

[AI Host 2] (10:08 - 10:09)

Oh, the endless pinging.

[AI Host 1] (10:10 - 10:23)

Exactly. That endless pinging is category one, the AI-suitable load. This is the highly repetitive, predictable stuff.

Routine member questions, findability problems where someone just can't locate a PDF on the website, policy lookups, and basic event logistics.

[AI Host 2] (10:23 - 10:30)

It's the endless triage. I mean, the questions where the answer is literally on page four of the member handbook, but nobody reads page four.

[AI Host 1] (10:30 - 10:31)

Nobody ever reads page four.

[AI Host 2] (10:31 - 10:36)

And it requires absolutely zero human empathy to tell someone the keynote speech starts at 9 a.m., right?

[AI Host 1] (10:36 - 10:46)

Precisely. That load is perfectly suited for governed AI. Now, category two is where the real danger lies.

This is the hidden implementation load.

[AI Host 2] (10:46 - 10:47)

The babysitting work.

[AI Host 1] (10:47 - 11:09)

Yes. This is all the internal work required to keep the new AI tool functioning. It's the time spent formatting documents so the system can read them, figuring out source ownership, meaning who actually updates the policy document when the rules change.

Oh, that's a big one. It's huge. And it's also establishing review cadences and managing the launch communication to your members.

[AI Host 2] (11:10 - 11:21)

So if category one is the robot's job and category two is the very real work of managing the robot, then category three has to be the actual human empathy, like the high-level strategic stuff, right?

[AI Host 1] (11:21 - 11:33)

That's it. Category three is the human value load. This is the work we fiercely protect.

It is interpretation. It is judgment. It's relationship building, member coaching, complex content improvement, and strategic decisions.

[AI Host 2] (11:33 - 11:34)

The stuff a machine just can't do.

[AI Host 1] (11:34 - 11:47)

Exactly. AI cannot build a relationship with that member who is struggling with a career transition. AI cannot look at political trends and decide which legislative policy your association should advocate for next year.

[AI Host 2] (11:48 - 11:51)

You know, it reminds me of a triage nurse at an emergency room.

[AI Host 1] (11:51 - 11:51)

Oh, I like that.

[AI Host 2] (11:52 - 12:12)

Right. The governed AI is the triage nurse. It handles the intake.

It points people to the right waiting room. It answers the basic questions about visiting hours. And by absorbing all of that, the actual doctors and specialists, your staff, can dedicate all their focus to the complex surgeries and patient care, which is that human value load.

[AI Host 1] (12:12 - 12:25)

That is a perfect analogy. The goal isn't to replace the doctors. It's to give them incredible leverage.

They spend less time being, you know, human search engines and more time doing the work that actually requires a human pulse.

[AI Host 2] (12:26 - 12:40)

Because we are talking about visualizing and separating this workload, this is the perfect time to mention a companion asset we have for you in the show notes. OK, let's unpack this. If you're driving or walking the dog, make a mental note to check the show notes later for the staff capacity impact worksheet.

[AI Host 1] (12:40 - 12:41)

It's such a crucial tool.

[AI Host 2] (12:41 - 13:00)

It really is the essential tool for exactly what we are talking about. It walks you through how to map out your current load, identify exactly what belongs in that AI-suitable category, and strictly define your human-only work. It's the bridge that takes us from a vibes conversation to an actual concrete operational plan.

[AI Host 1] (13:01 - 13:12)

And it forces clarity. Because without that worksheet, without doing the really hard work of mapping those three categories, you are just guessing. And guessing is how you accidentally build a second job for your team.

[AI Host 2] (13:12 - 13:22)

OK, so knowing what to map on a worksheet is great. But how do we prove this actually works in the real world? How do we know we aren't just adding to that hidden implementation load we just talked about?

[AI Host 1] (13:23 - 13:41)

We have to look at the mathematical truth of a healthy implementation. It requires a very careful proof of concept. The equation here is simple, but it is ruthless.

The AI-suitable load that's Category 1, the repetitive questions it must shrink by a significantly larger margin than the hidden implementation load adds to the staff's plate.

[AI Host 2] (13:41 - 13:53)

So, if the AI saves you 10 hours a week of answering routine emails about event parking, but it takes your team 12 hours a week to review the AI's logs and update the software, you've failed.

[AI Host 1] (13:53 - 13:59)

You've failed entirely. You have decreased your staff's capacity. You've built a liability, not an asset.

[AI Host 2] (14:00 - 14:24)

OK, let's ground this with a practical example from the source material. The analyses frequently reference Betty. And they don't talk about Betty as some, you know, sci-fi, all-knowing magic wand.

Betty is presented as a very grounded, association-specific governed AI. And I want to pause on that word, governed. For anyone who isn't living in the tech world day-to-day, what makes a governed AI different from just opening up chat GPT?

[AI Host 1] (14:25 - 14:40)

It is the most important distinction we can make. Governed means it exists inside a walled garden. A governed AI like Betty is entirely source-bound.

It is physically incapable of going out to the open internet, reading some random Reddit thread, and hallucinating a creative answer for your members.

[AI Host 2] (14:40 - 14:41)

Which is terrifying for associations.

[AI Host 1] (14:42 - 15:02)

Exactly. It only activates the existing, approved knowledge that you explicitly hand to it. It strictly reads your association's approved handbooks, your specific policies, and your official event guides.

Every single answer it gives is completely traceable back to a source document you control.

[AI Host 2] (15:02 - 15:26)

Wait, let me stop you right there. Because if Betty is just generating answers based on our PDFs, surely a human still has to proofread every single one of those answers before it goes out to a member, right? I mean, we can't just let a system speak for the association unmonitored.

But if I have to read the AI's answer, verify it against the PDF, and then hit send where's the time savings, aren't we just trading the time spent writing an email for the time spent editing a robot's draft?

[AI Host 1] (15:26 - 15:33)

That is the smartest, most common, and most valid pushback we see from senior staff. If I have to check its work, I might as well do it myself.

[AI Host 2] (15:33 - 15:35)

Right. It feels like double the work.

[AI Host 1] (15:35 - 15:47)

It does. And the answer to that pushback comes down to establishing a rigorous operating rhythm. If you don't build this rhythm into your daily operations, you absolutely will dump hidden work onto your staff, and they will hate the tool.

[AI Host 2] (15:48 - 15:54)

Okay, lay out this operating rhythm. How do we trust this system without micromanaging it to death?

[AI Host 1] (15:54 - 16:07)

It's a structured five-step rhythm. Step one, establish a baseline before you launch. You do not launch AI for your entire association at once.

You pick one single workflow with a high repeat volume.

[AI Host 2] (16:07 - 16:07)

Give me an example.

[AI Host 1] (16:07 - 16:17)

Let's say the annual certification renewal process. You baseline exactly how much time staff currently spends answering emails about that one specific process.

[AI Host 2] (16:17 - 16:18)

Okay, so you put a fence around it. Step two.

[AI Host 1] (16:19 - 16:21)

Naming source owners and escalation paths.

[AI Host 2] (16:22 - 16:22)

Yeah.

[AI Host 1] (16:22 - 16:28)

Because it is a governed AI, there will be questions it simply does not know the answer to, because the answer just isn't in your documents.

[AI Host 2] (16:28 - 16:28)

Right.

[AI Host 1] (16:29 - 16:39)

So when Betty hits a wall, who does that question escalate to? Is it the membership inbox? Is it a specific coordinator?

You must define exactly who handles the exceptions. There can be no ambiguity.

[AI Host 2] (16:40 - 16:47)

Right, because if it's ambiguous, it just sits in a general inbox, a member gets angry, and it creates massive internal anxiety for the staff.

[AI Host 1] (16:47 - 17:03)

Exactly. Now, step three is the crucial one. This directly answers your pushback about having to proofread every single email.

Step three is the review cadence. The key here is that staff must review answer patterns, not individual responses.

[AI Host 2] (17:03 - 17:13)

Okay, slow down on that. Reviewing patterns instead of individual responses. Explain the actual mechanism of that.

How does an association professional actually do that on a Friday afternoon?

[AI Host 1] (17:13 - 17:22)

Think about how you manage a brilliant new human employee. In their first week, you might ask to be CC'd on every email they send, just to make sure they have the right tone.

[AI Host 2] (17:22 - 17:22)

Sure.

[AI Host 1] (17:22 - 17:39)

But by month two, doing that would be insane. You'd never get your own work done. Instead, you look at their overall performance.

With a governed AI, you aren't reading a transcript of every single chat it had with every single member. Instead, you log into a dashboard and you look at group data and word clusters.

[AI Host 2] (17:39 - 17:42)

So the system is categorizing the interactions for you.

[AI Host 1] (17:42 - 17:58)

Yes. You log in, and the dashboard shows you, hey, out of 500 interactions this week, 70 people asked about the new late fee policy. And the AI flagged those conversations with a low confidence score.

You aren't reading 70 emails. You are reading a single trend.

[AI Host 2] (17:58 - 17:59)

That makes so much more sense.

[AI Host 1] (17:59 - 18:06)

The pattern tells you where the friction is. Is it consistently struggling with the late fee questions? If yes, that leads directly to step four.

[AI Host 2] (18:06 - 18:08)

Which is the correction and coaching loop.

[AI Host 1] (18:08 - 18:26)

Yes. You need a mechanism for staff and subject matter experts to coach the AI. But here's the secret.

Because a governed AI is source bound, coaching it rarely means reprogramming the bot. It almost always means updating your own messy documents.

[AI Host 2] (18:26 - 18:35)

Oh, wow. That is fascinating. So if the bot gives a confusing answer, it's not because the bot is stupid.

It's because the bot is accurately reading a confusing document.

[AI Host 1] (18:36 - 18:47)

Precisely. Human beings are great at navigating bad documents. Yeah.

If we read a PDF from 2019 that contradicts itself, we use human intuition to figure out what it probably means. And AI doesn't do that. It takes it literally.

[AI Host 2] (18:47 - 18:48)

It's incredibly literal.

[AI Host 1] (18:48 - 19:02)

Right. So if the AI is confused, it's holding up a mirror to your content. To coach the AI, you just have to go rewrite that paragraph in the 2019 PDF, so it actually makes sense.

You upload the clean version and the AI instantly gets smarter.

[AI Host 2] (19:03 - 19:14)

It exposes the cracks in your knowledge management. We love to blame the technology, but really, our foundation was just cracked. Which I imagine leads perfectly into the final step of the operating rhythm.

[AI Host 1] (19:14 - 19:22)

Yes. The Insights Review. This is the goldmine.

This is where you use the system to identify content gaps and surface demand signals from your members.

[AI Host 2] (19:22 - 19:23)

How does that work?

[AI Host 1] (19:23 - 19:38)

You look in the dashboard to see what members are asking that the AI couldn't answer because the source material didn't exist. That data tells your marketing and content teams exactly what they need to write next week. You stop guessing what members want, and you just look at what they are actively searching for.

[AI Host 2] (19:38 - 19:49)

Okay. So that operating rhythm baselining, establishing escalation paths, reviewing patterns instead of emails, coaching the source documents and pulling insights, that is the mechanism that keeps the math positive.

[AI Host 1] (19:49 - 19:49)

That's the engine.

[AI Host 2] (19:49 - 20:04)

That is how we ensure the AI saves more time than it costs to manage. But, you know, an operating rhythm is just a process on paper. It only survives if the team culture actually supports it.

Which brings us to the human element, change management.

[AI Host 1] (20:04 - 20:09)

The invisible rock upon which so many brilliant tech implementations break.

[AI Host 2] (20:09 - 20:19)

Let's talk about a massive pitfall the sources highlight regarding how teams design adoption. It's the trap of the enthusiastic staffer. We have all seen this play out.

[AI Host 1] (20:19 - 20:20)

In every organization.

[AI Host 2] (20:20 - 20:32)

Right. One person on the team goes to a tech conference, reads all the articles, and gets highly caffeinated and excited about AI. They bring it back to the office.

They champion it. They build the workflows. They set it all up.

And what happens to them?

[AI Host 1] (20:32 - 20:49)

They become the permanent bottleneck. And honestly, they are punished for their enthusiasm. If only one person on your staff knows how to review those patterns, or only one person knows how to update the source documents when a policy changes, you haven't built a sustainable system.

You have just built a massive dependency.

[AI Host 2] (20:49 - 20:58)

And that enthusiastic person is going to be answering Slack messages at 9pm on a Saturday because, quote unquote, the bot is acting weird.

[AI Host 1] (20:58 - 21:13)

And they will burn out rapidly. The system quickly turns into hidden labor for them. And when they eventually go on vacation or take a new job at a different association, the entire AI implementation grinds to a complete halt.

It becomes abandoned software.

[AI Host 2] (21:13 - 21:14)

And it just dies on the vine.

[AI Host 1] (21:14 - 21:32)

Completely. Adoption and source ownership must be designed across the entire team from day one. It has to be distributed.

The membership director owns the membership sources. The events coordinator owns the event sources. The education team owns the certification PDFs.

It cannot be siloed into one single AI person.

[AI Host 2] (21:33 - 21:48)

So what does this all mean? We've talked about worksheets, operating rhythms, triage nurses, and distributed ownership. Let's ask the big question.

Does a governed AI just magically fix a stressed out office? Are we saying this is the cure for the modern association's workload crisis?

[AI Host 1] (21:48 - 21:57)

I think we need to deliver a very grounded, deeply honest beat right now. We have to be incredibly careful with the word burnout. That word is thrown around way too casually in tech marketing.

[AI Host 2] (21:57 - 21:58)

Let's be real about it.

[AI Host 1] (21:58 - 22:18)

Tools like Betty, these governed AI systems, they can radically reduce specific tangible workload vectors. They're fantastic at eliminating repetitive triage. They eliminate the friction of internal search.

They give your staff incredible visibility into content gaps. But they absolutely do not solve burnout caused by deeper institutional issues.

[AI Host 2] (22:18 - 22:21)

Say more about that. What can't a governed AI fix?

[AI Host 1] (22:22 - 22:33)

AI cannot fix burnout caused by a toxic leadership culture. It cannot fix institutional uncertainty, where your board of directors keeps changing the strategic direction every six months and giving the staff whiplash.

[AI Host 2] (22:33 - 22:34)

Which happens all the time.

[AI Host 1] (22:34 - 22:50)

Exactly. It cannot fix chronic understaffing. If you have a team of three people trying to do the work of 10 people, AI is not going to suddenly make that workload sustainable.

And it certainly cannot fix poor strategic priorities. AI is simply a layer that activates your existing knowledge. It cannot fix a broken culture.

[AI Host 2] (22:51 - 23:11)

That is such a vital distinction to make. It is the difference between treating a symptom and treating the disease. Governed AI is essentially an incredibly efficient pain tiller for a very specific type of operational pain, the pain of repetition and findability.

But it is not a cure for a broken leg. Not a cure for structural dysfunction.

[AI Host 1] (23:12 - 23:25)

Exactly. And selling it to your team as a magical cure-all is a guaranteed way to lose their trust before you even launch. You have to be deeply honest with your staff about what load you are removing and what load will always remain human.

[AI Host 2] (23:25 - 23:43)

That brings us right back to our core thesis for this deep dive. Let's recap this. AI should reduce specific work, not create a second job.

The responsible implementation question isn't, how do we get our staff to use AI? It's what operating rhythm keeps the system healthy without dumping hidden work onto our people.

[AI Host 1] (23:43 - 23:46)

It is entirely about preserving human energy for human value.

[AI Host 2] (23:46 - 23:56)

Which is why I want to point the listener back to the show notes one more time. Go grab the Staff Capacity Impact Worksheet. This isn't just a fluffy PDF download.

It is a practical tool.

[AI Host 1] (23:56 - 23:57)

That's where you start.

[AI Host 2] (23:57 - 24:15)

Use this worksheet to choose just one workflow. Don't try to boil the ocean and reinvent your whole association by Friday. Pick one area.

Baseline the burden. Define exactly what should be deflected to a governed AI. And rigorously protect your staff's time for higher value work.

Start there.

[AI Host 1] (24:15 - 24:33)

If we connect this to the bigger picture, consider this one last thread to pull on. We spend all this time worrying about whether the AI is giving the perfect answer to our members. We worry about the wording and the tone.

But what if the greatest value of governed AI isn't actually answering your members' questions at all?

[AI Host 2] (24:33 - 24:36)

Wait, if it's not answering questions, what is it doing?

[AI Host 1] (24:36 - 24:55)

What if its highest purpose is acting as capacity intelligence? Think about it. What if the most valuable thing the system does isn't deflecting emails, but showing you exactly what your members are trying and failing to find?

What if the real return on investment is finally having a map that allows you to fix the root of the friction rather than just endlessly managing the symptoms?

[AI Host 2] (24:55 - 25:08)

Wow. Think about your own workflows tomorrow morning. What if the repetitive questions you are so tired of answering are exactly the map you need to fix the member experience once and for all?

We will leave you to mull that over.

[Rob Barnes] (25:14 - 26:38)

So the recap today, the big question was, how do we do this without burning out our staff? And of all the episodes so far, Thomas, I feel like the companion asset today has some real practical value. And I want to start there.

The Staff Capacity Impact Worksheet being applied to a particular workflow, a business process, we know a lot of them in associations. We have plenty of them around membership and marketing and events and advocacy that they live there. Applying this is super important because one of the things that our AI hosts talked about is if a new AI capability, a tool, saves us 10 hours a week of answering routine emails about our conference, as an example, but it takes 12 hours a week to review the logs and update the software to keep it on task, then there's a failure there because the math just doesn't math.

And I'd love to see more people doing this kind of math in advance of procuring AI or looking at applying and doing some experimentation. To get a real sense of, I think, what everybody was listening to in the episode that we were to is about this hidden work, this hidden layer of work.

[Thomas Altman] (26:40 - 29:01)

Yeah, no, exactly. I think the work, the thing that I've seen differentiate AI implementations that have added work to a staff's plate, right, like that creates a problem versus the ones that actually are more successful in reducing workload, or at least workload of menial tasks that are kind of soul draining, right, is thinking through and being aware that, yes, you may be replacing certain workflows, like doing the analysis of what workflows exist, where can AI fit in, what new work can be done that couldn't be done, and we can use AI for that. Understanding that part's very, very important.

But a lot of times what's hidden is the work of managing that in an ongoing way. Everyone kind of thinks of the work to get it up and running in the first place. That's usually not hidden.

What is hidden is how do we kind of manage this system going forward, right? And when that work is greater than the work saved, or even if it's the same, but more kind of soul draining, you're in a lose-lose situation, right? You're no longer kind of freeing up these people to do the human value work.

And I think that's the important part here. That's the thing that I want people to think about is first, what are the workflows, right? Think of the jobs to be done.

It's a framework we use internally a lot of the times that your AI system can do. Where does an AI doing that save your people time, right? And then once you understand that, think deeply about what it takes to manage that system on an ongoing basis and do the math there, right?

That's make the unhidden, make the hidden unhidden. Make the hidden transparent and available for everybody to look at so that you can actually be pretty proactive around reducing time on those other tasks, not overcompensating with the time managing system. And then repurposing those people.

I love this line kind of towards the end where it was like, it's not trying to get people to use AI. It's creating an operating system that keeps the system healthy without dumping hidden work onto our people. It's about preserving human energy for human value.

That idea, that idea of taking the human energy and using it to create human value without creating extra work. You'll know that better if you do this work ahead of time. And it's really important.

[Rob Barnes] (29:02 - 30:29)

And that culture within which the AI needs to live within becomes incredibly important as well. A governed AI, a well-applied, a well-implemented AI is still not going to address the very human issues of stuff, capacity and stuff, capability and stuff, understaffing, poor strategic alignment to priorities. It's not going to fix all of that.

You're right, that quote that you like so much about creating human value really could be applied much more broadly across the culture of an organisation than just as it relates to AI. And that's the challenge, right? I think for our listeners is to really look into the root cause of some of the, perhaps it's an inefficiency or it's the vibe of staff capacity, it's those sorts of things, look at it from a value perspective first, and then arrive back at some of the solutions that will help create that environment in that particular culture at that time will become self-evident.

And that form follows function kind of approach seems to be very applicable here. So thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question.

We just bought a new AMS, why now?

What this helps your team answer

Why new work from AI doesn't have to mean extra work — and the three buckets to sort your workflows into: what AI can take on, what it takes to manage that ongoing, and what should stay entirely human (relationship-building, judgment, interpretation).
The hidden cost most teams miss: the ongoing work of managing and reviewing an AI system can outweigh the time it saves, so the math has to be done before you implement, not after.
What governed AI genuinely can't fix — toxic culture, chronic understaffing, shifting strategic priorities — and why honesty about that upfront builds trust instead of setting your team up to feel burned again.

Companion Asset

Staff Capacity Impact Worksheet

AMS vs. AI Layer

We Just Bought a New AMS — Why Would We Take On an AI Project Now?

Why "we just spent millions on a new AMS" isn't a reason to wait on AI — a framework for separating your system of record (AMS/CMS/LMS/FMS) from your system of knowledge (the AI layer), and how to decide if now's the time or if you need to stabilize first.

31:33

0:00 31:33

[Thomas Altman] (0:04 - 0:09)
Today's big question, we just bought a new AMS, why would we take on an AI project now?

[Rob Barnes] (0:10 - 1:06)
We are Thomas Altman and Rob Barnes and this is episode seven of the Association Intelligence Podcast. Before we get into today's episode, the companion asset is the AI layer versus platform decision brief. It's linked in the show notes, so grab it now if you want to follow along or download it after and put it to work.

Thomas, the episode's central distinction here is the idea of a system of record, an association management software, colloquially known to all of us as an AMS, but for this purpose, CMS, LMS, financial management system, we'll just talk about it a little bit more broadly, versus a system of knowledge, this idea of an AI layer. Is that analogy strong enough to survive, particularly the skeptical CFO who's probably just spent $200,000 or even $2 million on a new association management software?

[Thomas Altman] (1:07 - 2:56)
I don't know if any framework is strong enough to survive that CFO, but I do think it has a lot of merit to it. The idea of a system of record versus a system of knowledge is a very key distinction and I think increasingly you're going to need both and you're record is going to be important. AI is never going to be your system of record.

AI is going to be the interface governed by that system of record. What we mean by that and the way we see this kind of being super important is that your AMS, for example, it's going to have information, transactional information around what level of membership does a individual have right now. Based off of that level of membership, including not a member at all, what permissions do they have to access certain levels of content?

How should they be treated differently based off of what kind of member they are? Are they going to be getting full access to the journals or use of the standards or are they going to be sort of coached to join or take a class? All of these things are system of record that the AI should be aware of and should govern how the AI interacts with their beforehand.

So it is important to have both and I think if you've just gone through a rough implementation, it might make sense to sort of take a deep breath, stabilize that before you take on a new project, but it doesn't mean you don't do it. Or if you're kind of judging between two, there is a strong argument to be made for putting the AI system in front because those interactions will actually govern what the AMS, the FMS, the LMS should do behind the scenes. So that's kind of how I saw this happening.

[Rob Barnes] (2:57 - 4:25)
It reminded me a little of the pre-AI era where federated search kind of came in as an idea that I can search for stuff, mostly content, not so much data, but across multiple platforms and bring it together kind of in one place for a user, either an internal user or an external user. These days it's more, I guess we then morphed into these ideas that we used to have people talked about data lakes being a central repository of all the stuff coming from the different management systems. And now there is the advent, of course, of the AI-enabled CDPs and what they can do because they are ready and they were purpose-built for the AI layer to live where the user needs to live to get across all of those kind of systems of record.

And still, you're right, the readiness of an organization who has just done, in some cases, pretty troubling either AMS migration or in some cases a CMS migration for a website, there is a point where a really mindful or thoughtful perhaps AI vendor might say, you know what, talk to us again in six months time or eight months time or nine months time. And there's very good reasons for doing that, right?

[Thomas Altman] (4:26 - 5:53)
Yeah, absolutely. I think making sure, again, we're all about governed AI here, right? So and the system of records are going to be the source from which AI is governed and its ability to connect to any of your MSs, your management systems, is going to inform the way that the AI kind of works.

And the important distinction here, I think, is making sure that the governing piece is available because that's going to lead to AI success, right? Making sure that that is correct and working how you want it to, but not depending on that to sort of revolutionize your member engagement strategies, right? That is probably not going to happen.

Your AMS may have some bells and whistles and may do this or that different from the previous one, but the actual experience of interacting with your association as a member probably fundamentally won't change. That is not necessarily true of the AI. So if your goal isn't to build up the member engagement strategies and different avenues and interfaces to that governed by these transactional databases, which are any of your management systems, I think that combination is what really matters.

So yes, it does make sense if you've just gone through a nightmare AMS implementation. I've been there myself. And the last thing you want to do is throw a new tech project on top of your staff.

Yeah, take a breath, stabilize, make sure it's working the way you want it to, but don't use it as an excuse to kick the can down the road. You're going to want to do it sooner rather than later.

[Rob Barnes] (5:54 - 6:15)
So before we hand this over to our AI podcast host, quick reminder that the following is generated using Notebook LM with heavy involvement from Thomas, myself, and our Betty team to ensure that the content generated is valuable to all of our listeners. For a deeper dive on our approach, please go back and listen to episode zero. And now over to the AI hosts.

[AI Host 1] (6:19 - 6:30)
Imagine this scenario for just a second. You've just spent two years and easily two million dollars completely overhauling your association's tech stack.

[AI Host 2] (6:30 - 6:31)
Oh man, the classic nightmare.

[AI Host 1] (6:32 - 6:45)
Right. You've survived like 18 months of painful data mapping. You've lived through the late nights, eating cold pizza in some random conference room while your team tried to figure out why a thousand member records just didn't port over correctly.

[AI Host 2] (6:45 - 6:46)
Yeah, pulling your hair out.

[AI Host 1] (6:46 - 7:02)
Exactly. But you did it. The launch party is over.

The congratulatory emails have gone out. But then the very next day, a member logs onto your shiny new portal, types a highly specific urgent question into the search bar, and they get, well, absolutely zero results.

[AI Host 2] (7:02 - 7:06)
Or worse, right? They get a disjointed list of 10 totally irrelevant links from like 2018.

[AI Host 1] (7:06 - 7:17)
Yes. It is the ultimate nightmare scenario, but it's happening every single day. And that exact disconnect is what brings us to the core of what we were unpacking in today's deep dive.

[AI Host 2] (7:17 - 7:26)
Because if you are sitting in a leadership role at an association right now, evaluating your tech stack, you have absolutely heard this exact objection echoed in your board rooms.

[AI Host 1] (7:27 - 7:34)
Oh, 100 percent. And the exact objection is this, quote, We just bought a new AMS. Why now? End quote.

[AI Host 2] (7:35 - 7:45)
It is the single most common and honestly, the most understandable reaction you can get from an executive team or a board of directors. I mean, we have to validate that feeling immediately.

[AI Host 1] (7:45 - 7:46)
Yeah, they're just exhausted.

[AI Host 2] (7:46 - 8:04)
Totally exhausted. Association leaders are suffering from this profound platform spend fatigue right now. They've got endless vendor evaluation cycles, implementation burnout.

The psychological toll of a massive association management system migration is just immense.

[AI Host 1] (8:04 - 8:14)
So immense. You finally cross the finish line of this marathon. You are bruised.

Your IT team hasn't slept a full weekend in six months. And then someone brings up adding an AI knowledge layer.

[AI Host 2] (8:14 - 8:23)
Right. When you've just dragged an entire organization through reengineering every operational workflow, the absolute last thing anyone wants to hear about is another piece of technology.

[AI Host 1] (8:23 - 8:29)
The immediate reaction is going to be incredibly defensive. It's a highly legitimate concern to just raise your hands and say enough.

[AI Host 2] (8:29 - 8:34)
Exactly. So how do we reframe this? What does this all actually mean for you?

[AI Host 1] (8:34 - 8:53)
Well, the goal of this deep dive is absolutely not to ask you to buy another platform. We are not doing that today. Definitely not.

Instead, the question you should be asking yourself and your team is this. Did our new AMS actually change how members find answers across our association's entire knowledge estate?

[AI Host 2] (8:53 - 9:00)
And that is the precise pivot we need to make here. We are shifting the conversation away from traditional software categories.

[AI Host 1] (9:01 - 9:02)
Right. The typical buckets.

[AI Host 2] (9:02 - 9:14)
Exactly. And we're moving toward the concept of jobs to be done, because when you look at it through that lens, you realize the migration might have solved an operational crisis, but it left the member knowledge crisis completely untouched.

[AI Host 1] (9:15 - 9:25)
Yeah, untouched. And to help you navigate that exact shift and the sources we're looking at today, there is a really fantastic resource. It's called the AI Layer versus Platform Decision Brief.

[AI Host 2] (9:25 - 9:26)
Oh, that braid is gold.

[AI Host 1] (9:26 - 9:31)
It really is. And I want to mention this right up top, because it is essentially the roadmap for this entire conversation.

[AI Host 2] (9:32 - 9:42)
Yeah. You can use it to definitively decide what actually belongs in your AMS, what goes in your CMS, your LMS, your community platforms, and what operations belong in a governed AI layer.

[AI Host 1] (9:42 - 9:55)
Right. It maps out those jobs perfectly, so you aren't just guessing. So let's just start right there.

To understand why a new platform doesn't automatically mean better answers for your members, we have to look at the fundamentally different jobs these systems were hired to do.

[AI Host 2] (9:56 - 9:59)
Let's break down the actual job of an AMS. Yeah.

[AI Host 1] (9:59 - 10:00)
What is it actually doing?

[AI Host 2] (10:00 - 10:09)
The best way to look at an AMS is as a system of record. That is its whole identity. It is designed from the ground up to handle transactions.

[AI Host 1] (10:09 - 10:11)
Okay. So like processing payments and stuff.

[AI Host 2] (10:11 - 10:20)
Exactly. It manages the definitive state of your membership data. It's constantly asking binary questions, like, has this person paid their dues?

Yes or no?

[AI Host 1] (10:20 - 10:24)
Right. Are they up for renewal? Have they registered for the annual conference?

[AI Host 2] (10:24 - 10:33)
Yes. What specific geographic segment do they belong to? It handles all those complex operational workflows and serves as the central hub for integrations.

[AI Host 1] (10:34 - 10:40)
So it's essentially a massive, highly structured relational database optimized for operational truth.

[AI Host 2] (10:40 - 10:45)
That's a great way to put it. It is brilliant at what it does, but it is definitely not a brain.

[AI Host 1] (10:46 - 10:54)
Right. A system of record. Okay.

But the AI knowledge layer, the job we are talking about today, its job is completely different.

[AI Host 2] (10:54 - 10:55)
Entirely different.

[AI Host 1] (10:55 - 11:03)
It is a system of knowledge. It isn't trying to process a credit card for an event registration. Its job is to provide approved source answers.

[AI Host 2] (11:04 - 11:07)
Yeah. It retrieves information across the entire fragmented system.

[AI Host 1] (11:07 - 11:15)
And it understands audience access tiers, right? Meaning it knows how to differentiate what a staff member is allowed to see versus what a general member is allowed to see.

[AI Host 2] (11:15 - 11:27)
Exactly. Plus, it provides exact citations back to your source material. It allows for staff coaching of the AI.

And crucially, it gives you insights into the actual questions your members are asking in real time.

[AI Host 1] (11:27 - 11:31)
Okay. So to put it simply, the system of record manages the member's status.

[AI Host 2] (11:31 - 11:38)
Right. And the system of knowledge manages the member's curiosity, their immediate problems, their need for reliable expertise.

[AI Host 1] (11:39 - 11:43)
Okay. Let's unpack this a bit more because analogies always help me visualize this stuff.

[AI Host 2] (11:43 - 11:44)
I love a good analogy. Lay it on me.

[AI Host 1] (11:44 - 11:52)
Okay. So if my association is this massive, sprawling library, it sounds like the AMS is the front desk.

[AI Host 2] (11:52 - 11:54)
Okay. Yeah. Tracking who comes in.

[AI Host 1] (11:54 - 12:09)
Right. It's checking people in, making sure their library card is valid, collecting late fees, and keeping a ledger of who walked in the door. But it's not the librarian who actually helps you find the nuanced information hidden inside the books.

Is that right?

[AI Host 2] (12:09 - 12:20)
That is a phenomenal way to look at the broader ecosystem here. The front desk versus the librarian. Right.

And to push your library analogy a bit further, think about the other platforms in your tech stack.

[AI Host 1] (12:21 - 12:21)
Like what?

[AI Host 2] (12:21 - 12:29)
Well, your CMS, which is your website content management system, or your LMS, your learning management system, or even your community forum platforms.

[AI Host 1] (12:30 - 12:31)
Oh, okay. So what are they in this library?

[AI Host 2] (12:31 - 12:38)
Those are just the bookshelves. They are content containers and experience surfaces. They physically hold the information.

Right.

[AI Host 1] (12:39 - 12:44)
The continuing education courses, the peer-reviewed articles, the heated forum threads about new regulations.

[AI Host 2] (12:44 - 12:50)
Exactly. But they don't inherently make that information conversational. They don't make it easily extractable.

[AI Host 1] (12:50 - 13:00)
Yeah. A member still has to walk down the aisle, find the right shelf, pull the book, and then speed read through a 40-page PDF to find their one specific answer.

[AI Host 2] (13:01 - 13:09)
Exactly. The AMS checks their library card, the CMS holds the book, but neither of them reads the book for the member to answer their specific question.

[AI Host 1] (13:09 - 13:10)
It makes so much sense.

[AI Host 2] (13:10 - 13:20)
And an AI knowledge layer acting as that super librarian doesn't just point to the aisle. It can instantly read all 10,000 books in the building simultaneously.

[AI Host 1] (13:20 - 13:21)
Wow.

[AI Host 2] (13:21 - 13:28)
Right. And extract the exact paragraph you need, synthesize a clear answer, and hand it to you with a footnote pointing right back to the original text.

[AI Host 1] (13:29 - 13:36)
Okay. That librarian analogy actually explains a very expensive mistake I see leaders making every day, which brings us to this massive trap.

[AI Host 2] (13:37 - 13:38)
The migration trap.

[AI Host 1] (13:38 - 13:53)
Yes. Because if the AMS is purely a system of record, a front desk, why do so many smart, capable people think that updating it is going to magically fix their content and knowledge findability problems?

[AI Host 2] (13:53 - 13:59)
It is an incredibly common mistake. And honestly, it's usually driven by vendor promises of an all-in-one solution.

[AI Host 1] (13:59 - 14:02)
Oh yeah. The we can do it all pitch.

[AI Host 2] (14:02 - 14:16)
Right. Leaders assume that a platform migration automatically fixes the member experience or knowledge findability. They think, well, we're moving off this clunky legacy server and buying a modern cloud platform.

So of course it will have modern search.

[AI Host 1] (14:16 - 14:17)
Right. It's new, so it must be better at everything.

[AI Host 2] (14:18 - 14:34)
But AMS modernization ends exactly where true knowledge access begins. You can have the most beautifully architected brand new AMS in the world, and your members will still be utterly frustrated when they try to search for a specific continuing education requirement or some nuanced industry guideline. Exactly.

[AI Host 1] (14:34 - 14:43)
Okay. I had to play devil's advocate here for a second, because if I just wrote a massive check to an AMS vendor, I'd be absolutely furious if you told me I need another layer on top of it.

[AI Host 2] (14:43 - 14:44)
Oh, I'd be mad too.

[AI Host 1] (14:44 - 14:54)
Right. Shouldn't their search function just magically be better now? I mean, I just spent millions of dollars.

Why doesn't the AMS just build this conversational search directly into their platform?

[AI Host 2] (14:54 - 15:04)
And that is the exact question every board asks, but you have to look at the underlying architecture. The search within the AMS might be slightly better than what you had before.

[AI Host 1] (15:04 - 15:05)
Like it might load faster.

[AI Host 2] (15:06 - 15:25)
Sure. But here's the critical distinction. A shiny new AMS only searches the AMS. Oh, only itself. Right. It is searching its own internal database of structured records and whatever limited structured content it directly hosts. It does not natively bridge the fragmented stack.

[AI Host 1] (15:25 - 15:27)
Because it doesn't own the other bookshelves.

[AI Host 2] (15:27 - 15:39)
Bingo. Let's use a highly specific example. Imagine a member comes to your site and asks, can I transfer my type B nursing certification credits from Ohio to meet the new 2026 state board requirements?

[AI Host 1] (15:39 - 15:40)
Okay, tough question.

[AI Host 2] (15:40 - 15:44)
Right. Now the AMS knows who the member is. It knows they live in Ohio.

[AI Host 1] (15:44 - 15:45)
Because it's the system of record.

[AI Host 2] (15:46 - 15:57)
Exactly. But the actual answer to that question might require a piece of data from a certification course hidden inside the LMS, a policy document sitting on the CMS, and maybe a recent discussion thread in the community platform. Oh, wow.

[AI Host 1] (15:57 - 15:58)
Yeah.

[AI Host 2] (15:58 - 16:06)
The AMS search cannot reach into those other silos. Furthermore, even if it could, standard keyword search is incredibly fragile.

[AI Host 1] (16:06 - 16:09)
Wait, fragile how? Like if they misspell a word?

[AI Host 2] (16:09 - 16:23)
Yes. Or if they use a synonym. Traditional search looks for exact string matches.

So if the member searches for type B credits, but your policy document calls them category two CEUs, a standard search engine returns zero results.

[AI Host 1] (16:23 - 16:25)
Zero results. The worst screen on the internet.

[AI Host 2] (16:26 - 16:43)
Exactly. Findability across the whole estate using semantic understanding. You know, where the system actually understands the intent and the context of the question rather than just matching words.

That is a totally separate job to be done. Right. It is a job the AMS was never designed to do because it is busy processing transactions and keeping your data secure.

[AI Host 1] (16:44 - 16:51)
That is such a light bulb moment. Semantic search changes the game because it actually understands what the member means, not just what they typed.

[AI Host 2] (16:51 - 16:52)
Precisely.

[AI Host 1] (16:52 - 17:02)
You just built a beautifully organized front desk, but the bookshelves in the back are still a maze. And now we understand why the front desk clerk can't help navigate the maze.

[AI Host 2] (17:02 - 17:03)
Yeah, they're stuck behind the desk.

[AI Host 1] (17:04 - 17:10)
So if we establish that the AMS migration doesn't solve knowledge access, we need to address.

[AI Host 2] (17:10 - 17:23)
Think about the state of the organization right after a launch. The organization's system requirements are fresh in everyone's minds. You just spent a year meticulously documenting exactly what every system does.

[AI Host 1] (17:23 - 17:25)
Right. You actually know what you have for once.

[AI Host 2] (17:25 - 17:39)
System ownership is clear. You finally know exactly who manages the LMS and who manages the CMS. And most importantly, your integrations and data architecture are clearer and cleaner than they have ever been.

[AI Host 1] (17:40 - 17:41)
Because you just mapped everything.

[AI Host 2] (17:41 - 17:45)
Exactly. The bodies are buried, but you know exactly where the graves are.

[AI Host 1] (17:45 - 17:53)
Here's where it gets really interesting to me, too. The migration itself likely surfaced exactly where knowledge and member experience are still fragmented.

[AI Host 2] (17:53 - 17:54)
Oh, 100%.

[AI Host 1] (17:54 - 18:06)
Because you had to look at all your data to move it. The content gaps are highly visible right now. You know exactly what didn't fit neatly into the new AMS. You know about that messy SharePoint folder that everyone is too afraid to touch.

[AI Host 2] (18:06 - 18:28)
You've audited the house. You know where the messy closets are. But there is another more strategic reason why the timing is right.

And it goes back to that executive pressure we talked about earlier. Think about the boardroom. Right after a massive platform spend, leaders are desperate to prove to their board of directors that the new AMS created actual, tangible member value.

[AI Host 1] (18:28 - 18:32)
Because the board is going, we just spent $2 million. What did we get?

[AI Host 2] (18:32 - 18:46)
Exactly. And a governed AI layer can be deployed relatively quickly to sit across that newly organized stack. It can immediately begin revealing what members still cannot find.

It acts as a post-migration signal to prove ROI.

[AI Host 1] (18:46 - 18:52)
Oh, wow. So instead of just saying, hey, our database is faster now, you can actually show them a whole new member experience.

[AI Host 2] (18:52 - 19:05)
Yes. It shows the board, look, not only did we upgrade our operational infrastructure to keep our data secure, but we also unlocked a conversational layer that is directly answering member questions using our proprietary vetted content.

[AI Host 1] (19:05 - 19:06)
That's huge.

[AI Host 2] (19:06 - 19:13)
It bridges the gap between an invisible back office upgrade and a highly visible front office experience. It is the ultimate proof of value.

[AI Host 1] (19:14 - 19:24)
That is a brilliant way to frame it to a board. It changes the narrative from, we bought a massive database, to we unlocked our institutional knowledge and made it accessible.

[AI Host 2] (19:24 - 19:26)
Completely shifts the perspective.

[AI Host 1] (19:26 - 19:41)
But as much as we love the potential here, and as much as we love the idea of semantic search, we have to be completely realistic. We are looking at these implementation playbooks and they are very clear that there are times when you should absolutely hit the brakes.

[AI Host 2] (19:41 - 19:42)
Yes, you have to read the room.

[AI Host 1] (19:42 - 19:50)
As much as an AI layer helps, throwing it at a team that is currently drowning is a recipe for disaster. So when should an association wait?

[AI Host 2] (19:50 - 19:55)
You must delay looking at an AI layer if your team is still in pure stabilization mode.

[AI Host 1] (19:55 - 19:56)
Okay. Like triage.

[AI Host 2] (19:56 - 20:04)
Right. If the AMS just launched last week and the finance team can't figure out how to process a simple refund, stop.

[AI Host 1] (20:04 - 20:05)
Yeah, that's not the time.

[AI Host 2] (20:05 - 20:13)
If the membership team is still dealing with data migration cleanup because thousands of records are duplicated, stop. Do not pass go.

[AI Host 1] (20:13 - 20:13)
Good advice.

[AI Host 2] (20:14 - 20:30)
If you are struggling with severe adoption failures and your staff are practically revolting in the hallways against the new system, do not introduce a new technology. You will break your team's spirit. Do not overload an exhausted team.

Let them stabilize the ship first.

[AI Host 1] (20:31 - 20:40)
That makes total sense. I mean, you can't build the penthouse if the foundation is currently crumbling. If people are crying in the break room over spreadsheet mapping, do not pitch them an AI project.

[AI Host 2] (20:40 - 20:55)
Precisely. Empathy for the team has to come first. And there is one more scenario where you should wait.

And this one is a bit more insidious. What's that? I warn against leaders trying to use AI to distract from unresolved platform governance.

[AI Host 1] (20:56 - 20:57)
Interesting.

[AI Host 2] (20:57 - 21:08)
Sometimes an executive will see that their content strategy is a total disaster. Their taxonomy is non-existent. No one knows who owns what document.

And they have 10 versions of the same policy floating around.

[AI Host 1] (21:08 - 21:09)
A total mess.

[AI Host 2] (21:09 - 21:15)
And they think, let's just buy an AI to search through the mess so we don't have to clean it up. AI is not a bandaid for terrible governance.

[AI Host 1] (21:15 - 21:38)
OK, let's pull on that thread and be really real with the listener. The honest beat here. If someone is dealing with messy member data or a botched migration, will a tool like Betty just swoop in and fix it?

Because if you listen to the marketing out there in the broader AI world, it sounds like absolute magic. Right, the magic wand. It sounds like you just plug it in and it fixes all your operational problems.

[AI Host 2] (21:38 - 21:43)
That is a crucial question. And let's just use Betty as our clarifying example here to cut through the hype.

[AI Host 1] (21:43 - 21:43)
OK.

[AI Host 2] (21:44 - 21:57)
Betty is a governed, source-bound knowledge assistant. It is a fantastic implementation pattern for associations looking to create that AI layer. But let me be perfectly clear about what it does not do.

[AI Host 1] (21:57 - 21:58)
What doesn't it do?

[AI Host 2] (21:58 - 22:02)
It does not replace your AMS. It does not fix your bad member record data.

[AI Host 1] (22:02 - 22:03)
So if there are duplicates.

[AI Host 2] (22:03 - 22:13)
Right. If Jane Smith has three conflicting profiles in your database with three different email addresses, Betty is not going to magically merge them and figure out which one is correct.

[AI Host 1] (22:13 - 22:13)
Good to know.

[AI Host 2] (22:14 - 22:20)
It does not clean up bad migration work. And it certainly does not solve your underlying integration architecture.

[AI Host 1] (22:20 - 22:25)
So it's not the IT department in a box. It's not going to write your API connections for you.

[AI Host 2] (22:25 - 22:56)
No, it is not. Betty is simply a governed layer that sits above your tech stack. It only works when the sources and the access rules are already reasonably clear.

Got it. It activates your existing knowledge. It doesn't invent a new operational reality.

It relies heavily on the AMS to know who the member is, what their tier is, and what they have paid for. It relies on the CMS to hold the actual documents. It simply acts as the intelligent conversational retrieval layer across those systems.

[AI Host 1] (22:56 - 23:08)
I really love that distinction. It really respects the heavy lifting that an AMS does. It acknowledges that the AMS is the unsung hero keeping the association running while being completely honest about what it can't do natively.

[AI Host 2] (23:08 - 23:10)
Exactly. Give the AMS its due.

[AI Host 1] (23:10 - 23:18)
So if we pull all of this together, all the strategy notes, all the vendor realities, what is the core thesis we want you, the listener, to walk away with today?

[AI Host 2] (23:19 - 23:26)
It is simply this. Buying an AMS does not automatically make your knowledge conversational or findable. It just doesn't.

[AI Host 1] (23:26 - 23:33)
The real question you need to be debating with your executive team is not AMS or AI. That is a false dichotomy that will only lead to frustration.

[AI Host 2] (23:34 - 23:41)
Absolutely. The question you should be asking is, what job did the AMS solve and what knowledge access job remains?

[AI Host 1] (23:41 - 23:52)
That is the exact strategic framework every association should adopt. You hired the AMS to be your system of record. It is securing your transactions and managing your data.

[AI Host 2] (23:52 - 24:01)
Now you need to deliberately decide how you are going to handle the system of knowledge. How are you going to get the brilliance out of your bookshelves and into the hands of your members?

[AI Host 1] (24:01 - 24:10)
And to help you do exactly that, I'm going to remind you one more time about the resource we mentioned earlier. Look in the show notes for the AI layer versus platform decision brief.

[AI Host 2] (24:10 - 24:12)
Don't skip that brief. It's so helpful.

[AI Host 1] (24:12 - 24:27)
This is the tool you need for your next internal meeting. We want you to use this brief to map out the job to be done. It will help you separate the concerns that an AI layer can beautifully address from the core operations that your team still absolutely owns and must manage within your platforms.

[AI Host 2] (24:28 - 24:33)
Do this exercise before you decide to add an AI layer or before you decide to delay it.

[AI Host 1] (24:33 - 24:40)
Yeah, it will bring incredible clarity to your roadmap. And honestly, it might just save you from making a very expensive miscalculation.

[AI Host 2] (24:40 - 24:59)
And as you go into those roadmap meetings, I want to leave you with one final provocative thought. Think about the search bar on your association's website right now, not a year from now, but today. Imagine a member right at this very second just typed in a complex, multi-layered question about a pressing professional issue.

[AI Host 1] (24:59 - 25:01)
Like the nursing credits example.

[AI Host 2] (25:01 - 25:25)
Exactly. A question that requires pulling from a course, a policy, and a forum. And imagine your brand new multi-million dollar AMS couldn't give them a conversational cited answer from your approved sources.

Ouch. They hit search and get a disjointed list of irrelevant links or worse, zero results. What is the hidden cost of that member walking away and being handed today?

[Rob Barnes] (25:30 - 27:06)
The question today was, we just bought a new AMS. Why would we consider an AI project now? And Thomas, the first thing that came to my mind after listening to the episode today was, I feel for the association finance manager, CFO or CEO. Because in the cases where we have this sunk cost now into one of the four association management softwares, whether it's the AMS, the CMS, the LMS, right now I am being presented with, I was going to say bombarded by, but that's probably not fair, but I'm presented with an AI tool that is now free for anyone that uses this AMS. And same for the financial management software. Why won't I just go in and use the free tool that's part of the software that I've already paid for and I already own and hope that it's going to solve some of the challenges that we talked about today. And it reminds me a little of episode three, where again, it was ChatGPT co-pilot versus a governed AI specifically for an association and that decision-making tree. We're kind of here again now.

Is this a new problem because of these AI bolt-ons to legacy platforms or new management platforms? Or does this have a new layer on it because it is AI? Either way, I don't envy the decision, the CFO right now.

[Thomas Altman] (27:07 - 30:37)
I think it has a lot of overlap, honestly, with episode three, which was the ChatGPT co-pilot versus governed AI. And I think the answer is going to be fairly similar, is maybe you do use it for certain things, right? And if the question is why now, sometimes the answer, the best answer would be not now, right?

Maybe we wait a little bit and we see what these systems do and what their limitations are, right? And we use what we have. But you should do so in a way that is kind of informed, kind of driving to some underlying understanding of how these systems work more deeply.

And I think that's the framework that today's companion asset, I think, helps you walk as well. But to kind of talk through that a bit, I think the way I would think about this is, yes, we've got an AMS. Do we do it? Do we add in a governed AI interface, an AI, a knowledge platform kind of in front of that as an interface to the member to improve member experience?

Or are we using internal AI that helps you use that product better? That's generally the distinction. What I think would probably fail is trying to hack some internal system that's meant to help you use a specific platform better to exist as this front-end kind of interface that kind of is the ghost in the machine behind everything else and helps the member and the public and whoever's going to be interacting with your association from the outside get to where they need to go, have the best experience possible, accomplish more of what they set out to accomplish and help you achieve your mission. And the reason for that is a lot of these MSs, the management systems, is they're at their core transactional databases.

Something happened, we have a record of it that changed something else about your person record or your company record or whatever it might be. And being able to use that information to inform how an AI system in front of that interacts and deepens engagement, I think it's a different tool, right? So why now?

Maybe not now. Maybe you wait when you still have some cleanup to do, right? The migration didn't go perfectly.

You went from an old AMS or an old LMS to a new one and things aren't clean. You've got data to duplication to do all that. Or maybe you haven't figured out how the process of refund, right, in the new AMS. That happens all the time. Those kinds of things, you've got to get stable first. It could be now if you were bringing on this AMS as a way to revolutionize the member engagement strategies, right? Because the AMS isn't going to do that.

The sort of AI system in front of that will or should. And being able to take advantage of that, the last kind of key point I kind of want to jump into is if you do that part well, right, if you have this AI in front, what it can do is make the ROI on your investment to the AMS higher and you get that ROI faster, right? Because it's actually taking advantage of a lot of the work that went into that in a way that deepens engagement with the members, right?

Revolutionizing that member engagement strategy I think is the core sort of benefit of adopting an AMS. And it's not going to do it on its own. So you want to adopt that AI strategy when it's time to kind of take advantage of what you already invested in.

[Rob Barnes] (30:38 - 31:27)
That's a really good way to round out this episode, Thomas. I agree. I think that there's some really interesting ways to think about this, which is slightly different now because it's the AI era, if you like, but they're certainly very common challenges to be faced.

So the more the listeners today and our association executive friends can start thinking about that using today's downloadable, which I say one more time before you go today, download the AI layer versus platform decision briefing paper from the show notes. It's the fastest way to actually put today's episode to work. And a very important episode it was.

Thank you for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question. What if it's wrong about something that matters?

What this helps your team answer

Why an AMS (or any management system) is a system of record, not a system of knowledge — and why AI should never try to replace that record, only sit in front of it as a governed interface.
How to tell whether "not now" is the right call (a messy migration, unresolved data cleanup, unstable processes) versus a stall tactic — and why waiting shouldn't mean waiting indefinitely.
Why a governed AI layer can actually increase the ROI on the AMS you already bought, by deepening member engagement in ways the AMS alone was never built to do.

Companion Asset

AI Layer vs. Platform Decision Brief

The Association Intelligence Podcast

The AI Questions Every Association Needs to Answer

Start with the question your team is stuck on.

How we built it

Are we ready for AI?

What do we tell the board?

Why Not Just Use ChatGPT or Copilot?

Should We Build This Ourselves?

We Don't Have a Tech Team — Can We Still Do This?

How Do We Do This Without Burning Out Our Staff?

We Just Bought a New AMS — Why Would We Take On an AI Project Now?

Want to pressure-test your association's AI strategy?