Lynn Boyden is an Information Architect at USC Information Technology Services. She also teaches Information Architecture at UCLA’s library school and is co-founder of LA UX Meetup, the largest local meetup for UX Professionals in the country.
In this episode hosts Lara Fedoroff and Chris Chandler talk with Lynn about Quantifying Qualitative Decisions. Lynn’s expertise with spreadsheets and decision making have taken on a life of their own because “Ain’t no party like a nerd party.” Learn how to use her methodology to objectively come to the right decision.
TRANSCRIPT
Welcome to UX Radio. The podcast that generates collaborative discussion about information architecture, user experience, and design.
UX Radio and host, Lara Fedoroff are happy to announce Chris Chandler as the new co-host. Their first guest is Lynn Boyden who is an information architect at USC information technology services. Lynn also teaches information architecture at UCLA’s library school, and is co-founder of LA UX Meetup, the largest local meetup for UX professionals in the country.
Now we join Lara Fedoroff and Chris Chandler as they welcome Lynn Boyden.
Hi Lynn, Thanks for being with us today.
Thanks for having me.
Lynn. I’d like to know how you got started as an information architect. Not many people have that title anymore.
I graduated Library school and I was pregnant and no one will hire you when you’re pregnant. It’s just. That’s how it is. I ended up taking a job as an admin within the department and I worked there for a couple years and then one day I got a film call from a very dear friend.
Who might be sitting at this table.
Maybe.
Yes.
Chris called me on a Monday, and said “get this book and read it, your interview is at 3 on Thursday” and the book was the information architecture for the world wide web, the polar bear book and I read it and it was like being struck by lightning. This was what I wanted to do. This was exactly what I wanted to do. I finished the book by Tuesday night and went to the interview on Thursday and aced the interview and the rest is history.
Amazing. Tell me a little bit about the job at USC. You’ve been there for a while.
I’ve been there since 2010, and we word as an internal agency, so we only do USC projects and at USC you can, if you want a website you can hire a graduate student or you can hire an external agency, or you can hire us and so we bring a local expertise as well as depth in understanding what the goals and drivers are for the USC experience.
You’re probably one of the only people on both payrolls for USC and UCLA. What’s that like.
It’s kind of, I feel a little bit like a ninja. It helps that I don’t care at all football. I just regard all of that as kind of amusing side event. But both USC and UCLA are happy to have the collaboration across the campuses, and so I just feel like it makes the whole thing a little stronger.
So we should explain that at UCLA you’re a professor and you have been teaching for many years, an information architecture class, right? In the library of Science department.
Yeah, It’s more morphed into user experience design. It’s an elective in their two-year master’s degree program and I teach it as a lab. I recruit nonprofits who need strategy work, user experience design work of some sort and I invite them to pitch their projects to my class and then the students organize into groups and it’s ten weeks and at the end of the ten weeks they deliver recommendations to the client and they come out of the class with a portfolio piece, client recommendations, project experience. I have some students who repeat the class, and that’s really great. They get additional readings and I asked them to take a leadership role in the group projects and I’ve been teaching since 2001, and it’s kind of terrifying and exciting, but sometime it’s wonderful too. I go to UX events in Los Angeles and see a range of my students from over the years and it’s a serious warm fuzzy. They are active and engaged and they are out there leading actions. And it’s just, it’s wonderful to see.
And then we get to hire them.
And then, yes. I have an extensive network as one might imagine.
One of them is working here at Philosophy at the moment, Sarah.
One of the best and brightest.
The three of us here, actually had a good experience with a client that we shared between Philosophy and your class and so you worked on a project of your own.
Right! With UCLA. It was called LYNX. That specific project was targeting African-American males between the ages of 18 and 35. UCLA research team had realized that they weren’t receiving the right information about being HIV positive and so what was happening was they weren’t getting the treatment.
The users you mean. The users were not getting the right information.
Yes.
My students took the next phase of that, of research on that project and made a lot of recommendations for the research team and I think that they are going to work with us again this year. So I’m excited about that.
Yeah we get 4, 6 week pilot with them. Right? To build a quick app to help them get out that information in the way that they really needed it.
Right.
And then the hand-off you came to, I think a demo night that we did here where Lara presented the results and since it was on the UCLA campus and I saw. I think I might have seen the light bulb go off over your head, like I should reach out to them. Anyway that gives me the warm fuzzies that we were able to collaborate like that. I know that they were super appreciative of the work that the students did. How do you think the UX community in Los Angeles has changed over the time?
It has gotten bigger and younger. But I think that’s me not them.
Bigger and older. Sorry, that’s how I’m working it
They’ve also gotten older too. When we had that first UX Meetup that we organized off of the SIGIA list*, we held it in Molly Malones and it was. We expected maybe a dozen people to turn up and there were about 40 or 45 that came and the bar was very angry with us that we hadn’t made plans in advance to host our party and that was very exhilarating to understand that wow there were 40 of us out there because Chris and I kind of thought that we were just doing it by ourselves.
Or that we knew all of the other 5 people who might be doing it in Los Angeles. and I like your throwback term there. the SIGIA, that’s an old old mailing list. an email list, the Special Interest Group in Information Architecture hosted by ASIS. As I recall.
And now the LA UX meetup has about, I think over 5 thousand people in it. Which is pretty impressive.
Yeah, it’s absolutely amazing. I mean I think that’s one of the things about Los Angeles that people who don’t live here don’t appreciate is just how vast it is. I remember one of our early jokes in the meet-up days. Was we had to move the meet-up around because no matter where you have the meet-up in Los Angeles. Somebody has to drive an hour to come to your meet-up. So that was a conscious decision we made, it’s like most of the tech events in Los Angeles happen on the west side. But we wanted to make sure we did things downtown. I remember some good ones at LA Times, the Valley, in the NorthEast, Los Angeles in terms of Glendale and Pasadena. I think looking back that that was actually one of the keys to success for that group, the geographical movement.
One of the things that I remember with a great deal of fun was the UX bus. When I drove the VW bus in 2008 to, i think I drove it to UX meetups just for a few years. But when the meetings were on the East side of town, I would load up the bus with UXers and I had seat-belts for six, not including the driver and we would just network relentlessly and snark and chat in the bus on the way there and then we would go to the even and then we all bus back and that was so much fun and we had such a good time.
Swaying back and forth to the music, waving to all the people who were admiring your bus
Right. All of the sudden this is an episode of the Californians.
Oh! So you take the four to the four-o-five to Wilshire down to Los Cienaga.
I don’t think people outside of our little bubble can appreciate the long trek it is from the west side to the other parts of LA.
It was nice to have people, like minded people to talk to as I made that drive.
Your current project is quantifying qualitative decisions. Can you tell me a little bit about that?
So one of the hardest things that we have to do as User experience designers is to make decisions. It’s hard in any kind of process. It’s hard in your personal life, it’s hard in your professional life, it’s hard individually, it’s hard in groups and I’ve got 20 years of experience in this field and I’ve adapted a variety of methods that help me arrive at decisions and so I’ve been for the last couple of years working on a method that I call quantifying qualitative decisions and that really gets to the crux of it. The business always wants numbers around decision making and decision making in user experience design is very difficult because they are qualitative decisions. So how do you bring that to business in a way that allows them to say, Yes! This is a good decision, I can see this.
I just wanna say. I’ve known you for a long time, Lynn. I think one of the things that you are just genius at is that operationalizing of decision making. I mean I can recall an Information Architecture Summit several years ago where you brought a deliverable to share which was a giant spreadsheet that you had printed out, that you had posted on the walls to help that consensus group decision making and to walk people through your process. I’ve always been really impressed by your lack of fear when it comes to spreadsheets.
I love spreadsheets. I do everything in spreadsheets. I was experiencing with a thing that I call dirty deliverables. Where I had done a bunch of user experience research, usability testing for the UCLA marketing communications team about the UCLA homepage and I wanted to present my results in something that had more impact than powerpoint and so I created that giant spread and when Chris says giant spreadsheet, we’re talking the cells were regular print out size cells, but the spreadsheet itself was about 12 feet long by 6 feet tall. I had mapped out every question that we had asked them and every outcome and then I had taken highlighter and I went through and highlighted all the things that were of interest and I color-coded things so that they could be identified at a distance and it was really great because you could back off of the spreadsheet and say wow! All those places that were highlighted in light blue was where somebody mentioned that the font-size was too small but then you can zoom in and look at it and see exactly what they had said. So it was both a summary and granular data all rolled up in one deliverable and I was very nervous when I presented it to the client because, you know it’s marketing communications for one of the major research universities in the world, and I’m going to come in with this messy spreadsheet. I brought in print-outs of recommendations and observations, sort of an executive summary and shared that with them, but then I unfurled this and put it up on the wall and the rest of the meeting was over. That was all we wanted to do was dive deep into the data and that was very gratifying because I understood that they wanted that data, they were desperate for that hard concrete qualitative information and we made some really good decisions around the future directions for the design for the home-page that they implemented, so that was very satisfying. My new tool is participatory decision making process and we use it to come to decisions where there are alot of different criteria that enter into the decision, a lot of different stakeholders with different priorities and different viewpoints that need to sort of be accomodated and heard and also where the decision often needs to be communicated out to other groups so there are a lot of different methods that will incorporate that, both the participatory nature and the multi-factor nature. But the way I’ve structured this process also leaves with you with an artifact that you can share with people. That people can look at and make sense out of. So if you like I could just jump right in and describe how this tool works.
Yeah! That’d be great.
So I use a spreadsheet to do it but you can do whatever you want. post-its are often also a useful tool so there are sort of three phases. The first phase is to identify all of the qualities of a successful solution. So, the example that I like to use is choosing a house. You’ve been out to all the open houses and you’ve looked at them all and you’ve narrowed it down to four different houses that you want, you need to decide between. They need some base criteria and you have multiple stakeholders. You have you, you have your spouse, you have your kids, everybody has an opinion. Some opinions are more important that others, some qualities are more important than others when you’re buying a house. For example, is it located in a good school district, is a very important quality for a house purchase. Does it have fruit trees in the backyard is important but less important. Does it have a backyard? Very important, Right? so there is different levels of things. Some qualities may be more important to others, to some stakeholders rather than others.
By the way. This is literally how Lynn chose her current house is with a spreadsheet that is very much like the one she is describing.
It’s true, we did do that. As a team everybody who is participating in the decision identifies the qualities that are important to a successful outcome, a successful decision, Right? So, in the house example. What is a house that we would like to live in for a long time, right? and everybody contributes. There is no filtering, there is no nay-saying, there is no dissing. At this point everybody contributes. You put all of the qualities out together either on a table or on a spreadsheet and once you’ve agreed that you’ve hit them all and that’s not a binding decision. You can add more later. Then you begin to evaluate them, you group them, or you roll some up, you break others apart. You want to keep the qualities positive in nature because it helps with the process later on if they are all positive. So instead of saying, it’s not in a bad neighborhood you would say, it’s in a good neighborhood, right? So once you’ve broken up the qualities you want to make sure that there is a good distribution of them, that you’re not including too many qualities in one area and not enough in another. As you’re doing this you may come up with more. Feel free to add them, basically at this point you are just gathering all the qualities. Once everybody has agreed that these are the qualities that are important for making a decision, then you move onto the next phase and that phase is to weight those qualities. So when I weight a quality, I’m talking about each stakeholder, each person in the room looks at each quality and assigns a value to its importance to a successful decision or a successful outcome. So in the house question, let me clarify the weighting system is 0,1,2. I do it that way because you’re making a lot of weighing decisions in this process and if you bump it up to 3 or 5, you end up with a lot of decision fatigue. So we keep it very simple 0,1,2 it’s a quick and easy decision to make. You spend a lot less time discussing what the outcomes mean.
So for the scales is 0 not important at all?
0 is not important, 1 is important but not the most important thing, and 2 is essential. I guess 2 is a must-have and 1 is nice to have and 0 is I don’t care.
So brainstorm the qualities. Everybody has input into the weighing of the qualities.
Right, so each person weights each quality 0,1,2 and then you sum those weights into an aggregate weight.
So again, live in a spreadsheet you’re having people enter data and having that all set up so that it’s calculated live.
Yes and you can do it remote. We did it remote actually using a google doc. So everybody was assigned a column and they entered their score into the thing and I had set it up mathematically so that it would weigh them automatically and there is a lot of fun that you can have with google docs in multiple locations with multiple people and especially if you preload equations and stuff into them.
Ain’t no party like a nerd party.
That sounds like so much fun!
I’m a spreadsheet geek and I love that. So, you go through the qualities, you weigh them. Each person contributes a weight. You aggregate the weights. So if you have four people in the room making the decision. Your highest possible weight will be 8 on any given quality, right? So you’ve got your qualities, you’ve got them weighted. The next step is to score each candidate on each quality.
What do you mean by candidate.
Let’s say I’m looking at four different houses. One of them is a condo in the Highlands, one of them is a little cottage in Santa Monica, one of them is a townhouse in Culver city. Let’s say we’re just looking at three.
We’re living large.
So we look at just the condo in the Highlands and we look at the first quality. We ignore the weights for now. We look at the first quality. So I give it a score on that first quality, has fruit trees in the backyard. Doesn’t even have a backyard, it gets a 0. So, we are scoring the candidate on how well it meets that quality. Again 0,1,2 0 it doesn’t meet it at all, 1 it sort of hits it, 2 it’s a winner. Each participant assigns a 0,1,2 score for each quality in a candidate. Once each candidate has been scored by all the participants on each quality. You aggregate those scores, you multiply them by that qualities weight and you get a weighted score for that quality. You sum each quality’s score for the candidate and you end up with a weighted score for that candidate and then you do it for the next candidate. So that cottage in Santa Monica, then you would evaluate has fruit trees in the backyard. Yes! there is a whole grove of mandarin trees back there. It hits a two, right? Has a hot tub, it’s a 0.
Because they are hard to maintain.
So each person scores the candidates. And what happens is the weights give the more important qualities higher importance in the scoring and the differences in at the end blow way out and it’s really clear to see which is the important candidate. You can also use it for prioritizing things. I u-
So let me ask you one thing, which is an interesting underlying assumption here is that everybody’s opinion here is equal. I love that. Do you run into any issues with that? If you’ve got a big spread of senior and junior and leadership and stuff in the room do people feel like that’s appropriate for-
I have never encountered any negative outcome of that. I have encountered some amazing positive outcome of that. We used this method with my, then 14 year old daughter to choose a high school. We had a couple of options to us. Lycée Français, which is a fairly expensive private school in Los Angeles and the local public high school and her father and I and she all sat down and we generated a list of qualities of what makes a good high school education and we agreed on them and separated them out. We weighted them and then we scored each candidate on those qualities and totalled them up and the score was overwhelmingly in favor of the public school and she was fine with that decision because she had participated at the same level as the bigs, and she saw that the decision was not just, it’s too expensive we can’t afford it you can’t do this. So, she was on board with the decision and it worked out really well.
You also used this method at USC recently, right?
I did. We were working on, I’ve used it several times to great success. One was we were working on a mobile app. It was going to be the first mobile app ITS had ever done and I was the UX person and we had 12 developers who had joined the project and we had met six times without the developers being able to identify which platform they wanted to develop this app on and they hadn’t yet decided what the app was going to do and I knew, bless developers I love them so. We can’t do the good work that we do without them. They have some personality quirks and one of them is they like to argue about platforms. So we had six meetings in which we did not decide which platform we were going to build this on and so I proposed to the project leader that we implement this method so we did and at the end of an hour and a half meeting, we had made a decision about which platform we were going to develop it on and then we could go ahead and talk about what the app was supposed to do. And the funny thing was at the end of it, once we had decided what the app was supposed to do we ended up choosing a different platform. So, the decision was non binding but it helped us get through that block to make a decision so that we could then move the project forward so that was one success that we had and the app is live in the App store today which is very exciting and another project that we used it on, there was a team at USC, we implemented workday as our campus-wide HR software and it had been in place for about a year and we had a couple of teams working in the field with the stakeholders in the field on the customer side as well as stakeholders in ITS to identify what kinds of enhancements we were going to do to the implementation, what we would do next and they worked for about nine months gathering data, doing interviews, doing research and they had all of this research and they needed the top-level decision makers to put aside their political differences and come together and rank these 16 projects so that they would know where to allocate resources and how to move forward and they had heard of my success with the software team with the developers and asked if I could implement this project, this method with them. So, we gave it a shot and we had 5 senior leadership members participating in this and we did a few things in advance. We seeded the qualities with things that we already knew were important to them. When we presented to them the qualities, and then we solicited from them additional qualities and feedback on what we had and I have to say kudos to the team. They pretty much nailed the qualities, they only added two more and that was pretty great and we really emphasized keeping the university’s mission as kind of the goal that would drive their recommendations around weighting and scoring these. So, bless their souls, they worked through 16 different software-enhancement projects evaluating each one on eleven different qualities. We have them auction paddles that said 0, 1 ,2 so that they could just hold them up, they didn’t have to speak it out loud.
Wow!
They were all kind of checking each other out to see what it was so first they weighed the qualities then they scored all 16 candidates and at the end of it we had a beautiful spreadsheet with all of the scores. The scores weren’t identified as to who had scored it what, which is I think a feature.
Well I was going to say, I think that method of having auction paddles is brilliant because everybody does the reveal at the same time so nobody is looking to see what did the highest paid person look for, right?
Yes, yes and they were so pleased by the outcome, we had a ranked prioritized list of these 16 software enhancement projects. They were so pleased that they asked me if I would conduct the same process with the larger campus administrative stakeholder group. They have an advisory counsel with people from all of the schools and divisions. and so, we did another round of it the following week. That was very exciting, pooling it all together sort of adapting the method for a much larger group.
A much larger group right?
Yes. So in this case we ended up sending out the existing qualities as a survey with 0, 1, 2 for the weighting. We did the weighting via survey in advance. We included some empty text boxes at the end as well as the weighting controls for them to add qualities that we may have missed. We sent it out to 50 participants, we had three qualities that were added. One of them was, I have empty text box so I’m going to complain and the other two were, pretty much we had already addressed them in the earlier qualities. We didn’t end up adding any other qualities. Then when we got the larger group together in the room, we walked through the weights, so that they would see how they had weighted things and in this case the numbers were much bigger because 50 people so the highest weight was 100 and the lowest weight was, I think it was in the 50’s somewhere. We gave each participant a clicker. You may not know what clickers are. They use them in large survey classes. You can set them to record. They’re basically voting tools. So professors use them to administer quizzes. there are clicker apps that people can put on their phones if people want to do it that way. We ran a little test quiz just to make sure all the clickers were working. We had the preprogrammed google spreadsheet. Somebody in the back was tallying them up and we entered them live and again, we worked through 16 different software enhancement packages scoring them on 11 different qualities. The entire room participated. The outcome was, the top 5 projects were more or less the same, there was a fair amount of disagreement below that. But we came out with people who were satisfied with the way the decision had been made and a very clear direction on how apply resources and to move forward to improve the workday experience for the entire campus. That was really exciting. I felt pretty happy about that.
So do you think this method is best used when you have multiple actors and a complex decision to make?
It works really well that way. Also, you can do it yourself. I have used it to choose between job offers. I literally list out the qualities of what’s a good job and then I weight those to the importance to me and if they’re 0 then I just cross them off the list. It’s easy enough and then I score each candidate on each quality, and it’s satisfying.
I’ll say. I watched you do this over the years as I said. It’s been pretty impressive. What’s coming up next for you?
I’m teaching again in the Spring term and so I’m gathering nonprofit projects for that as I am every Winter.
By the way. If any of you out there are listening and you work at a nonprofit and you’d be interested in having Lynn’s student work on you project then please contact her.
Yes, please do hit me up. I’m almost always looking for projects. I work with the client to scope the project so that they’re appropriate for the duration of the class as well as the skill level of the participants and I’ve almost always revised my syllabus too much every year. Just because it helps keep me sane and keep me on top of things. Keeps my cred good and then the other thing that I’m super excited about, is the IA Summit is coming in the middle of March and I’m working with an amazing team again this year to pull together a slam project. So, I can’t say anything more about what the story is, but it will be wacky.
That’s interesting, a slam you say!
Yes, in 2004 a bunch of us got together and decided that we wanted to do something at the IA Summit but that we were bored all of your classic, my IA is the one true IA presentations that we were seeing. The other kind of presentations that we were seeing were case studies which were interesting but how many case studies can you sit through in a conference?
Or the worst, a panel discussion.
So we created this little role-playing, problem solving game kind of based on a mashup of poetry slam and a murder mystery dinner and we present as a client with a problem and then we arbitrarily divide the participants into groups and they have a fixed time to solve the problem and present the results back to us. Problems that we have solved in the past have been a self updating interface for a time-travel machine. We did a problem where a company called Bal-Mart bought a company called Fordstrom and wanted to provide a consistent user experience across all platforms. We did that long before cross platform was a thing. You can google us up and there is all kinds of crazy pictures and things. I have some wonderful stories from the slam that I’m happy to share over a cocktail.
That’s right. Ask Lynn about the slam when you meet her.
And you have an article coming out soon.
I hope so. I’m working on writing up this article quantifying qualitative decisions and my goal is to have it published soon so look for that.
Wonderful. We’ll make sure to post all the information on the website. Thank you so much for being here with us today.
Thank you very much.
I’m grateful for the opportunity.
Thanks to Steve Crosby for digital development. An original score piece by Cameron Meshell.
This episode is sponsored by WeWork. “Meaningful conversations are essential to the success of every entrepreneur, freelancer, and small business owner. At WeWork, we consistently strive to make meeting new people and having interesting conversations natural and effortless. From the design of our workspace to the events at our buildings we do everything we can to support the idea that if one of us is successful we all benefit. Every WeWork location is staffed with community managers who work directly with members to understand their business needs, struggles, and growth plans and connect them to other members who can help. Events are an integral part of the WeWork experience. From product launches to elevator pitches. Whether you’re asking for advice, looking for product feedback, or just meeting like minded entrepreneurs. WeWork.com is a seamless extension to community.” For more information go to WeWork.com. That’s WeWork.com. Go to WeWork.com/UX-Radio to receive a discount now.
UX-Radio is produced by Lara Fedoroff. If you want more UX Radio, you can subscribe to our free podcast on ITunes or go to UX-Radio.com where you will find podcasts, resources, and more.
