Conduct better designer portfolio reviews with this tool

TL;DR: Here’s a Portfolio Review–Set-up and Assessment Tool for you to use.


A couple years ago I contracted with Capital One to help bring some order to their rapidly expanding design organization. I focused on recruiting and hiring practices–when you’ve got ~40 open reqs, you want to make those processes as efficient and effective as possible.

Key to any design recruiting process is the portfolio review, where a designer walks people through a selection of their work, and the thinking and activity that went into creating it.

Every interview loop at Capital One includes two behavioral interviews (BIs), which have a formal and repeatable structure, and require training to administer. It turned out the topics of one of the behavioral interviews was similar to what you’d get out of a portfolio review, and in an effort to reduce the time on-site (which could get up to 7 or 8 hours) I had hoped to replace a BI with the portfolio review.

I worked with Capital One’s HR team on this, and learned that, to be a worthwhile tool, it needed rigor and repeatability. This was key to removing bias from the process, focusing it on skills and experience, not personality and camaraderie.

I had never conducted ‘rigorous’ portfolio reviews. I’d always just had a candidate show their work, and ask some questions to clarify the candidate’s role, and that was that. Such a loose approach was not going to fly here.

The goal was to make something that ensured fairness in the process, and that, regardless of who was providing feedback, the assessment would be the same.

I asked the folks at Capital One if I could share the tool, and they said yes. So here you go:

Portfolio Review – Set Up and Assessment Tool

The idea is to not just have it be a free for all. The candidate preparation helps candidates know what to expect and how to shape their presentation. The suggested prompts provide a guide to the interview team for how to probe in a productive fashion

And key to making this work is the assessment tool. When I first drafted it, for each skill (visual design, interaction design, communication, etc.) you could score someone 1 to 5. However, there was no guide as to what a “1” would be or a “5”, and so it was too open for interpretation.

So I worked with craft leaders throughout Capital One to come up with language for these skills, to provide clear guidance in scoring candidates. That, for me, was the key ‘innovation’ in this approach.

I share it with the hopes that this helps make designer interview processes better everywhere. Feel free to copy it and make it your own!

If I were to write a second edition of Org Design for Design Orgs…

It’s been about two years since Org Design For Design Orgs came out. After having worked with it, taught it, and spoken with many design leaders and seen many design orgs, there’s a list of things I know I’d want to address if we wrote a second edition (note: no plans to do so).

Make dual-track career growth more explicit. In the book, we presented a single levels framework, with the idea that it could support career growth either as a manager, or an individual contributor. In retrospect, those paths are different enough that it warrants calling out, as I have in the levels framework we developed at Snagajob.  I’m also intrigued with the work Athenahealth did on establishing “Dual Track Leadership.”

In the “evolution of design organizations,” go beyond stage 5, to at least stage 7. We charted 5 stages of organizational evolution, from the “initial pair,” all the way to “distributed leadership,” where there’s about 70-80 people on the team. We yadda-yadda’d beyond that, saying, “just keep doing this, but more.” Since writing the book, there have been an increasing number of design teams that go beyond 100, and it’s clear that there are patterns in that development. It’s worth addressing what comes into place when the team hits 150 (stage 6), as  that’s when Design Operations / Design Management becomes quite robust, and again when it gets to about 250 (stage 7), where it can support deeper craft leadership, “principal” or “distinguished” designers, and also when it’s time to consider–should it remain as a single centralized org?

Dig into the crucial role of the Team Lead. In chapter 4, we dedicate a page to the role of the Team Lead, including the line, “the best team leads are a combination of coach, diplomat, and salesman.” That line became the seed for my talk on design leadership, and the process of writing that talk, and sharing these ideas at conferences and inside companies, has shown me that there’s much more to share about this crucial role. In fact, I consider it the most important role in a design organization, more than any VP or Director.  (That will be the subject of a future post on this site).

Go deeper on Design Operations / Design Management as a role and practice. Though the book has been called “the bible of design operations,” we don’t really tackle Design Operations / Design Management head on in a thorough way, particularly around matters of Program Management (budgeting, scheduling, coordinating efforts), Education (internal training and skills building), and Measurement (tools, systems, and approaches for understanding the impact of the work).

Do a better job distinguishing between Product and Communication Design. In our utopian desire to merge all design activities under the rubric of “service design,” and have product designers and communication designers working side-by-side on design teams, we neglected to delve into the very real differences between delivering product design and communication design. They operate on different cadences, work with different parts of the organization, and most of their time, simply don’t interact. That said, there is real value in having product and communication design on the same team (it was essential when we rebranded Snag). This is still a point of contention for many design orgs, and so warrants more honest, pointed discussion rather than our hand-waving of “it’ll be great”.

Soft power as a tool for distributed teams. However much I believe a centralized design organization to be the right way to go for, like, 95% of design teams, the reality is that many function in some kind of distributed, federated, siloed fashion. For those organizations, I’ve been applying the notion of “soft power” as a tool to get these distinct design teams aligned with a common goal, purpose, and set of practices.

Even more about recruiting and hiring–Portfolio Assessment Tool and design exercises. Even though it’s the longest chapter in the book, it turns out there’s still more to say about recruiting and hiring practices for design. The community still is at odds about the value of design exercises (though we’re not). Also, since writing the book, I’ve had the opportunity to craft a Portfolio Assessment Tool that brings a needed level of rigor to the practice–a clearer set of prompts to guide the discussion, and a guided worksheet to aid people in assessing a portfolio as to remove bias and focus on the content of the work.

What do you think?

So, these are the ideas I’ve had. And I’m sure Kristin has a bunch of things she’s considering. And I am wondering: for those who’ve read the book, what more could we address that would help you?

 

 

Design Exercises are a Bad Interviewing Practice

Recruiting and hiring is among the most difficult and time-consuming aspects of a design manager’s job, and wherever they gather and share experiences, the subject of design exercises inevitably comes up. We wrote about it briefly in our book:

Design Tests?

A topic of some controversy within product design circles is whether candidate interviews should involve some kind of design test or challenge akin to what happens in engineering interviews. Our firm, resolute response to this is “no.” Design tests set up an unhealthy power dynamic in the interview environment, when instead you should be fostering collegiality. The context in which the challenge is given (typically narrowly time-boxed and with only a little information and little support) is wholly artificial—and so whether a candidate succeeds or fails is not a meaningful indicator of actual practice. There is nothing you will find out in such a test that you couldn’t better learn through probing the candidate about their portfolio.

I had hoped that this would be sufficient and never need to be discussed again. Judging by lengthy multi-party threads on Twitter, I was wrong. Forthwith, a lengthier set of reasons for why design exercises are bad interview practice.

Design Firms Don’t Do Them

At Adaptive Path, we hired world-class designers without ever having them conduct a challenge. Same thing back at the first design firm I worked for, Studio Archetype, which was a standard-bearer for early digital design. These are companies whose sole purpose was the delivery of superlative design, and where the value was the talent of the people on staff. How were we able to assess their abilities? As alluded to in the passage above, portfolio reviews, including discussions of how they tackled design challenges.

They are a waste of time

If there’s nothing you can get from a design exercise that you can’t get from a portfolio review and a well-structured thoughtful interview, then it follows they are a waste of time. I call this out because recruiting and hiring is already monumentally time consuming, and anything that needlessly takes up time should be excised from the process.

Design Is Not Engineering

I can’t say for certain, as I haven’t done the research as to where design exercises emerged as an interviewing practice (it’s not from traditional design practice), but my guess is that they came about in technology companies where software engineering was the dominant practice. Design had to overcome its perception as squishy, soft, “make it pretty,” by demonstrating rigor, relying on data, and generally making the practice of design operate more like engineering.

And engineering hiring interviews involve technical exercises (coding challenges and the like), so shouldn’t design hiring interviews?

The thing is, coding challenges are waaaaay more straightforward than design exercises. There are demonstrably better ways to solve engineering problems. And in most coding exercises, the outcome is predetermined — it’s a matter of how would you realize it?

The same is not at all true for design. You’re not applying process to realize an already known outcome. You’re taking in a massive amount of input in order to navigate your way through the problem space. Unlike engineers, you need to consider business context, user needs, goals, and capabilities, brand concerns, technical constraints, channels of use, and god knows what else. And good designers know that there are many potential solutions to a problem, and require testing and iteration to get to anything like a good solution.

Design Exercises Bias Towards Facile Problem-Solving

Designers don’t all solve problems the same way. Some take in a lot of data, go off into a cave, noodle on it for a while, and come out with something great. Others iterate and prototype almost from the get go, uncovering solutions through refinement. Some require thinking out loud, and deep collaboration to get their best work. A great design organization has people with a variety of problem-solving modes and approaches, which enables the organization to better tackle a wide array of challenges.

The artificial constraints of design exercises (typically time-limited; a problem that the candidate isn’t prior familiar with, but which the interviewers are; performing under the scrutiny of others) biases toward a narrow range of problem solving.

A design exercise, by its very nature, is inclined towards facile solutions, and so biases teams towards facile designers. There’s not really any room for grokking depth.

Design Exercises Exacerbate An Already Problematic Power Dynamic

Design exercises ask candidates to perform on demand. In the context of a job interview, this only heightens the fraught power dynamic between an employer and prospective candidate. Even in markets where talent is in high demand, job interviews place candidates in a vulnerable situation. Being expected to perform on demand only adds to the candidate’s stress and anxiety, and makes for a suboptimal candidate experience. This Twitter exchange between my friends Ryan and Jared touches on this…

twitter

 

As Jared notes later in the thread, design exercises introduce cultural bias, too:

What about take-home exercises?

This is often the response to my ranty diatribes against design exercises. What if they’re take-home? Then people have all the time they need, and it’s the pressure cooker of performing-on-demand is.

Beyond the obvious problem, that’s still at the root of all of my issues with design exercises (for the people in the back: THEY ARE ARTIFICIAL CONSTRUCTS THAT DON’T REFLECT HOW DESIGN ACTUALLY HAPPENS), they introduce new issues… Namely, now you’re asking this person to do unpaid work. Young people with savings (i.e., don’t need money) and free time will be able to put a lot more effort into take-home exercises than, say, a single parent whose at-home time is focused on their children, and can only do “homework” after the kids asleep and when they’re likely exhausted.

Recognizing this, some companies do offer to pay people for taking the time to do a take-home exercise (which can help defray costs like child care), and that’s better than not doing so, but even better…? No exercises. Because you don’t need them. Because they add nothing to the recruiting and hiring process that can’t be figured out through thoughtful, experienced-based interviews, a savvy portfolio review, and speaking with people the candidate has worked with.

Design recruiting, portfolios, and showing work

Jared Spool tweeted about hiring designers and requiring portfolios a couple days ago:

As a design hiring manager. I require a portfolio. In Org Design for Design Orgs, I wrote:

The most important representation of a designer’s career is not their résumé, but their portfolio. Design managers end up reviewing dozens, if not hundreds, of portfolios for any role. For those designers who do not have public portfolios, ask to see one. Any designer under consideration must have a portfolio. No portfolio means no job.

As with any discussion started on Twitter, there exists greater nuance than can be comfortably communicated.

His blanket statement that “the best designers don’t have them” is wrong–many great designers do. But, yes, many great designers don’t have an updated portfolio handy, because they’re busy.

As a hiring manager, I am willing to start the recruiting process without a portfolio, if there are other positive indicators in play—their resume shows they’ve worked for companies with high design standards, or there’s a strong personal referral.

I advocate for 2 screening interviews before an on-site. In the case of someone without a portfolio, my first discussion would be to vet them as professionals, as people, their career, their trajectory, and if they seem to be a fit for the role I’m looking to fill (appropriate skills, seniority level, interest). The second discussion would be a detailed discussion of their work. While this wouldn’t need to be a formal portfolio, they would need to be able to walk me through 1-2 projects, even if that means pulling up working docs from hard drive folders.

But, and this is the key point: no designer is coming on-site without having shown their work in the screening process.

It’s also worth mentioning that when they ARE on-site, I do expect a more formal portfolio presentation, to be given to the interview panel and possibly other members of the design team. So if this person is serious about the role, they will need to find time in their ‘busy’ schedule to pull this material together.

If they claim to be too busy to do so, that’s a sign of disinterest, and it’s best to move on.