Trend 7: Accountability for Anticipating Design Outcomes
Designers must justify their research and adapt their methods to conform to rigorous standards expected of other business activities.
Meredith Davis and Deborah Littlejohn
Trend 7: Accountability for Anticipating Design Outcomes
Designers must justify their research and adapt their methods to conform to rigorous standards expected of other business activities.
Meredith Davis and Deborah Littlejohn

Trend 7: Accountability for Anticipating Design Outcomes

Download the PDF version with competencies.

Research informs practice and is an essential activity in many design offices. As a strategic voice in product development, marketing, and service, evidence-based design research has been asked to conform to rigorous standards and be measured by the same metrics as other primary business activities. Designers must justify research in terms of its continuing value, not purely on “See what we found!” Design researchers must adapt methods borrowed from other disciplines to design problems and define acceptable evidence for practical applications.

In a professional survey of AIGA members regarding their interests for continuing education, what was not mentioned was as important as what was said. As expected, business and technology ranked high in members’ priorities, but research was apparently below members’ radar. A 2005 study by Metropolis magazine of 1,051 design practitioners and academics also showed confusion regarding the perceived value of research.1 Some respondents thought research was choosing colors for a project, while others cited deep studies of user behavior. When asked what topics should receive the greatest research attention from the field, Metropolis respondents listed sustainability, culture, and technology. At the same time, however, they ranked systems theory, anthropology, and computer science among the topics least relevant to research. It is difficult to research issues of sustainability without an understanding of systems, to study culture without use of ethnographic methods, or to influence the development of technology without knowing a little bit about computer programming.

Further, for many students and practitioners, research is something that happens before design, not throughout the design and evaluation process. Students often confuse project research with understanding the subject matter of communication or with Internet searches and the library, not with issues of audience and context. And for professionals and students alike, the obligation to evaluate design success often ends with production. In these cases, there is no disciplined, systematic approach to understanding the drivers and consequences of design decisions.

Design researchers examine a number of important aspects of practice, seeking knowledge that can be applied to specific projects and/or generalized to the work of the field:

  • How designers think;
  • What people need and want (issues of usefulness, usability, and desirability; the roles information, products, and services play in people’s lives);
  • What context demands (issues of sustainability, social equity, cultural appropriateness, technological feasibility, and economic viability);
  • How design is planned, produced, and distributed;
  • How the field evaluates the outcomes of design (including throughout history and in culture); and
  • Methods for studying these things.

Design today is practiced in an interconnected world with increasing potential for significant social, cultural, environmental, technological, and economic effects (see also Trend: Complex Problems). While studying people’s physical and cognitive interactions with design in the context of use is important, gaining insight into how their motives and behavior are connected to complex systems is central to designing for human experience. In a counting-and-measuring culture that demands that evidence supports design decisions of expanding consequence, design professionals increasingly rely on research expertise. Undergraduate students who expect to practice during the next half century must be “research receptive”—able to read and apply research findings in their day-to-day work. And graduate students who expect to teach at the college level or lead professional offices must arrive “research ready.”

The most interesting research advances in many disciplines, not just design, occur at the intersections of differing fields, when one profession finds knowledge relevant in another. By introducing new expertise in the work of design—from organizational studies, behavior economics, neuroscience, cognitive psychology, anthropology, data science, urban planning, macroeconomics, and more—designers address some of the factors that produce outcomes for individuals, groups, organizations, and society. And as the scale and complexity of design problems grow, design research emerges as a new service offering in design practices, and attracts necessary collaborators from other fields.

Research is not only necessary in making convincing arguments for design decisions, but also in establishing evidence-based criteria that justify design practice in general (see also Trend: Resilient Organizations). In a 2005 study of more than 400 North American offices, Phinney Bischoff director David Miller concluded that design firms need to demonstrate measurable performance evidence of clients’ return on investment in design services. Miller claimed that, in many cases, evidence of successful performance outcomes was more important to clients and stakeholders than the design itself.2

Evidence of the trend in practice

User-centered research — Quantitatively and qualitatively, design research informs expectations of potential effects for users and the ecologies in which new technological products and services live. While nineteenth-century technology was invented to meet existing needs, today’s technologies often arrive before people know what they are good for. Rather than rely exclusively on self-reporting in interviews and focus groups that require respondents to imagine things that don’t yet exist, design researchers study patterns in what people actually do in their interactions with both current systems and prototypes of new products. They give voice to unmet user needs and wants, to hazy impressions not captured by survey questions. Extreme users—both novices and experts—represent a variety of contrasting motives and behaviors, making them especially useful in guiding the design of features and functions.

Further, research suggests that today’s technology users increasingly want to be producers of content and form, secondary “designers” of systems and services that adapt to their use. In response, professional designers shift from crafting discrete objects to developing tools and systems through which others create their own experiences and subsequently contribute to evolving versions of technology (see also Trend: Making Sense in the Data Economy). Technologist Gerhard Fischer describes a problem in thinking of people as passive “consumers,” arguing instead that people want to be designers in activities that are personally meaningful and important, and consumers only in routine activities they consider insignificant.3 Learning what people find meaningful is a task for design research.

For example, software designers at Xerox PARC developed Eureka, a communal knowledge sharing system for field service technicians, using design research. From field observations in ethnographically grounded case studies, researchers found that what technicians really needed was help with problems for which no standard solution was adequate.4 Technicians routinely invented solutions to these service anomalies and shared them informally with other technicians (see also Trend: Bridging Digital and Physical Experiences). Researchers also found that “fame” motivated sharing. As a result, Eureka vets service “tips” through the community’s most trusted peers and attributes them to specific technician authors. The company could have simply made changes to the standards manual, but discovered through research that recognition for innovative troubleshooting would encourage further invention and collaboration. The system saved Xerox more than $100 million in service costs over 12 years.

Many of today’s design research methods employ user-friendly approaches not only in determining people’s preferences for particular features and functions, but also, for what to make in the first place. Cognitive psychologist and design researcher Liz Sanders pioneered MakeTools, a collection of open-ended materials and research activities through which users envision new products and services. Using Sanders’ method, graduate researcher Michelle Wong asked seventh grade students to design an imaginary remote control device for interacting with her computer program on the anatomy of the human heart. Student designs included buttons that changed the “owner” of the heart, telling the researcher that they wanted to compare human anatomy to other species. Another button was for animating the heart under a variety of conditions—stress, exercise, and a heart attack—visualizations that don’t appear in middle school science textbooks. None of these ideas was part of Wong’s original design and would not have arisen had the researcher simply asked students to test the prototype.

Intelligent research tools — Wired magazine reported that from the beginning of recorded time until 2003, humans had created 5 billion gigabytes of information.5 IBM estimated that in 2017, 2.5 quintillion bytes of information were created every day. The United States alone generates 2,657,700 gigabytes of Internet data every minute. Many established research strategies break down under this volume and velocity of information.

Design researchers use data science and analytics to complement more traditional qualitative research methods (observation, ethnography, interviews, etc.) for insights into how people use designed systems (see also Trend: Resilient Organizations and Trend: Making Sense in the Data Economy). Intelligent tools collect evidence of users’ journeys and the frequency with which they interact with particular information, products, environments, and services. Large databases are a research resource if designers know the right questions to ask. Information collected through the Nest Learning Thermostat, for example, tells owners on demand about their energy usage, but by sensing when and where people are in the home, can also inform researchers about how people occupy buildings. Keywords used in search engines tell researchers about content that interests users at particular times of the day, week, or year. Merriam-Webster arrived at “feminism” as the 2017 word-of-the-year by following keyword searches. And better than surveys, the big data sets generated by search algorithms allow advertisers to match the content of ads to specific consumers and publishers to predict political bias in what people like to read.

Designers are both users and developers of intelligent systems. Machine learning and artificial intelligence give computers the ability to learn and respond to users’ queries without being explicitly programmed for particular answers. System intelligence grows as users interact, adding content and behavior patterns to massive databases (see also Trend: Aggregation and Curation and Trend: Making Sense in the Data Economy). Data mining searches for these patterns and transforms them into usable structures. A task for design researchers is to determine the right questions to ask of all of this information; to make sense of what design researcher Rick E. Robinson calls the “faces, places, and traces” of big data.

Quantone Music uses IBM’s Watson to collect and organize qualitative data on musical influence and popular opinion. Watson analyzes unstructured data from music blogs and articles for making recommendations of new music to users through Quantone’s app, MusicGeek. Talkspace is a global online platform that connects people with a therapist. The system uses Watson’s Personality Insights API to match mental health professionals to individual users with particular traits. Through an analysis of the user’s authored text, Watson categorizes social characteristics, thinking style, and emotional stress.

Sentiment analysis uses natural language processing, textual analysis, and biometric data to understand the emotional content of people’s responses in surveys and social media. Design researchers employ sentiment analysis in analyzing consumers’ reviews of products and services, as well as responses to branding and political messages. Microsoft’s Outlook add-in ToneDetector identifies the positive or negative tone of an email as the user types, and IBM’s Tone Analyzer detects different emotive states, such as “anger” or “agreeableness,” in business correspondence.

Faculty and graduate student research — While some design research is useful in addressing specific projects or settings, other research seeks to explain the fundamental nature of phenomena that are generalizable across a variety of applications or contexts. Because the attention to design research is relatively recent in comparison to work in other academic fields, there is ambiguity regarding definitions of design faculty scholarship and the curricular content of design research at the graduate level. In the United States there are only a handful of Ph.D. programs in design that prepare practitioners and faculty in evidence-based research paradigms and methods; that is, in posing questions and applying methods that yield new findings for the discipline. To date, the United States has resisted titling practice-based doctoral degrees as Ph.D.s. By contrast, a reclassification of European universities in the 1990s created confusion over a single degree title (Ph.D.), which now stands for both research programs that generate new knowledge and those based in personal reflections on professional practice.

On the other hand, master’s programs that simply refine skills students acquired, or even missed, in undergraduate study seem less economically viable than in the past. This argues for a new, more ambitious curricular agenda that differentiates advanced degrees from undergraduate study, prepares the professoriate for the scholarship demands of research universities, and qualifies graduates for leadership positions in research-oriented design offices and companies. To deliver on this promise, however, programs must take a systematic approach to teaching research methods and help students in framing meaningful investigations.

Donald Norman is former vice president of the Advanced Technology Group at Apple, cofounder of the Nielsen Norman Group, and director of The Design Lab, a design research center at the University of California, San Diego. In a 2010 article for Core 77, Norman decried designers’ educational shortfall in dealing with the social and political issues inherent in interaction, service, and experience design research. He described new areas of practice as being more like the applied social and behavioral sciences, but unfortunately are under the direction of designers who “think they know but don’t” and who are unconsciously influenced by their biases.6 Norman called for design education to change by teaching about technology within a political and business context and by establishing new courses in the behavioral sciences that are appropriate to the applied requirements of design practice and research. He distinguished between academic research and research in practice, with the latter looking for large effects, needing immediate results, and using simple methods and small sample sizes.

It is unlikely that the Ph.D. will become the terminal degree for American design faculty in the near future, but this is no reason to delay the development of research programs that support growing demand for knowledge in the field. Design faculty must partner with colleagues from more established research disciplines and connect their investigations to issues valued by practice. Undergraduate students must graduate knowing how to read and apply the findings of more experienced investigators. Graduate students must know how to structure research investigations and write summaries for dissemination. Research undertaken by professional offices is apt to remain proprietary—trade secrets until the findings no longer represent a competitive advantage. Universities, on the other hand, have an obligation to share research results broadly with businesses and the public through publications and presentations. Therefore, it is in the best interest of all to build a healthy research culture for the field.

Core concepts and principles

Phases of design research — Designers have a variety of research tools at their disposal, some unique to design and some borrowed from other disciplines. Just as the design process is creative and iterative, so is the research process. It is important to choose the right tool for the right phase of the design process. Martin and Hanington align five stages of research with the design process:7

  • Planning, scoping, and definition are where the research explores and defines the overall project.
  • Exploration, synthesis, and implications make use of exploratory methods, such as ethnography, to gain deep understanding of the problem context and to suggest possible design implications.
  • Concept generation and early prototype iteration comprise a stage in which design benefits from participatory and generative methods that involve users and stakeholder input.
  • Evaluation, refinement, and production add to knowledge gained in earlier stages through iterative testing and feedback from a variety of sources.
  • Launch and monitoring take place after production, when the design is completed but not yet released into the marketplace. Research at this stage includes quality assurance testing to confirm that the design is ready for the public. Once the design is released, ongoing reviews can signal if revisions are necessary—a feedback loop for continuous improvement.

Formative research — Formative research is exploratory and typically informs the beginning stages of a project. It focuses on process during development activities preceding a proposed solution to a design problem. Designers use formative studies to gain insight into the problem setting, stakeholders, and activities. For example, design researchers may use observational methods to identify user “pain points” in an existing service system. 

Evaluative research — Summative or conclusion research helps the designer frame the outcomes of an investigative process. This type of research is typically used at the end of a project to determine success and overall effects, including those not anticipated by the designer. It can gauge user satisfaction and determine future research directions.

Qualitative versus quantitative research — This refers to the type of data collected, as well as how the data is reported. As a form of formative or exploratory research, data collected using qualitative methods often take the form of “rich” or “thick” descriptions of people and settings.8 Analysis involves searching for patterns or shared concepts. Observational studies and interviews (or a combination of both) are common methods that form in-depth understanding of human behavior, as well as the meanings that motivate behavior and the places where it occurs.9

The goal of a quantitative study is often to predict outcomes or to determine that there is or is not a relationship among variables. Researchers typically collect quantitative data as numbers or categories—for example, how often an event takes place or groupings of people by demographic indicators. Quantitative research generally uses statistics to report outcomes. Quantitative research is useful when studying large numbers of people or cumulative results in something over time, but when participants are few or situations are unstable, statistics can mislead.

Ethnographic studies — Ethnographic study observes participants in natural settings, rather than in labs. The researcher immerses him/herself in the culture and records observations in field notes and images. The goal of this research is to understand the participants’ points of view and behavior. Ethnographic research is useful in scoping and defining a problem from a user perspective. Knowing what people want to do with technology and how it fits into their lives, for example, is very different from testing their interactions with a particular device or program in a lab.

IDEO researchers studied the concept of “luxury.” In an interview, a female participant described herself as not really a consumer of luxury products or services. When re-interviewed after being observed by researchers making weekly appointments to have her nails done, she replied, “Well, that is a necessity, not a luxury.” In other words, the research challenged the designers’ original perceptions of concept boundaries.

Experimental studies — In experimental research, the researcher systematically manipulates one problem variable at a time to determine its effect on other variables. Experimental research may seek proof of a cause, or it may reach conclusions about interdependence among many variables.

Donald Meeker, working with graphic designer Chris O’Hara and type designer James Montalbano, proposed Clearview typeface for highway signage. In contrast to the traditional Highway Gothic, the design cleaned up the intersections of letterform strokes to mitigate a halo effect at night, increased the size of counterspaces, and differentiated the form of the lowercase “i” and “l” without increasing the size of the sign. Each of these visual variables could be tested individually for an effect on the speed and accuracy of drivers’ interpretations. Although research indicated improved legibility under these changes, the federal highway authority unfortunately returned to its original font, claiming that the positive outcomes of Clearview were the result of test signs being newer with less wear than existing signs. Unfortunately, the government felt it could not assign improvement to the typeface alone because another variable (sign age) compromised the experimental method.

Case studies — Case study research examines a phenomenon in a particular context. It is a holistic approach that is useful when the boundaries of the phenomenon are unclear and multiple types of evidence are available.10 Case studies are used to determine how theories or models actually work in real life, to report results that inform future action. User satisfaction with products developed through their participation, or how interdisciplinary teams collaborate in software development are examples of topics that might be pursued through case studies. The case study method is often used in business with the hope that what is found in one situation might be successful when applied to another.

Challenges for designers

In academia, faculty must find more experienced research partners and sources of funding that expand the influence of design beyond the arts. They must fill in gaps in their understanding of research methods and frame investigations that matter to practice and society. They must argue for research-sensitive standards that guide the evaluation of students and faculty and for forms of dissemination that may be at odds with the typical expectations of art departments. Given the importance of user-centered issues in design research, they must become familiar with ethics and institutional review practices related to research with human subjects.

As the speed with which design offices and technology firms turn around projects accelerates, design professionals must develop research methods that match the velocity and variety of research challenges. Rapid prototyping allows designers to test solutions under a variety of conditions and scenarios of use. Increasingly, development methods include launching technology for public use as a way of troubleshooting.

The current context for professional practice challenges designers to demonstrate how their solutions to design problems perform in ways that are measurable and generalizable to other problems and contexts. Marketing is data-driven and human factors experts have metrics for describing technological performance. While all research need not be quantitative, business expects designers to demonstrate economic as well as social performance. Questions frequently arise regarding, “Has design made a difference and by how much?” Design researchers need to develop convincing stories that validate the contributions design makes to solving problems and measures that are both authentic to the issues of design and compelling evidence for others.

Resources

Bennett, A. and Heller, S. (2006). Design Studies: Theory and Research in Graphic Design. New York, NY: Princeton Architectural Press.

Cresswell, J. and Cresswell, D. (2018). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage Publications.

Cresswell, J. and Poth, C. (2017). Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Thousand Oaks, CA: Sage Publications.

Groat, L. and Wang, D. (2013). Architectural Research Methods. Hoboken, NJ: John Wiley and Sons.

Hanington, Bruce and Bella Martin. (2012). Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Berkley, MA: Rockport Publishers.

IBM Big Data & Analytics Hub. Retrieved in January 2018.

Ignatow, G. and Mihalcea, R. (2017). An Introduction to Text Mining: Research Design, Data Collection, and Analysis. Thousand Oaks, CA: Sage Publications.

Kumar, V. (2012). 101 Methods: A Structured Approach to Driving Innovation in Your Organization. Hoboken, NJ: John Wiley and Sons.

Laurel, B. and Lunenfeld, P. (2003). Design Research: Methods and Perspectives. Cambridge, MA: MIT Press.

Muratovski, G. (2015). Research for Design: A Guide to Methods and Practice. Thousand Oaks, CA: Sage Publications.

Rodgers, P. and Yee, J. (2014). The Routledge Companion to Design Research. New York, NY: Routledge.

Sanders, L. and Stappers, P. (2013). Convivial Toolbox: Generative Research for the Front End of Design. Amsterdam, Netherlands: BIS Publishers.

Visocky O’Grady, J. and Visocky O’Grady, K. (2017). A Designer’s Research Manual. Berkley, MA: Rockport Publishers.

Yin, R. (2017). Case Study Research and Applications: Design and Methods. Thousand Oaks, CA: Sage Publications.

Research-oriented design journals

Design Issues

Design Studies

Human-Computer Interaction

International Journal of Design

Journal of Design Research

She Ji: The Journal of Design, Economics, and Innovation

Visible Language

1.Manfra, L. (2005). “School survey 2005: Research– Its Role in North American Design Education.” Metropolis, 25(1), 132–134.

2. Miller, D.C. “Design in the Age of Accountability.” Communication Arts. Retrieved in January 2018.

3. Fischer, G. (2002). “Beyond Couch Potatoes: From Consumer to Designers and Active Contributors.” First Monday. Retrieved in January 2018.

4. Szymanski, M. and Whalen, J. “Communal Knowledge Sharing: The Eureka Story.” Xerox Parc. Retrieved in January 2018.

5. Hoffer, D. “What Does Big Data Look Like? Visualization is Key for Humans.” Wired magazine. Retrieved in January 2018.

6. Norman, D. (2010). “Why Design Education Must Change.” Core 77. Retrieved in January 2018.

7. Hanington, Bruce and Bella Martin. (2012). Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Berkley, MA: Rockport Publishers.

8. Geertz, C. (1973). “Thick Description: Toward an Interpretive Theory of Culture.” In The Interpretation of Cultures: Selected Essays. New York: Basic Books, 3–30.

9. Groat, L. and Wang, D. (2013). Architectural Research Methods. Hoboken, NJ: John Wiley and Sons.

10. Yin, R. (2017). Case Study Research and Applications: Design and Methods. Thousand Oaks, CA: Sage Publications.

Users increasingly want to be producers of content and form, secondary “designers” of systems and services.
A task for design researchers is to determine the right questions to ask of all of this information.
Design researchers need to develop convincing stories that validate the contributions design makes to solving problems.

Tags data science research