Testing Visual Documentation
Table of Contents
Alex spent three weeks creating the perfect diagram of his teamâs AI recommendation system. It was a work of artâcolorful, comprehensive, with perfect layout and typography. He proudly added it to the documentation and waited for the praise to roll in.
Instead, he got confusion. âWhat does this arrow mean?â âI donât understand how the data flows.â âWhich part represents the model training?â
The painful truth hit him: a visualization that makes perfect sense to its creator might be completely baffling to everyone else.
Sound familiar? Youâre not alone. Weâve all been Alex at some point, creating visuals that make perfect sense in our heads but confuse our audience. The good news? With the right testing approach, you can catch these issues before they cause problems.
In this module, weâll explore how to test your visual documentation to ensure it actually accomplishes what you need: transferring understanding from your brain to someone elseâs.
Why Even Beautiful Visualizations Can Fail Miserably
Letâs start with a simple truth: your brain is playing tricks on you. When you create a visualization, you bring along all your background knowledge and context. You see things in your diagram that others literally cannot see.
This phenomenon has a name: the curse of knowledge. Once you know something, itâs nearly impossible to imagine what itâs like not to know it. This creates a massive blind spot when creating visualizations for others.
Common visualization failures include:
- Assumed knowledge gaps: âObviously everyone knows what a transformer architecture is!â (Narrator: They donât.)
- Unclear flow: Arrows pointing everywhere like a plate of visual spaghetti
- Symbol confusion: Using icons or symbols that make sense to you but are cryptic to others
- Cognitive overload: Cramming so much information into one diagram that viewers donât know where to look
- Missing context: Failing to explain what the visualization is showing and why it matters
The truth is, weâre all terrible judges of our own visualizationsâ clarity. Thatâs why testing isnât optionalâitâs essential.
The Visual Testing Toolkit: 5 Methods to Validate Your Diagrams
1. The âFresh Eyesâ Test: Getting Immediate Reactions
This is your first line of defense against unclear visualizations. Show your diagram to someone who hasnât seen it before and ask:
What to do:
- Find a colleague who isnât familiar with your visualization
- Show them the image for 30 seconds (no more!)
- Take it away and ask: âWhat do you think this is showing?â
- Listen carefully without interrupting or defending your work
What youâll learn:
- Whether the main point is immediately clear
- What elements draw attention first (which might not be what you intended)
- What generates confusion or questions
The reality check: If they canât explain the main point back to you in simple terms, your visualization needs workâno matter how beautiful or technically accurate it is.
2. The âFive Secondâ Test: Testing Immediate Comprehension
This test measures whether viewers can grasp the key information in your visualization almost instantly.
What to do:
- Show your visualization to a tester for exactly five seconds
- Hide it and ask a specific question: âWhat was the relationship between component A and component B?â or âWhat was the main trend shown?â
What youâll learn:
- Whether key relationships are visually obvious
- If important information stands out appropriately
- How well your visual hierarchy (whatâs emphasized) works
Quick tip: Use a free online tool like UsabilityHub to run this test with multiple people remotely.
3. The âDescribe and Drawâ Test: Checking for Mental Model Transfer
This powerful test reveals whether your visualization successfully transfers your mental model to the viewer.
What to do:
- Show your visualization to someone for 60 seconds
- Take it away and give them a blank piece of paper
- Ask them to draw what they remember, focusing on the relationships and structure (not artistic quality!)
- Compare their drawing to your original
What youâll learn:
- Which elements were memorable enough to be reproduced
- How the relationships between components were understood
- What was completely missed or misunderstood
The humbling truth: Prepare to be surprised by what people actually take away from your visualization versus what you thought you were communicating!
4. The âPrediction Testâ: Validating Understanding Through Questions
This test checks whether viewers can use your visualization to reason about the system it represents.
What to do:
- Show your visualization and ask âwhat ifâ questions:
- âWhat would happen in the system if component X failed?â
- âHow would the data flow change if we added another step here?â
- âWhich part would need to change if we wanted to add feature Z?â
What youâll learn:
- Whether your visualization conveys the systemâs behavior, not just its structure
- If cause-and-effect relationships are clear
- How well viewers can use the visualization to reason about changes
Make it concrete: For an AI system flowchart, you might ask: âIf we wanted to add more training data, where would it enter the system?â
5. The âExpert vs. Noviceâ Test: Ensuring Appropriateness for Your Audience
Different audiences need different levels of detail. This test ensures your visualization works for its intended audience.
What to do:
- Show your visualization to both an expert in the subject and a relative novice
- Ask both the same questions about what they understand
- Compare their responses
What youâll learn:
- Whether your visualization is too technical or too simplified
- If youâve found the right balance for your target audience
- How to adjust based on who will actually use the documentation
The balancing act: If experts say âthis is oversimplifiedâ but novices say âI canât follow this,â you might need multiple visualizations for different audience levels.
The Testing Process: From Creation to Validation
Letâs walk through a practical process for incorporating testing into your visual documentation workflow:
1. Test Early with Low-Fidelity Mockups
Donât wait until youâve created a polished visualization to start testing. Begin with simple sketches.
Why this works:
- People are more comfortable giving honest feedback on something that looks unfinished
- Youâll waste less time redoing polished work
- You can test multiple approaches quickly
Pro tip: Even a paper sketch or whiteboard drawing can be tested. Take a photo and show it to colleagues for quick reactions.
2. Incorporate Feedback Systematically
Develop a system for collecting and acting on visualization feedback.
Simple feedback template:
- What was clear about this visualization?
- What was confusing or raised questions?
- What information seemed to be missing?
- What felt unnecessary or distracting?
The documentation hack: Keep a âvisualization feedback logâ where you record common issues. Youâll start seeing patterns that will improve all your future visualizations.
3. Test with Your Actual Target Audience
Whenever possible, test with people who actually represent your end users.
Who to include:
- New team members (great proxies for external developers)
- People from adjacent teams who need to understand your system
- Actual customers or users (if you can access them)
- People with different technical backgrounds
Reality check: Your fellow AI engineers are almost certainly not representative of most of your documentation users. They know too much!
4. Check for Accessibility Issues
Make sure your visualizations work for everyone, including people with color blindness or other visual impairments.
Quick checks:
- Run your visualization through a color blindness simulator like Coblis
- Ensure text within visualizations has sufficient contrast
- Check that information isnât conveyed by color alone
The simple test: Convert your image to grayscale. If you lose important information, you need to fix it.
5. Iterate Based on Testing Results
Testing isnât a one-time event. Plan for multiple rounds of improvement.
Iterative testing checklist:
- Make one major change at a time so you can tell what improved
- Test with fresh participants (previous testers are now âcontaminatedâ with knowledge)
- Document what changed and how results improved
- Know when to stopâperfect is the enemy of good enough!
Common Problems and Their Visual Solutions
Based on thousands of tested visualizations, here are the most common problems that emerge during testing and how to fix them:
Problem: âI donât know where to start lookingâ
Solution: Add a clear visual entry point, usually at the top left for Western readers. Use size, color, or position to create a clear starting point for the viewerâs journey through your visualization.
Problem: âI donât understand what these arrows meanâ
Solution: Be consistent with connector meanings. Consider adding a small legend explaining different types of connections (data flow, control flow, dependencies).
Problem: âThis is too complicated to understandâ
Solution: Layer your information. Start with a simplified high-level view, then allow viewers to âdrill downâ into more complex layers, either through interactive elements or through a series of increasingly detailed visualizations.
Problem: âI canât tell which parts are most importantâ
Solution: Use visual hierarchy principles. The most important elements should be:
- Larger
- More colorful/higher contrast
- More centrally located
- Surrounded by more white space
Problem: âI donât see how this relates to what Iâm trying to doâ
Solution: Add practical examples or use cases directly in your visualization. Show how the diagram connects to real-world usage.
Case Study: Before and After Testing
Letâs look at a real example of how testing transformed an AI system visualization:
The original diagram was a complex flowchart of a machine learning pipeline that the team thought was crystal clear. It showed every component of the system with careful labeling.
The test results were eye-opening:
- 7 out of 8 testers couldnât identify the overall purpose of the system
- Most couldnât trace the path of data through the system
- Several mistook training components for inference components
The redesign broke the visualization into three simpler diagrams:
- A high-level overview showing the major system components
- A detailed training pipeline visualization
- A separate inference pipeline visualization
The results after redesign:
- 90% of testers could correctly describe the systemâs purpose
- Users could accurately trace data flows through each subsystem
- New team members reported using the diagrams as reference during their first weeks
The lesson: Sometimes the best fix for a confusing visualization is to break it into multiple, simpler visualizations with a clear narrative connecting them.
DIY User Testing Exercise: Test Your Own Visualization
Time to put these principles into practice with your own visualization:
The mission: Run a simple test on one of your existing AI system visualizations (or create a new one if needed).
Your testing plan:
- Select a visualization from your documentation
- Find 2-3 people who arenât familiar with it (colleagues from other teams work great)
- Run the âFresh Eyesâ test described above
- Document what you learn
- Make at least one improvement based on the feedback
- Test again to see if comprehension improves
Reflection questions:
- Were you surprised by what people did or didnât understand?
- What assumptions had you made that werenât obvious to others?
- How might this impact your approach to future visualizations?
Advanced Testing Approaches: When Stakes Are High
For critical documentation where clarity is essential, consider these more formal testing approaches:
Eye-Tracking Analysis
Eye-tracking technology shows exactly where people look when viewing your visualization and in what order.
When to use it: For extremely important documentation where you need to ensure users follow a specific visual path to understand complex systems.
How to do it simply: While professional eye-tracking is expensive, you can approximate it by having someone talk through what theyâre looking at in real-time as they view your visualization.
A/B Testing Different Approaches
When you have multiple possible visualization approaches, test them against each other.
Simple method:
- Create two different visualizations of the same information
- Show each to a similar group of testers
- Ask the same questions of both groups
- Compare comprehension rates and accuracy of understanding
The data advantage: This gives you quantifiable data about which approach works better, not just subjective opinions.
Longitudinal Testing: The Long-Term Memory Test
This test checks whether your visualization creates lasting understanding.
How it works:
- Show testers your visualization and ask comprehension questions
- One week later, ask the same questions without showing the visualization again
- See how much theyâve retained
Why it matters: Sometimes visualizations that seem clear in the moment donât create lasting mental modelsâthis test reveals which explanations stick.
Resources to Deepen Your Testing Toolkit
Books and Articles
- Donât Make Me Think by Steve Krug - The classic on usability testing
- Visualizing Data by Andy Kirk - Includes excellent sections on evaluation
- User Testing Visual Design by Nielsen Norman Group
Testing Tools
- UsabilityHub - Run remote five-second tests and more
- Optimal Workshop - Tools for testing information architecture
- Maze - User testing platform with visual reporting
Communities for Feedback
- UX Stack Exchange - Get expert feedback on visualization approaches
- DataViz Society - Community of data visualization professionals
- Information is Beautiful Awards - See exemplary visualizations that have been thoroughly tested
Learning from the Pros: Resources for Further Study
For those who want to dive deeper into visualization testing methodologies:
- Nielsen Norman Group on Visualization Usability - Research-based guidance on testing information graphics
- Microsoftâs Guidelines for AI UI Design - Includes guidance on testing visualizations for AI interfaces
- Googleâs People + AI Guidebook - Guidance on explaining and visualizing AI
- Fernanda Viégas and Martin Wattenberg - Pioneers in the testing and evaluation of data visualization
- DataViz Society - Community of data visualization professionals
- Information is Beautiful Awards - See exemplary visualizations that have been thoroughly tested
Frequently Asked Questions
Get answers to common questions about testing visual documentation for AI systems, including methods for effective testing, approaches for different audiences, and strategies for improving visualizations based on test results.
Testing Fundamentals
Testing visual documentation for AI systems is critical because: 1) AI concepts are inherently complex and abstract, making visualization errors more likely and more problematic; 2) The âcurse of knowledgeâ is particularly severe in AIâexperts often fail to recognize when their visualizations assume specialized knowledge; 3) Visualization misunderstandings can lead to incorrect mental models about how AI systems work, potentially causing misuse or safety issues; 4) AI documentation often crosses technical boundaries, requiring visualizations that work for diverse audiences with varying technical backgrounds; 5) Many AI concepts (like neural networks or embedding spaces) have no physical real-world analog, making intuitive visualization especially challenging; and 6) The stakes are higherâin domains like healthcare, finance, or autonomous systems, misunderstanding AI behavior due to poor visualization can have serious consequences. Unlike documentation for more familiar technologies, AI visualizations often need to bridge significant knowledge gaps between creators and users, making testing essential to ensure they actually transfer understanding rather than creating confusion.
The most common problems revealed during testing of AI system visualizations include: 1) Excessive complexityâtoo many components or connections shown simultaneously, overwhelming viewers; 2) Unclear data flowâusers cannot trace how information moves through the system; 3) Undefined visual languageâinconsistent use of shapes, colors, and connection types without clear meaning; 4) Missing contextâfailing to explain what the visualization represents within the larger system; 5) Terminology disconnectsâusing technical terms or acronyms that arenât explained; 6) Invisible assumptionsâmaking conceptual leaps that seem obvious to experts but confuse others; 7) Poor visual hierarchyâfailing to emphasize the most important elements; 8) Accessibility issuesâusing color as the sole differentiator between important elements; 9) Abstraction problemsâvisualizations that are either too abstract (vague boxes and arrows) or too detailed (showing every computation); and 10) Audience mismatchâcreating visuals too technical for business users or too simplified for technical implementers. Many of these issues stem from the AI fieldâs tendency to use specialized language and concepts that arenât widely understood outside expert circles, making testing with representative users particularly valuable.
For most AI documentation visualization testing, you need surprisingly few testers to identify the majority of problems: 1) 3-5 participants will typically reveal about 80% of the major usability issuesâthis follows Jakob Nielsenâs well-established user testing research; 2) However, for AI visualizations specifically, you should ensure representation across different user types (e.g., data scientists, software engineers, business stakeholders) since each brings different knowledge and expectations; 3) For initial testing, even 1-2 people who arenât familiar with your specific AI system can provide invaluable feedback; 4) Rather than testing with many people at once, itâs more effective to run multiple small testing rounds with iterative improvements between them; 5) For critical, high-stakes AI documentation (like safety-critical systems), consider increasing to 7-10 testers per round to catch more edge cases; and 6) The quality of testers matters more than quantityâone tester who truly represents your target audience provides more valuable feedback than several who donât. The most important factor is selecting testers who match your documentationâs actual audience in terms of technical background and domain knowledge, rather than simply testing with conveniently available colleagues.
Testing Methods and Approaches
If you have very limited time, the âFresh Eyesâ test provides the most valuable quick feedback for AI visualizations: 1) Find someone unfamiliar with your specific visualization (a colleague from another team is ideal); 2) Show them the visualization for exactly 30 seconds; 3) Remove it from view and ask: âWhat do you think this is showing?â and âWhat would you say is the main point?â; 4) Listen without interrupting or defending your work; 5) Ask what elements they remember and what questions they have. This approach takes less than 5 minutes but reveals whether your visualization communicates its core message effectively. For AI documentation specifically, also ask one follow-up question related to system behavior: âBased on what you saw, how do you think this AI system processes data?â This reveals whether your visualization creates an accurate mental model, not just superficial understanding. The key to this testâs effectiveness is selecting someone with a technical background similar to your target audience but without prior knowledge of your specific visualizationâthis combination provides the most relevant feedback in the shortest time.
To test AI visualizations for both technical and non-technical audiences: 1) Conduct parallel testing with at least two people from each audience group; 2) Use the same testing protocol but customize questions to reflect different usage needsâtechnical users might need to implement the system, while business users need to explain it to stakeholders; 3) Compare responses to identify comprehension gapsâlook for concepts that technical audiences understand but non-technical audiences miss; 4) Pay special attention to terminology confusion among non-technical testers and oversimplification concerns from technical testers; 5) Use a âlayered explanationâ approachâshow a high-level visualization first, then reveal more technical details, and test how well each layer works for different audiences; 6) For mixed audiences, try the âexplain it backâ testâask a technical person to view the visualization and explain it to a non-technical person, observing where communication breaks down; and 7) Consider creating separate but visually consistent visualizations for different audiences if testing reveals significant comprehension gaps. The most effective AI documentation often uses a progressive disclosure approach, where core concepts work for all audiences while additional visual layers provide the technical depth that specialists require.
Testing interactive AI visualizations requires additional considerations beyond static testing: 1) Observe without guidance firstâwatch users interact without providing instructions to identify whatâs naturally discoverable; 2) Track the exploration pathânote which interactive elements users discover and in what order; 3) Identify unused featuresâpay attention to interactive elements users never discover or use; 4) Test on actual deployment platformsâensure the interactivity works on all intended devices and browsers; 5) Use think-aloud protocolsâask users to verbalize their thoughts as they explore to understand their mental process; 6) Test for interaction errorsâobserve if users attempt interactions that donât exist but seem intuitive to them; 7) Measure time to insightâcompare how quickly users answer key questions with interactive versus static versions; 8) Check for cognitive loadâassess whether the interactivity enhances understanding or becomes a distraction; 9) Test accessibilityâensure interactive elements work with keyboard navigation and screen readers; and 10) Conduct retention testingâafter using the interactive visualization, test what information users remember later. For AI-specific interactive visualizations, also verify that users can correctly predict system behavior after manipulating parameters, which reveals whether the interaction created accurate mental models of the underlying AI system.
Testing Improvements and Iteration
After testing AI visualizations, prioritize fixes in this order: 1) Fundamental comprehension blockersâissues that prevent users from understanding the basic purpose or function of the AI system, such as completely misinterpreted data flows or relationships; 2) Safety and risk misunderstandingsâerrors that could lead to incorrect usage or false assumptions about system capabilities or limitations; 3) Key concept confusionsâmisinterpretations of core AI concepts that form the foundation for other understanding; 4) Cognitive overload issuesâexcessive complexity that overwhelms users and prevents them from extracting meaning; 5) Navigation and flow problemsâconfusion about how to read or follow the visualizationâs intended sequence; 6) Missing contextâabsent explanations or references that leave users unable to connect the visualization to its purpose; 7) Technical accuracy concernsâcorrectly understood but technically inaccurate representations that could mislead experienced users; 8) Accessibility barriersâissues that prevent certain users from accessing the information; 9) Visual hierarchy weaknessesâimportant elements that donât stand out appropriately; and 10) Aesthetic and professional appearanceâissues of finish and polish. For AI documentation specifically, prioritize fixes that address the âblack box problemââensure your visualization helps users develop accurate mental models of how the AI system processes information and makes decisions, as this is often the most critical function of AI visualization.
The decision between redesigning or incrementally improving an AI visualization should be based on these factors: 1) Start with a complete redesign if: multiple testers cannot identify the basic purpose of the visualization; users develop dangerously incorrect mental models of the AI system; the visualization is trying to show too many aspects simultaneously; or technical experts identify fundamental conceptual errors in the representation; 2) Choose incremental improvements when: the basic concept is understood but specific elements cause confusion; the structure works but terminology or labeling needs clarification; the visualization is mostly effective but has specific points of confusion; or when time constraints make a full redesign impractical; 3) A hybrid approach often works bestâmaintain the effective core structure while completely reworking problematic sections; 4) Quantify the issuesâif more than 30% of the visualization elements are problematic, a redesign is usually more efficient than numerous small fixes; 5) Consider audience adaptationâsometimes what appears to need a redesign simply needs to be adapted for a different audience rather than fundamentally changed. For AI visualizations specifically, be more willing to redesign when the visualization fails to accurately represent the systemâs actual behavior or decision-making process, as this can lead to dangerous misunderstandings about AI capabilities and limitations.
To objectively measure improvement in your revised AI visualization: 1) Define specific success metrics before testingâsuch as time to comprehension, accuracy of understanding key concepts, or ability to complete specific tasks using the information; 2) Conduct A/B testingâshow some users the original and others the revised version, using identical testing protocols to enable direct comparison; 3) Use task completion scenariosââUsing this visualization, explain how this AI system would handle [specific scenario]â and measure accuracy; 4) Implement comprehension scoringâcreate a list of key points users should understand from the visualization and score how many they correctly grasp; 5) Track eye movement patternsâeven informal observation of where users look first and how they scan the visualization can reveal improvements in visual flow; 6) Measure time to first correct insightâhow quickly can users extract meaningful information; 7) Count questions and confusionsâtrack the number of clarifying questions users ask with each version; 8) Assess confidence levelsâask users to rate their confidence in their understanding after viewing each version; and 9) Conduct delayed recall testsâcheck what users remember about the system days after viewing the visualization. For AI-specific visualizations, also measure the accuracy of usersâ predictions about how the AI system would behave in novel situations, as this reveals whether the visualization has created an accurate mental model of the underlying system mechanics.
Test Your Knowledge
Test your understanding of visual documentation testing principles and techniques with this quiz!
Testing Visual Documentation Quiz
According to the chapter, what cognitive bias makes creators poor judges of their own visualization's clarity?
Wrapping Up: From Testing to Excellence
Testing your visualizations isnât just about avoiding embarrassing mistakesâitâs about developing a fundamental skill: the ability to see your work through othersâ eyes.
With each test, youâll build stronger instincts for what works and what doesnât. Youâll start anticipating confusion points before they happen. Youâll develop a sixth sense for clarity.
Remember: the goal isnât perfect visualizations (thereâs no such thing). The goal is visualizations that successfully transfer understanding to your specific audience with minimal friction.
In our next module, weâll explore the broader process of documentation review, building on these testing principles to ensure your entire documentation system works together seamlessly.
As the visualization expert Alberto Cairo puts it: âA good visualization isnât about showing the dataâitâs about enabling understanding.â With the testing approaches in this module, youâre now equipped to ensure your visualizations do exactly that, turning complex AI concepts into clear, accessible knowledge for everyone who needs it.
Found value in the course?
Your support fuels my work!
Have an issue?
Help improve the course by reporting an issue
