Prof. Judy Fan: Cognitive Tools for Making the Invisible Visible
Summary
This talk explores the human capacity for cognitive innovation, specifically how we create and utilize 'cognitive tools' like the number line and data visualizations. The research emphasizes the interplay between visual abstraction, communication, and engineering, arguing that understanding these processes is crucial for explaining human progress. The speaker presents studies on how people use drawings to convey information, distinguishing between depictions and explanations, and investigates the capabilities of AI in understanding and generating sketches. Finally, the talk delves into data visualization, examining how humans and AI reason with graphical data and the challenges in developing effective and interpretable AI models for understanding visualizations.
Key Insights
Visual abstraction is a key human cognitive tool used to communicate knowledge by highlighting relevant aspects of information.
These examples of making the invisible visible leverage 'visual abstraction' to communicate what we see and know. This abstraction highlights what is relevant to notice, whether through detailed representations like Darwin's finches or more schematic ones. This ability to reformulate understanding in terms of useful abstractions drives technological progress and the re-engineering of the physical world.
Explaining human innovation requires accounting for both cognitive tools (abstraction) and engineering (creation).
Traditional cognitive psychology primarily focuses on how people process external information and social cognition involves multi-individual interactions. However, to understand human discovery and invention, two crucial elements are missing: 1) cognitive tools (material objects encoding information to shape thought) and 2) engineering (leveraging understanding to create new, useful things). Embracing both abstraction and engineering is necessary to explain the world's development.
Research aims to bridge the gap between discovering useful abstractions and applying them to create new things.
The core research goal is to develop psychological theories that explain both how humans discover useful abstractions about the world and how they apply these abstractions to create new things. This involves closing the loop between understanding and creation.
Visual explanations prioritize mechanistic information over visual fidelity.
Comparing visual depictions and explanations, studies found that explanations emphasize causal parts and symbolic displays of motion/interaction more than depictions. Depictions, conversely, emphasize background elements to aid object identification.
Data visualization is a powerful tool for resolving patterns that are too large, noisy, or slow to observe directly.
Data visualization, like scientific instruments, helps reveal aspects of the world not directly observable. Plots can expose patterns in data that are too vast, complex, or time-consuming to grasp through direct perception, making them indispensable in science, business, and decision-making.
Existing tests of visualization understanding may not accurately characterize the underlying skills being measured.
An analysis of common visualization tests (GGR, VLAT) indicates that item difficulty is convergent across different participant samples, and performance is correlated between tests. However, performance is not consistently linked to plot type or question type, suggesting underlying factors beyond these categories drive error patterns.
Cognitive tools like the number line are human inventions, not natural, enabling abstract thought and mathematical discovery.
The speaker introduces the concept of cognitive tools, using the number line as a familiar example. This tool, invented by humans, extends beyond natural forms to facilitate abstract reasoning. The invention of rectangular coordinates by Descartes is highlighted as a revolutionary advancement that linked algebra and geometry, solving complex mathematical problems like the Delian problem (doubling the volume of a cube). This demonstrates how such tools become indispensable in education and scientific progress.
Sections
Introduction and the Concept of Cognitive Tools
Cognitive tools like the number line are human inventions, not natural, enabling abstract thought and mathematical discovery.
The speaker introduces the concept of cognitive tools, using the number line as a familiar example. This tool, invented by humans, extends beyond natural forms to facilitate abstract reasoning. The invention of rectangular coordinates by Descartes is highlighted as a revolutionary advancement that linked algebra and geometry, solving complex mathematical problems like the Delian problem (doubling the volume of a cube). This demonstrates how such tools become indispensable in education and scientific progress.
Human history is marked by technologies that make the invisible visible, fostering learning and discovery.
The evolution of human learning and discovery is intertwined with the development of technologies that render the unseen visible. Examples include Darwin's illustrations of finches to highlight variation, Galileo's telescope for astronomical observation, Ramón y Cajal's drawings of the nervous system, and Feynman diagrams for visualizing subatomic particles. These tools expand our understanding of the world by presenting information in ways that highlight relevant features.
Visual abstraction is a key human cognitive tool used to communicate knowledge by highlighting relevant aspects of information.
These examples of making the invisible visible leverage 'visual abstraction' to communicate what we see and know. This abstraction highlights what is relevant to notice, whether through detailed representations like Darwin's finches or more schematic ones. This ability to reformulate understanding in terms of useful abstractions drives technological progress and the re-engineering of the physical world.
Explaining human innovation requires accounting for both cognitive tools (abstraction) and engineering (creation).
Traditional cognitive psychology primarily focuses on how people process external information and social cognition involves multi-individual interactions. However, to understand human discovery and invention, two crucial elements are missing: 1) cognitive tools (material objects encoding information to shape thought) and 2) engineering (leveraging understanding to create new, useful things). Embracing both abstraction and engineering is necessary to explain the world's development.
Research aims to bridge the gap between discovering useful abstractions and applying them to create new things.
The core research goal is to develop psychological theories that explain both how humans discover useful abstractions about the world and how they apply these abstractions to create new things. This involves closing the loop between understanding and creation.
Part 1: Visual Abstraction in Communication
Understanding visual explanations requires considering perception, production, and communication of graphical elements.
The process of using visual abstraction to communicate involves three key behaviors: visual perception (transforming sensory input into meaningful experiences), visual production (generating markings), and visual communication (arranging graphical elements to impact others). These three components are essential for creating and interpreting drawings.
Models of visual processing can generalize to sketches, supporting a resemblance-based account of pictorial meaning.
Research using neural networks trained on natural photographs showed they could generalize to sparse sketches, suggesting that pictorial meaning and resemblance can be understood through models of the ventral stream of visual processing.
Spatial constraints govern how sketch elements correspond to real-world object parts, supporting robust sketch understanding.
Further work involved training a decoder to map sketch elements to photograph elements, allowing for warping but not tearing. The success of this approach indicates that strong spatial constraints guide the correspondence between sketch parts and real object parts.
People adjust drawing fidelity based on context to effectively communicate identity.
In a drawing game, sketchers produced more detailed drawings when distractors were similar (close trials) and sparser drawings when distractors were dissimilar (far trials). This adaptation, using less ink and time, still achieved high accuracy in communicating the target object's identity, indicating context-sensitivity in depiction.
Both visual abstraction capacity and context sensitivity are crucial for appropriate communication through drawings.
A computational model revealed that both a visual encoder's capacity for abstraction and a decision-making module's sensitivity to context are critical for people to communicate about objects at the appropriate level of abstraction.
Shared history can lead to the emergence of new graphical conventions over time.
Recent work explores how memory of previous interactions influences communication, leading people to produce more abstract, potentially proto-symbolic tokens whose meaning relies heavily on shared history.
Visual explanations prioritize mechanistic information over visual fidelity.
Comparing visual depictions and explanations, studies found that explanations emphasize causal parts and symbolic displays of motion/interaction more than depictions. Depictions, conversely, emphasize background elements to aid object identification.
People possess intuitive understandings of what constitutes a visual explanation, sacrificing fidelity for clarity of mechanism.
Participants consistently created drawings that emphasized mechanistic information over visual appearance when asked to explain how a contraption worked, even without prior instruction. This suggests people share intuitions about the goal of visual explanations, prioritizing explanatory content over strict visual accuracy.
Communicative context and goals fundamentally shape how people draw.
This research highlights the critical role of communicative context and objectives in determining drawing strategies and the resulting appearance of depictions. It provides tools to characterize how people communicate goal- and context-relevant visual information.
A benchmark (SEVA) was created to test AI's ability to understand and generate abstract human-like sketches.
The SEVA benchmark, comprising 90,000 sketches of 128 concepts under varying time constraints, was developed to evaluate algorithms' robustness to sparsity and semantic ambiguity in sketches. It challenges AI to represent how different sketches of the same object (like Picasso's bulls) can be equally understood by humans.
A significant gap exists between human and AI performance in sketch understanding and generation.
While AI models improve with more drawing time, human performance consistently surpasses AI in recognition accuracy and exhibits more nuanced uncertainty. Even models like CLIP-trained ones, though performing better, still lag significantly behind humans in capturing sketch understanding patterns.
Humans and AI sparsify drawings differently, revealing functional distinctions in communication.
Although human and CLIPasso-generated sketches with similar stroke counts were similarly recognizable, they differed in how they conveyed meaning across different production budgets. Humans and AI exhibit distinct strategies in simplifying drawings, impacting the distribution of evoked meanings.
Part 2: Data Visualization and Statistical Reasoning
Data visualization is a powerful tool for resolving patterns that are too large, noisy, or slow to observe directly.
Data visualization, like scientific instruments, helps reveal aspects of the world not directly observable. Plots can expose patterns in data that are too vast, complex, or time-consuming to grasp through direct perception, making them indispensable in science, business, and decision-making.
Learning to interpret data visualizations is a crucial skill for navigating a complex world.
The ability to read, interpret, and create graphs is a critical goal of STEM education. Proficiency in data visualization acts as a superpower, enabling individuals to distill complex information into understandable narratives and calibrate beliefs in a complicated world.
AI models show a meaningful gap compared to humans in understanding data visualizations, particularly in error patterns.
A benchmarking effort comparing humans and various AI systems (Blip2, Lava, Matcha, GPT-4V) on graph-based reasoning tests revealed a consistent gap in performance. Crucially, AI models, including GPT-4V, did not replicate human-like error patterns, indicating a difference in their underlying reasoning processes.
Understanding data visualization involves selecting the appropriate plot for a given epistemic goal.
Research explores how people choose plots to satisfy specific questions or 'epistemic goals' regarding a dataset. The intuition is that different questions necessitate different visualization types to effectively shift beliefs and convey information.
People are sensitive to plot features relevant to answering specific questions, not just plot type preferences.
Analysis of plot choices suggests that people are not simply adherents to specific plot types (e.g., bar plot purists). Instead, they tend to select plots based on features that are genuinely useful for answering the question at hand, a strategy that predicts performance across various tasks.
Existing tests of visualization understanding may not accurately characterize the underlying skills being measured.
An analysis of common visualization tests (GGR, VLAT) indicates that item difficulty is convergent across different participant samples, and performance is correlated between tests. However, performance is not consistently linked to plot type or question type, suggesting underlying factors beyond these categories drive error patterns.
Developing improved measures of visualization understanding is crucial for effective educational application.
The current assessments of visualization understanding may not be optimally designed. The research stresses the importance of developing better measurement tools to accurately capture and characterize visualization skills, with the ultimate goal of improving educational outcomes and helping people calibrate their understanding of the world.
Conclusion
Cognitive technologies are central to education and human progress, enabling continuous innovation.
The talk concludes by reiterating that cognitive tools and technologies are fundamental to education and human generative activities. Understanding how these tools work and how to improve them is key to enabling each generation to build upon the last and to continually reimagine and improve the world.
Ask a Question
*Uses 1 Wisdom coin from your coin balance
