虛擬環境現在用於各種應用，從專業培訓到室內設計。在許多情況下，例如醫學訓練和基於空間性規則的活動（例如團隊運動），獲得的空間知識準確性是一個重要的部分。虛擬環境中出現空間認知的問題是它提供給人類視覺資訊的不足，導致人對環境空間特徵無法有精確判斷。這甚至存在於虛擬實境中，其中距離估計的準確性仍然表現的比實體世界低。在我們調查其他感官對空間認知的支持之後，聽覺刺激似乎是大多數科技最易於整合的選擇，因為它提供距離和方向的信息。我們分別進行了兩個實驗來確認指向性聲音對空間知識獲得之影響，接下來再用針對性的空間音頻線索和聲波化中表現出來，增強人在雙人合作情境下對任務相關信息的接收。實驗結果告知未來的設計該如何整合指向性聲音去支持空間性的虛擬活動，也可以針對每一種情境裡最傾向的感官來增強活動裡的感受。

Virtual environments are now used in a diverse array of applications, from professional training to interior design. In many cases, such as medical training and rule-based spatial activity (e.g. team sports), accuracy of acquired spatial knowledge is an important component of the activity. An issue that arises for spatial cognition in virtual environments is the inadequacy of visual affordances for humans to form precise judgments about spatial features in the environment. This exists even in virtual reality, where distance estimation still demonstrates declined accuracy compared to in the physical world. After surveying the disposition of other sensory modalities to spatial cognition, auditory stimuli appeared to be the most holistic and easily integrable option for most technologies, due to its provision of distance and orientational information. We performed a set of two lab experiments to confirm, respectively, the impact of spatial auditory cues on spatial knowledge acquisition, and then in a collaborative context to explore the effect of interlocutor-controlled spatial auditory cues on addressee understanding of communicated information. The findings show that targeted spatial audio cues and sonification do enhance reception of task-relevant information, and inform how future design of technology-mediated spatial activity may target perceptual enhancement to accommodate an activity’s sensory predisposition.

Acknowledgments ii
Abstract iii
Table of Contents 1
Chapter 1 Introduction 3
Chapter 2 Background 8
2.1 Spatial Perception, the Vergence-Accommodation Conflict, and Audition 8
2.2 Audiovisual Integration of Spatio-temporal and Spatial Relational Processing 9
2.2.1Auditory and Visual Integration in Semantic and Spatio-temporal processing 9
2.2.2 Auditory Processing of Spatial Relational References 11
Chapter 3 — Experiment 1 16
3.1 Method 16
3.1.1 Experiment Design 16
3.1.2 Experiment Materials and Virtual Reality System 17
3.1.3 Participants 19
3.1.4 Procedure 19
3.1.5 Measurement 20
3.2 Results 21
3.3 Discussion 22
Chapter 4 — Experiment 2 23
4.1 Method 24
4.1.1 Experiment Design 24
4.1.2 AuralTrace Prototype: Pointer Trajectory Pitch Sonification 26
4.1.3 Experiment Stages and Tasks 29
4.1.4 Participants 33
4.1.5 Procedure 33
4.1.6 Measurement 35
4.2 Results 39
4.2.1 Post-Communication SRI Task Performance 39
4.2.2 Subjective Evaluation of Pointer Effects 43
4.3 Discussion 48
4.3.1 Effects of Pitch Perception on Understanding Higher-level Spatial Information 48
4.3.2 Perceptual Context for Communication and Collaboration 50
4.3.3 AuralTrace Effects: Explicit or Implicit? 52
4.3.4 Future Work 53
Chapter 5 — General Discussion & Conclusion 56
References 58

1. Miguel A. Alonso-Arevalo, Simon Shelley, Dik Hermes, Jacqueline Hollowood, Michael Pettitt, Sarah Sharples, and Armin Kohlrausch. 2012. Curve shape and curvature perception through interactive sonification. ACM Transactions on Applied Perception 9, 4: 1–19. https://doi.org/10.1145/2355598.2355600
2. Giuliano Benelli, Maurizio Caporali, Antonio Rizzo, and Elisa Rubegni. 2001. Design concepts for learning spatial relationships. In Proceedings of the 19th annual international conference on Computer documentation  - SIGDOC ’01, 22. https://doi.org/10.1145/501516.501522
3. M. Billinghurst and S. Weghorst. The use of sketch maps to measure cognitive maps of virtual environments. In Proceedings Virtual Reality Annual International Symposium ’95, 40–47. https://doi.org/10.1109/VRAIS.1995.512478
4. Jennifer K Bizley and Yale E Cohen. 2013. The what, where and how of auditory-object perception. Nature reviews. Neuroscience 14, 10: 693–707. https://doi.org/10.1038/nrn3565
5. Jennifer K. Bizley and Andrew J. King. 2008. Visual–auditory spatial processing in auditory cortical neurons. Brain Research 1242: 24–36. https://doi.org/10.1016/J.BRAINRES.2008.02.087
6. GEORGE M. BODNER and ROLAND B. GUAY. 1997. The Purdue Visualization of Rotations Test. The Chemical Educator 2, 4: 1–17. https://doi.org/10.1007/s00897970138a
7. James Broderick, Jim Duggan, and Sam Redfern. 2018. The Importance of Spatial Audio in Modern Games and Virtual Environments. In 2018 IEEE Games, Entertainment, Media Conference (GEM), 1–9. https://doi.org/10.1109/GEM.2018.8516445
8. Yuan-Chia Chang, Hao-Chuan Wang, Hung-kuo Chu, Shung-Ying Lin, and Shuo-Ping Wang. 2017. AlphaRead. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW ’17, 2246–2259. https://doi.org/10.1145/2998181.2998258
9. Alan R. Cohen, Subash Lohani, Sunil Manjila, Suriya Natsupakpong, Nathan Brown, and M. Cenk Cavusoglu. 2013. Virtual reality simulation: basic concepts and use in endoscopic neurosurgery training. Child’s Nervous System 29, 8: 1235–1244. https://doi.org/10.1007/s00381-013-2139-z
10. Christopher M. Conway and Morten H. Christiansen. 2005. Modality-Constrained Statistical Learning of Tactile, Visual, and Auditory Sequences. Journal of Experimental Psychology: Learning, Memory, and Cognition 31, 1: 24–39. https://doi.org/10.1037/0278-7393.31.1.24
11. H. Couclelis, R. G. Golledge, N. Gale, and W. Tobler. 1987. Exploring the anchor-point hypothesis of spatial cognition. Journal of Environmental Psychology 7, 2: 99–122. https://doi.org/10.1016/S0272-4944(87)80020-8
12. A. Cummings, R. Čeponienė, A. Koyama, A.P. Saygin, J. Townsend, and F. Dick. 2006. Auditory semantic networks for words and natural sounds. Brain Research 1115, 1: 92–107. https://doi.org/10.1016/j.brainres.2006.07.050
13. Sarah D’Angelo and Darren Gergle. 2016. Gazed and Confused. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16, 2492–2496. https://doi.org/10.1145/2858036.2858499
14. H.Q. Dinh, N. Walker, L.F. Hodges, Chang Song, and A. Kobayashi. Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings IEEE Virtual Reality (Cat. No. 99CB36316), 222–228. https://doi.org/10.1109/VR.1999.756955
15. Zohar Eitan, Asi Schupak, Alex Gotler, and Lawrence E. Marks. 2014. Lower Pitch Is Larger, Yet Falling Pitches Shrink. Experimental Psychology 61, 4: 273–284. https://doi.org/10.1027/1618-3169/a000246
16. Jing Feng, Ian Spence, and Jay Pratt. 2007. Playing an Action Video Game Reduces Gender Differences in Spatial Cognition. Psychological Science 18, 10: 850–855. https://doi.org/10.1111/j.1467-9280.2007.01990.x
17. Daniel J. Finnegan, Eamonn O’Neill, and Michael J. Proulx. 2016. Compensating for Distance Compression in Audiovisual Virtual Environments Using Incongruence. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16, 200–212. https://doi.org/10.1145/2858036.2858065
18. Susan R. Fussell, Robert E. Kraut, and Jane Siegel. 2000. Coordination of communication. In Proceedings of the 2000 ACM conference on Computer supported cooperative work  - CSCW ’00, 21–30. https://doi.org/10.1145/358916.358947
19. Darren Gergle, Robert E Kraut, and Susan R Fussell. 2013. Using Visual Information for Grounding and Awareness in Collaborative Tasks Using Visual Inform
20. Michele Geronazzo, Alberto Bedin, Luca Brayda, Claudio Campus, and Federico Avanzini. 2016. Interactive spatial sonification for non-visual exploration of virtual maps. International Journal of Human-Computer Studies 85: 4–15. https://doi.org/10.1016/J.IJHCS.2015.08.004
21. Timothy D. Griffiths, Christophe Micheyl, and Tobias Overath. 2012. Auditory Object Analysis. . Springer, New York, NY, 199–223. https://doi.org/10.1007/978-1-4614-2314-0_8
22. Souta Hidaka, Wataru Teramoto, and Yoichi Sugita. 2015. Spatiotemporal Processing in Crossmodal Interactions for Perception of the External World: A Review. Frontiers in integrative neuroscience 9: 62. https://doi.org/10.3389/fnint.2015.00062
23. Michael Janner, Karthik Narasimhan, and Regina Barzilay. 2018. Representation Learning for Grounded Spatial Reasoning. Transactions of the Association for Computational Linguistics 6: 49–61. https://doi.org/10.1162/tacl_a_00004
24. Ute Jekosch. 2005. Assigning Meaning to Sounds — Semiotics in the Context of Product-Sound Design. In Communication Acoustics. Springer-Verlag, Berlin/Heidelberg, 193–221. https://doi.org/10.1007/3-540-27437-5_8
25. Adam Jones, J. Edward Swan, Gurjot Singh, and Eric Kolstad. 2008. The effects of virtual reality, augmented reality, and motion parallax on egocentric depth perception. In Proceedings - IEEE Virtual Reality, 267–268. https://doi.org/10.1109/VR.2008.4480794
26. Marcel Adam Just and Patricia A. Carpenter. 1985. Cognitive Coordinate Systems. Accounts of Mental Rotation and Individual Differences in Spatial Ability. Psychological Review 92, 2: 137–172. https://doi.org/10.1037/0033-295X.92.2.137
27. Andrew J King. 2009. Visual influences on auditory spatial learning. Philosophical transactions of the Royal Society of London. Series B, Biological sciences 364, 1515: 331–9. https://doi.org/10.1098/rstb.2008.0230
28. Robert Konrad. 2015. What is the vergence-accommodation conflict and how do we fix it? XRDS: Crossroads, The ACM Magazine for Students 22, 1: 52–55. https://doi.org/10.1145/2810048
29. Maria Kozhevnikov and Andre Garcia. 2011. Visual-Spatial Learning and Training in Collaborative Design in Virtual Environments. In Collaborative Design in Virtual Environments. Springer Netherlands, Dordrecht, 17–26. https://doi.org/10.1007/978-94-007-0605-7_2
30. Robert E. Kraut, Susan R. Fussell, and Jane Siegel. 2003. Visual Information as a Conversational Resource in Collaborative Physical Tasks. Human–Computer Interaction 18, 1–2: 13–49. https://doi.org/10.1207/S15327051HCI1812_2
31. Mei-Po Kwan and Jiyeong Lee. 2005. Emergency response after 9/11: the potential of real-time 3D GIS for quick emergency response in micro-spatial environments. Computers, Environment and Urban Systems 29, 2: 93–113. https://doi.org/10.1016/J.COMPENVURBSYS.2003.08.002
32. Elinda Ai-Lim Lee and Kok Wai Wong. 2014. Learning with desktop virtual reality: Low spatial ability learners are more positively affected. Computers & Education 79: 49–58. https://doi.org/10.1016/J.COMPEDU.2014.07.010
33. Fumiko Maeda, Ryota Kanai, and Shinsuke Shimojo. 2004. Changing pitch induced visual motion illusion. Current biology : CB 14, 23: R990-1. https://doi.org/10.1016/j.cub.2004.11.018
34. Lawrence E. Marks. 1974. On Associations of Light and Sound: The Mediation of Brightness, Pitch, and Loudness. The American Journal of Psychology 87, 1/2: 173. https://doi.org/10.2307/1422011
35. Adrien Merer, Sølvi Ystad, Richard Kronland-Martinet, and Mitsuko Aramaki. 2011. Abstract Sounds and Their Applications in Audio and Perception Research. . Springer, Berlin, Heidelberg, 176–187. https://doi.org/10.1007/978-3-642-23126-1_12
36. Cecilie Møller, Andreas Højlund, Klaus B. Bærentsen, Niels Chr. Hansen, Joshua C. Skewes, and Peter Vuust. 2018. Visually induced gains in pitch discrimination: Linking audio-visual processing with auditory abilities. Attention, Perception, & Psychophysics 80, 4: 999–1010. https://doi.org/10.3758/s13414-017-1481-8
37. Cesare V Parise, Katharina Knorre, and Marc O Ernst. 2014. Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences of the United States of America 111, 16: 6104–8. https://doi.org/10.1073/pnas.1322705111
38. Stacey Parrott, Emmanuel Guzman-Martinez, Laura Orte, Marcia Grabowecky, Mark D Huntington, and Satoru Suzuki. 2015. Direction of Auditory Pitch-Change Influences Visual Search for Slope From Graphs. Perception 44, 7: 764–78. https://doi.org/10.1177/0301006615596904
39. Robert Earl Patterson, Byron J. Pierce, Herbert H. Bell, and Gary Klein. 2010. Implicit Learning, Tacit Knowledge, Expertise Development, and Naturalistic Decision Making. Journal of Cognitive Engineering and Decision Making 4, 4: 289–303. https://doi.org/10.1177/155534341000400403
40. G. Pingali, A. Opalach, Y. Jean, and I. Carlbom. 2001. Visualization of sports using motion trajectories: providing insights into performance, style, and strategy. In Proceedings Visualization, 2001. VIS ’01., 75–544. https://doi.org/10.1109/VISUAL.2001.964496
41. C. C. Pratt. 1930. The spatial character of high and low tones. Journal of Experimental Psychology 13, 3: 278–285. https://doi.org/10.1037/h0072651
42. B. Razavi, W. E. O’Neill, and G. D. Paige. 2007. Auditory Spatial Perception Dynamically Realigns with Changing Eye Position. Journal of Neuroscience 27, 38: 10249–10258. https://doi.org/10.1523/JNEUROSCI.0938-07.2007
43. Arthur S. Reber. 1989. Implicit learning and tacit knowledge. Journal of Experimental Psychology: General 118, 3: 219–235. https://doi.org/10.1037/0096-3445.118.3.219
44. Hessam Roodaki, Navid Navab, Abouzar Eslami, Christopher Stapleton, and Nassir Navab. 2017. SonifEye: Sonification of Visual Information Using Physical Modeling Sound Synthesis. IEEE Transactions on Visualization and Computer Graphics 23, 11: 2366–2371. https://doi.org/10.1109/TVCG.2017.2734327
45. J R Saffran, E K Johnson, R N Aslin, and E L Newport. 1999. Statistical learning of tone sequences by human infants and adults. Cognition 70, 1: 27–52. Retrieved January 14, 2019 from http://www.ncbi.nlm.nih.gov/pubmed/10193055
46. Alejandro Salgado-Montejo, Fernando Marmolejo-Ramos, Jorge A. Alvarado, Juan Camilo Arboleda, Daniel R. Suarez, and Charles Spence. 2016. Drawing sounds: representing tones and chords spatially. Experimental Brain Research 234, 12: 3509–3522. https://doi.org/10.1007/s00221-016-4747-9
47. R N Shepard. 1982. Geometrical approximations to the structure of musical pitch. Psychological review 89, 4: 305–33. Retrieved January 14, 2019 from http://www.ncbi.nlm.nih.gov/pubmed/7134331
48. Jonathan Z Simon. 2015. The encoding of auditory objects in auditory cortex: insights from magnetoencephalography. International journal of psychophysiology : official journal of the International Organization of Psychophysiology 95, 2: 184–90. https://doi.org/10.1016/j.ijpsycho.2014.05.005
49. Maya Visser, Elizabeth Jefferies, Karl V. Embleton, and Matthew A. Lambon Ralph. 2012. Both the Middle Temporal Gyrus and the Ventral Anterior Temporal Area Are Crucial for Multimodal Semantic Processing: Distortion-corrected fMRI Evidence for a Double Gradient of Information Convergence in the Temporal Lobes. Journal of Cognitive Neuroscience 24, 8: 1766–1778. https://doi.org/10.1162/jocn_a_00244
50. Ilana B Witten and Eric I Knudsen. 2005. Why seeing is believing: merging auditory and visual worlds. Neuron 48, 3: 489–96. https://doi.org/10.1016/j.neuron.2005.10.020
51. Seraphina Yong and Hao-Chuan Wang. 2018. Using Spatialized Audio to Improve Human Spatial Knowledge Acquisition in Virtual Reality. In Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion - IUI 18, 1–2. https://doi.org/10.1145/3180308.3180360