Signals and Symbols
(language and other processing)
CUNY Graduate Center – Spring 2024


Language is perhaps the best window we have into cognition. Humans' knowledge of language consists not only in grammatical representation, but in the processes which operate over such representation. Thus, how we are able to convert gradient, continuous, ephemeral perceptual signals into discrete, mental symbols (and vice versa) is of fundamental importance to work toward a fuller understanding the cognitive system of language.

This discussion-based seminar will provide a wide-ranging but in-depth overview of topics in the real-time processing of language, speech, and related perceptual domains. Special attention will be devoted to the use of simple algorithmic models and related experiments in order to explore the specific mechanisms involves in the mental representation and use of language.

Goals and Objectives

  • Gain familiarity with the core questions, debates, and methods common in contemporary research in language processing and psycholinguistics
  • Foster an understanding of the ways in which processing interfaces with the core sub-fields of linguistics (phonology, syntax, etc.). And practice transferring insights and inspirations from one field into adjacent areas.
  • Be able to read and critique the primary literature in processing and psycholinguistics, including the empirical methods and primary data common in the field.
  • Materials

    While there is no official textbook for the course, we will almost exclusively read primary papers from the relevant literature. All readings assigned throughout the term (both required and optional) will be posted to the course schedule/website.

    With the goal of fostering focus, discussion, and the exchange of ideas, the use of screens (laptops, phones, etc.) will not be permitted during class. Any visual aids will be provided via physical materials (e.g. print outs) in class.


    As a paper/discussion-centric seminar, the primary component of course grades (50%) will be calculated based on active participation and attendance throughout the term. As part of this effort, each student will lead paper/topic discussion in the course at least once per term (which will count towards the participation grade). This will include preparing a hand-out or discussion-aid to distribute in class.

    Students will also be required to submit a short document (less than a page) each week before class with notes/reflections from the assigned readings (this will constitute 30% of the overall grade).

    Finally, students will write a final term paper (consisting of a research project proposal)—the final paper will make up the remaining 20% of the grade. There will however, be no HWs quizzes or tests assigned throughout the term.


    The instructor will attempt to provide all reasonable accommodations to students upon request. If you believe you are covered under the Americans With Disabilities Act, please direct accommodations requests to Vice President for Student Affairs Matthew G. Schoengood.


    The course takes place (in person) at the Graduate Center, and students are expected to attend all classes (in person). However, students who have reason to believe they may be contagious for COVID-19 or other infectious diseases should contact the instructor and stay home. Other absences will not be excused, and the instructor reserves the right to tie grades to attendance records.


    In line with the Student Handbook policies on plagiarism, students are expected to complete their own work. The general ethos of the integrity policy is that actions which shortcut the learning process are forbidden while actions which promote learning are encouraged. Studying and discussing notes, papers, and ideas together provides a fruitful avenue for learning and is encouraged. Using a classmate’s solution or text in your submitted notes or having someone else write a portion of your presentation, however, is prohibited because it circumvents the learning process. If you have any questions about what is or is not permissible, please contact your instructor.

    The instructor reserves the right to refer violations to the Academic Integrity Officer.

    Weekly Schedule

    (Please note that this is subject to change.) That said, we'll plan to discuss at least some subset of: Marr's three-levels of analysis for information processing systems (Computational, Algorithmic, and Implementation), continuous vs. discrete representations in speech and vision, perception as a form of ``sampling,'' syntactic parsing/processing, categorical perception, pragmatic processing, the relationship between knowledge and perception (linguistic relativity), the time-course of spoken-word recognition, plasticity in both speech and syntactic processing, word learning, eye-movements in reading and perception; language production, ``frequency effects,'' and the influence of processing on the lexicon.

    If you have a particularly strong interest in a topic not mentioned, just come talk to me and we may be able to add something in.

    Week 0 (1/25) – Intro; Processing and Representation

    Handout (Overview)

    (No readings to prep, but be sure to sign up for GitHub if you don't already have an account)

    Week 1 (2/01) – Marr's Levels; Visual Representation of Images

    Handout (Marr)

    Marr, D. 1982 Vision: A computational investigation into the human representation and processing of visual information. (Ch. 1-2) MIT press.

    Quine, W.V.O. 2013 Word and object (Ch. 3) MIT press.
    Quine, W.V.O. 1957 The scope and language of science. British Journal for the philosophy of Science, 8(29), 1-17.

    Week 2 (2/08) – Speech Perception; Competition and Timecourse of Spoken Word Recognition

    Handout (Speech part I)

    Liberman, A. M., Harris, K. S., Hoffman, H. S., & Griffith, B. C. 1957 The discrimination of speech sounds within and across phoneme boundaries. Journal of experimental psychology. 54(5), 358.
    "A Quick and Dirty summary of the Cohort Model" Excerpt from Wikipedia
    McClelland, J. L., & Elman, J. L. 1986 The TRACE model of speech perception. Cognitive psychology. 18(1), 1-86.
    Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. 1998 Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of memory and language. 38(4), 419-439.

    Marslen-Wilson, W., & Tyler, L. K. 1980 The temporal structure of spoken language understanding Cognition. 8(1), 1-71.

    Week 3 (2/15) Maintenance of speech information

    Handout (Speech part II)

    Bushong, W., & Jaeger, T. F. 2017 Maintenance of perceptual information in speech perception. Proceedings of the 39th Annual Meeting of the Cognitive Science Society. pg 86–191
    Samuel, A. G., & Kraljic, T. 2009 Perceptual learning for speech. Attention, Perception, & Psychophysics. 71(6), 1207-1218.
    Caplan, S., Hafri, A., & Trueswell, J. C. 2021 Now you hear me, later you don’t: The immediacy of linguistic computation and the representation of speech. Psychological Science. 32(3), 410-423.

    McMurray, B. 2022 The myth of categorical perception. The Journal of the Acoustical Society of America. 152(6), 3819-3842.

    Week 4 (2/22) – No class (GC on a Monday schedule)

    Week 5 (2/29) Perceptual learning; What exactly are we maintaining

    Handout (Intermediate Representations)

    Kleinschmidt, D. F., & Jaeger, T. F. 2016 Re-examining selective adaptation: Fatiguing feature detectors, or distributional learning? Psychonomic bulletin & review. 23, 678-691.
    Toscano, J. C., Anderson, N. D., Fabiani, M., Gratton, G., & Garnsey, S. M. 2018 The time-course of cortical responses to speech revealed by fast optical imaging. Brain and Language. 184, 32-42.

    Week 6 (3/07) Sentence Parsing; Modularity

    Handout (Parsing I -- Modularity)

    Van Gompel, R. P., & Pickering, M. J. 2007 Syntactic parsing. The Oxford handbook of psycholinguistics. pg 289-307.

    Pick one of these four:
    Fodor, J. D. 1998 Learning to parse? Journal of psycholinguistic research. 27, 285-319.
    Trueswell, J. C., Tanenhaus, M. K., & Garnsey, S. M. 1994 Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of memory and language. 33(3), 285-318.
    Frazier, L., & Fodor, J. D. 1978 The sausage machine: A new two-stage parsing model. Cognition. 6(4), 291-325.
    Frazier, L., & Rayner, K. 1982 Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences.. Cognitive psychology. 14(2), 178-210.

    Week 7 (3/14🥧) Parallel Parsing; Dependency-Length

    Handout (Parsing II -- Semantic and other Effects)
    Slides (Parsing as an Engineering Problem)

    Discussion lead: James

    Gibson, E. 2000 The dependency locality theory: A distance-based theory of linguistic complexity Image, language, brain. 95-126.
    Charniak, E. & Johnson, M. Computational Linguistics (Unpublished manuscript chapter)

    Konieczny, L. 2000 Locality and parsing complexity. Journal of psycholinguistic research. 29, 627-645.
    Hale, J. 2001 A probabilistic Earley parser as a psycholinguistic model. Second meeting of the north american chapter of the association for computational linguistics.

    Week 8 (3/21) Expectation-based Parsing; Beyond Surprisal

    Discussion lead: Zhilang

    Levy, R. 2008 Expectation-based syntactic comprehension Cognition. 106(3), 1126-1177.
    Huang, K. J., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. 2024 Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty Journal of Memory and Language. 137, 104510.

    Smith, N. J., & Levy, R. 2013 The effect of word predictability on reading time is logarithmic. Cognition. 128(3), 302-319.
    Luke, S. G., & Christianson, K. 2016 Limits on lexical prediction during reading. Cognitive psychology. 88, 22-60.

    Week 9 (3/28) A trip down the Kindergarten-Path

    Handout (Parsing IV -- Surprisal, Kindergarden Path Effect)

    Trueswell, J. C., Sekerina, I., Hill, N. M., & Logrip, M. L. 1999 The kindergarten-path effect: Studying on-line sentence processing in young children. Cognition. 73(2), 89-134.
    Woodard, K., Pozzan, L., & Trueswell, J. C. 2016 Taking your own path: Individual differences in executive function and language processing skills in child learners. Journal of experimental child psychology. 141, 187-209.

    Additional (Syntactic Adaptation):
    Fine, A. B., Jaeger, T. F., Farmer, T. A., & Qian, T. 2013 Rapid expectation adaptation during syntactic comprehension. PloS one. 8(10), e77661.
    Harrington Stack, C. M., James, A. N., & Watson, D. G. 2018 A failure to replicate rapid syntactic adaptation in comprehension. Memory & cognition. 46, 864-877.

    Additional (Localization of Cognitive Control -- Left Inferior Frontal Gyrus):
    Novick, J. M., Kan, I. P., Trueswell, J. C., & Thompson-Schill, S. L. 2009 A case for conflict across multiple domains: Memory and language impairments following damage to ventrolateral prefrontal cortex. Cognitive neuropsychology. 26(6), 527-567.
    Novick, J. M., Trueswell, J. C., & Thompson‐Schill, S. L. 2010 Broca’s area and language processing: Evidence for the cognitive control connection Language and Linguistics Compass. 4(10), 906-924.

    Week 10 (4/04) Morphology (with guest speaker Jordan Kodner)

    Stockall, L., & Marantz, A. 2006 A single route, full decomposition model of morphological complexity: MEG evidence. The mental lexicon. 1(1), 85-123.
    Ussishkin, A., & Wedel, A. 2002 Neighborhood density and the root-affix distinction. Proceedings of the North East Linguistics Society (NELS). Vol. 32, No. 2, p. 15

    Additional (background material):
    Halle, M., & Marantz, A. 1994 Some key features of Distributed Morphology. MIT working papers in linguistics 21(275), 88.
    Audring, J. 2022 Advances in morphological theory: construction morphology and relational morphology Annual Review of Linguistics 8, 39-58.

    Week 11 (4/11) Language Production (Part I)

    Dell, G. 1995 Speaking and misspeaking. An invitation to cognitive science: Language. 2. Vol. 1. Ch. 7 pg. 183–208.
    Bock, K., & Levelt, W. J. 1994 Language production: Grammatical encoding. Academic Press

    Bock, K. 1986 Syntactic Persistence in Language Production. Cognitive psychology 18(3), 355-387.
    Ferreira, V., & Dell, G. 2000 Effect of ambiguity and lexical availability on syntactic and lexical production Cognitive psychology 40(4), 296-340.
    Gleitman, L., January, D., Nappa, R., & Trueswell, J. C. 2007 On the give and take between event apprehension and utterance formulation. Journal of memory and language 57(4), 544-569.

    Week 12 (4/18) Language Production (Part II)

    Discussion leads: Selin & Asmaa

    Jaeger, T. F. 2010 Redundancy and reduction: Speakers manage syntactic information density. Cognitive psychology 61(1), 23-62.
    Caplan, S. 2021 The Incremental Mechanisms of Functional Design. Immediacy of Linguistics Computation (UPenn Doctoral Dissertation) Ch. 5

    Week 13 (4/25) – No class (Spring Break☀️😎)

    Week 14 (5/02) Word Learning

    Discussion lead: Elliot

    Trueswell, J. C., Medina, T. N., Hafri, A., & Gleitman, L. R. 2013 Propose but verify: Fast mapping meets cross-situational word learning. Cognitive psychology. 66(1), 126-156.

    Yu, C., & Smith, L. B. 2007 Rapid word learning under uncertainty via cross-situational statistics. Psychological Science. 18(5), 414-420.
    Stevens, J. S., Gleitman, L. R., Trueswell, J. C., & Yang, C. 2017 The pursuit of word meanings. Cognitive science. 41, 638-676.

    Week 15 (5/09) More Word Learning; Categories and Concepts

    Discussion lead: Griffin

    Gleitman, L. R., Cassidy, K., Nappa, R., Papafragou, A., & Trueswell, J. C. 2005 Hard words. Language learning and development. 1(1), 23-64.
    Xu, F., & Tenenbaum, J. B. 2008 Word learning as Bayesian inference. Psychological review. 114(2), 245.

    Caplan, S. 2021 Word Learning as Category Formation Immediacy of Linguistics Computation (UPenn Doctoral Dissertation). Ch. 3
    Rehder, B., & Hoffman, A. B. 2005 Eyetracking and selective attention in category learning Cognitive psychology. 51(1), 1-41.

    Week 16 (5/16 – "exam period") – Squib Presentations

    (w/ customary licorice)

    Squib Details

    Some useful resources

  • The CHILDES database
  • The SUBTLEX Corpus
  • CELEX2 corpus
  • English Lexicon Project
  • jTRACE model for speech perception
  • Self-paced reading time data
  • Spraakbanken Swedish text corpora
  • The CMU Pronouncing Dictionary
  • Python Natural Language Toolkit