Storage vs. Computation of Multimodal Language

Instructors:
Liina Pylkkanen & Alec Marantz

Time & Place:
THU 2pm – 4:45pm 10 Wash Pl Rm 103

Description:

This advanced seminar focusses on the following two themes:

The multimodality of language. Ours brains are capable of taking in language in vastly different ways, ranging from the phoneme-by-phoneme progression of spoken language to our ability to comprehend entire sentences from just a 200ms flash-like presentation of a written expression. How should we conceive of the “higher” level components of language, syntax and semantics, given the drastic differences at the “lower” levels?

Storage vs. computation. Linguists claim that speakers use grammars to store the infinite set of potential words, phrases and sentences of their language. Because these grammars generate the infinite sets from a finite set of pieces (phonemes, morphemes…), psycho- and neuro-linguists often assume that speakers must use grammars in language processing by generating the forms they they process, and that there is a distinction between “memorized” complex forms, such as irregular past tenses and idioms, and generated forms. We will challenge these assumptions and explore the possibilities that generative grammars are used in processing in a different way and that there is no categorical difference between memorized and generated forms.

For the first five weeks of the semester, these themes will be discussed through the work of the job candidates in our currently ongoing faculty search in Language.

Requirements:
Attend, participate, present,  final paper.

Deadlines:
Feb 28 – Mar 4: Meet with LP or AM about your final project.
Mar 10 (midnight): 1-page project plan due
May 10 (midnight): Final papers due

Schedule:

Jan-27 Job candidate journal club 1: Schotter

  • Antúnez, M., Milligan, S., Hernández‐Cabrera, J. A., Barber, H. A., & Schotter, E. R. (2021). Semantic parafoveal processing in natural reading: Insight from fixation‐related potentials & eye movements. Psychophysiology, e13986.
    Link.

  • Brooks, T., Gramfort, A., Marantz, A. “Investigating Stages in Word Recognition with Concurrent Eye-tracking and MEG Recordings.” Available on Brightspace.

Feb-03 Job candidate journal club 2: Kocab

  • Kocab, A., Snedeker, J., and Davidson, K. (submitted). Quantifiers in Nicaraguan Sign Language. On Brightspace.
  • Blanco-Elorrieta, E., & Pylkkänen, L. (2016). Composition of complex numbers: Delineating the computational role of the left anterior temporal lobe. NeuroImage, 124, 194-203. Link.
  • Optional: Zhang, L., & Pylkkänen, L. (2018). Semantic composition of sentences word by word: MEG evidence for shared processing of conceptual and logical elements. Neuropsychologia, 119, 392-404. Link.

Feb-10 Job candidate journal club 3: Blanco-Elorrieta

  • Blanco-Elorrieta, E., & Caramazza, A. (2021). A common selection mechanism at each linguistic level in bilingual and monolingual language production. Cognition, 104625. Link.
  • Phillips, S. F., & Pylkkanen, L. (2021). Composition within and between Languages in the Bilingual Mind: MEG Evidence from Korean/English Bilinguals. Eneuro, 8(6). Link.

Feb-17 Job candidate journal club 4: Gwilliams

  • Gwilliams, L., King, JR., *Marantz, A. & *Poeppel, D. (Submitted). The dynamics of phoneme sequence processing. Link.
  • Snell, J., & Grainger, J. (2017). The sentence superiority effect revisited. Cognition, 168, 217-221. Link.

Feb-24 Poster slam idea lab for final projects

Mar-03 Storage vs. Computation: Phonology

  • Brodbeck, C., Bhattasali, S., Heredia, A. C., Resnik, P., Simon, J. Z., & Lau, E. (2021). Parallel processing in speech perception: Local and global representations of linguistic context. bioRxiv. Link.
  • Fedorenko, E., Hsieh, P. J., Nieto-Castañón, A., Whitfield-Gabrieli, S., & Kanwisher, N. (2010). New method for fMRI investigations of language: defining ROIs functionally in individual subjects. Journal of neurophysiology, 104(2), 1177-1194. Link.

Mar-10 Storage vs. Computation: Morphology

  • Oseki, Y., & Marantz, A. (2020). Modeling Morphological Processing in Human Magnetoencephalography. Proceedings of the Society for Computation in Linguistics, 3(1), 209-219. Link.
  • Carota, F., Bozic, M., & Marslen-Wilson, W. (2016). Decompositional representation of morphological complexity: Multivariate fMRI evidence from Italian. Journal of Cognitive Neuroscience, 28(12), 1878-1896. Link.

Mar-17 NYU spring break

Mar-24 Storage vs. Computation: Syntax

  • Linzen, T., & Jaeger, T. F. (2016). Uncertainty and expectation in sentence processing: Evidence from subcategorization distributions. Cognitive Science, 40(6), 1382-1411. Link.
  • Brennan, J. R., Dyer, C., Kuncoro, A., & Hale, J. T. (2020). Localizing syntactic predictions using recurrent neural network grammars. Neuropsychologia, 146, 107479. Link.

Mar-31 Sentence superiority: Visual

  • Wen, Y., Mirault, J., & Grainger, J. (2021). Fast syntax in the brain: Electrophysiological evidence from the rapid parallel visual presentation paradigm (RPVP). Journal of Experimental Psychology: Learning, Memory, and Cognition, 47(1), 99. Link.
  • Massol, S., Mirault, J., & Grainger, J. (2021). The contribution of semantics to the sentence superiority effect. Scientific reports, 11(1), 1-6. Link.

Apr-07 Sentence superiority: Auditory

  • Bonhage, C. E., Fiebach, C. J., Bahlmann, J., & Mueller, J. L. (2014). Brain signature of working memory for sentence structure: enriched encoding and facilitated maintenance. Journal of Cognitive Neuroscience, 26(8), 1654-1671. Link.
  • Roverud, E., Bradlow, A. R., & Kidd, G. (2020). Examining the sentence superiority effect for sentences presented and reported in forwards or backwards order. Applied psycholinguistics, 41(2), 381-400. Link.

Apr-14 Structure superiority in visual scenes

  • Kaiser, D., Quek, G. L., Cichy, R. M., & Peelen, M. V. (2019). Object vision in a structured world. Trends in cognitive sciences, 23(8), 672-685. Link.
  • Võ, M. L. H., Boettcher, S. E., & Draschkow, D. (2019). Reading scenes: how scene grammar guides attention and aids perception in real-world environments. Current opinion in psychology, 29, 205-210. Link.

Apr-21 Asynchronous. 10-minute first-draft-videos of final presentations.
Apr-28 Project presentations
May-05 Project presentations