CS224U

Lessons learned about benchmarking, adversarial testing, the dangers of over- and under-claiming, and AI alignment.

Transcript: https://web.stanford.edu/class/cs224u/podcast/bowman/


Sam's website

Sam on Twitter

NYU Linguistics

NYU Data Science

NYU Computer Science

Anthropic

SNLI paper: A large annotated corpus for learning natural language inference

SNLI leaderboard

FraCaS

SICK

A SICK cure for the evaluation of compositional distributional semantic models

SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

RTE Knowledge Resources

Richard Socher

Chris Manning

Andrew Ng

Ray Kurtzweil

SQuAD

Gabor Angeli

Adina Williams

Adina Williams podcast episode

MultiNLI paper: A broad-coverage challenge corpus for sentence understanding through inference

MultiNLI leaderboards

Twitter discussion of LLMs and negation

GLUE

SuperGLUE

DecaNLP

GPT-3 paper: Language Models are Few-Shot Learners

FLAN

Winograd schema challenges

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

JSALT: General-Purpose Sentence Representation Learning

Ellie Pavlick

Ellie Pavlick podcast episode

Tal Linzen

Ian Tenney

Dipanjan Das

Yoav Goldberg

Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

Big Bench

Upwork

Surge AI

Dynabench

Douwe Kiela

Douwe Kiela podcast episode

Ethan Perez

NYU Alignment Research Group

Eliezer Shlomo Yudkowsky

Alignment Research Center

Redwood Research

Percy Liang podcast episode

Richard Socher podcast episode

AI and social science, the causal revolution in economics, predictions about the impact of AI, teaching MBAs, productizing AI, and a journey from Tel Aviv to Princeton to Stanford.

Transcript: https://web.stanford.edu/class/cs224u/podcast/goldberg/


Amir's website

Amir on Twitter

Computational Culture Lab

ChatGPT

Laura Nelson

Bart Bonikowski

Chris Winship

Bernie Koch

Treebanks

BIG-bench

Guido Imbens

Endogeneity

Susan Athey

Cambridge Analytica

Prediction Machines

Speech and Language Processing

DALL-E 2

Midjourney

Stable Diffusion

Postmodernism, or, the Cultural Logic of Late Capitalism

Turing test

Matt Salganik

Paul DiMaggio

Leaving Ohio, being back in Belgium, organizing NAACL 2022, reviewing at NLP-scale, universal dependencies, and doing NLU before it was cool.

Transcript: https://web.stanford.edu/class/cs224u/podcast/demarneffe/


Marie's website

Generating Typed Dependency Parses from Phrase Structure Parses

Universal Dependencies project

OSU Linguistics

NAACL 2022

Dan Jurafsky

Dan Roth

Chris Manning

ARR

Priscilla Rasmussen

Transactions of the ACL

Finding Contradictions in Text

Not a simple yes or no: Uncertainty in indirect answers

Recognizing Textual Entailment

Anna Rafferty

Scott Grimm

"Was It Good? It Was Provocative." Learning the Meaning of Scalar Adjectives

Did It Happen? The Pragmatic Complexity of Veridicality Assessment

Yejin Choi

Yejin Choi's ACl 2022 talk

Barbara Plank

Linguistically debatable or just plain wrong?

Jesse Dodge

Reproducibility badges at NAACL 2022

Stanford Sentiment Treebank

Judith Tonhauser

Nan-Jiang Jiang

Lauri Karttunen

Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

Microsoft DeBERTa surpasses human performance on the SuperGLUE benchmark

Daniel Zeman

Marta Recasens

Coding puzzles, practices, and education, structured prediction, the culture of Hugging Face, large models, and the energy of New York.

Transcript: https://web.stanford.edu/class/cs224u/podcast/rush/


Sasha's website

Sasha on Twitter

Sasha on the Humans of AI podcast

Sasha on The Thesis Review Podcast with Sean Welleck

Sasha on the Talking Machines Podcast

Sasha interviewed by Sayak Paul

Hugging Face

PyTorch

The Annotated Transformer

The Annotated Alice

The Annotated S4

Sasha and Dan Oneață's declarative graphics library Chalk

Drawing Big Ben in Chalk

OpenNMT

Ken Shan

Blog post by Ken and Dylan Thurston

Edward Z. Yang

Stuart Shieber

Literate programming

Soumith Chintala

Lua Torch

TensorFlow

Graham Neubig

Chris Dyer

DyNet

JAX

jax.vmap

Matt Johnson

Finale Doshi-Velez, whose undergrad ML course inspired and informed Sasha's

Tensor Puzzles

GPU Puzzles

A tweet that Chris added to his CV

Adam Paszke

Dougal MacLaurin

Dex

Named Tensor notation

Named Tensors in PyTorch

TorchDim

Mini Torch

Torch-Struct

Sarah Hooker's paper 'The hardware lottery'

Jacob Andreas

Kevin Ellis

Hugging Face transformers library

Hugging Face datasets library

Hugging Face diffusers library

Hugging Face evaluate library

scikit-learn

Big Science blog

BLOOM

The Technology Behind BLOOM Training

CRFM

Eleuther

T0 and PromptSource

Washington Post: Big Tech builds AI with bad data. So scientists sought better data

The bet: Is Attention All You Need?

Democratizing access to large-scale language models with OPT-175B

Epic OPT-175 Logbook

Google's PaLM

United's shares plunge 76% on bogus bankruptcy report

Imagen

Albert Gu

Bell Labs

Moving to Stanford, linguistic and social variation, interventional studies, and shared stories and lessons learned from an ACL Young Rising Star.

Transcript: https://web.stanford.edu/class/cs224u/podcast/yang/


Diyi's website

Diyi on Twitter

Dan Jurafsky

The Stanford NLP Group

Buford Highway in Atlanta

Sweet tea

VALUE paper

AAE

GLUE

Negative concord

Exploring the role of grammar and word choice in bias toward African American English (AAE) in hate speech classification

Inducing positive perspectives with text reframing

Dynabench

Datasheets for datasets

MTurk

Upwork

Prolific

Seekers, Providers, Welcomers, and Storytellers: Modeling Social Roles in Online Health Communities

ToTTo: A controlled table-to-text generation dataset

Six questions for socially aware language technologies

The importance of modeling social factors of language: Theory and practice

Dirk Hovy

Workshop on Shared Stories and Lessons Learned EMNLP 2022

Workshop on Shared Stories and Lessons Learned ICCV 2021

Jeff Hancock