Christopher Hazard

This is a project I have been working on that has gone through multiple iterations. Originally inspired by Ask Me Anything: (https://arxiv.org/abs/1511.06973 ), the idea of the project is to incorporate knowledge from an external knowledge base into a visual question answering process. Most VQA approaches focus on the purely visual aspect of the task, with little focus on incorporating common sense knowledge ("human is an animal", "a bus is for transportation", etc.) into reasoning tasks to build a truly multi-modal system.  Training a machine to think properly about these types of questions is difficult because datasets are noisy, the correct reasoning process is unknown a priori, and most importantly, the best way to represent knowledge and fuse it is still undetermined.

My goal is to develop a human interpretable deep learning model that can perform first order logic based reasoning as well as utilize common knowledge to answer questions about an image. To address this I have gone through several approaches, including compositional models (e.g. neural module networks:  https://arxiv.org/abs/1511.02799 ) with additional query options for information retrieval, combining a neural turing machine ( https://arxiv.org/abs/1410.5401 ) with an iterative querier based on a scene graph (inspired by http://vision.stanford.edu/pdf/zhu2017cvpr.pdf ) and a gated graph neural network (https://arxiv.org/abs/1511.05493 ).

My most recent approach has been to work with scene graphs extracted from an image to reason about objects on a more abstract level, then train a neural network model to label the graph nodes with encodings to mimic assignment of logical symbols, followed by gated graph neural network propagation. Using this type of model as a backbone, another module is tasked with learning to retrieve database information with regards to nodes it iteratively selects to build a more complete knowledge graph based on the question.

In collaboration with Jean Oh (http://www.cs.cmu.edu/~./jeanoh/ )

Relevent datasets:

visual genome:  https://visualgenome.org/static/paper/Visual_Genome.pdf 

visual 7w:  https://arxiv.org/abs/1511.03416 

CLEVR dataset for semantic reasoning: CLEVR_A_Diagnostic_CVPR_2017_paper.pdf 

The idea of this project is to simulate a game of basketball by learning team strategies through self play in an environment that mimics the game of basketball at a high level. The individual players are represented as dots on a 2d-grid resembling a court and are tasked with learning controllers that tell the players where to move according to realistic yet simple physics constraints. Outcomes of player interactions such as stealing, pass success rates, etc. are determined by regressions on average nba statistics for these events to make a simple simulation of the game. 

Using this simulation framework, we learn multi-agent strategies for coordinated movement via a social attention network backbone ( https://arxiv.org/abs/1710.04689 ) using PPO (proximal policy optimization) with reward shaping to encourage legal play. 

In the future, I plan to customize the team strategy according to the individual players' skills (represented as coefficients in play outcome event regressions based on that player's official nba record) using a meta-learning framework such as model agnostic meta learning (MAML: https://arxiv.org/pdf/1703.03400.pdf) on top of the baseline model for a generic team. This sort of system could potentially be used to judge match ups between teams to determine fair betting odds.

In this project, I developed a state space simulation model for football games that uses stochastic transitions based on historical NFL data and taking into account the players involved in each in-game event.  To calculate accurate probability distributions for in game event outcomes like "probability of kicking a field goal" or "number of yards in a passing play", I use Bayesian regressions with features drawn from the game state and priors drawn over the historical play by play NFL records of the players. Player skills are represented as coefficients in these regression models (represented as Bayes nets), which we can then use to calculate explicit posterior distributions over play outcomes.

Via dynamic programming, I calculate the value of each game state (including yard line, down, time on clock, etc.), and use that to calculate Nash equilibria for calling plays (e.g. whether the team should run, pass, kick, or punt) to simulate high level coaching decisions. With this framework, we can repeatedly simulate entire games to estimate the distribution of score spreads and compare them to posted game odds.  

This book inspired my interest in the topic: Wayne Winston "Mathletics"  (https://www.amazon.com/Mathletics)