Data Collection and Analysis Tools
As part of our project to construct comprehensive data sets and to empirically test hypotheses related to social and political behavior and attitudes, we have developed a suite of tools and modeling processes now available for broader dissemination. Below, we list the tools, describe their functionality and outline how interested parties can best use them.
pysmap is a package for manipulating and analyzing data sets containing large numbers of tweets. The package uses python 3 , a high-level programming language, but it is user-friendly and intuitive. it is designed to be used by someone with minimal programming experience. you can visit the documentation on github to learn how to install pysmap and use it. it requires a very basic working knowledge of python. pysmap can use many different data sources like bson, json, and csv files as well as MongoDB as a back-end data source. pysmap can be used for doing basic kinds of analyses like counting tweet features (like hashtags, mentions, etc), making basic graphs counting tweets, it can be used to filter on a set of pre-made filters, like language (detected or twitter ascribed), geo or non-geo enabled tweets. if you want a filter to be considered feel free to submit a github issue with your feature request.
smappdragon is a low level data parser designed for parsing twitter data. it is meant to be used by those with more experience programming in python (or programming in general). you can read the documentation on github. smappdragon can be used to write your own methods and filters for complex twitter data. it can be used to reduce the size of a dataset (stripping out unnecessary fields on entire data sets). it can be used to check individual fields in tweet objects easily. if you find a bug or have any suggestions for improvements we appreciate all feedback; submit a github issue.
All packages outlined here are python-based, and require that the user’s computer meet certain basic requirements. The lab has established a protocol for ensuring that your computer is properly set up before attempting to use the various tools. It includes both required and suggested steps, and walks even introductory-level users through the process step-by-step. The protocol can be found here. If you have additional questions, please feel free to email the lab.