Abstract:
With new generations of astronomical surveys coming on-line, covering many decades of the electromagnetic spectrum, we have many opportunities to learn about the properties of stars and galaxies and how they depend on environment. If we can identify the one-in-a-million sources within these data sets we are, effectively, probing the tails of the source distributions. These extreme sources have the potential to challenge our understanding of the physics of galaxy formation. While it is the size and dimensionality of the new surveys that enables us to identify the unusual, it is this same volume of data that presents many interesting computational and statistical challenges. In this talk I will discuss new ways of analyzing data streams, looking for both common relationships and identifying unusual or transient sources. I will focus on ways for reducing the complexity of data to simplify an analysis and how we can use techniques, developed by internet companies to index the web, to make this science tractable.