The potential of gamma-ray bursts (GRBs) to act as high redshift probes has been recognized since they were first unveiled at cosmological distances more than a decade ago. Since this time we have used them to locate star forming galaxies at progressively higher redshift, culimating in the discovery of GRB 090423 at a spectroscopically confirmed redshift of 8.2 last year, with a further burst now possible at z~9. I will present the discovery of these high redshift GRBs, the study of their host galaxies with the refurbished Hubble Space Telescope, and the implications they have for star formation in the early Universe.