The 'final frontier' in studies of cosmic structure formation is the epoch of cosmic reionization, when the cold neutral Intergalactic Medium (IGM) was heated and reionized by primordial galaxies, a few hundred million years after the Big Bang. I propose that a large fraction of the first generations of massive stars in primordial galaxies ended as black holes and neutron stars in High Mass X-ray Binaries, and that besides the ultraviolet radiation from their massive stellar progenitors, feedback from accreting compact objects in binaries was an additional, important source of heating and reionization of the IGM. X-rays from the large populations of HMXBs have less optical depth than UVs, and determined the early thermal history of the universe maintaining it ionized over large volumes of space. This has a direct impact on the properties of the faintest galaxies at high redshifts, the smallest dwarf galaxies in the local universe, and on the existing and future surveys at radio wavelengths of atomic hydrogen in the early universe.