No matter in what direction we look, we always look towards the big bang, because, since light travels at a finite speed, the ‘farther out’ we look, the more to the past we look, and the big bang is the one event to the past of everything in the universe.
Think about the balloon analogy for the universe’s expansion. Galaxies are little dots painted on the surface of an expanding balloon; that surface represents a two dimensional version of our three dimensional universe – one dimension is left out for clarity. If you take this universe at any given point in its existence, you thus have a sphere of a certain size; if you then reverse time, that sphere will shrink, until it contracts down to a point – the big bang singularity. Now it’s important to notice that this singularity is not anywhere in space, rather, it is all of space – there’s no point on the balloon to which you could point and say: ‘there, that’s where the big bang happened’. In a manner of speaking, it happened everywhere.
Now, pick one of those galaxies on the surface of the balloon as ours. If you look from that galaxy to a nearby one, you will not see it as it is ‘right now’, but rather, as it was at the time it emitted the light you now see – so if, say, it is a million light years away, you’ll see it as it was a million years ago. Consequently, everything you see a million light years away will appear as it was a million years ago. But a million years ago, the universe was slightly smaller than it is now – so in whatever direction you look, if you look at the parts of the universe a million light years away, you’ll look at a slightly smaller universe.
You can extend this game to things ever farther away – something a billion light years away will appear to you as it was a billion years ago, and the universe a billion years ago was smaller than it is today. It’s clear where this goes – no matter in what direction you look, if you look far enough, you will see the universe as it was a long time ago, so theoretically, if you could look far enough, you would see the universe as it was at the big bang in every direction.
However, there’s a catch to this – the early universe was not, in fact, transparent; it was opaque until about 380,000 years after the big bang, so this is the furthest back you can look using ordinary light (neutrinos or gravity waves could be used to look back yet further than that). It’s from this time that the cosmic microwave background hails.