What caused dense matter to exist and proceed to the big bang?

The short answer to your question is that no one knows how the Big Bang came about. And matter, as we know it, didn't come into existence until well after the Big Bang had banged. We can change the question a bit and ask "What was there in that superdense space that exploded and became the Big Bang?" - no one knows that either.

In trying to find an internet resource to answer your question, I happened upon this link and I think it's probably one of the best explanations of what we know and what we don't about the Big Bang: http://www.space.com/31192-what-triggered-the-big-bang.html (it does get a bit snarky in places but the explanation is excellent).

To summarize what the article says: the theory of the Big Bang is a theory that does not actually tell us how the Universe came into being. It is instead a theory that tries to describe the history of the Universe after the Big Bang.

This distinction is vital. Scientists, through observations, theories, modelling, and all the other ways the theory can be checked and validated, feel they can accurately describe what the Universe looked like at #10^-36# seconds when it was the size of a small fruit. Beyond that, they simply don't know.