The Pingdom report at tools.pingdom.com will give you the time required to do the DNS lookup, the wait time for the server to respond with the HTML content, and the time it takes to download all the files on the page.
If you want your site to remain secret, you could take a screenshot of the test results, hovering your mouse over the bar chart on the first line of the output so we can see the times (which is your page HTML, DNS lookup time, etc.), edit out the URL then upload it so we can take a look at it. Then we can get an idea how long the server wait time is, which is when the server should be loading all the files to generate the HTML output before sending to the browser.
Do a test on Pingdom then a couple minutes later run another test. It loads much faster the second time, right?
I wanted to mention that I, too, have noticed that when Wordpress has not been accessed for a while it takes considerably longer to load. It doesn't take as long for me as it does for you (mine is 1 - 2 seconds when it hasn't been accessed for a while, 600 - 700 ms when it has been recently accessed and I'm on shared hosting). One possible reason may be that when the site is accessed regularly, the code is cached in memory and is loaded from there instead of being read from disk. A few years back I did a test where I created a PHP file that included another PHP file and set a timer to see how long it took to load the include file. The first access always took magnitudes longer than subsequent accesses, leading me to believe the code was cached in RAM. But I am no expert in the workings of web servers.
As a matter of fact, I was running a Pingdom test on a Wordpress site I have under development prior to coming to Sitepoint and reading your post. As the site is currently being developed, I have almost no traffic from humans or bots. For the first Pingdom test, it took 1.65 seconds to load the page (using the default Twenty Twelve theme) with a wait time of 1.09 seconds which is not something I find impressive. The server load at the time was 1.64 on a shared server with 12 physical cores (two hex core CPUs) so there is no issue with server load being too high. 27 minutes later I ran another Pingdom test and it showed a load time of 604 milliseconds with a wait time of 208 milliseconds. Much faster the second time.
You didn't mention if you were using any type of PHP accelerator. The most time consuming part of generating a web page is reading data from the hard drive. With as many file includes as there are in Wordpress, not to mention any plugins you may be using, it can take some time to generate a page. But not 10 - 15 seconds. Something isn't right.