High response time on varnish

I faced to one server with varnish that had very high response time. Testing from tool such as GTMetrix, it reported up to 26ms. After stop varnish, back to Apache, it’s back to normal.

I am just testing varnish, with default vcl, nothing more. Any idea why it took so high response time?