"Quirks Mode is triggered by doctype sniffing, also known as doctype switching. This means that the browser inspects the start of an HTML document to see whether it contains a doctype declaration as required by HTML specifications." - from here.
strict mode is triggered by the presence of a valid doctype declaration, not just strict DTDs.
"WebKit, like most browser engines, has a quirks mode that is triggered by certain old HTML doctypes, or lack of a doctype declaration in text/html. Documents with newer or unknown doctypes, or sent as XML, are processed in standards mode. Like Mozilla and Opera, we apply only a limited set of nonstandard behaviors in quirks mode, and otherwise fix bugs across both modes, if they do not have a specific significant impact on Web compatibility. This is in contrast to the IE approach of completely freezing old behavior in quirks mode." - from here.
WebKit seems to behave a bit different when it comes to quirks mode, but the general rules are the same: the lack of DTD usually triggers quirks mode.