A 🐔 headless browser is a browser that can be used without a graphical interface. It can be controlled programmatically to automate harmless tasks, such as doing QA (Quality Assurance) tests.
But Headless browsers are more commonly used to automate web scraping, fake user numbers and advertisement impressions or to look for vulnerabilities on a website.
That is why you probably don't want them on your site. And that is why I am going to show you all the Headless Detection tests I used to build HeadlessDetectJS and in the end I am going to give you the combined source code you can use to get a score of how likely a user it to be using a headless browser, so you can just copy that to your site.
Disclaimer: I'm only going to test for HEADLESS BROWSERS and NOT BROWSER AUTOMATION in general using things like the navigator.webdriver flag or other things I mentioned in my article about how to make Selenium and Browser automation undetectable, and all this detection can theoretically be easily bypassed by just not using the Headless Browser for browser automation as recommended in the article.
Another Heads Up: In some tests, I will use the following JavaScript Challenge to detect what the User's Browser is and then do the test or not:
eval.toString().length
The following Browsers will return the following values:
- Firefox: 37
- Safari: 37
- Chrome: 33
- Internet Explorer: 39
User agent
Let's start with the user agent - this is the attribute commonly used to detect both the OS and the user's browser.
testUserAgent() {
if (/HeadlessChrome/.test(window.navigator.userAgent)) {
// Headless
return 1;
} else {
// Not Headless
return 0;
}
}
The User Agent can also be obtained from the HTTP headers. That's why you could also add another server side detection in PHP or similar to perform the same test there (Which i am not going to talk about because i am focusing on JavaScript but it should not be too hard to write).
Window.Chrome
Window.chrome is an object that provides features for developers of Chrome extensions. Although available in normal mode, it is not available in headless mode. That's why you can easily test the following code - I also used the JavaScript challenge that I explained at the beginning here to do this test only for chrome users.
testChromeWindow() {
if (eval.toString().length == 33 && !window.chrome) {
// Headless
return 1;
} else {
// Not Headless
return 0;
}
}
Notification Permissions
Permissions cannot be handled in headless mode. This leads to an inconsistent state where conflicting values are reported by Notification.permission and navigator.permissions.query.
Furthermore, this test is a bit special because it has to be done using callbacks instead of normal returns.
testNotificationPermissions(callback) {
navigator.permissions.query({
name: 'notifications'
}).then(function(permissionStatus) {
if (Notification.permission === 'denied' && permissionStatus.state === 'prompt') {
// Headless
callback(1);
} else {
// Not Headless
callback(0);
}
});
}
No Plugins
Headless browsers don't have plugins. That's why you are able to test how many plugins a browser has to detect a rogue headless browser because a headless browser would return an array that contains no plugins.
Some Browser like Chrome have default plugins, such as Chrome PDF viewer or Google Native Client, but some browsers don't have default plugins, so if the user didn't install any himself, it would also look like a headless browser.
In addition, Firefox will always show that you have zero plugins installed, so if someone uses Firefox, it would also look like a headless browser.
This is an oversight I'm just going to ignore because it doesn't matter that much when you add it to all the other test results and average a final score. (+ Most users have at least 1 plugin installed anyway)
function testPlugins(resultBlock) {
let length = navigator.plugins.length;
return length === 0 ? 1 : 0;
}
App Version
Browsers running with puppeteer in headless mode will include "Headless" in their app version.
function testAppVersion() {
let appVersion = navigator.appVersion;
return /headless/i.test(appVersion) ? 1 : 0;
}
Connection Rtt
The navigator.connection.rtt attribute has a value of 0 in the headless browser-but sometimes it doesn't even exist, which is why I'm just going to return "Not Headless" if that's the case.
function testConnectionRtt() {
let connection & nbsp; = navigator.connection;
let connectionRtt = connection ? connection.rtt : undefined;
if (connectionRtt === undefined) {
return 0; // Flag doesn't even exists so just return NOT HEADLESS
} else {
return connectionRtt === 0 ? 1 : 0;
}
}
Conclusion
At the end of the day, this is an infinite game of cat and mouse. Scrapers will always find new ways to remain invisible, websites will always find new ways to make scrapers visible, and so on...
The fact that I wrote this article to help websites probably helps scrapers even more because it probably taught them new methods, and now they're going to try to work around them (although some things like the user agent are very basic and well known and should already be considered by most scrapers)
If you still want this script in a summarized and finished form, just click here to see the GitHub page where you will also be explained how you can insert and use it on your website.
(By the way, if you know any detection methods I didn't talk about in this article, it would be nice if you were to open a new issue or even a Pull request on GitHub to help this list and other people.)