We were getting a number of requests for the file /.well-known/traffic-advice in our Nginx web server logs for all our websites. Since the file did not exist at that time, the logs recorded several 404 not found errors. This blog post shows you how to handle those kind of requests.
Multiple /.well-known/traffic-advice HTTP requests in your web server logs
What is in Nginx access logs
There were many of these entries in all our major websites, ArulJohn.com, PinoyMix.com, etc everyday. They all had a user-agent Chrome Privacy Preserving Prefetch Proxy.
This is a sample of the Nginx access.log entries corresponding to /.well-known/traffic-advice.
64.233.173.98 - - [29/Sep/2023:00:14:33 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "13.229.44.106"
64.233.173.98 - - [29/Sep/2023:01:17:15 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "13.228.24.30"
64.233.173.101 - - [29/Sep/2023:03:00:09 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "54.179.154.97"
64.233.173.99 - - [29/Sep/2023:04:06:35 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "18.139.225.3"
64.233.173.97 - - [29/Sep/2023:05:20:07 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "3.0.184.240"
64.233.173.100 - - [29/Sep/2023:06:26:18 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "54.251.179.149"
64.233.173.101 - - [29/Sep/2023:07:27:01 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "13.250.25.74"
66.249.93.14 - - [29/Sep/2023:08:31:02 -0400] "GET /.well-known/traffic-advice HTTP/1.1" 404 58 "-" "Chrome Privacy Preserving Prefetch Proxy X-Middleton/1" "15.237.42.181"
Chrome Privacy Preserving Prefetch Proxy bot
The closest and earliest information related to this is this draft specification and Google's webpage about Private prefetch proxy in Chrome.
According to Google, it is feature of Chrome that prefetches cross-origin content without exposing user information, so that if a user visited a prefetched URL, it would load faster. Google found 20% to 30% faster Largest Contentful Paint on prefetched navigations.
What is the traffic-advice file?
The traffic-advice file, which resides under /.well-known/ directory is a configuration file in the JSON format. It does not have the extension .json, perhaps for flexibility in future.
Only Chrome uses it for now. The traffic-advice file is used by prefetch proxies visiting your website to let them know whether to prefetch your pages or not, and how much. The percentage of the "how much" is a float between 0.0 (0) and 1.0 (100%).
How to add traffic-advice to Nginx
If you are using Nginx web server, these are steps on how to configure Nginx to include the /.well-known/traffic-advice file.
First, create this directory structure:
This is assuming that your Nginx home directory is at /usr/share/nginx/html
sudo mkdir -p /usr/share/nginx/html/.well-known
Next, create the file traffic-advice:
sudo vi /usr/share/nginx/html/.well-known/traffic-advice
Add this content to traffic-advice:
[{
"user_agent": "prefetch-proxy",
"google_prefetch_proxy_eap": {
"fraction": 1.0
}
}]
Now, configure Nginx to return this file with a custom MIME content-type application/trafficadvice+json.
Edit /etc/nginx/nginx.conf and add these lines just before the end of the server block:
location ^~ /.well-known/traffic-advice {
allow all;
types { } default_type "application/trafficadvice+json; charset=utf-8";
alias /usr/share/nginx/html/.well-known/traffic-advice;
}
Finally, reload Nginx:
sudo systemctl reload nginx.service
Try to visit your website using curl DOMAIN.COM/.well-known/traffic-advice and it should return the custom MIME type and the contents:
This is the configuration setting for aruljohn.com:
$ curl -i https://aruljohn.com/.well-known/traffic-advice
HTTP/2 200
server: nginx
date: Sat, 30 Sep 2023 19:22:37 GMT
content-type: application/trafficadvice+json; charset=utf-8
content-length: 98
last-modified: Fri, 29 Sep 2023 13:15:23 GMT
etag: "6516cdeb-62"
expires: Fri, 29 Sep 2023 19:22:37 GMT
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubDomains
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
referrer-policy: no-referrer
permissions-policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
cache-control: public, immutable
accept-ranges: bytes
[{
"user_agent": "prefetch-proxy",
"google_prefetch_proxy_eap": {
"fraction": 1.0
}
}]
How to add traffic-advice to Apache
If you are using Apache web server, these are steps on how to configure Nginx to include the /.well-known/traffic-advice file.
First, create this directory structure:
This is assuming that your Nginx home directory is at /var/www/html
sudo mkdir -p /var/www/html/.well-known
Next, create the file traffic-advice:
sudo vi /var/www/html/.well-known/traffic-advice
Add this content to traffic-advice:
[{
"user_agent": "prefetch-proxy",
"google_prefetch_proxy_eap": {
"fraction": 1.0
}
}]
Now, configure Apache to return this file with a custom MIME content-type application/trafficadvice+json.
Edit the main .htaccess and add these lines:
RewriteRule "^/\.well-known/traffic-advice$" - [T=application/trafficadvice+json,END]
and this block too:
<FilesMatch "traffic-advice">
Header set Content-Type "application/trafficadvice+json"
</FilesMatch>
Finally, reload Apache:
sudo systemctl reload apache2.service
Try to visit your website using curl DOMAIN.COM/.well-known/traffic-advice and it should return the custom MIME type.
How to allow only a percentage to prefetch
If you want to control a specific amount of traffic, you can set that in traffic-advice. For example, if you want to allow only 20% of traffic (0.2) to be prefetched, convert the percentage to its fraction equivalent:
[{
"user_agent": "prefetch-proxy",
"fraction": 0.2
}]
How to not be prefetched
If you do not want to reject all the prefetch requests, this is the configuration:
[{
"user_agent": "prefetch-proxy",
"disallow": true
}]
This scenario can be applied to cases where due to a sudden spike in requests, you decide to temporarily return a 503 status Service Unavailable (Under Maintenance) and temperarily reject prefetch requests. You can also add the header Cache-Control: no-store to not store the request and Retry-After: header.
Conclusion
If this worked for you, awesome! If it hasn't worked or you need more information about configuring your /.well-known/traffic-advice, feel free to contact me.
Related Posts
If you have any questions, please contact me at arulbOsutkNiqlzziyties@gNqmaizl.bkcom. You can also post questions in our Facebook group. Thank you.