On Sat, Jul 27, 2013 at 10:05:54PM -0500, Daniel Miller wrote:
Perhaps the script could be generalized to show the differences in headers
(minus the Date header) that are returned given the different user agent
strings? This would cover the Location header, as it currently does, but
also could cover different HTTP status codes, content length, or other
quirks of the server/application.
I was experimenting with this idea the last few days.
The main problem i encountered was that of many redirections. For example,
assume a site that first redirects everything to https and then if it encounters
an unsupported User-Agent it redirects the request to a special page. Facebook,
Twitter and probably more apps behave like that.
So we perform two different requests, one with a valid and one with an
unsupported User-Agent header. Notice that on the first request we probably
won't have any notable differences on the responses since both requests are just
redirected to https. But after that, all the headers are encrypted so we can't
perform any comparison.
Even if it wasn't the SSL thing, we would have to compare the headers on every
single redirection (assuming we have more than one). But i think this is way too
complicated for this feature and i'm not even sure it's even possible with the
current implementation of http library.
If you know any cases of hosts behaving differently (apart from redirecting)
based on User-Agents please let me know. Maybe we could tackle these cases in a