Nerd Talk
I recently fixed an issue in the server that’s related to lighttpd 404 handler. The problem is when a URL is accessed with weird characters lighttpd returns a 404 page. For example, you saw something like this in your page views log and it returned a 404 error:
The example above results to “http://agyuku.net ” in the browser and returns Page Not Found. Notice the space after .net. The weird character “%C2%A0” is translated by the browser as a space but the server actually sees it as something else. Adding a space after the domain doesn’t really cause the server to react and treat it as a missing page. A solution that should do the work is to add a rewrite rule in lighttpd’s conf file:
The regular expression matches the weird characters at the end of the URL. When lighttpd receives a page request it will only read the URL without the weird stuff. $1 is a back reference to the regular expression pattern which means use those that matches the first pattern, in this case it’s (.+?). This should actually work but there’s a case due to different server settings, this may not solve the issue as lighttpd doesn’t see “%C2%A0″ as-is when the URL is accessed, it may be something else.
If the above method didn’t work then enabling lighttpd’s debug variables will give a clue to the issue. The debug variables are disabled by default, somewhere in lighttpd.conf set these to be enabled:
debug.log-request-header = "enable"
debug.log-response-header = "enable"
debug.log-request-handling = "enable"
debug.log-file-not-found = "enable"
Once those are set save lighttpd.conf and restart the server. Access the URL with weird characters in the browser to trigger the error and it should write something in the server’s error log. Do a tail to the error log to check the last reported error:
tail -n 150 /path/to/your/error.log
The command above displays the last 150 lines in the error log. Check the displayed errors and there should be something like:
(response.c.221) -- splitting Request-URI
(response.c.222) Request-URI : /%C2%A0
(response.c.223) URI-scheme : http
(response.c.224) URI-authority: agyuku.net
(response.c.225) URI-path : /%C2%A0
(response.c.226) URI-query :
(response.c.254) -- sanatising URI
(response.c.255) URI-path : /Â
(mod_access.c.135) -- mod_access_uri_handler called
(response.c.391) -- before doc_root
(response.c.392) Doc-Root : /some/document/root/path
(response.c.393) Rel-Path : /Â
(response.c.394) Path :
(response.c.442) -- after doc_root
(response.c.443) Doc-Root : /some/document/root/path
(response.c.444) Rel-Path : /Â
(response.c.445) Path : /some/document/root/path/Â
(response.c.462) -- logical -> physical
(response.c.463) Doc-Root : /some/document/root/path
(response.c.464) Rel-Path : /Â
(response.c.465) Path : /some/document/root/path/Â
(response.c.482) -- handling physical path
(response.c.483) Path : /some/document/root/path/Â
(response.c.539) -- file not found
(response.c.540) Path : /some/document/root/path/Â
Notice /%C2%A0 translated as /Â. The browser converts the weird characters as a space but the server treated it differently and converted it to “”. A 404 response is sent by the server because it’s trying to find /some/document/root/path/ which doesn’t exist. We will use the findings above to fix the issue, modify the URL rewrite to something like:
"(.+?)(%C2%A0|Â)$" => "$1"
The pattern now includes the weird character in the URL (just to be sure we handle it as well) and the one that the server see internally (the “”). Save lighttpd and restart the server. Access the URL and it should load up the page without an error.
The above scenario is something I experienced and may be different from issues you may have but I hope it helps someone. And wow, it’s 4 in the morning, I should get some sleep.
More on lighttpd debug variables here.
I used http://agyuku.net an example domain only and it’s not running in lighttpd, it runs in Apache.
I like it, in fact I’m starting to get addicted to it :)
