Letās start off with a quick question!
Can you spot the difference between the two rows below?
/images/raĢksmoĢrgaĢs.jpg
/images/rƤksmƶrgƄs.jpg
I couldnāt.
My browser however insisted that there was no ārƤksmƶrgĆ„s.jpgā on the webserver ā a file that from my point of view clearly was there.
Since the error only occurred with filenames containing the letters Ć„,Ƥ & ƶ I at first suspected that there was an issue with mixing up UTF-8 and ISO-8859-1, however, this wasnāt the case.
My next course of action was to urlencode the requested filename and the filename from the server, and this is when I found something interesting!
ra%CC%88ksmo%CC%88rga%CC%8As
r%C3%A4ksm%C3%B6rg%C3%A5s
Now you see the difference, right?
The reason behind the difference is that there are multiple ways to represent the common Swedish letters Ć„, Ƥ and ƶ (and other non-ascii letters aswell – but for readabiltiy, letās keep it short).
If we look at the char codes for three letters that were causing trouble in my case:
Letter | Mac OSX | Linux
-------+-----------+------
Ć„ | 97 + 778 | 228
Ƥ | 97 + 776 | 229
ƶ | 111 + 776 | 246
Notice the pattern?
Mac uses āaā (97) and āoā (111) and then adds the circle (778) or the dots (776). Linux however has a diffrent char entirely.
There are multiple standards for representing characters in unicode, the competing normal forms here are āCanonical Decompositionā (NFD) and āCanonical Compositionā (NFC) ā and I needed to convert between the two.
My solution
I had this error on a server where files had been stored on a Mac and then re-uploaded to a Linux server. I didnāt have shell access to the server so I fixed it by using the following PHP-code that looped through all affected files and updated their names:
<?php
// Normalizes all filenames in folder
foreach(glob("*", 2) as $file){
$after = Normalizer::normalize($file, Normalizer::FORM_C);
if($file !== $after){
rename($file, $after);
}
}
You could probably use iconv or similar tools to achieve the same thing easier if youāve got shell-access (or php exec is enabled).
Fun fact: RƤksmƶrgĆ„s is a commonly used Swedish word used for testing that the non-ascii Ć
ĆĆ is working correctly.
