[Interest] QDir::entry(Info)List on macos

[Manner Róbert]

Hi,

((
My local encoding is utf-8, so the source file seems to be the same as well:
> file main.cpp
main.cpp: c program text, UTF-8 Unicode text
))

I think I have found the bug which causes this:
https://bugreports.qt.io/browse/QTBUG-70732

I am currently working on a workaround, I am now able to list the files
now using <dirent.h>:

        QStringList results;
        DIR *dir;
        struct dirent *ent;
        QByteArray bpath = path.toUtf8();
        if ((dir = opendir (bpath.constData())) != NULL) {
          /* print all the files and directories within directory */
          while ((ent = readdir (dir)) != NULL) {
            results << ent->d_name;
          }
          closedir (dir);
        } else {
          /* could not open directory */
          perror ("");
        }
        return results;

This gives back the files successfully, the downside is that they are in
UTF-8 normalization format D (UTF-8-mac) which represents accented
characters like O" format, so the original character (O) + the accent
("). Similarly as "ls -1" does.

And having a QString like that does not match "normal" (==
"normalization format C") UTF-8 strings. Eg.:

  QString ch1("\u00D6");   // this is normal representation of "Ö"
  QString ch2("O\u0308");  // this specifies O with an accent
  qDebug() << ch1 << ch2 << (ch1 == ch2 ? "matches!" : "does not match");

This outputs: "Ö" "Ö" "does not match" not only on mac, but even on linux.

So I also needed to convert "utf8 to utf8" :) for my complete
workaround. Unfortunately I did not find a way to do so with QTextCodec.
(Is there?) So I am trying now iconv... this seems to do the trick, but
quite ugly and inefficient.

QByteArray
utf8_unmac(const QByteArray &utf8_mac)
{
  iconv_t conv = iconv_open("UTF-8-mac", "UTF-8");
  if (conv == (iconv_t)-1)
    {
      RAISE(TestRunnerException(QString("Iconv open failed: ") +
sys_errlist[errno]));
    }

  char *inp = const_cast<char *>(utf8_mac.constData());  // iconv is
moving these thats why the duplication
  size_t inp_len = utf8_mac.size();

  size_t out_len = inp_len * 2;

  QByteArray utf8;
  utf8.resize(out_len);

  char *out = utf8.data();

  if (iconv(conv, &inp, &inp_len, &out, &out_len) == (size_t)-1)
    {
      RAISE(TestRunnerException(QString("Iconv convert failed: ") +
sys_errlist[errno]));
    }

  utf8.chop(out_len);
  iconv_close(conv);
  return utf8;
}

Hope it is useful for someone. To be honest, I always imagined that if
everyone would use UTF-8 life would be much better, now I'm unsure ;)

Br,

Robert


On 11/15/18 9:49 AM, Olivier B. wrote:
> What is the encoding of your source file?
> QString constructors interprets char* as if they are UTF-8. If the
> source file is encoded in your local encoding, the QString created for
> QFile constructor will have a wrong unicode storage of your wanted
> filename, then will try to convert what it thinks is UTF into your
> local encoding to pass the filename to the system calls. Maybe the mac
> explorer can work around this and adjust the displayed name, but the
> Filesystem interface of Qt can't because that puts forbidden
> characters in the real name?
> Le jeu. 15 nov. 2018 à 08:55, Manner Róbert <rmanni at gmail.com> a écrit :
>> On 11/14/18 5:34 PM, Thiago Macieira wrote:
>>> On Wednesday, 14 November 2018 01:34:17 PST Manner Róbert wrote:
>>>> int main() {
>>>>   qDebug() << QDir(".").entryList();
>>> QCoreApplication missing. Try again with it.
>>>
>> Tried, without success, still does not display these files. Even tried
>> with QDirIterator, that is also working the same (skipping these files).
>>
>> With further checking I noticed that the files are only not displaying
>> if I do not create them with Qt. Eg I created with "touch filenamé". If
>> I create the same file with QFile, that seems to be found by these dir
>> lists. I know it sounds insane.
>>
>>
>> Locale is utf8:
>>
>>> locale
>> LANG="en_US.UTF-8"
>> LC_COLLATE="en_US.UTF-8"
>> LC_CTYPE="en_US.UTF-8"
>> LC_MESSAGES="en_US.UTF-8"
>> LC_MONETARY="en_US.UTF-8"
>> LC_NUMERIC="en_US.UTF-8"
>> LC_TIME="en_US.UTF-8"
>> LC_ALL="en_US.UTF-8"
>>
>> The modified source code:
>>
>>> cat main.cpp
>> #include <QDirIterator>
>> #include <QDebug>
>> #include <QFileInfoList>
>> #include <QCoreApplication>
>> #include <QFile>
>>
>> int main(int argc, char** argv) {
>>   QFile file("éáőú");  // This file is visible in the list! But not any
>> other I create with touch for example.
>>   file.open(QFile::WriteOnly);
>>   file.close();
>>
>>   QCoreApplication app(argc, argv);
>>   QDirIterator iterator(".", QDir::Files, QDirIterator::Subdirectories);
>>   while (iterator.hasNext())
>>     {
>>       qDebug() << "QDiriterator" << iterator.next();
>>     }
>>
>>   qDebug() << QDir(".").entryList(QDir::System | QDir::AllEntries |
>> QDir::Hidden);
>>
>>   qDebug() << QDir(".").entryInfoList(QDir::System | QDir::AllEntries |
>> QDir::Hidden);
>> }
>>
>> Thanks in advance for any idea.
>>
>> Robert
>>
>> _______________________________________________
>> Interest mailing list
>> Interest at qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/interest