[Development] To improve UX of the online installer

[Sze Howe Koh]

3 broad issues impede new users who want to start using Qt and
frustrate old users who want to update Qt:

    (A) Downloading metadata is very time-consuming.
    (B) The automatic mirror selection algorithm doesn't always pick
the best mirror.
    (C) (New users) It's not clear how to choose what to install.


======================================
Factors that contribute to (A) include:

* The metadata downloader opens lots of short-duration TCP
connections. Wireshark revealed 2391 HTTP GET requests from
77.86.229.90 (download.qt.io) on Windows. This process is less painful
on Linux (1205 GETs -- almost half of Windows') as there are fewer
compilers/platforms.


* Downloading many small files is very inefficient compared to
downloading a few large files. While GET-ting metadata from
download.qt.io, my network traffic crawls at ~10 KiB/s (yes, ten
kibibytes). After this, the actual binaries download from my
designated mirror at ~1 MiB/s, which shows that metadata downloader
doesn't make good use of available bandwidth. I'm in Australia.


* MaintenanceTool downloads metadata too frequently (QTIFW-975).


* If the metadata download fails halfway (e.g. one file times out),
the whole process needs to start from scratch. There is no
retry/resume. This makes the online installer nigh unusable if the
user's connection to download.qt.io is flaky:
https://forum.qt.io/post/450076


* Currently, there is a way to manually avoid downloading unwanted
metadata: Deselecting the unwanted Temporary Repositories. However,
all 91 Repositories must be unchecked one-by-one, and this process
must be repeated each time MaintenanceTool is restarted:
https://forum.qt.io/post/450076


======================================
Evidence for (B) include:

* Geographical proximity does not imply mirror quality. E.g. a user
from China finds Chinese mirrors too slow, and opts for non-Chinese
mirrors instead:
http://lists.qt-project.org/pipermail/interest/2013-December/010336.html


* The algorithm doesn't pick the closest mirror anyway. E.g. a user
from Israel was given a Japanese mirror instead of a European mirror:
https://forum.qt.io/topic/87221/


======================================
Examples of (C) include:

* Some users don't realize there are multiple kits per Qt version.
They select "Qt 5.9.4" and then wonder why Qt is huge (35 GB):
https://forum.qt.io/topic/87608/windows-install-download-size


* Some users don't realize Qt Creator != Qt. They install Qt Creator
without Qt and end up unable to build anything:
https://forum.qt.io/topic/84198/no-valid-kits-found


* Some users don't realize they need to install a compiler separately.
They also end up unable to build anything:
https://forum.qt.io/topic/79532/msvc2015-64bit-compiler-kit-installation-requirements


======================================
Ideas for improvement (some render others unnecessary):

1. Amalgamate and compress all metadata into one file (or at least one
file per "group").

2. Cache metadata locally and use a hash to identify them; avoid
re-downloading metadata that's already available (Mitch Curtis
suggested this at QTIFW-975)

3. Avoid downloading metadata for archived/advanced packages by
default (Iikka Eklund said this is planned in QTIFW-975)

4a. Remember the which Repositories the user has deselected before,
and ignore them for future sessions.

4b. Let the user choose broad "groups" that they are interested in,
and remember this choice. For example, there is no point getting
metadata for Android/UWP packages or showing these packages in the
"Selection Tree" if I'm only interested in Desktop development.

5. Add a "Select/Deselect All" button to the Repositories page.

6. Allow retry/resume if the metadata download fails halfway.

7. Allow users to manually pick a mirror for binary download.

8a. Before the user is asked to "Select the components to install",
show a short page that to explain the difference between Qt and Qt
Creator and explain that a compiler must be installed separately
unless MinGW is used.

8b. Before the user is asked to "Select the components to install",
have a small wizard to ask a new user some questions -- Which
version(s): Latest version, LTS version, or older version? Which
target platforms? Which compiler? Or do they not care/know and want us
to suggest a good default? Use the answers to pre-fill the "Selection
Tree" and remind the user to download an external compiler if
necessary.

9. If no Qt version is selected (or if too many are selected), pop-up
a confirmation dialog before installing.

10. Expand the documentation at http://doc.qt.io/qt-5/gettingstarted.html


Since (3) is already being planned, please consider (4) which is a
generalization of (3).

It would be fantastic to see a holistic approach that addresses (A) +
(B) + (C). For example, (8b) helps the user make the right selection
at the first try. At the same time, users' answers in (8b) feed into
(4b) to download fewer metadata files and trim the "Selection Tree".
This is further sped up by (1) + (2): Each "group" has its own
amalgamated metadata file, and these files are cached locally to avoid
re-downloading in the future. Once the selection is complete, (7)
ensures that the use can get binaries in a timely manner even if the
auto-selected mirror is no good.


Regards,
Sze-Howe