Menu
▾
▴
#2941 Launcher fails to load aircraft catalog images when Ibiblio mirror is unavailable
Sorry for the incomplete bug report, but I didn't want to miss this one.
Our primary mirror at http://mirrors.ibiblio.org/flightgear/ftp/ appears to be down at the moment. This causes issues in the launcher, as no additional aircraft can be installed or browsed.
In an attempt to work around this I added an alternative catalog from a different mirror to the launcher. It gives me the following suspicious console output:
[WARN]:general package downloaded from:https://kompot.snt.utwente.nl/pub/software/flightgear/ftp/Aircraft-trunk/catalog.xml is now at: http://mirrors.ibiblio.org/flightgear/ftp/Aircraft-trunk/catalog.xml
Suspicious, because it still refers to the ibiblio catalog (which is not available). The launcher does list and install aircraft now, but all previews and thumbnails fail to load because for some reason the launcher tries to obtain them from the Ibiblio mirror. E.g.,:
[WARN]:io CURL Result:6 Couldn't resolve host name
[WARN]:io HTTP request: set failure:6 reason Couldn't resolve host name
[WARN]:gui void __cdecl PreviewImageItem::onDownloadError(enum QNetworkReply::NetworkError) failed to download: QUrl("http://mirrors.ibiblio.org/flightgear/ftp/Aircraft-trunk/previews/c182s_Previews/cockpit_c182s-1.png")
[WARN]:gui "Host mirrors.ibiblio.org not found"
[WARN]:gui void __cdecl PreviewImageItem::onFinished(void) failed to read image data from QUrl("http://mirrors.ibiblio.org/flightgear/ftp/Aircraft-trunk/previews/c182s_Previews/cockpit_c182s-1.png")
Again, we see that the launcher is using Ibiblio instead of the UTwente mirror.
Looking at the catalog.xml format, I think the issue is that we have hardcoded paths to the Ibiblio mirror in there :-(
This is primarily an issue for the previews and thumbnails, because they only have one hardcoded path. However, the download urls also seem to be hardcoded to just three mirrors. This means that the UTwente mirror for example is never used to download any data, apart from the catalog.xml file...
The mirrors aren't hard-coded, they're part of each catalog: each catalog gets to decide what mirror it uses. And we have multiple sources for both previews/thumbnails and the actual downloads.
If you look at https://gentoo.snt.utwente.nl/pub/software/flightgear/ftp/Aircraft-trunk/catalog.xml you'll see that it lists them like:
So the problem is that all catalog mirros will use the same mirror. Therefore they don't spread the load.
Last edit: Gijs 2024-11-29
Eg see:
https://gitlab.com/flightgear/fgmeta/-/blob/next/catalog/stable-2020-catalog/catalog.config.xml?ref_type=heads
The problem we maybe have is that UKmirror is down, and I'm not Nia still runs cdn.merspeiler : so if Ibiblio is down, yeah it will fail.
To use a different server, you would need to make a new catalog confguration directory (template and catalog_config.xml) and then run the update_catalog.py script to produce such a catalog.
I guess it has never been updated when I lost the domain and moved all the mirrors to the de1mirror.flightgear.org domain... it's running here https://de1mirror.flightgear.org/fgaddon/
I did correct it in the catalog.xml which got copied to ibiblio... but it never made it into upstream git repos it seems
I guess 'it should' work, and I thought it was working for the past N years (where N > 6, I think) ... so yeah maybe it's never worked and Ibilbio is mostly quite reliable?
Right, Ibiblio has been very reliable, so I guess we just never noticed this. There's a single point of failure for the images (Nia's mirror and UKMirror aren't used for them either, so whether or not they're up doesn't matter in this case. It does for the file downloads of course).
Wouldn't it be better if we were to add a top-level list of mirrors and only use relative paths for the thumbnails, previews and downloads? The launcher can then check if an url returns a valid response, and skip to the next url down the list if it doesn't.
This way we can also use a randomizer to spread the load over the various mirrors. And it saves some bytes in the catalog.xml as we don't need to repeat the full urls a million times.
The idea was that downloads don't have to live on the same server as the catalog: using a full URL is deliberate choice. And eg the downloads could be at some hashed URL that doesn't relate the the generated structure.
Basically I was trying to allow as much flexibility for catalog authors as possible :)
WShat we coudl do is maybe allow relative URIs as well , since of course the common case for 'our' catalogs would work. But that might need a code change, I'll have to check.
And again, the idea was the catalog does contain multiple mirrors for previews & thumbnails: it's right there in the confguration file! So I'm not sure where the alternatives aren't being generated.
Ah sorry, I missed the fact that there were multiple urls for the thumbnails (but not for the previews?). Seems like this doesn't work yeah. And even if it did, we'd have an issue right now because all three mirrors are down (and the ones that are still up aren't listed).
BTW I'm confused by 'the ones that are still up' .... if people want to add a mirror, I need to add it to the catalog config.xml .... that's the only way to add a mirror that actually works, AFAIK.
(Assuming these are mirrors of the official / stable catalog: different if people have their own hangar they maintain, with its own mirrors)
I don't have a complete list, but at least these have been available for many years (and were listed on the website and/or legacy website) and are still up to date:
Last edit: Gijs 2024-11-29
Well it's the first time I learn any of these exist.! The problem is they're apparently mirroring what is on Ibiblio? But of course the generated catalog doesn't know they exist .... this is not how the system was designed to work :(
So you can manually download from these, but the launcher package manager can't use them, since the catalog XML won't ever download from them.
I think they're mirroring Ibiblio yes (so in the end that leaves us with the same SPF if Ibiblio ever ceases to exist). Ideally they would mirror from a domain that we own, so we could easily point to a different ground source.
Speaking of which, cannot we use the DNS stuff from TerraSync to dynamically serve a catalog to the launcher that's guarenteed to be online? If we then ensure that we add/remove all current mirrors to the catalog.xml file and fix the preview/thumbnail issue, we've got everything covered (I think). That is, provided that Ibiblio comes back online of course so the mirros get updated.
Last edit: Gijs 2024-11-29
There's a few things going on :)
Given all of that, I think we can do some solutions but some help would be appreciated, since my ToDo list is quite long....
Other thing is, I am planning to make some changes to what's on Ibilbio, possibly adding a TerraSync mirror (and remove some of the ancient aircraft catalogs). It would be very interesting to know if these 'hidden' mirrors will just mirror 100% what is on Ibiblio, in that case.
Addressing #3 will ensure that we can cope with future temporary outages of Ibiblio (at least for users who have the catalog in their cache, or who can update their settings to point to a catalog at one of the mirrors), while #1 and #2 will ensure that we can deal with Ibiblio going down either permanently or for a prolonged time.
I'd say that on the short term only #3 has a high priority, provided that Ibibilio will soon come back online of course...
I'll have plenty of time from January onwards to work on the other issues.