Inquiry Regarding Abnormal Tile Conversion When Setting MaxZoom to 16

[magicianlim]

I have a question regarding a project I'm currently developing, which utilizes tile zoom level 16. Previously, on a Mac Mini M1 using version 2.4, I configured the config.json file to set the maximum zoom to 16, allowing the creation of mbtiles files without any issues. However, on a newly purchased MacBook M3 with tilemaker version 3.0, tiles are only being correctly generated up to zoom level 15, and at 16, the tiles are not being generated properly. Why might this be happening?

When attempting to generate tiles, at zoom level 16, it results in 0 outputs. The converted tiles are significantly smaller in size than expected and appear to be missing most of the data. What should I try to resolve this issue?

Here are some metrics from the tile conversion process: collecting tiles: 326ms, filtering tiles: z0 (1, 0ms) z1 (1, 0ms) z2 (1, 0ms) z3 (1, 0ms) z4 (1, 0ms) z5 (1, 0ms) z6 (4, 0ms) z7 (8, 0ms) z8 (20, 0ms) z9 (59, 2ms) z10 (203, 9ms) z11 (719, 38ms) z12 (2612, 152ms) z13 (9651, 612ms) z14 (36803, 2458ms) z15 (142103, 9823ms) z16 (0, 13668ms).

ps...
Please continue testing. The issue persists at zoom level 15. The settings are the same as when it was set to zoom level 16.
Tile conversion works correctly up to zoom level 14.

[cldellow]

Sounds like something I might have broken.

I'm travelling for a couple of weeks, but can have a look in March if the issue is still around.

[magicianlim]

I'll be waiting for the issue to be resolved quickly. Thank you.

[magicianlim]

config-openmaptiles.json
I have attached the config.json file used for the tile conversion. Interestingly, the same issue occurs on the Mac Mini M1 with version 2.4. I tried converting files after installing version 3.0, but when it didn't work, I attempted again with version 2.4, and the same issue started occurring. Could it be that the installation of version 3.0 caused a version upgrade of other dependent software, which might be the problem? Please consider this. Thank you.

[systemed]

Just checking the obvious, but can I check you're building with the FAT_TILE_INDEX option?

[magicianlim]

FAT_TILE_INDEX is not being used. This option is believed to be employed for zoom levels above 17. Even though this option wasn't utilized when functioning correctly in version 2.4, it was feasible to produce normal tiles at zoom level 16.

I will provide information about the MacBook I am using.
MacBook pro16 - Apple m3 max
memory: 48G
Os: Sonoma 14.2.1.

Here is the version information of the installed software.
boost 1.84.0
lua 5.4.6
reinstall 1.6.0
rapidjson 1.1.0

I am attaching the log.
log.txt

Original pbf file. 191mb
https://mega.nz/file/MsYRCayT#EBK7AuYMTgUJHk7tp8PaqnCkWHkT3CULhClzQE3gNs8

16zoom converted file. 83mb
https://mega.nz/file/4wQSlKgK#xnOLYEL5A-6a-hP31NcePGMe2TUDa3DaGSNqVa-vfa8

I have compressed the folder where the tile maker used is installed.
https://mega.nz/file/c5ZXGQTQ#M8WO89J549TREoWZ1RJOJEGQCLvI0NTqfL782jZ3IQI

14zoom converted screenshot.

16zoom converted screenshot.

[cldellow]

tl;dr: this is indeed a bug that I introduced. A workaround in the short term is to build with -DFAT_TILE_INDEX.

I can reproduce the issue on master and confirm it's not present in v2.4.0.

It is quite broken in master - changing just the maxzoom/basezoom of the default config to 15 from 14 seems to break lower zooms (!).

The non-technical explanation is that I did not understand FAT_TILE_INDEX correctly. I thought stock tilemaker only supported up to z14, and that zooms above that required FAT_TILE_INDEX. However, as @magicianlim points out, stock tilemaker ought to support up to z16.

The technical explanation is that I used a uint8 for Z6Offset when FAT_TILE_INDEX is not set.

This means that when we add OutputObjects to the index (so that we might retrieve them when writing them out later), we'll overflow and end up transposing them thousands of kilometres away.

I see a few possible paths forward:

1. Require FAT_TILE_INDEX for zooms >= 15. Pro: no code changes. Con: breaks status quo.
2. Use a uint16_t for Z6Offset when FAT_TILE_INDEX is not set. Pro: restores the status quo. Con: uses an extra byte per output object. Not the end of the world, but the memory reductions were hard-won and I'd like to keep them.
3. When basezoom is > 14, index them at the z14 level. Pro: restores status quo, no extra memory. Con: there'll be a runtime hit, as we'll be relying on the later clipping logic to filter out false positives (since a single z14 tile covers 4 z15 tiles, but the item may only actually be present in a single z15 tile).

I think I like option 3. @systemed do you see any problems with that?

Separately, I notice that output_object.h only allocates 4 bits for minzoom, so it's not currently possible to express a minzoom of z16 or higher. I'm not familiar enough with how people use higher zooms - is that a problem that should also be fixed?

[systemed]

Thanks for tracking this down!

Option 3 works for me and has the advantage that we can get rid of the compile-time option. Generating z15 tiles is reasonably common, but I think that most people using z16+ will only be doing it for a small area so the runtime hit shouldn't be massive.

We probably still want to retain the runtime warning if you're generating z16+, as there is a lot of misunderstanding about the whole "overzooming" concept and people do tend to think that they need to generate z18 tiles as they would with rasters.

[cldellow]

Hi @magicianlim,

I've put together a patch that tries to fix this up. It's the disc-682-1 branch on my fork or PR #700 here.

Can you give it a try and see if it resolves your issue?

[magicianlim]

I have confirmed that the conversion was successful at maxzoom 16.
Thank you very much.

[cldellow]

Great, thank you for testing it.