Make SemanticMediaWiki compatible with new ElasticSearch

[marijnvanwezel]

This pull request makes Semantic MediaWiki compatible with new ElasticSearch (8.x tested). It is currently still a draft, since I have not yet updated the tests. It is working in my personal testing environment.

There are a number of major changes in the ElasticSearch API, as well as some breaking changes in the ElasticSearch PHP package, that this pull request addresses. Most notably:

• Mapping types have been removed (https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html);
• Elasticsearch PHP methods now return Promise objects, which need to be explicitly converted to a type;
• The cat API no longer returns a JSON by default;
• The associative array syntax for hosts is no longer supported (Elastic\Transport\NodePool\Node::__construct(): Argument #1 ($host) must be of type string, array given elastic/elasticsearch-php#1214). This required the addition of a configuration variable, $smwgElasticsearchCredentials;
• doc_values is no longer supported on properties of type text.

[marijnvanwezel]

I apologise for the somewhat messy commit history. If/when this is merged, a squash commit might be appropriate.

[jaideraf]

I would suggest you to "Make SemanticMediaWiki compatible with ElasticSearch 7.10.x" wich is probably the last version of ElasticSearch Wikimedia will support[0]. After that, a migration to OpenSearch[1][2] will follow because of license issues[3].

[0] https://www.mediawiki.org/wiki/Extension:CirrusSearch
[1] https://phabricator.wikimedia.org/T280482
[2] https://phabricator.wikimedia.org/T325028
[3] https://en.wikipedia.org/wiki/OpenSearch_(software)#OpenSearch

EDIT: or "Make SemanticMediaWiki compatible with OpenSearch X.x"

[marijnvanwezel]

I have not tested it, but I think this PR is also compatible with ElasticSearch 7.x, or requires minimal changes to make it compatible.

[marijnvanwezel]

I will try to see if I can get a test wiki running with ElasticSearch 7.10.2 and this PR somewhere in the coming week (probably Friday).

[malberts]

Thanks for starting with this! Another compatibility question is whether this will still work with ES 6.x, or if these changes can accommodate it. If not, then this will have to be part of SMW 5.x.

IMO if it's very difficult/time-consuming, then it is probably not worth it to complicate the code to support MW 1.35-era ElasticSearch 6.x. Perhaps SMW 5.x is a good time to raise SMW's minimum versions to MW 1.39 and ES 7.x.

[marijnvanwezel]

I have confirmed that file ingestion is now also working with this pull request.

[krabina]

See https://phabricator.wikimedia.org/T280482 for reference

[marijnvanwezel]

Ad asked Cindy and she will ask the Wikimedia Foundation what their stance will be. For now, it is probably best to:

1. Make Semantic MediaWiki compatible with the latest open-source version of ElasticSearch (7.10.x);
2. Wait until Wikimedia updates the minimum requirements of CirrusSearch;
3. Make Semantic MediaWiki compatible with the versions of ElasticSearch or OpenSearch (depending on the outcome of point 2) that are supported by CirrusSearch;
4. Try to keep Semantic MediaWiki and CirrusSearch compatible.

@krabina @jaideraf What do you think?

I didn't have time to test this patch on 7.10 unfortunately, but hopefully I will have time for it next week.

[krabina]

I totally agree!

[jaideraf]

I totally agree, too!

[marijnvanwezel]

@JeroenDeDauw I am also wondering what you think. In particular, see this comment by jaideraf and this comment by me. Thank you in advance for taking the time to look at this!

[JeroenDeDauw]

What you wrote sounds reasonable to me

[cicalese]

I was able to test this patch successfully with the following versions:

MediaWiki: 1.39.1
PHP: 7.4.33
Elasticsearch: 7.10.2

[krabina]

Great, thank you for testing! Is this now ready for integration, @JeroenDeDauw, @gesinn-it-gea ?

[wgevaert]

Some time ago, we tested the branch from marijnvanwezel/SemanticMediaWiki and one of our testers did find some issues. We have then reverted to a version with ES 6.8, so we do not have full details of the issues.

issues:

1. Sort parameter does not seem to work. Tested the following wikicode: {{#ask:[[Modification date::+]] |?Modification date |sort=Modification date }}. Does not give results. Without sort it does give results
2. Running rebuildData gave "errors" and after that "nothing worked anymore" (no results from ask queries. (I do not have more info because we moved back to ES6.8 and testers did not make screenshots).

[krabina]

@cicalese and @marijnvanwezel can you reproduce the sorting problem?
I guess problem 2 would have been noticed.

[marijnvanwezel]

I was unable to reproduce problem 2, but I was able to reproduce problem 1. ES reports the following syntax error in queries with sort:

org.elasticsearch.common.ParsingException: Failed to parse; org.elasticsearch.xcontent.XContentParseException: [1:112] [bool] failed to parse field [must]

I am looking into why this happens.

[marijnvanwezel]

Query without sort:

{
    "index": "smw-data-mediawiki",
    "body": {
        "_source": false,
        "from": 0,
        "size": 51,
        "query": {
            "constant_score": {
                "filter": {
                    "bool": {
                        "filter": {
                            "exists": {
                                "field": "P:29.datField"
                            }
                        }
                    }
                }
            }
        },
        "sort": [
            {
                "subject.sortkey.sort": {
                    "order": "asc"
                },
                "subject.title.sort": {
                    "order": "asc"
                }
            }
        ]
    }
}

Query with sort:

{
    "index": "smw-data-mediawiki",
    "body": {
        "_source": false,
        "from": 0,
        "size": 51,
        "query": {
            "constant_score": {
                "filter": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "filter": {
                                        "exists": {
                                            "field": "P:29.datField"
                                        }
                                    }
                                }
                            },
                            [
                                {
                                    "exists": {
                                        "field": "P:29.datField"
                                    }
                                }
                            ]
                        ]
                    }
                }
            }
        },
        "sort": [
            {
                "P:29.datField": {
                    "order": "asc"
                }
            }
        ]
    }
}

[marijnvanwezel]

The sorting issue is fixed in the above PR.

[bertrandgorge]

I've added some documentation here: #5614 (hope I got it right)