lp:maria

Author: Oleg Smirnov
Author Date: 2024-05-18 14:34:40 UTC

MDEV-32041 Fix code review comments

Author: Sergei Golubchik
Author Date: 2024-05-03 17:19:03 UTC

MDEV-28671 post-testing fixes

Various help message improvements:
* MySQL->MariaDB, mysqld->mariadbd, "mysqld daemon" -> "mariadbd process"
* typos
* don't specify defaults directly in the help message
* don't say that an option is deprecated, mark is as such
* missing spaces in the middle of the text
etc

Author: Julius Goryavsky
Author Date: 2024-05-17 16:01:21 UTC

galera: rpl_galera_to_mariadb_clone_slave_using_mariadb-backup test fix for ssl support

Author: Julius Goryavsky
Author Date: 2024-05-17 16:01:21 UTC

galera: rpl_galera_to_mariadb_clone_slave_using_mariadb-backup test fix for ssl support

Author: Oleg Smirnov
Author Date: 2024-03-10 10:03:04 UTC

MDEV-33281 Implement optimizer hints

This commit introduces a parser for optimizer hints.
Mainly it is a port from MySQL
(commit SHA a8b622b9eee3380ce66a2be45cfaee72dfb38165), WL#8016.

High level requirements:
F-1: Hints must be enclosed into /*+ */ comment.
F-2: Hints must be specified after
     SELECT|INSERT|REPLACE|UPDATE|DELETE key words.
F-3: EXPLAIN must understand hints.
F-4: System identifiers must be visible in EXPLAIN FORMAT=JSON.

Author: Robin Newhouse
Author Date: 2023-07-25 20:13:33 UTC

Minimize unsafe C functions with safe_strcpy()

Similar to #2480.
567b681 introduced safe_strcpy() to minimize the use of C with
potentially unsafe memory overflow with strcpy() whose use is
discouraged.
Replace instances of strcpy() with safe_strcpy() where possible, limited
here to files in the `sql/` directory.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.

Author: Anson Chung
Author Date: 2024-04-25 17:46:33 UTC

Add GitLab MTR runs to ensure support past 2038

A new GitLab MTR run is added which uses libfaketime to simulate
starting the server and running the test suite in the future.
The job simulates running in two times in the year 2038,
once before the signed 32 bit UNIX limit and once after. This change
helps to ensure that all future changes preserve the ability for the
engine start and function past the year 2038.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.

Fixes #3228

Author: Monty
Author Date: 2024-05-17 11:09:31 UTC

fixup! 9831521fc852f72317024778c723e28d121635e6

Author: Alexander Barkov
Author Date: 2024-05-14 05:19:34 UTC

MDEV-34123 CONCAT Function Returns Unexpected Empty Set in Query

Search conditions were evaluated using val_int(), which was wrong.
Fixing the code to use val_bool() instead.

Details:
- Adding a new item_base_t::IS_COND flag which marks Items used
  as <search condition> in WHERE, HAVING, JOIN ON, CASE WHEN clauses.
  The flag is at the parse time.
  These expressions must be evaluated using val_bool() rather than val_int().

  Note, the optimizer creates more Items which are used as search conditions.
  Most of these items are not marked with IS_COND yet. This is OK for now,
  but eventually these Items can also be fixed to have the flag.

- Adding a method Item::is_cond() which tests if the Item has the IS_COND flag.

- Implementing Item_cache_bool. It evaluates the cached expression using
  val_bool() rather than val_int().
  Overriding Type_handler_bool::Item_get_cache() to create Item_cache_bool.

- Implementing Item::save_bool_in_field(). It uses val_bool() rather than
  val_int() to evaluate the expression.

- Implementing Type_handler_bool::Item_save_in_field()
  using Item::save_bool_in_field().

- Fixing all Item_bool_func descendants to implement a virtual val_bool()
  rather than a virtual val_int().

- To find places where val_int() should be fixed to val_bool(), a few
  DBUG_ASSERT(!is_cond()) where added into val_int() implementations
  of selected (most frequent) classes:

  Item_field
  Item_str_func
  Item_datefunc
  Item_timefunc
  Item_datetimefunc
  Item_cache_bool
  Item_bool_func
  Item_func_hybrid_field_type
  Item_basic_constant descendants

- Fixing all places where DBUG_ASSERT() happened during an "mtr" run
  to use val_bool() instead of val_int().

Author: Sergey Petrunia
Author Date: 2024-05-12 16:10:47 UTC

MDEV-34125: ANALYZE FORMAT=JSON: r_engine_stats.pages_read_time_ms has wrong scale

- Change the comments in class ha_handler_stats to say the members
  are in ticks, not milliseconds.
- In sql_explain.cc, adjust the scale to print milliseconds.

Author: Vicențiu Ciorbaru
Author Date: 2024-05-16 15:00:08 UTC

Add EF_SEARCH and EF_CONSTRUCTION to intermediate layers too

Author: Dave Gosselin
Author Date: 2024-03-18 17:03:33 UTC

MDEV-32777 DDL operations very slow under macOS

macOS is a development platform for our product but calls to
fcntl(F_FULLFSYNC) are too slow for developer needs. macOS offers
at least three different disk flush calls, making tradeoffs between
reliability and performance (see analysis notes on this MDEV).
On macOS only and as a compromise, we will back out the change to
call fcntl(F_FULLFSYNC), sacrificing a full flush in favor of more
performant writes.

Author: Sergei Golubchik
Author Date: 2024-05-16 10:12:03 UTC

11.6 branch

Author: Lena Startseva
Author Date: 2024-05-16 08:09:11 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 11.4

Author: mariadb-DebarunBanerjee
Author Date: 2024-05-16 07:40:44 UTC

MDEV-34167 We fail to terminate transaction early with ER_LOCK_TABLE_FULL when lock memory is growing

This regression is introduced in 10.6 by following commit.
commit b6a2472489accf0ae9ac3655ffe9b2997ab267ba
MDEV-27891: SIGSEGV in InnoDB buffer pool resize

During DML, we check if buffer pool is running out of data pages in
buf_pool_t::running_out. Here is 75% of the buffer pool is occupied by
non-data pages we rollback the current transaction and exit with
ER_LOCK_TABLE_FULL.

The integer division (n_chunks_new / 4) becomes zero whenever the total
number of chunks are < 4 making the check completely ineffective for
such cases. Also the check is inaccurate for larger chunks.

Fix-1: Correct the check in buf_pool_t::running_out.

Fix-2: While waiting for free page, check for
buf_LRU_check_size_of_non_data_objects.

Author: Lena Startseva
Author Date: 2024-05-16 06:40:58 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 11.1

Author: Lena Startseva
Author Date: 2024-05-16 04:13:03 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 11.0

Author: Lena Startseva
Author Date: 2024-04-24 06:19:35 UTC

Removing MDEV-27871 from tastcases because it is not a bug
Fix for v.11.0

Author: Lena Startseva
Author Date: 2024-05-16 03:27:31 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 10.11

Author: Lena Startseva
Author Date: 2024-05-15 10:43:48 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 10.6

Author: VladislavVaintroub
Author Date: 2024-05-15 22:24:01 UTC

fix embedded

Author: Brandon Nesterenko
Author Date: 2024-05-15 19:51:01 UTC

Fix rpl.rpl_semi_sync_cond_var_per_thd

rpl.rpl_semi_sync_cond_var_per_thd could sporadically fail with
ER_NET_READ_ERROR, which was not suppressed in the test.

This patch adds the test suppression for that error.

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:55:43 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:55:43 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:54:58 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:54:58 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:54:18 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:54:18 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:53:41 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:53:41 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-05-15 14:52:16 UTC

bump the VERSION

Author: mariadb-DebarunBanerjee
Author Date: 2024-05-15 14:17:56 UTC

MDEV-28800 SIGABRT due to running out of memory for InnoDB locks

This regression is introduced in 10.6 by following commit.
commit 898dcf93a886a68450dce7713208fdad82f9178e
(Cleanup the lock creation)

It removed one important optimization for lock bitmap pre-allocation.
We pre-allocate about 8 byte extra space along with every lock object to
adjust for similar locks on newly created records on the same page by
same transaction. When it is exhausted, a new lock object is created
with similar 8 byte pre-allocation. With this optimization removed we
are left with only 1 byte pre-allocation. When large number of records
are inserted and locked in a single page, we end up creating too many
new locks almost in n^2 order.

Fix-1: Bring back LOCK_PAGE_BITMAP_MARGIN for pre-allocation.

Fix-2: Use the extra space (40 bytes) for bitmap in trx->lock.rec_pool.

Author: Dave Gosselin
Author Date: 2024-03-07 20:31:26 UTC

MTR flag to mark tests as incompatible with macOS

Introduces a new MTR include, not_mac.inc, which when included
at the top of a test, prevents that test from running on macOS.

Author: Sergei Golubchik
Author Date: 2024-05-15 13:26:20 UTC

post testing 3

Author: Hugo Wen
Author Date: 2024-05-07 18:28:21 UTC

MDEV-34112 Replace one operator name keyword

Alternative operator name keywords like `and`, `or`, `xor`, etc., are
uncommon in MariaDB and can cause obscure build errors when the GCC
flag `-fno-operator-names` is applied.

Description of `-fno-operator-names`:
https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Dialect-Options.html
> Do not treat the operator name keywords `and`, `bitand`, `bitor`,
> `compl`, `not`, `or` and `xor` as synonyms as keywords.

Part of the build errors:

    /local/p4clients/pkgbuild-LdLa_/workspace/src/RDSMariaDB/sql/sql_select.cc:11171:28: error: expected ‘)’ before ‘and’
    11171 | DBUG_ASSERT(sel >= 0.0 and sel <= 1.00001);
          | ^~~
    /local/p4clients/pkgbuild-LdLa_/workspace/src/RDSMariaDB/include/my_global.h:372:44: note: in definition of macro ‘unlikely’
      372 | #define unlikely(x) __builtin_expect(((x) != 0),0)
          | ^
    ...

The build failure is caused by using alternative operator name keywords
`and` introduced in commit b66cdbd1e.
Replace the `and` keyword with `&&` and target on MariaDB 11.0+ branches
which include the commit.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.

Author: Kristian Nielsen
Author Date: 2024-05-15 12:55:57 UTC

MDEV-32020: Promote pending XA PREPARE on slave

When a slave needs to be promoted as a new master, can now explicitly
instantiate pending XA PREPAREd transactions into prepared, committed, or
rolled back state.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-15 12:53:23 UTC

MDEV-34175 mtr_t::log_close() warning should change the shutdown condition

- InnoDB should print the warning message saying "Shutdown is in progress"
only when shutdown state is greater than SRV_SHUTDOWN_INITIATED.

Author: mariadb-DebarunBanerjee
Author Date: 2024-05-15 12:00:52 UTC

MDEV-34166 Server could hang with BP < 80M under stress

BUF_LRU_MIN_LEN (256) is too high value for low buffer pool(BP) size.
For example, for BP size lower than 80M and 16 K page size, the limit is
more than 5% of total BP and for lowest BP 5M, it is 80% of the BP.
Non-data objects like explicit locks could occupy part of the BP pool
reducing the pages available for LRU. If LRU reaches minimum limit and
if no free pages are available, server would hang with page cleaner not
able to free any more pages.

Fix: To avoid such hang, we adjust the LRU limit lower than the limit
for data objects as checked in buf_LRU_check_size_of_non_data_objects()
i.e. one page less than 5% of BP.

Author: Rucha Deodhar
Author Date: 2024-05-15 11:36:20 UTC

MDEV-34143: Server crashes when executing JSON_EXTRACT after setting
non-default collation_connection

Analysis:
Due to different collation, the string has nothing to chop off.

Fix:
Got rid of chop(), only append " ," only when we have more elements to
add to the result.

Author: Monty
Author Date: 2024-05-15 09:20:27 UTC

MDEV-33582 Add more warnings to be able to better diagnose network issues

Changed the logged messages from errors to warnings
Also changed 'remain' to 'read_length' in the warning to make it more readable.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-15 10:03:02 UTC

MDEV-34619 Don't allow innodb_open_files to be lesser than
  number of non-user tablespace.

- InnoDB only closes the user tablespace when the number of open
files exceeds innodb_open_files limit. In that case, InnoDB should
make sure that innodb_open_files value should be greater
than number of undo tablespace, system and temporary tablespace.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-15 06:26:25 UTC

MDEV-33087 ALTER TABLE...ALGORITHM=COPY should build indexes more efficiently

- During copy algorithm, InnoDB should use bulk insert operation
for row by row insert operation. By doing this, copy algorithm
can effectively build indexes. This optimization is disabled
for temporary table, versioning table and table which has
foreign key relation.

Introduced the variable innodb_alter_copy_bulk to allow
the bulk insert operation for copy alter operation
inside InnoDB. This is enabled by default

ha_innobase::extra(): HA_EXTRA_END_ALTER_COPY mode tries to apply
the buffered bulk insert operation, updates the non-persistent
table stats.

row_merge_bulk_t::write_to_index(): Update stat_n_rows after
applying the bulk insert operation

row_ins_clust_index_entry_low(): In case of copy algorithm,
switch to bulk insert operation.

copy_data_error_ignore(): Handles the error while copying
the data from source to target file.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-11 16:31:47 UTC

MDEV-34057 Inconsistent FTS state in concurrent scenarios

Problem:
=======
- This commit is a merge of mysql commit 129ee47ef994652081a11ee9040c0488e5275b14.
InnoDB FTS can be in inconsistent state when sync operation
terminates the server before committing the operation. This
could lead to incorrect synced doc id and incorrect query results.

Solution:
========
- During sync commit operation, InnoDB should pass
the sync transaction to update the max doc id
in the config table.

Author: Dave Gosselin
Author Date: 2024-04-26 16:13:31 UTC

MDEV-33971 NAME_CONST in WHERE clause replaced by inner item

Improve performance of queries like
  SELECT * FROM t1 WHERE field = NAME_CONST('a', 4);
by, in this example, replacing the WHERE clause with field = 4
in the case of ref access.

The rewrite is done during fix_fields and we disambiguate this
case from other cases of NAME_CONST by inspecting where we are
in parsing. We rely on THD::where to accomplish this. To
improve performance there, we change the type of THD::where to
be an enumeration, so we can avoid string comparisons during
Item_name_const::fix_fields. Consequently, this patch also
changes all usages of THD::where to conform likewise.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-14 15:27:58 UTC

MDEV-34156 InnoDB fails to apply the redo log for compressed tablespace

Problem:
=======
- During recovery, InnoDB fails to apply the redo log
for compressed tablespace. The reason is that
InnoDB assumes that pages has been freed while applying
the redo log for it. InnoDB does multiple scan due to
small buffer pool size. Problematic page has been freed and
reinitialize multiple times. InnoDB stores the freed page
information before it ran out of memory.
But InnoDB assigns the freed page ranges to tablespace in
recv_init_crash_recovery_spaces() even though
InnoDB doesn't have complete freed range information.

Solution:
=========
rec_init_crash_recovery_spaces(): Don't add the freed ranges
to tablespace if InnoDB has to read the redo log again.

Author: Julius Goryavsky
Author Date: 2024-05-14 02:52:53 UTC

MDEV-32633: additional post-merge changes for 10.5+

Author: Sergei Golubchik
Author Date: 2024-05-14 11:31:53 UTC

cleanup: compile with -fno-operator-names in maintainer mode

Author: Julius Goryavsky
Author Date: 2024-05-14 11:18:33 UTC

galera: wsrep-lib submodule update

Author: Denis Protivensky
Author Date: 2024-05-07 13:42:13 UTC

MDEV-33952: Fix flaky galera_create_table_as_select test with debug sync

The test that triggers multi-master conflict between two CTAS commands
uses LOCK/UNLOCK TABLES to block local CTAS from progress. It could
result in a race when UNLOCK TABLES command is issued a bit earlier
then needed, causing local CTAS to run further and change wsrep
transaction state, so that a different code path is taken later and
the original error gets overridden, causing the test to fail.
The solution is to replace LOCK/UNLOCK TABLES with debug sync points.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Lena Startseva
Author Date: 2023-08-30 05:22:07 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 10.5

Author: midenok
Author Date: 2024-05-13 17:58:51 UTC

Error codes

Author: Yuchen Pei
Author Date: 2024-05-07 03:49:17 UTC

MDEV-28105 Return error in ha_spider::write_row() if info(HA_STATUS_AUTO) fails

Spider calls info with HA_STATUS_AUTO to update auto increment info,
which may attempt to connect the data node. If the connection fails,
it may emit an error and return the same error. This error should not
be of lower priority than any possible error from the later call to
handler::update_auto_increment().

Without this change, certain errors from update_auto_increment() such
as HA_ERR_AUTOINC_ERANGE may get ignored, causing my_insert() to call
my_ok(), which fails the assertion because the error was emitted in
the info() call (Diagnostics_area::is_set() returns true).

Author: Dave Gosselin
Author Date: 2024-04-03 17:16:25 UTC

MDEV-32397, MDEV-32403 Crashes during join processing.

Queries having the following form may cause a crash
  SELECT t1.a FROM ( SELECT a AS a1 FROM t1 ) dt
  JOIN t1 ON a1 LIKE EXISTS ( SELECT a + RAND () FROM t1 UNION SELECT a FROM t1);
because the table t1 has some JOIN cleanup operations performed prematurely
during the subselect.

In this particular case, the presence of RAND() makes the subquery
uncacheable, necessitating the need to execute the subquery multiple
times during join record evaluation. Each time the subquery runs, it
creates its own JOIN structure which has references to the table t1.
When the subquery completes, JOIN::cleanup and functions called by it
result in ha_end_keyread() being called on table t1. However, we are
not done with table t1 because the upper level `select t1.a from...`
query requires table t1 to be open. To solve this, we make the executor
aware of when we're in subqueries like this and delay JOIN cleanup
until the end of the query.

Author: Denis Protivensky
Author Date: 2024-03-25 13:40:55 UTC

MDEV-32633: Fix Galera cluster <-> native replication interaction

It's possible to establish Galera multi-cluster setups connected
through the native replication when every Galera cluster is configured
to have a separate domain ID.
For this setup to work, we need to replace domain ID values in generated
GTID events when they are written at transaction commit to the values
configured by Wsrep replication.

At the same time, it's possible that the GTID event already contains
a correct domain ID if it comes through the native replication from
another Galera cluster.
In this case, when such an event is applied either through a native
replication slave thread or through Wsrep applier, we write GTID event
on transaction start and avoid writing it during transaction commit.

The code contained multiple problems that were fixed:
- applying GTID events didn't work because it's applied without a
running server transaction and Wsrep transaction was not started
- GTID event generation on transaction start didn't contain proper
"standalone" and "is_transactional" flags that the original applied
GTID event contained
- condition determining that GTID event is written on transaction start
to avoid writing it on commit relied on the fact that the GTID event
is the first found in transaction/statement caches, which wasn't the
case and resulted in duplicate GTID events written
- instead of relying on the caches to find a GTID event, a simple check
is introduced that follows the exact rules for checking if event is
written at transaction start as described above
- the test case is improved to check that exact GTID events are
applied after two Galera clusters have synced.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2024-05-13 20:31:45 UTC

MDEV-21322: report slave progress to the master

This patch extends the command `SHOW REPLICA HOSTS` with three
columns:

 1) Gtid_State_Sent. This represents that latest GTIDs sent to the
replica in each domain. It will always be populated, regardless of
the semi-sync status (i.e. asynchronous connections will still
update this column with the latest GTID state sent to the replica).

 2) Gtid_State_Ack. For semi-synchronous connections (only), this
column represents the last GTID in each domain that the replica has
acknowledged.

 3) Sync_Status. This value represents the synchronization status of
the replica, and is used to help determine how to interpret the
Gtid_State_Ack column. There are four possible values:

   3.1) Initializing. This means the binlog dump thread is still
initializing, and has not yet determined the synchronization status
of the replica.

   3.2) Asynchronous: This means the replica is not configured for
semi-sync replication, and thereby, Gtid_State_Ack should always be
empty.

   3.3) Semi-sync Stale: This means the replica is configured for
semi-sync replication, however, connected using an old state, and is
not readily able to send ACKs for new transactions. Functionally,
this means that the primary will try to catch the replica up-to-date
by sending transactions which will not be ACKed. Additionally, the
value shown by Gtid_State_Ack will be empty until the replica
catches up and ACKs its first transaction.

   3.4) Semi-sync Active: This means the replica is configured for
semi-sync replication, and is readily sending ACKs for new
transactions it receives. It is possible for Gtid_State_Ack to be
empty while Sync_Status is "Semi-sync Active" if no new transactions
have been executed on the primary since the replica has connected.

Additionally, this patch creates a new semantic for the
configuration rpl_semi_sync_master_timeout=0. That is, now when 0,
1) new transactions will not attempt to wait for an ACK before
completing, and 2) the primary will still request ACKs from the
replica for new transactions. This means that Gtid_State_Ack will be
updated for each ACK from the replica and Sync_Status will read as
"Semi-sync Active". Effectively, this creates a mode to mimic the
asynchronous connection behavior, while allowing one to monitor the
progress at which the primary is sending transactions to the replica
via the new columns Gtid_State_Sent and Gtid_State_Ack.

Also note that a new error message was added to account for the case
that Gtid_State_(Sent/Ack) represents a binary log file that was
purged/cannot be found.

The overall implementation is rather simple. It leverages the
existing semi-sync framework, where the replica uses binlog file:pos
to ACK transactions, in order to infer GTID state by performing a
binlog lookup at the time `SHOW REPLICA HOSTS` is executed. In
particular, the Slave_info struct is extended to store 1) the binlog
file:pos pair of the transaction which was last sent to the replica,
2) the binlog file:pos pair that was last ACKed by the replica, and
3) and enum to represent the Sync_Status.

This patch was initially started by @JackSlateur in PR#1427, where
it was then transferred to @an3l who buffed it out in PR#2374, and
final touches were put on by @bnestere.

Reviewed By:
============
<TODO>

Author: Dave Gosselin
Author Date: 2024-05-13 14:36:11 UTC

MDEV-34129 mariadb-install-db appears to hang on macOS

A bug in signal_handler thread initialization led to an overcomplicated
implementation of wait_for_signal_handler_to_end, namely that we could
fail to initialize the signal handler but mark it as active anyway.
This meant that we could wait for it to terminate when it didn't exist
in the first place.

Additionally, let's immediately close down the signal handler loop when
we decide to break connections--it's the start of process termination
anyway, and there's no need to wait once we've invoked break_connections.

Author: Sergey Petrunia
Author Date: 2024-04-16 09:40:47 UTC

MDEV-33752: EXPLAIN UPDATE acquires locks on table rows

Variant#2: modify thd_sql_command().

Make EXPLAIN commands behave like SELECTs when it comes to locking:
- In the SQL parser, make EXPLAIN DELETE and EXPLAIN UPDATE not to
  request write MDL or table locks.
- Same as above in multi_delete_set_locks_and_link_aux_tables().
- InnoDB analyzes the return value of thd_sql_command(). Make it return
  SQLCOM_SELECT for EXPLAINs.

Note that before this patch, EXPLAIN UPDATE over InnoDB table would read
rows using READ-COMMITTED, like UPDATE does. After this patch, EXPLAIN
UPDATE over InnoDB table would read rows from transaction's snapshot.

Author: Dmitry Shulga
Author Date: 2024-05-06 13:10:06 UTC

MDEV-33769: Memory leak found in the test main.rownum run with --ps-protocol against a server built with the option -DWITH_PROTECT_STATEMENT_MEMROOT

A memory leak happens on the second execution of a query that run in PS mode
and uses the function ROWNUM().

A memory leak took place on allocation of an instance of the class Item_int
for storing a limit value that is performed at the function set_limit_for_unit
indirectly called from JOIN::optimize_inner. Typical trace to the place where
the memory leak occurred is below:
 JOIN::optimize_inner
  optimize_rownum
   process_direct_rownum_comparison
    set_limit_for_unit
     new (thd->mem_root) Item_int(thd, lim, MAX_BIGINT_WIDTH);

To fix this memory leak, calling of the function optimize_rownum()
has to be performed only once on first execution and never called
after that. To control it, the new data member
  first_rownum_optimization
added into the structure st_select_lex.

Author: Sergei Golubchik
Author Date: 2024-05-13 09:09:55 UTC

main.alter_table_online fails in --view

disable view protocol (DROP VIEW doesn't work very well with transactions)
and cosmetic cleanups

Author: Sergey Petrunia
Author Date: 2024-05-03 10:40:06 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

(Based on original patch by Oleksandr Byelkin)

Multi-table DELETE can execute via "buffered" mode: at phase #1 it collects
rowids of rows to be deleted, then at phase #2 in multi_delete::do_deletes()
it calls handler->rnd_pos() to read rows to be deleted and deletes them.

The problem occurred when phase #1 used Rowid Filter on the table that
phase #2 would be deleting from.
In InnoDB, h->rnd_init(scan=false) and h->rnd_pos() is an index scan over PK
under the hood. So, at phase #2 ha_innobase::rnd_init() would try to use the
Rowid Filter and hit an assertion inside ha_innobase::rnd_init().

Note that multi-table UPDATE works similarly but was not affected, because
patch for MDEV-7487 added code to disable rowid filter for phase #2 in
multi_update::do_updates().

This patch changes the approach:
- It makes InnoDB not use Rowid Filter in rnd_pos() scans: it is disabled in
  ha_innobase::rnd_init() and enabled back in ha_innobase::rnd_end().
- multi_update::do_updates() no longer disables Rowid Filter for phase#2 as
  it is no longer necessary.

Author: Yuchen Pei
Author Date: 2024-05-13 07:22:20 UTC

MDEV-32627 Spider: use CONNECTION string in SQLDriverConnect

Author: Yuchen Pei
Author Date: 2024-05-12 23:15:14 UTC

MDEV-32487 Check plugin is ready when resolving storage engine

This handles the situation when one thread is still initiating a
storage engine plugin, while another is creating a table using it.

Author: Sergei Golubchik
Author Date: 2024-05-12 09:25:32 UTC

sporadic failures of galera.galera_sst_mariabackup

the test failed almost always in release (but not in debug) builds with

--- galera_sst_mariabackup.result
+++ galera_sst_mariabackup.reject
@@ -5,7 +5,7 @@
 connection node_1;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces
-0
+3
 connection node_2;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces

and

[Warning] InnoDB: Cannot change innodb_undo_tablespaces=0 because previous shutdown was not with innodb_fast_shutdown=0

because mariadbd *before this test* wasn't using innodb_fast_shutdown=0

Fix the bootstrap to use innodb_fast_shutdown=0 (and the bootstrap
creates a starting point for any test that uses a .cnf file)

followup for cac0fc97cca4

also, remove redundant include/have_innodb.inc

Author: Sergei Golubchik
Author Date: 2024-05-12 09:25:32 UTC

sporadic failures of galera.galera_sst_mariabackup

the test failed almost always in release (but not in debug) builds with

--- galera_sst_mariabackup.result
+++ galera_sst_mariabackup.reject
@@ -5,7 +5,7 @@
 connection node_1;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces
-0
+3
 connection node_2;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces

and

2024-05-11 14:35:36 0 [Warning] InnoDB: Cannot change innodb_undo_tablespaces=0 because previous shutdown was not with innodb_fast_shutdown=0

because mariadbd *before this test* wasn't using innodb_fast_shutdown=0

Fix the bootstrap to use innodb_fast_shutdown=0 (and the bootstrap
creates a starting point for any test that uses a .cnf file)

followup for cac0fc97cca4

also, remove redundant include/have_innodb.inc

Author: Sergei Golubchik
Author Date: 2024-05-09 18:37:40 UTC

MDEV-33852 start the server after deb installation

and fix installation of mysql.service on buster

followup for ec09c034d84f

Author: VladislavVaintroub
Author Date: 2024-05-10 15:30:51 UTC

MDEV-33748 : use C++ thread_local, also for mysys TLS variable.

This avoids compiler-specific definitions, or C11 requirement.

Author: Yuchen Pei
Author Date: 2024-05-10 00:01:15 UTC

MDEV-33490 MENT-1504 Fix some english strings in spider.

Author: Rex Johnston
Author Date: 2024-03-18 20:50:19 UTC

MDEV-32583 UUID() should be treated as stochastic for the purposes of forcing query materialization

RAND() and UUID() are treated differently with respect to subquery
materialization both should be marked as uncacheable, forcing materialization.
Altered Create_func_uuid(_short)::create_builder()

Author: Rex Johnston
Author Date: 2024-05-09 18:30:26 UTC

MDEV-28621 group by optimization incorrectly removing subquery where subject buried in a function

Do not remove the unit from the query graph. Group by item removal happens during prepare phase.
Revert change after execution.

Author: Vicențiu Ciorbaru
Author Date: 2024-02-17 15:03:30 UTC

Initial HNSW implementation

This commit includes the work done in collaboration with Hugo Wen from
Amazon:

    MDEV-33408 Alter HNSW graph storage and fix memory leak

    This commit changes the way HNSW graph information is stored in the
    second table. Instead of storing connections as separate records, it now
    stores neighbors for each node, leading to significant performance
    improvements and storage savings.

    Comparing with the previous approach, the insert speed is 5 times faster,
    search speed improves by 23%, and storage usage is reduced by 73%, based
    on ann-benchmark tests with random-xs-20-euclidean and
    random-s-100-euclidean datasets.

    Additionally, in previous code, vector objects were not released after
    use, resulting in excessive memory consumption (over 20GB for building
    the index with 90,000 records), preventing tests with large datasets.
    Now ensure that vectors are released appropriately during the insert and
    search functions. Note there are still some vectors that need to be
    cleaned up after search query completion. Needs to be addressed in a
    future commit.

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

As well as the commit:

    Introduce session variables to manage HNSW index parameters

    Three variables:

    hnsw_max_connection_per_layer
    hnsw_ef_constructor
    hnsw_ef_search

    ann-benchmark tool is also updated to support these variables in commit
    https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
    https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

Author: Vicențiu Ciorbaru
Author Date: 2024-02-17 15:03:30 UTC

Initial HNSW implementation

This commit includes the work done in collaboration with Hugo Wen from
Amazon:

    MDEV-33408 Alter HNSW graph storage and fix memory leak

    This commit changes the way HNSW graph information is stored in the
    second table. Instead of storing connections as separate records, it now
    stores neighbors for each node, leading to significant performance
    improvements and storage savings.

    Comparing with the previous approach, the insert speed is 5 times faster,
    search speed improves by 23%, and storage usage is reduced by 73%, based
    on ann-benchmark tests with random-xs-20-euclidean and
    random-s-100-euclidean datasets.

    Additionally, in previous code, vector objects were not released after
    use, resulting in excessive memory consumption (over 20GB for building
    the index with 90,000 records), preventing tests with large datasets.
    Now ensure that vectors are released appropriately during the insert and
    search functions. Note there are still some vectors that need to be
    cleaned up after search query completion. Needs to be addressed in a
    future commit.

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

As well as the commit:

    Introduce session variables to manage HNSW index parameters

    Three variables:

    hnsw_max_connection_per_layer
    hnsw_ef_constructor
    hnsw_ef_search

    ann-benchmark tool is also updated to support these variables in commit
    https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
    https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

Author: Yuchen Pei
Author Date: 2024-05-09 03:56:26 UTC

MDEV-32180 [demo] drop functions in spider deinit

This will not work because of MDEV-32796: if spider is used in any
way (e.g. create table ... engine=Spider), any attempt to uninstall it
will be deferred to server shutdown, by which time it would be
impossible to send queries over local sql service, because
close_connection() sets thd->killed to KILL_CONNECTION

Author: Yuchen Pei
Author Date: 2024-05-09 01:16:12 UTC

MDEV-34064 [demo] return error from ha_commit_trans() in rpl_slave_state::truncate_state_table(THD *thd)

just a demo - a similar fix to MDEV-28105 and MDEV-27186.

Whether the 12607 error makes sense is perhaps a different matter:

mysqltest: At line 7: query 'CHANGE MASTER TO master_host='a',master_port=1,master_user='a',master_demote_to_slave=1' failed: <Unknown> (12607): This xid is not exist

Author: Yuchen Pei
Author Date: 2024-05-09 01:08:14 UTC

MDEV-29027 ASAN errors in spider_db_free_result after partition DDL

Spider calls ha_spider::close() at least twice on ALTER TABLE ... ADD
PARTITION. The first call frees wide_handler and the second call
accesses wide_handler->trx->thd (heap-use-after-free).

In general, there seems to be no problem with using THD obtained by
the macro current_thd() except in background threads. Thus, we simply
replace wide_handler->trx->thd with current_thd().

Original author: Nayuta Yanagasawa

Author: Yuchen Pei
Author Date: 2024-05-09 01:02:35 UTC

MDEV-27186 spider/partition: Report error on info() failure

Like MDEV-28105, spider may attempt to connect to remote server in
info(), and it may emit an error upon failure to connect. In this
case, the downstream caller ha_partition::open() should return the
error to avoid inconsistency.

This fixes MDEV-27186, MDEV-27237, MDEV-27334, MDEV-28241, MDEV-34101.

Author: Yuchen Pei
Author Date: 2024-05-09 00:28:55 UTC

MDEV-27186 spider/partition: Report error on info() failure

Like MDEV-28105, spider may attempt to connect to remote server in
info(), and it may emit an error upon failure to connect. In this
case, the downstream caller ha_partition::open() should return the
error to avoid inconsistency.

This fixes MDEV-27186, MDEV-27237, MDEV-27334, MDEV-28241, MDEV-34101.

Author: Julius Goryavsky
Author Date: 2024-05-09 00:18:16 UTC

MDEV-33677 : Port Performance Schema for Galera from ES

Added performance schema tables for Galera.
This is a port of the code from ES 10.6.

Refs:
MENT-346 commit 86c06851
MENT-346 commit 10d0fcaf
MENT-1333 commit 2005adc3
MENT-347 commit aaca43b4
MENT-1370 commit 1f460e6b

Author: Igor Babaev
Author Date: 2024-05-08 22:30:15 UTC

MDEV-30073 Wrong result on 2nd execution of PS for query with NOT EXISTS

This is a preliminary patch to check possible failures in buildbot.
Variant 7

Author: Sergei Golubchik
Author Date: 2024-05-08 22:04:20 UTC

columnstore 6.4.8-2

Author: Rex Johnston
Author Date: 2024-04-29 23:39:20 UTC

MDEV-19520 pushdown_from_having_into_where() not calling fix_fields() correctly

fix_fields() called in pushdown_from_having_into_where() assuming that the
item is immutable.
normalize_cond() extended to include (not f)=>(f == 0) to avoid the
issue where NOT(field) was being transformed to (field == 0) during
this fix_fields call.

Author: Sergei Golubchik
Author Date: 2024-05-08 07:31:10 UTC

test needs to cleanup after itself

Author: Sergey Petrunia
Author Date: 2024-05-03 10:40:06 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

(Based on original patch by Oleksandr Byelkin)

Multi-table DELETE can execute via "buffered" mode: at phase #1 it collects
rowids of rows to be deleted, then at phase #2 in multi_delete::do_deletes()
it calls handler->rnd_pos() to read rows to be deleted and deletes them.

The problem occurred when phase #1 used Rowid Filter on the table that
phase #2 would be deleting from.
In InnoDB, h->rnd_init(scan=false) and h->rnd_pos() is an index scan over PK
under the hood. So, at phase #2 ha_innobase::rnd_init() would try to use the
Rowid Filter and hit an assertion inside ha_innobase::rnd_init().

Note that multi-table UPDATE works similarly but was not affected, because
patch for MDEV-7487 added code to disable rowid filter for phase #2 in
multi_update::do_updates().

This patch changes the approach:
- It makes InnoDB not use Rowid Filter in rnd_pos() scans: it is disabled in
  ha_innobase::rnd_init() and enabled back in ha_innobase::rnd_end().
- multi_update::do_updates() no longer disables Rowid Filter for phase#2 as
  it is no longer necessary.

Author: Sergei Golubchik
Author Date: 2024-05-08 17:01:38 UTC

Merge branch 'bb-10.5-serg' into bb-10.6-serg

Author: Dave Gosselin
Author Date: 2023-11-30 16:18:17 UTC

MDEV-28509 Dereferenced null pointer of type 'struct JOIN_TAB' in add_key_field

This patch fixes a crash when calculating join statistics during query
optimization for queries with dangling WINDOW references. Put another way,
the system may crash when a query defines a WINDOW but doesn't then refer to
it.

Item::marker is overloaded for different uses, the most typical refer to it as
a bit field. However, the setup_group function uses it to mark that a field
was found when traversing a GROUP BY. Originally, this marking set the
Item::marker field to 1 to indicate that it was found. Later on in setup_group
(and only when SQL mode ONLY_FULL_GROUP_BY is enabled), we would skip any such
marked fields when checking that fields only referenced those found in the
GROUP BY; otherwise, it would be silly to find fields of the GROUP BY within
the GROUP BY field itself. Setting Item::marker to 1 seemed mostly harmless
at that point in time. But later, in git sha 4d143a6ff6, we introduced
several changes: (1) the value of marker in setup_group was changed from 1 to
UNDEF_POS, (2) Item::marker was changed from uint8 to int8 (which has since
been changed to an int), and (3) UNDEF_POS which is defined to be -1 was also
added.

Queries that define WINDOWs internally will setup groups and orders as part
of query processing via the setup_group function. Consequently because of
the behavior described earlier above, such queries may have items with markers
as UNDEF_POS (-1, or 0xffffffff) which is the same as marking all of the flag
bits as set. This is disastrous for those users of Item::marker which refer
to it as a bit field as some of those bits are mutually exclusive in meaning,
and in many places we don't mask the bits we're interested in, we just
compare the value of the field as a whole to some flag value with direct
comparison. In particular, the method
Item_direct_view_ref::grouping_field_transformer_for_where would
incorrectly see that the ref's marker was set for substitution when it was
actually -1, all bits set, taking the wrong execution path leading to the
crash.

Upon return from the setup_group function, we set the value of the marker
flag to zero if we set it to -1. This preserves the marker behavior for
'full group by' (if configured) while not otherwise allowing the marker
flag state to leak outside of this function.

Author: Dave Gosselin
Author Date: 2024-02-02 19:29:08 UTC

MDEV-32424 Crash in subselect engine

According to MDEV-29363, recompute the with_subquery() attribute
in the pushdown code to avoid writing to freed memory.

Author: Dave Gosselin
Author Date: 2024-04-15 19:17:30 UTC

MDEV-33616 Tests failing on macOS

These tests rely on THR_KEY_mysys but it is not initialized. On
Linux, the corresponding thread variable is null, but on macOS it has a
nonzero value. In all cases, initialize the variable explicitly. We could
call MY_INIT, however some memory is leaked on test program exit in that
case. Instead, factor just enough to be able to initialize THR_KEY_mysys
from the test code (as well as from the library code).

Author: VladislavVaintroub
Author Date: 2024-05-04 08:06:11 UTC

MDEV-29955 post-fix - add CMake policy CMP0074, so ZLIB_ROOT is not ignored

Author: Monty
Author Date: 2024-05-07 19:03:38 UTC

fixup! 9831521fc852f72317024778c723e28d121635e6

Author: Yuchen Pei
Author Date: 2024-05-08 04:03:08 UTC

MDEV-28105 Return error in ha_spider::write_row() if info(HA_STATUS_AUTO) fails

Spider calls info with HA_STATUS_AUTO to update auto increment info,
which may attempt to connect the data node. If the connection fails,
it may emit an error and return the same error. This error should not
be of lower priority than any possible error from the later call to
handler::update_auto_increment().

Without this change, certain errors from update_auto_increment() such
as HA_ERR_AUTOINC_ERANGE may get ignored, causing my_insert() to call
my_ok(), which fails the assertion because the error was emitted in
the info() call (Diagnostics_area::is_set() returns true).

Author: Monty
Author Date: 2024-05-07 21:38:59 UTC

fixup! f551fa75e13dd57b057117c570f2e2414ec6d500

Author: Monty
Author Date: 2024-05-07 21:38:59 UTC

fixup! f551fa75e13dd57b057117c570f2e2414ec6d500

Author: Monty
Author Date: 2024-05-07 14:21:01 UTC

MDEV-34055 Assertion '...' failure or corruption errors upon REPAIR on Aria tables

The problem was two fold:
- REPAIR TABLE t1 USE_FRM did not work for transactional
  Aria tables (Table was thought to be repaired, which it was not) which
  caused issues in later usage of the table.
- When swapping tmp_data file to data file, sort_info files where not
  updated. This caused problems if there was several unique keys and
  there was a duplicate for the second key.

Author: Sergey Petrunia
Author Date: 2024-05-07 11:10:35 UTC

MDEV-28621: group by optimization incorrectly removing subquery where subject buried in a function

Workaround patch: Do not remove GROUP BY clause when it has
subquer(ies) in it.

remove_redundant_subquery_clauses() removes redundant GROUP BY clause
from queries in form:
  expr IN (SELECT no_aggregates GROUP BY ...)
  expr {CMP} {ALL|ANY|SOME} (SELECT no_aggregates GROUP BY ...)
This hits problems when the GROUP BY clause itself has subquer(y/ies).

This patch is just a workaround: it disables removal of GROUP BY clause
if the clause has one or more subqueries in it.

Tests:
- subselect_elimination.test has all known crashing cases.
- subselect4.result, insert_select.result are updated.
Note that in some cases results of SELECT are changed too (not just
EXPLAINs). These are caused by non-deterministic SQL: when running a
query like:

  x > ANY( SELECT col1 FROM t1 GROUP BY constant_expression)

without removing the GROUP BY, the executor is free to pick the value
of t1.col1 from any row in the GROUP BY group (denote it $COL1_VAL).
Then, it computes x > ANY(SELECT $COL1_VAL).

When running the same query and removing the GROUP BY:

   x > ANY( SELECT col1 FROM t1)

the executor will actually check all rows of t1.

Author: Galina Shalygina
Author Date: 2024-04-30 19:31:47 UTC

MDEV-23878 Wrong result with semi-join and splittable derived table

Due to this bug a wrong result might be expected from queries with
an IN subquery predicate in the WHERE clause and a derived table in the
FROM clause to which split optimization could be applied.

The function JOIN::fix_all_splittings_in_plan() used the value of the
bitmap JOIN::sjm_lookup_tables() such as it had been left after the
search for the best plan for the select containing the splittable
derived table. That value could not be guaranteed to be correct. So the
recalculation of this bitmap is needed to exclude the plans with key
accesses from SJM lookup tables.

Approved by Igor Babaev <igor@maridb.com>

Author: Monty
Author Date: 2024-05-07 07:57:43 UTC

fixup! 9c20f04ef024cd4cb42e884fecae0b540453f4dc

Author: Monty
Author Date: 2024-05-07 07:45:27 UTC

fixup! 3a7a3e9b3be3504e1bfc32c931e14ed47a317d9b

Author: Sergei Golubchik
Author Date: 2023-11-04 09:11:51 UTC

don't disable lto in DEB builds

Author: Kristian Nielsen
Author Date: 2024-05-07 06:45:31 UTC

Merge 10.6 to 10.11

Author: Kristian Nielsen
Author Date: 2024-05-07 06:45:31 UTC

Merge 10.6 to 10.11

Author: Yuchen Pei
Author Date: 2024-05-06 04:46:18 UTC

MDEV-32640 Reset thd->lex->mi.connection_name.str towards the end of mysql_execute_command

Reset the connection_name to contain a null string, if the pointer
points to the same space as that of the system variable
default_master_connection.

We do this because the system variable may be updated which could free
the pointer and create a new one, causing use-after-free for
re-execution of prepared statements and stored procedures where the
LEX may be reused.

This allows connection_name to be set again be to the system variable
pointer in the next call of this function (see earlier in this
function), after any possible updates to the system variable.