But doing this messes everything with Joomla Cassiopeia standard template and Bootstrap 5

That's impossible. CBs CSS is scoped to CB exclusively. It cannot style outside of CB. Our demo site is Joomla 5 with Cassiopeia and it displays perfectly fine.

As CB is used nearly everywhere to connect, bootstrap is loaded twice, this makes the pages bigger and messes everything.

It's not enough data to be a concern for todays internet. It's a few KB in a world of 500MB JS pages. You can also replace both with a CDN if you like. You can apply your own tree shaking, diff with Bootstrap 5 and only include the Bootstrap 4 classes that don't exist in 5, etc.. this is why we let you turn CBs off so you can replace it with your own.

Could be good to have a solution for only Bootstrap 5 to not overload pages ? or at least to deal with the changes such as ps ms and so on

You can map Bootstrap 5 classes to Bootstrap 4 if you want. We're going to switch to Tailwind in CB 3.x. So I'm not going to redesign all of our UI to then turn around and immediately have to do it again.

Could you just explain : You can apply your own tree shaking, diff with Bootstrap 5 and only include the Bootstrap 4 classes that don't exist in 5 ?

With what can I achieve this ? I tried to use the compiler to modify bootstrap 4 with the plugin custom twitter bootstrap but it does not seem to work well

If you don't know how to do any of these please hire a freelance developer to assist you.

You do not need to really be doing this though. Continue to load CBs Bootstrap 4 and if you need to override its colors you can do so using CB Template Changer and its Template Builder feature. You're going to have multiple CSS libraries regardless, because with CB 3.x Tailwind will be loaded in, but again it's a few kb (around 200) and is completely negligible. These type of hyper optimizations can exhaust a crazy amount of time and amount to no real performance improvements.