feat: enhance deduplication rules provisioning with provider support and environment configuration

[tuantran0910]

Closes #4401

📑 Description

To make the provider configurations to be updated, so I think we should:

1. First, delete all existing provisioned providers.
2. Then, provision new providers from either the environment variable or the directory. Combine provisioning providers and deduplication rules together.
3. If any error occurs, rollback everything to the original state.

So each time the backend starts, restarts, ... The progress re-provisioning providers will happen. What if each time the backend restarts, the configurations not change, so I think we first calculate the hash of the configurations, then save it in either Redis or secret manager.

✅ Checks

• My pull request adheres to the code style of this project
• My code requires changes to the documentation
• I have updated the documentation as required
• All the tests have passed

[vercel]

@tuantran0910 is attempting to deploy a commit to the KeepHQ Team on Vercel.

A member of the Team first needs to authorize it.

[tuantran0910]

Hmm, storing the hash value when provisioning in local file storage might not be a good way when we have many backend replicas as mentioned in this issue #4398. Solved by use other SecretManager type like k8s :D

[shahargl]

hey @tuantran0910 - although we want to get this functionality, we feel it's a bit over-engineering here. let's sync if see if there is simpler solution for that.

[tuantran0910]

hey @tuantran0910 - although we want to get this functionality, we feel it's a bit over-engineering here. let's sync if see if there is simpler solution for that.

Hi @shahargl, I completely agree with you. We always aim for the simplest solution to our problems. I don’t have much experience before, so I would really appreciate any advice you could share. Thanks! :D