Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure Stack Support #5532

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Azure Stack Support #5532

wants to merge 10 commits into from

Conversation

patrickdillon
Copy link

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR adds infrastructure provisioning support on Azure Stack. Currently Azure Stack is completely unsupported, but with the changes in this PR I was able to fully provision an OpenShift cluster.

This PR adds a new field armEndpoint to the cluster spec, and extends azureEnvironment to accept a new value, HybridEnvironment, to indicate installation to Azure Stack:

armEndpoint: https://management.ashRegion.ashInstance.com
azureEnvironment: HybridEnvironment

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
#5201

Special notes for your reviewer:

This is a large PR, which I know is not preferred, but I have laid out the commits logically and with messages so they should be easy to follow in that manner. I would be happy to break it up into smaller PRs if that would help.

Furthermore, there were some significant challenges in this implementation. Particularly: I could not get tag reconciliation using the tagging service to work: an inscrutable 500 error was returned. Therefore, 88fc6ea skips adding the tagging service for azure stack. Perhaps I should do the same for MachinePool?

I was pretty satisfied with how other challenges were addressed, but definitely happy to discuss them. Thanks!

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests
  • cherry-pick candidate

Release note:

Adds support for installing to Azure Stack environments. Users can specify `cloudEnvironment: HybridEnvironment` and specify `armEndpoint` in the cluster spec.

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Mar 31, 2025
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 31, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sbueringer for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Contributor

Welcome @patrickdillon!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-azure 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-azure has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

Hi @patrickdillon. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 31, 2025
@k8s-ci-robot k8s-ci-robot requested review from Jont828 and nojnhuh March 31, 2025 18:46
@damdo
Copy link
Member

damdo commented Apr 1, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 1, 2025
@patrickdillon
Add Azure Stack as a valid environment
59750a5 
Adds AzureStack as a valid cloud environment. The value
"HybridEnvironment" is the value provided by the Azure autorest
package. It would be possible to have a different user-facing
value, such as AzureStackCloud, but internally within the code
it is necessary to check for the value "HybridEnvironment"
returned by autorest. This commit opts to use a single value,
rather than separate user-facing and internal values.
@patrickdillon
Add ARMEndpoint to Cluster Class
11f3e63 
Adds the ARMEndpoint field for specifying the ARM Resource Manager
Endpoint for use with Azure Stack deployments. The endpoint is
used to configure the environment as well as manage resources.
@patrickdillon
Populate AzureClients & Authorizer for Azure Stack
bc9c9ea 
Uses ARMEndpoint from Cluster scope to configure Azure Stack settings
for Azure Client and Authorizer, which will be used to configure
ARM options for the V2 SDK.
@patrickdillon
Set ARMClientOptions for Azure Stack
69a52d6 
Sets ARM Client Options when using the Azure Stack environment. Extends
ARMClientOptions to accept an ARMEndpoint, which can be obtained from
the authorizer interface, the same source the cloud environment.

Sets the APIVersion to a hybrid cloud profile to ensure compatibility
with hybrid environments.
@patrickdillon
Generate CRDs for AzureStack
db3664e
@patrickdillon
Skip privatedns zones on Azure Stack
271a656 
Azure Stack Hub does not support private dns zones, so skip them.
@patrickdillon
AzureStack: handle missing avail set sku cache
b64d4e6 
The Resource SKU API for availability sets may not be available in
an Azure Stack environment. The cache is used to determine the
fault domain count. For Azure Stack, we can default to 2. Future
work could potentially set this programatically or expose the
fault domain count in the API.
@patrickdillon
Skip tag reconciliation in Azure Stack
36ca4a4 
The tag service using the V2 SDK is not available in azure stack.
Skip tag reconciliation in Azure Stack environments.
@patrickdillon
Change Disk Client API Version for Azure Stack
bdbe2bc 
The standard 2020-06-01 API Version is not supported for disk
operations in Azure Stack, so change to the compatible 2018-06-01
profile.
@patrickdillon
Retry VM Delete without Force if Bad Request
4605c9e 
Azure Stack returns a 400 error when trying to delete a VM with
the force flag and the error message suggests retrying without
the flag.
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 5, 2025
@patrickdillon
Copy link
Author

Don't think anybody has looked at this yet so I went ahead and force pushed to rebase, and fixed the unit test failure (due to newly wrappd error).

@patrickdillon patrickdillon mentioned this pull request Apr 5, 2025
Merged
bryan-cox
bryan-cox reviewed Apr 7, 2025
View reviewed changes
api/v1beta1/types_class.go
@@ -48,6 +48,7 @@ type AzureClusterClassSpec struct {
// - GermanCloud: "AzureGermanCloud"
// - PublicCloud: "AzurePublicCloud"
// - USGovernmentCloud: "AzureUSGovernmentCloud"
// - StackCloud: "HybridEnvironment"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is Azure Stack officially supported? Autorest is a deprecated library. I don't see Azure Stack listed in the new Cloud library - https://github.com/Azure/azure-sdk-for-go/blob/d165f8083de58e12135bce2102205081eaade930/sdk/azcore/cloud/cloud.go#L10.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, the GermanCloud isn't supported anymore.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know there is an issue too to try and get rid of the autorest library implementations in CAPZ - #2974

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know there is an issue too to try and get rid of the autorest library implementations in CAPZ

Right, I discussed this a little in the issue #5201. AFAIK there is no available alternative.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea whether it is still supported, although I'm pretty sure it is still sold.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2025
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@patrickdillon patrickdillon mentioned this pull request Apr 8, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bryan-cox bryan-cox bryan-cox left review comments

@Jont828 Jont828 Awaiting requested review from Jont828

@nojnhuh nojnhuh Awaiting requested review from nojnhuh

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
CAPZ Planning
Status: Todo
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@patrickdillon @k8s-ci-robot @damdo @bryan-cox