[CI][Crossbow] Use apache/arrow instead of separated repository (e.g. ursacomputing/crossbow)

[kou]

Describe the enhancement requested

We don't have enough GitHub Actions capacity when we develop Crossbow. So using separated repository (and separated CI services such as Azure Pipelines and Travis CI) was reasonable.

But we have enough GitHub Actions capacity recently. Can we reconsider the Crossbow approach?

We may be able to use on.workflow_dispatch and gh wokrflow run --ref as an alternative of the current Crossbow approach.

Some inconvenient points for the current Crossbow approach:

• CI results use a GitHub comment (example) instead of GitHub status checks.
  • It may cause a long history.
• No direct link to artifacts.
  • Crossbow creates a new tag for each job and uses GitHub Release associated with the tag for artifacts. There is no direct link for the associated GitHub Release.
• We can't use Dependabot for dev/tasks/*/*.yml
• Difficult to automate job submission.
  • For example, we want to submit binary artifacts build jobs after source archive build is completed automatically in release process.
  • FYI: We already have secrets.GITHUB_CROSSBOW_TOKEN in apache/arrow GitHub Actions. So we will be able to implement it with the current Crossbow approach.
• ...

Component(s)

Continuous Integration, Developer Tools

[amoeba]

In case it helps, Datafusion just implemented something like this, see https://github.com/apache/datafusion/blob/main/.github/workflows/pr_comment_commands.yml for how they did it.

[kou]

Wow! I didn't know it!

[kszucs]

I have been thinking of the same recently, we could take advantage of additional GHA features though I have some questions:

• What if the apache/arrow GHA capacity decreases? I'm not sure what is our current concurrency limit.
• Are there crossbow jobs running on other CI systems which couldn't be ported?
• What if we need to support non-github hosted architectures? We would solve it using self-hosted runners?

In an ideal setup we could easily create a release by:

1. pressing a button, triggering jobs, uploading artifacts using https://github.com/actions/upload-artifact?
2. notifying the release manager or dev@ that the artifacts are available
3. release manager downloads, signs and uploads the artifacts under the release tag?
4. trigger the post-release tasks (wherever additional credentials are not required or machine tokens could be used)