Class: GitHubPackages Private

Inherits:
Object show all
Includes:
Context, SystemCommand::Mixin, Utils::Output::Mixin
Defined in:
github_packages.rb

Overview

This class is part of a private API. This class may only be used in the Homebrew/brew repository. Third parties should avoid using this class if possible, as it may be removed or changed without warning.

GitHub Packages client.

Constant Summary collapse

URL_DOMAIN =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

"ghcr.io"
URL_REGEX =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

%r{(?:#{Regexp.escape(URL_PREFIX)}|#{Regexp.escape(DOCKER_PREFIX)})([\w-]+)/([\w-]+)}
VALID_OCI_TAG_REGEX =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Valid OCI tag characters https://github.com/opencontainers/distribution-spec/blob/main/spec.md#workflow-categories

/^[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}$/
TAB_ARCH_TO_PLATFORM_ARCHITECTURE =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Translate Homebrew tab.arch to OCI platform.architecture

T.let(
  {
    "arm64"  => "arm64",
    "x86_64" => "amd64",
  }.freeze,
  T::Hash[String, String],
)
BUILT_ON_OS_TO_PLATFORM_OS =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Translate Homebrew built_on.os to OCI platform.os

T.let(
  {
    "Linux"     => "linux",
    "Macintosh" => "darwin",
  }.freeze,
  T::Hash[String, String],
)

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Utils::Output::Mixin

#issue_reporting_message, #odebug, #odeprecated, #odie, #odisabled, #ofail, #oh1, #oh1_title, #ohai, #ohai_title, #onoe, #opoo, #opoo_outside_github_actions, #opoo_without_github_actions_annotation, #pretty_deprecated, #pretty_disabled, #pretty_duration, #pretty_install_status, #pretty_installed, #pretty_uninstalled, #pretty_unmarked, #pretty_upgradable, #pretty_warning

Methods included from SystemCommand::Mixin

#system_command, #system_command!

Methods included from Context

current, current=, #debug?, #deferred_environment_expansion?, #quiet?, #verbose?, #with_context

Class Method Details

.image_formula_name(formula_name) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



127
128
129
130
131
132
133
 
# File 'github_packages.rb', line 127

def self.image_formula_name(formula_name)
  # Invalid docker name characters:
  # - `/` makes sense because we already use it to separate repository/formula.
  # - `x` makes sense because we already use it in `Formulary`.
  formula_name.tr("@", "/")
              .tr("+", "x")
end

.image_version_rebuild(version_rebuild) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



136
137
138
139
140
141
142
 
# File 'github_packages.rb', line 136

def self.image_version_rebuild(version_rebuild)
  unless version_rebuild.match?(VALID_OCI_TAG_REGEX)
    raise ArgumentError, "GitHub Packages versions must match #{VALID_OCI_TAG_REGEX.source}!"
  end

  version_rebuild
end

.repo_without_prefix(repo) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



103
104
105
106
 
# File 'github_packages.rb', line 103

def self.repo_without_prefix(repo)
  # Remove redundant repository prefix for a shorter name.
  repo.delete_prefix("homebrew-")
end

.root_url(org, repo, prefix = URL_PREFIX) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



109
110
111
112
113
114
 
# File 'github_packages.rb', line 109

def self.root_url(org, repo, prefix = URL_PREFIX)
  # `docker`/`skopeo` insist on lowercase organisation (“repository name”).
  org = org.downcase

  "#{prefix}#{org}/#{repo_without_prefix(repo)}"
end

.root_url_if_match(url) ⇒ String?

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



117
118
119
120
121
122
123
124
 
# File 'github_packages.rb', line 117

def self.root_url_if_match(url)
  return if url.blank?

  _, org, repo, = *url.to_s.match(URL_REGEX)
  return if org.blank? || repo.blank?

  root_url(org, repo)
end

.version_rebuild(version, rebuild, bottle_tag = nil) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'github_packages.rb', line 88

def self.version_rebuild(version, rebuild, bottle_tag = nil)
  bottle_tag = (".#{bottle_tag}" if bottle_tag.present?)

  rebuild = if rebuild.positive?
    if bottle_tag
      ".#{rebuild}"
    else
      "-#{rebuild}"
    end
  end

  "#{version}#{bottle_tag}#{rebuild}"
end

Instance Method Details

#upload_bottle(user, token, skopeo, formula_full_name, bottle_hash, keep_old:, dry_run:, warn_on_error:) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:



150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
 
# File 'github_packages.rb', line 150

def upload_bottle(user, token, skopeo, formula_full_name, bottle_hash, keep_old:, dry_run:, warn_on_error:)
  # We run the preupload check twice to prevent TOCTOU bugs.
  result = preupload_check(user, token, skopeo, formula_full_name, bottle_hash,
                           keep_old:, dry_run:, warn_on_error:)
  # Skip upload if preupload check returned early.
  return if result.nil?

  formula_name, org, repo, version, rebuild, version_rebuild, image_name, image_uri, keep_old = *result

  root = Pathname("#{formula_name}--#{version_rebuild}")
  FileUtils.rm_rf root
  root.mkpath

  if keep_old
    download(user, token, skopeo, image_uri, root, dry_run:)
  else
    write_image_layout(root)
  end

  blobs = root/"blobs/sha256"
  blobs.mkpath

  git_path = bottle_hash["formula"]["tap_git_path"]
  git_revision = bottle_hash["formula"]["tap_git_revision"]

  source_org_repo = "#{org}/#{repo}"
  source = "https://github.com/#{source_org_repo}/blob/#{git_revision.presence || "HEAD"}/#{git_path}"

  formula_core_tap = formula_full_name.exclude?("/")
  documentation = if formula_core_tap
    "https://formulae.brew.sh/formula/#{formula_name}"
  elsif (remote = bottle_hash["formula"]["tap_git_remote"]) && remote.start_with?("https://github.com/")
    remote
  end

  license = bottle_hash["formula"]["license"].to_s
  created_date = bottle_hash["bottle"]["date"]
  if keep_old
    index = JSON.parse((root/"index.json").read)
    image_index_sha256 = index["manifests"].first["digest"].delete_prefix("sha256:")
    image_index = JSON.parse((blobs/image_index_sha256).read)
    (blobs/image_index_sha256).unlink

    formula_annotations_hash = image_index["annotations"]
    manifests = image_index["manifests"]
  else
    require "utils/spdx"
    image_license = SPDX.truncate_license(license)

    formula_annotations_hash = {
      "com.github.package.type"                => GITHUB_PACKAGE_TYPE,
      "org.opencontainers.image.created"       => created_date,
      "org.opencontainers.image.description"   => bottle_hash["formula"]["desc"],
      "org.opencontainers.image.documentation" => documentation,
      "org.opencontainers.image.licenses"      => image_license,
      "org.opencontainers.image.ref.name"      => version_rebuild,
      "org.opencontainers.image.revision"      => git_revision,
      "org.opencontainers.image.source"        => source,
      "org.opencontainers.image.title"         => formula_full_name,
      "org.opencontainers.image.url"           => bottle_hash["formula"]["homepage"],
      "org.opencontainers.image.vendor"        => org,
      "org.opencontainers.image.version"       => version.to_s, # Schema accepts strings for version
    }.compact_blank
    manifests = []
  end

  processed_image_refs = Set.new
  manifests.each do |manifest|
    processed_image_refs << manifest["annotations"]["org.opencontainers.image.ref.name"]
  end

  require "sbom"

  manifests += bottle_hash["bottle"]["tags"].map do |bottle_tag, tag_hash|
    bottle_tag = Utils::Bottles::Tag.from_symbol(bottle_tag.to_sym)
    all_bottle = bottle_tag.to_sym == :all

    tag = GitHubPackages.version_rebuild(version, rebuild, bottle_tag.to_s)

    if processed_image_refs.include?(tag)
      puts
      odie "A bottle JSON for #{bottle_tag} is present, but it is already in the image index!"
    else
      processed_image_refs << tag
    end

    local_file = tag_hash["local_filename"]
    odebug "Uploading #{local_file}"

    tar_gz_sha256 = write_tar_gz(local_file, blobs)

    tab = tag_hash["tab"]
    architecture = TAB_ARCH_TO_PLATFORM_ARCHITECTURE[tab["arch"].presence || bottle_tag.standardized_arch.to_s]
    raise TypeError, "unknown tab['arch']: #{tab["arch"]}" if architecture.blank?

    os = if tab["built_on"].present? && tab["built_on"]["os"].present?
      BUILT_ON_OS_TO_PLATFORM_OS[tab["built_on"]["os"]]
    elsif bottle_tag.linux?
      "linux"
    else
      "darwin"
    end
    raise TypeError, "unknown tab['built_on']['os']: #{tab["built_on"]["os"]}" if os.blank?

    os_version = tab["built_on"]["os_version"].presence if tab["built_on"].present?
    case os
    when "darwin"
      os_version ||= "macOS #{bottle_tag.to_macos_version}"
    when "linux"
      os_version&.delete_suffix!(" LTS")
      os_version ||= OS::LINUX_CI_OS_VERSION
      glibc_version = tab["built_on"]["glibc_version"].presence if tab["built_on"].present?
      glibc_version ||= OS::LINUX_GLIBC_CI_VERSION
      cpu_variant = tab.dig("built_on", "oldest_cpu_family") || Hardware::CPU::INTEL_64BIT_OLDEST_CPU.to_s
    end

    platform_hash = {
      architecture:,
      os:,
      "os.version" => os_version,
    }.compact_blank

    tar_sha256 = Digest::SHA256.new
    Zlib::GzipReader.open(local_file) do |gz|
      while (data = gz.read(Utils::Gzip::GZIP_BUFFER_SIZE))
        tar_sha256 << data
      end
    end

    config_json_sha256, config_json_size = write_image_config(platform_hash, tar_sha256.hexdigest, blobs)

    documentation = "https://formulae.brew.sh/formula/#{formula_name}" if formula_core_tap

    local_file_size = File.size(local_file)

    path_exec_files_string = if (path_exec_files = tag_hash["path_exec_files"].presence)
      path_exec_files.join(",")
    end
    sbom_supplement_annotation = SBOM.github_packages_sbom_supplement_annotation(
      tag_hash["sbom"],
      formula_full_name:,
      formula_name:,
      version:,
      tar_gz_sha256:,
      root_url:          bottle_hash["bottle"]["root_url"],
      license:,
      created_date:,
    )

    descriptor_annotations_hash = {
      "org.opencontainers.image.ref.name" => tag,
      "sh.brew.bottle.cpu.variant"        => cpu_variant,
      "sh.brew.bottle.digest"             => tar_gz_sha256,
      "sh.brew.bottle.glibc.version"      => glibc_version,
      "sh.brew.bottle.size"               => local_file_size.to_s,
      "sh.brew.bottle.installed_size"     => tag_hash["installed_size"].to_s,
      "sh.brew.license"                   => license,
      "sh.brew.tab"                       => (all_bottle ? tab.except("arch", "built_on") : tab).to_json,
      "sh.brew.sbom.supplement"           => sbom_supplement_annotation,
      "sh.brew.path_exec_files"           => path_exec_files_string,
    }.compact_blank

    # TODO: upload/add tag_hash["all_files"] somewhere.

    annotations_hash = formula_annotations_hash.merge(descriptor_annotations_hash).merge(
      {
        "org.opencontainers.image.created"       => created_date,
        "org.opencontainers.image.documentation" => documentation,
        "org.opencontainers.image.title"         => "#{formula_full_name} #{tag}",
      },
    ).compact_blank.sort.to_h

    image_manifest = {
      schemaVersion: 2,
      config:        {
        mediaType: "application/vnd.oci.image.config.v1+json",
        digest:    "sha256:#{config_json_sha256}",
        size:      config_json_size,
      },
      layers:        [{
        mediaType:   "application/vnd.oci.image.layer.v1.tar+gzip",
        digest:      "sha256:#{tar_gz_sha256}",
        size:        File.size(local_file),
        annotations: {
          "org.opencontainers.image.title" => local_file,
        },
      }],
      annotations:   annotations_hash,
    }
    validate_schema!(IMAGE_MANIFEST_SCHEMA_URI, image_manifest)
    manifest_json_sha256, manifest_json_size = write_hash(blobs, image_manifest)

    {
      mediaType:   "application/vnd.oci.image.manifest.v1+json",
      digest:      "sha256:#{manifest_json_sha256}",
      size:        manifest_json_size,
      platform:    all_bottle ? nil : platform_hash,
      annotations: descriptor_annotations_hash,
    }.compact
  end

  index_json_sha256, index_json_size = write_image_index(manifests, blobs, formula_annotations_hash)
  raise "Image index too large!" if index_json_size >= 4 * 1024 * 1024 # GitHub will error 500 if too large

  write_index_json(index_json_sha256, index_json_size, root,
                   "org.opencontainers.image.ref.name" => version_rebuild)

  puts
  args = ["copy", "--retry-times=3", "--format=oci", "--all", "oci:#{root}", image_uri.to_s]
  if dry_run
    puts "#{skopeo} #{args.join(" ")} --dest-creds=#{user}:$HOMEBREW_GITHUB_PACKAGES_TOKEN"
  else
    args << "--dest-creds=#{user}:#{token}"
    retry_count = 0
    begin
      system_command!(skopeo, verbose: true, print_stdout: true, args:)
    rescue ErrorDuringExecution
      retry_count += 1
      odie "Cannot perform an upload to registry after retrying multiple times!" if retry_count >= 10
      sleep 2 ** retry_count
      retry
    end

    package_name = "#{GitHubPackages.repo_without_prefix(repo)}/#{image_name}"
    ohai "Uploaded to https://github.com/orgs/#{org}/packages/container/package/#{package_name}"
  end
end

#upload_bottles(bottles_hash, keep_old:, dry_run:, warn_on_error:) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:

  • bottles_hash (Hash{String => T.untyped})
  • keep_old (Boolean)
  • dry_run (Boolean)
  • warn_on_error (Boolean)

Raises:



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'github_packages.rb', line 57

def upload_bottles(bottles_hash, keep_old:, dry_run:, warn_on_error:)
  user = Homebrew::EnvConfig.github_packages_user
  token = Homebrew::EnvConfig.github_packages_token

  raise UsageError, "HOMEBREW_GITHUB_PACKAGES_USER is unset." if user.blank?
  raise UsageError, "HOMEBREW_GITHUB_PACKAGES_TOKEN is unset." if token.blank?

  skopeo = ensure_executable!("skopeo", reason: "upload")

  require "json_schemer"

  load_schemas!

  bottles_hash.each do |formula_full_name, bottle_hash|
    # First, check that we won't encounter an error in the middle of uploading bottles.
    preupload_check(user, token, skopeo, formula_full_name, bottle_hash,
                    keep_old:, dry_run:, warn_on_error:)
  end

  # We intentionally iterate over `bottles_hash` twice to
  # avoid erroring out in the middle of uploading bottles.
  # rubocop:disable Style/CombinableLoops
  bottles_hash.each do |formula_full_name, bottle_hash|
    # Next, upload the bottles after checking them all.
    upload_bottle(user, token, skopeo, formula_full_name, bottle_hash,
                  keep_old:, dry_run:, warn_on_error:)
  end
  # rubocop:enable Style/CombinableLoops
end