Class: GitHubPackages Private

Inherits:
Object show all
Includes:
Context, SystemCommand::Mixin, Utils::Output::Mixin
Defined in:
github_packages.rb

Overview

This class is part of a private API. This class may only be used in the Homebrew/brew repository. Third parties should avoid using this class if possible, as it may be removed or changed without warning.

GitHub Packages client.

Defined Under Namespace

Classes: Downloads

Constant Summary collapse

URL_DOMAIN =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

"ghcr.io"
URL_REGEX =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

%r{(?:#{Regexp.escape(URL_PREFIX)}|#{Regexp.escape(DOCKER_PREFIX)})([\w-]+)/([\w-]+)}
VALID_OCI_TAG_REGEX =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Valid OCI tag characters https://github.com/opencontainers/distribution-spec/blob/main/spec.md#workflow-categories

/^[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}$/
TAB_ARCH_TO_PLATFORM_ARCHITECTURE =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Translate Homebrew tab.arch to OCI platform.architecture

T.let(
  {
    "arm64"  => "arm64",
    "x86_64" => "amd64",
  }.freeze,
  T::Hash[String, String],
)
BUILT_ON_OS_TO_PLATFORM_OS =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Translate Homebrew built_on.os to OCI platform.os

T.let(
  {
    "Linux"     => "linux",
    "Macintosh" => "darwin",
  }.freeze,
  T::Hash[String, String],
)

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Utils::Output::Mixin

#issue_reporting_message, #odebug, #odeprecated, #odie, #odisabled, #ofail, #oh1, #oh1_title, #ohai, #ohai_title, #onoe, #opoo, #opoo_once, #opoo_outside_github_actions, #opoo_without_github_actions_annotation, #pretty_cannot_install, #pretty_deprecated, #pretty_disabled, #pretty_duration, #pretty_install_status, #pretty_installed, #pretty_removed, #pretty_uninstalled, #pretty_unmarked, #pretty_upgradable, #pretty_warning

Methods included from SystemCommand::Mixin

#command_sandbox, #system_command, #system_command!

Methods included from Context

current, current=, #debug?, #deferred_environment_expansion?, #quiet?, #verbose?, #with_context

Class Method Details

.downloads(formula_name) ⇒ Downloads?

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

GitHub's package overview includes exact totals and daily chart values. Cache successful snapshots per UTC day for CLI and API generation callers.

Parameters:

Returns:



39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'github_packages.rb', line 39

def self.downloads(formula_name)
  cache = HOMEBREW_CACHE/"github-packages-downloads/#{formula_name}.json"
  if cache.exist?
    begin
      cached = Downloads.from_hash(JSON.parse(cache.read))
      return cached if cached.fetched_at.start_with?(Time.now.utc.strftime("%F"))
    rescue JSON::ParserError, TypeError, ArgumentError
      nil
    end
  end

  url = "https://github.com/Homebrew/homebrew-core/pkgs/container/core%2F" \
        "#{image_formula_name(formula_name).gsub("/", "%2F")}"
  output = Utils::Curl.curl_output("--fail", "--compressed", url)
  return unless output.success?

  total = output.stdout[%r{Total downloads</span>\s*<h3[^>]*title="(\d+)"}, 1]
  chart = output.stdout[%r{aria-label="Downloads for the last 30 days">.*?</svg>}m]
  return if total.nil? || chart.nil?

  bars = chart.scan(/<rect\b[^>]*>/m).flatten
  daily = bars.filter_map do |bar|
    date = bar[/\bdata-date="(\d{4}-\d{2}-\d{2})"/, 1]
    count = bar[/\bdata-merge-count="(\d+)"/, 1]
    next if date.nil? || count.nil?

    [date, count.to_i]
  end.to_h
  return if daily.empty? || daily.length != bars.length

  dates = daily.keys.sort
  downloads = Downloads.new(source: url, all_time: total.to_i, last_30_days: daily.values.sum,
                            start_date: dates.fetch(0), end_date: dates.fetch(-1), fetched_at: Time.now.utc.iso8601)
  cache.dirname.mkpath
  cache.atomic_write(JSON.generate(downloads.serialize))
  downloads
end

.image_formula_name(formula_name) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



182
183
184
185
186
187
188
 
# File 'github_packages.rb', line 182

def self.image_formula_name(formula_name)
  # Invalid docker name characters:
  # - `/` makes sense because we already use it to separate repository/formula.
  # - `x` makes sense because we already use it in `Formulary`.
  formula_name.tr("@", "/")
              .tr("+", "x")
end

.image_version_rebuild(version_rebuild) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



191
192
193
194
195
196
197
 
# File 'github_packages.rb', line 191

def self.image_version_rebuild(version_rebuild)
  unless version_rebuild.match?(VALID_OCI_TAG_REGEX)
    raise ArgumentError, "GitHub Packages versions must match #{VALID_OCI_TAG_REGEX.source}!"
  end

  version_rebuild
end

.repo_without_prefix(repo) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



158
159
160
161
 
# File 'github_packages.rb', line 158

def self.repo_without_prefix(repo)
  # Remove redundant repository prefix for a shorter name.
  repo.delete_prefix("homebrew-")
end

.root_url(org, repo, prefix = URL_PREFIX) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



164
165
166
167
168
169
 
# File 'github_packages.rb', line 164

def self.root_url(org, repo, prefix = URL_PREFIX)
  # `docker`/`skopeo` insist on lowercase organisation (“repository name”).
  org = org.downcase

  "#{prefix}#{org}/#{repo_without_prefix(repo)}"
end

.root_url_if_match(url) ⇒ String?

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



172
173
174
175
176
177
178
179
 
# File 'github_packages.rb', line 172

def self.root_url_if_match(url)
  return if url.blank?

  _, org, repo, = *url.to_s.match(URL_REGEX)
  return if org.blank? || repo.blank?

  root_url(org, repo)
end

.version_rebuild(version, rebuild, bottle_tag = nil) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



143
144
145
146
147
148
149
150
151
152
153
154
155
 
# File 'github_packages.rb', line 143

def self.version_rebuild(version, rebuild, bottle_tag = nil)
  bottle_tag = (".#{bottle_tag}" if bottle_tag.present?)

  rebuild = if rebuild.positive?
    if bottle_tag
      ".#{rebuild}"
    else
      "-#{rebuild}"
    end
  end

  "#{version}#{bottle_tag}#{rebuild}"
end

Instance Method Details

#upload_bottle(user, token, skopeo, formula_full_name, bottle_hash, keep_old:, dry_run:, warn_on_error:) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:



205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
 
# File 'github_packages.rb', line 205

def upload_bottle(user, token, skopeo, formula_full_name, bottle_hash, keep_old:, dry_run:, warn_on_error:)
  # We run the preupload check twice to prevent TOCTOU bugs.
  result = preupload_check(user, token, skopeo, formula_full_name, bottle_hash,
                           keep_old:, dry_run:, warn_on_error:)
  # Skip upload if preupload check returned early.
  return if result.nil?

  formula_name, org, repo, version, rebuild, version_rebuild, image_name, image_uri, keep_old = *result

  root = Pathname("#{formula_name}--#{version_rebuild}")
  FileUtils.rm_rf root
  root.mkpath

  if keep_old
    download(user, token, skopeo, image_uri, root, dry_run:)
  else
    write_image_layout(root)
  end

  blobs = root/"blobs/sha256"
  blobs.mkpath

  git_path = bottle_hash["formula"]["tap_git_path"]
  git_revision = bottle_hash["formula"]["tap_git_revision"]

  source_org_repo = "#{org}/#{repo}"
  source = "https://github.com/#{source_org_repo}/blob/#{git_revision.presence || "HEAD"}/#{git_path}"

  formula_core_tap = formula_full_name.exclude?("/")
  documentation = if formula_core_tap
    "https://formulae.brew.sh/formula/#{formula_name}"
  elsif (remote = bottle_hash["formula"]["tap_git_remote"]) && remote.start_with?("https://github.com/")
    remote
  end

  license = bottle_hash["formula"]["license"].to_s
  created_date = bottle_hash["bottle"]["date"]
  if keep_old
    index = JSON.parse((root/"index.json").read)
    image_index_sha256 = index["manifests"].first["digest"].delete_prefix("sha256:")
    image_index = JSON.parse((blobs/image_index_sha256).read)
    (blobs/image_index_sha256).unlink

    formula_annotations_hash = image_index["annotations"]
    manifests = image_index["manifests"]
  else
    require "utils/spdx"
    image_license = SPDX.truncate_license(license)

    formula_annotations_hash = {
      "com.github.package.type"                => GITHUB_PACKAGE_TYPE,
      "org.opencontainers.image.created"       => created_date,
      "org.opencontainers.image.description"   => bottle_hash["formula"]["desc"],
      "org.opencontainers.image.documentation" => documentation,
      "org.opencontainers.image.licenses"      => image_license,
      "org.opencontainers.image.ref.name"      => version_rebuild,
      "org.opencontainers.image.revision"      => git_revision,
      "org.opencontainers.image.source"        => source,
      "org.opencontainers.image.title"         => formula_full_name,
      "org.opencontainers.image.url"           => bottle_hash["formula"]["homepage"],
      "org.opencontainers.image.vendor"        => org,
      "org.opencontainers.image.version"       => version.to_s, # Schema accepts strings for version
    }.compact_blank
    manifests = []
  end

  processed_image_refs = Set.new
  manifests.each do |manifest|
    processed_image_refs << manifest["annotations"]["org.opencontainers.image.ref.name"]
  end

  require "sbom"

  manifests += bottle_hash["bottle"]["tags"].map do |bottle_tag, tag_hash|
    bottle_tag = Utils::Bottles::Tag.from_symbol(bottle_tag.to_sym)
    all_bottle = bottle_tag.to_sym == :all

    tag = GitHubPackages.version_rebuild(version, rebuild, bottle_tag.to_s)

    if processed_image_refs.include?(tag)
      puts
      odie "A bottle JSON for #{bottle_tag} is present, but it is already in the image index!"
    else
      processed_image_refs << tag
    end

    local_file = tag_hash["local_filename"]
    odebug "Uploading #{local_file}"

    tar_gz_sha256 = write_tar_gz(local_file, blobs)

    tab = tag_hash["tab"]
    architecture = TAB_ARCH_TO_PLATFORM_ARCHITECTURE[tab["arch"].presence || bottle_tag.standardized_arch.to_s]
    raise TypeError, "unknown tab['arch']: #{tab["arch"]}" if architecture.blank?

    os = if tab["built_on"].present? && tab["built_on"]["os"].present?
      BUILT_ON_OS_TO_PLATFORM_OS[tab["built_on"]["os"]]
    elsif bottle_tag.linux?
      "linux"
    else
      "darwin"
    end
    raise TypeError, "unknown tab['built_on']['os']: #{tab["built_on"]["os"]}" if os.blank?

    os_version = tab["built_on"]["os_version"].presence if tab["built_on"].present?
    case os
    when "darwin"
      os_version ||= "macOS #{bottle_tag.to_macos_version}"
    when "linux"
      os_version&.delete_suffix!(" LTS")
      os_version ||= OS::LINUX_CI_OS_VERSION
      glibc_version = tab["built_on"]["glibc_version"].presence if tab["built_on"].present?
      glibc_version ||= OS::LINUX_GLIBC_CI_VERSION
      cpu_variant = tab.dig("built_on", "oldest_cpu_family") || Hardware::CPU::INTEL_64BIT_OLDEST_CPU.to_s
    end

    platform_hash = {
      architecture:,
      os:,
      "os.version" => os_version,
    }.compact_blank

    tar_sha256 = Digest::SHA256.new
    Zlib::GzipReader.open(local_file) do |gz|
      while (data = gz.read(Utils::Gzip::GZIP_BUFFER_SIZE))
        tar_sha256 << data
      end
    end

    config_json_sha256, config_json_size = write_image_config(platform_hash, tar_sha256.hexdigest, blobs)

    documentation = "https://formulae.brew.sh/formula/#{formula_name}" if formula_core_tap

    local_file_size = File.size(local_file)

    path_exec_files_string = if (path_exec_files = tag_hash["path_exec_files"].presence)
      path_exec_files.join(",")
    end
    sbom_supplement_annotation = SBOM.github_packages_sbom_supplement_annotation(
      tag_hash["sbom"],
      formula_full_name:,
      formula_name:,
      version:,
      tar_gz_sha256:,
      root_url:          bottle_hash["bottle"]["root_url"],
      license:,
      created_date:,
    )

    descriptor_annotations_hash = {
      "org.opencontainers.image.ref.name" => tag,
      "sh.brew.bottle.cpu.variant"        => cpu_variant,
      "sh.brew.bottle.digest"             => tar_gz_sha256,
      "sh.brew.bottle.glibc.version"      => glibc_version,
      "sh.brew.bottle.size"               => local_file_size.to_s,
      "sh.brew.bottle.installed_size"     => tag_hash["installed_size"].to_s,
      "sh.brew.license"                   => license,
      "sh.brew.tab"                       => (all_bottle ? tab.except("arch", "built_on") : tab).to_json,
      "sh.brew.sbom.supplement"           => sbom_supplement_annotation,
      "sh.brew.path_exec_files"           => path_exec_files_string,
    }.compact_blank

    # TODO: upload/add tag_hash["all_files"] somewhere.

    annotations_hash = formula_annotations_hash.merge(descriptor_annotations_hash).merge(
      {
        "org.opencontainers.image.created"       => created_date,
        "org.opencontainers.image.documentation" => documentation,
        "org.opencontainers.image.title"         => "#{formula_full_name} #{tag}",
      },
    ).compact_blank.sort.to_h

    image_manifest = {
      schemaVersion: 2,
      config:        {
        mediaType: "application/vnd.oci.image.config.v1+json",
        digest:    "sha256:#{config_json_sha256}",
        size:      config_json_size,
      },
      layers:        [{
        mediaType:   "application/vnd.oci.image.layer.v1.tar+gzip",
        digest:      "sha256:#{tar_gz_sha256}",
        size:        File.size(local_file),
        annotations: {
          "org.opencontainers.image.title" => local_file,
        },
      }],
      annotations:   annotations_hash,
    }
    validate_schema!(IMAGE_MANIFEST_SCHEMA_URI, image_manifest)
    manifest_json_sha256, manifest_json_size = write_hash(blobs, image_manifest)

    {
      mediaType:   "application/vnd.oci.image.manifest.v1+json",
      digest:      "sha256:#{manifest_json_sha256}",
      size:        manifest_json_size,
      platform:    all_bottle ? nil : platform_hash,
      annotations: descriptor_annotations_hash,
    }.compact
  end

  index_json_sha256, index_json_size = write_image_index(manifests, blobs, formula_annotations_hash)
  raise "Image index too large!" if index_json_size >= 4 * 1024 * 1024 # GitHub will error 500 if too large

  write_index_json(index_json_sha256, index_json_size, root,
                   "org.opencontainers.image.ref.name" => version_rebuild)

  puts
  args = ["copy", "--retry-times=3", "--format=oci", "--all", "oci:#{root}", image_uri.to_s]
  if dry_run
    puts "#{skopeo} #{args.join(" ")} --dest-creds=#{user}:$HOMEBREW_GITHUB_PACKAGES_TOKEN"
  else
    args << "--dest-creds=#{user}:#{token}"
    retry_count = 0
    begin
      system_command!(skopeo, verbose: true, print_stdout: true, args:)
    rescue ErrorDuringExecution
      retry_count += 1
      odie "Cannot perform an upload to registry after retrying multiple times!" if retry_count >= 10
      Utils.exponential_backoff_sleep(retry_count)
      retry
    end

    package_name = "#{GitHubPackages.repo_without_prefix(repo)}/#{image_name}"
    ohai "Uploaded to https://github.com/orgs/#{org}/packages/container/package/#{package_name}"
  end
end

#upload_bottles(bottles_hash, keep_old:, dry_run:, warn_on_error:) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:

  • bottles_hash (Hash{String => T.untyped})
  • keep_old (Boolean)
  • dry_run (Boolean)
  • warn_on_error (Boolean)

Raises:



109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
 
# File 'github_packages.rb', line 109

def upload_bottles(bottles_hash, keep_old:, dry_run:, warn_on_error:)
  user = Homebrew::EnvConfig.github_packages_user
  token = Homebrew::EnvConfig.github_packages_token

  raise UsageError, "HOMEBREW_GITHUB_PACKAGES_USER is unset." if user.blank?
  raise UsageError, "HOMEBREW_GITHUB_PACKAGES_TOKEN is unset." if token.blank?

  skopeo = Utils::Executable.ensure!("skopeo", reason: "upload")

  require "json_schemer"

  load_schemas!

  bottles_hash.each do |formula_full_name, bottle_hash|
    # First, check that we won't encounter an error in the middle of uploading bottles.
    preupload_check(user, token, skopeo, formula_full_name, bottle_hash,
                    keep_old:, dry_run:, warn_on_error:)
  end

  # We intentionally iterate over `bottles_hash` twice to
  # avoid erroring out in the middle of uploading bottles.
  bottle_count = bottles_hash.count
  bottles_hash.each_with_index do |(formula_full_name, bottle_hash), index|
    # Next, upload the bottles after checking them all.
    upload_bottle(user, token, skopeo, formula_full_name, bottle_hash,
                  keep_old:, dry_run:, warn_on_error:)
    next if bottle_count < 3

    uploaded_count = index + 1
    ohai "Upload progress: #{uploaded_count} formula(e) uploaded, #{bottle_count - uploaded_count} remaining"
  end
end