Module: PyPI Private

Extended by:
Utils::Output::Mixin
Defined in:
utils/pypi.rb

Overview

This module is part of a private API. This module may only be used in the Homebrew/brew repository. Third parties should avoid using this module if possible, as it may be removed or changed without warning.

Helper functions for updating PyPI resources.

Defined Under Namespace

Classes: Package

Class Method Summary collapse

Methods included from Utils::Output::Mixin

issue_reporting_message, odebug, odeprecated, odie, odisabled, ofail, oh1, oh1_title, ohai, ohai_title, onoe, opoo, opoo_outside_github_actions, opoo_without_github_actions_annotation, pretty_cannot_install, pretty_deprecated, pretty_disabled, pretty_duration, pretty_install_status, pretty_installed, pretty_uninstalled, pretty_unmarked, pretty_upgradable, pretty_warning

Class Method Details

.format_pip_command(command) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Render source resources as URLs in pip command diagnostics.

Parameters:

Returns:



499
500
501
 
# File 'utils/pypi.rb', line 499

def self.format_pip_command(command)
  command.map { |argument| argument.is_a?(Resource) ? argument.url : argument }.join(" ")
end

.normalize_python_package(name) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



491
492
493
494
495
 
# File 'utils/pypi.rb', line 491

def self.normalize_python_package(name)
  # This normalization is defined in the PyPA packaging specifications;
  # https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization
  name.gsub(/[-_.]+/, "-").downcase
end

.pip_output(command, print_stderr: false) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
 
# File 'utils/pypi.rb', line 504

def self.pip_output(command, print_stderr: false)
  Sandbox.ensure_sandbox_available!
  if Sandbox.avoid_nested_sandboxing?
    raise "Python metadata inspection needs Homebrew's sandbox, which cannot run inside another sandbox."
  end

  unless Sandbox.full_write_isolation?
    opoo <<~EOS
      The sandbox cannot restrict file permissions or ownership.
      Python metadata inspection uses the available sandbox protections.
    EOS
  end

  Dir.mktmpdir("homebrew-pypi", HOMEBREW_TEMP) do |directory|
    command = command.each_with_index.map do |argument, index|
      next argument unless argument.is_a?(Resource)

      argument.fetch(quiet: !print_stderr, skip_patches: true)
      source = Pathname(directory)/"source-#{index}"
      source.mkpath
      source.cd { argument.downloader.stage { source = Pathname.pwd } }
      source
    end

    sandbox = Sandbox.new
    sandbox.allow_write_path(directory)
    sandbox.deny_write_homebrew_repository
    sandbox.deny_read_home
    Tempfile.create("report", directory) do |report|
      sandbox_env = ENV.to_h.filter_map do |key, value|
        "#{key}=#{value}" if key.match?(/\A(?:HOMEBREW_(?:LIBRARY|PREFIX|GIT)|(?:https?|all|no)_proxy)\z/i)
      end
      sandbox.run "/usr/bin/env", "-i", "PATH=#{ENV.fetch("PATH")}", *sandbox_env, "HOME=#{directory}",
                  "TMPDIR=#{directory}", "PIP_CACHE_DIR=#{directory}/cache", "PIP_CONFIG_FILE=#{File::NULL}",
                  "PIP_REQUIRE_VIRTUALENV=false", "/bin/sh", "-c",
                  "report=$1; shift; exec \"$@\" > \"$report\"#{" 2>/dev/null" unless print_stderr}",
                  "brew-pypi", report.path, *command, passthrough_stdin: false
      report.rewind
      report.read
    end
  end
end

.pip_report(packages, python_name: "python", print_stderr: false, ignore_cooldown_package: nil) ⇒ Array<Package>

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

  • packages (Array<Package>)
  • python_name (String) (defaults to: "python")
  • print_stderr (Boolean) (defaults to: false)
  • ignore_cooldown_package (Package, nil) (defaults to: nil)

Returns:



553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
 
# File 'utils/pypi.rb', line 553

def self.pip_report(packages, python_name: "python", print_stderr: false, ignore_cooldown_package: nil)
  return [] if packages.blank?

  # Delay packages published in the last day so resource resolution is less
  # likely to pick a freshly compromised PyPI release. A cooldown-exempt main
  # package (third-party taps only) is passed by its direct sdist URL so pip's
  # index upload-time filter cannot hide a just-published release; its
  # dependencies stay index-resolved and cooled.
  requirements = packages.map do |package|
    exempt = ignore_cooldown_package && package == ignore_cooldown_package && package.valid_pypi_package?
    next package.requirement unless exempt

    name, sdist_url = package.pypi_info
    next package.to_s if sdist_url.blank?

    # PEP 508 direct reference. Any extras are preserved so their dependencies
    # still resolve, while the URL bypasses the index upload-time filter.
    extras = package.extras.presence
    next sdist_url unless extras

    "#{name}[#{extras.join(",")}] @ #{sdist_url}"
  end

  command = [
    Utils::Path.formula_opt_libexec(python_name)/"bin/python",
    "-m", "pip", "install", "-q", "--disable-pip-version-check",
    "--dry-run", "--ignore-installed",
    "--uploaded-prior-to=P#{Homebrew::RELEASE_COOLDOWN_DAYS}D",
    "--report=/dev/stdout", *requirements
  ]
  pip_output = begin
    PyPI.pip_output(command, print_stderr:)
  rescue ErrorDuringExecution
    odie <<~EOS
      Unable to determine dependencies for "#{packages.join(" ")}" because of a failure when running
      `#{format_pip_command(command)}`.
      Please update the resources manually.
    EOS
  end
  pip_report_to_packages(JSON.parse(pip_output)).uniq
end

.pip_report_to_packages(report) ⇒ Array<Package>

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



596
597
598
599
600
601
602
603
604
605
 
# File 'utils/pypi.rb', line 596

def self.pip_report_to_packages(report)
  return [] if report.blank?

  report["install"].filter_map do |package|
    name = normalize_python_package(package["metadata"]["name"])
    version = package["metadata"]["version"]

    Package.new "#{name}==#{version}"
  end
end

.resource_blocks_from_formula(contents) ⇒ Hash{String => String}

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
 
# File 'utils/pypi.rb', line 471

def self.resource_blocks_from_formula(contents)
  blocks = {}
  _processed_source, root_node = Utils::AST.process_source(contents)
  return blocks if root_node.nil?

  root_node.each_node(:block) do |node|
    next unless Utils::AST.call_node_match?(node, name: :resource, type: :block_call)

    send_node = node.send_node
    name_node = send_node.arguments.first
    next if name_node.blank? || !name_node.str_type?

    resource_name = name_node.str_content
    blocks[resource_name] = node.location.expression.source
  end

  blocks
end

.update_pypi_url(url, version) ⇒ String?

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



220
221
222
223
224
225
226
227
228
229
 
# File 'utils/pypi.rb', line 220

def self.update_pypi_url(url, version)
  package = Package.new url, is_url: true

  return unless package.valid_pypi_package?

  _, url = package.pypi_info(new_version: version)
  url
rescue ArgumentError
  nil
end

.update_python_resources!(formula, version: nil, package_name: nil, extra_packages: nil, exclude_packages: nil, dependencies: nil, install_dependencies: false, print_only: false, quiet: false, verbose: false, ignore_errors: false, ignore_non_pypi_packages: false, ignore_main_package_cooldown: false) ⇒ Boolean

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Return true if resources were checked (even if no change).

Parameters:

  • formula (Formula)
  • version (String, nil) (defaults to: nil)
  • package_name (String, nil) (defaults to: nil)
  • extra_packages (Array<String>, nil) (defaults to: nil)
  • exclude_packages (Array<String>, nil) (defaults to: nil)
  • dependencies (Array<String>, nil) (defaults to: nil)
  • install_dependencies (Boolean) (defaults to: false)
  • print_only (Boolean) (defaults to: false)
  • quiet (Boolean) (defaults to: false)
  • verbose (Boolean) (defaults to: false)
  • ignore_errors (Boolean) (defaults to: false)
  • ignore_non_pypi_packages (Boolean) (defaults to: false)
  • ignore_main_package_cooldown (Boolean) (defaults to: false)

Returns:

  • (Boolean) 


249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
 
# File 'utils/pypi.rb', line 249

def self.update_python_resources!(formula, version: nil, package_name: nil, extra_packages: nil,
                                  exclude_packages: nil, dependencies: nil, install_dependencies: false,
                                  print_only: false, quiet: false, verbose: false,
                                  ignore_errors: false, ignore_non_pypi_packages: false,
                                  ignore_main_package_cooldown: false)
  if [package_name, extra_packages, exclude_packages, dependencies].all?(&:blank?)
    list_entry = formula.pypi_packages_info

    package_name = list_entry.package_name
    extra_packages = list_entry.extra_packages
    exclude_packages = list_entry.exclude_packages
    dependencies = list_entry.dependencies
  end

  missing_dependencies = Array(dependencies).reject do |dependency|
    Formula[dependency].any_version_installed?
  rescue FormulaUnavailableError
    odie "Formula \"#{dependency}\" not found but it is a dependency to update \"#{formula.name}\" resources."
  end
  if missing_dependencies.present?
    missing_msg = "formulae required to update \"#{formula.name}\" resources: #{missing_dependencies.join(", ")}"
    odie "Missing #{missing_msg}" unless install_dependencies
    ohai "Installing #{missing_msg}"
    require "formula"
    missing_dependencies.each { |dep| Formula[dep].ensure_installed! }
  end

  python_deps = formula.deps
                       .select { |d| d.name.match?(/^python(@.+)?$/) }
                       .map(&:to_formula)
                       .sort_by(&:version)
                       .reverse
  python_name = if python_deps.empty?
    "python"
  else
    (python_deps.find(&:any_version_installed?) || python_deps.first).name
  end

  main_package = if package_name.present?
    package_string = package_name
    package_string += "==#{formula.version}" if version.blank? && formula.version.present?
    Package.new(package_string, python_name:)
  elsif package_name == ""
    nil
  else
    stable = formula.stable
    stable_url = stable&.url
    if stable.nil? || stable_url.nil?
      odie "#{formula.full_name} has no stable URL to determine the main Python package from."
    end

    Package.new(stable_url, is_url: true, python_name:,
                resource: (stable.resource if stable.downloader.is_a?(VCSDownloadStrategy)))
  end

  if main_package.nil?
    odie "The main package was skipped but no PyPI `extra_packages` were provided." if extra_packages.blank?
  elsif version.present?
    if main_package.valid_pypi_package?
      main_package.version = version
    else
      return false if ignore_non_pypi_packages

      odie "The main package is not a PyPI package, meaning that version-only updates cannot be \
        performed. Please update its URL manually."
    end
  end

  extra_packages = (extra_packages || []).map { |p| Package.new p }
  exclude_packages = (exclude_packages || []).map { |p| Package.new p }
  exclude_packages += %w[argparse pip wsgiref].map { |p| Package.new p }
  if (newest_python = python_deps.first) && newest_python.version < Version.new("3.12")
    exclude_packages.append(Package.new("setuptools"))
  end
  # remove packages from the exclude list if we've explicitly requested them as an extra package
  exclude_packages.delete_if { |package| extra_packages.include?(package) }

  input_packages = Array(main_package)
  extra_packages.each do |extra_package|
    if !extra_package.valid_pypi_package? && !ignore_non_pypi_packages
      odie "\"#{extra_package}\" is not available on PyPI."
    end

    input_packages.each do |existing_package|
      if existing_package.same_package?(extra_package) && existing_package.version != extra_package.version
        odie "Conflicting versions specified for the `#{extra_package.name}` package: " \
             "#{existing_package.version}, #{extra_package.version}"
      end
    end

    input_packages << extra_package unless input_packages.include? extra_package
  end

  non_pypi_resource_names = formula.resources.filter_map do |resource|
    next if resource.url.start_with?(PYTHONHOSTED_URL_PREFIX)
    next if resource.livecheck_defined?

    resource.name
  end.to_set

  existing_resources_by_name = formula.resources.to_h { |resource| [resource.name, resource] }
  formula_contents = formula.path.read
  existing_resource_blocks = resource_blocks_from_formula(formula_contents)

  require "formula"
  Formula[python_name].ensure_installed!

  # Resolve the dependency tree of all input packages
  show_info = !print_only && !quiet
  ohai "Retrieving PyPI dependencies for \"#{input_packages.join(" ")}\"..." if show_info

  print_stderr = verbose && show_info
  print_stderr ||= false

  ignore_cooldown_package = main_package if ignore_main_package_cooldown
  found_packages = pip_report(input_packages, python_name:, print_stderr:,
                              ignore_cooldown_package:)
  # Resolve the dependency tree of excluded packages to prune the above
  exclude_packages.delete_if { |package| found_packages.exclude? package }
  if exclude_packages.present?
    ohai "Retrieving PyPI dependencies for excluded \"#{exclude_packages.join(" ")}\"..." if show_info
    exclude_packages = pip_report(exclude_packages, python_name:, print_stderr:)
  end
  # Keep extra_packages even if they are dependencies of exclude_packages
  exclude_packages.delete_if { |package| extra_packages.include? package }
  if (main_package_name = main_package&.name)
    exclude_packages += [Package.new(main_package_name)]
  end

  new_resource_blocks = ""
  package_errors = ""
  found_packages.sort.each do |package|
    if exclude_packages.include? package
      ohai "Excluding \"#{package}\"" if show_info
      exclude_packages.delete package
      next
    end
    next if (package_name = package.name) && existing_resources_by_name[package_name]&.livecheck_defined?

    ohai "Getting PyPI info for \"#{package}\"" if show_info
    name, url, checksum, version, package_error = package.pypi_info(ignore_errors: ignore_errors)
    if package_error.blank?
      # Fail if unable to find name, url or checksum for any resource
      if name.blank?
        if ignore_errors
          package_error = "unknown failure"
        else
          odie "Unable to resolve some dependencies. Please update the resources for \"#{formula.name}\" manually."
        end
      elsif url.blank? || checksum.blank?
        if ignore_errors
          package_error = "unable to find URL and/or sha256"
        else
          odie <<~EOS
            Unable to find the URL and/or sha256 for the "#{name}" resource.
            Please update the resources for "#{formula.name}" manually.
          EOS
        end
      else
        existing_is_non_pypi = !non_pypi_resource_names.delete?(name).nil?

        if (existing_resource = existing_resources_by_name[name]) &&
           (existing_block = existing_resource_blocks[name]) &&
           ((existing_resource.url == url && existing_resource.checksum&.hexdigest == checksum) ||
            (existing_is_non_pypi && existing_resource.version.to_s == version))
          new_resource_blocks += <<-EOS
#{existing_block.dup}

          EOS
          next
        end
        # Append indented resource block
        new_resource_blocks += <<-EOS
resource "#{name}" do
  url "#{url}"
  sha256 "#{checksum}"
end

        EOS
      end
    end

    if package_error.present?
      # Leave a placeholder for formula author to investigate
      package_errors += "  # RESOURCE-ERROR: Unable to resolve \"#{package}\" (#{package_error})\n"
    end
  end

  package_errors += "\n" if package_errors.present?
  resource_section = "#{package_errors}#{new_resource_blocks}"

  odie "Excluded superfluous packages: #{exclude_packages.join(", ")}" if exclude_packages.any?

  if print_only
    puts resource_section.chomp
    return true
  end

  odie <<~EOS unless non_pypi_resource_names.empty?
    "#{formula.name}" contains non-PyPI resources: #{non_pypi_resource_names.sort.join(", ")}
    Please update the resources manually.
  EOS

  ohai "Updating resource blocks" unless quiet
  formula_ast = Utils::AST::FormulaAST.new(formula.path.read)
  if formula_ast.replace_resource_stanzas(
    resource_section,
    replace_existing:   formula.resources.any? { |resource| !resource.name.start_with?("homebrew-") },
    preserve_livecheck: true,
  ) == :multiple_groups
    odie "Unable to update resource blocks for \"#{formula.name}\" automatically. Please update them manually."
  end
  formula.path.atomic_write(formula_ast.process)

  if package_errors.present?
    ofail "Unable to resolve some dependencies. Please check #{formula.path} for RESOURCE-ERROR comments."
  end

  true
end