Class: Utils::Analytics::FormulaHistory Private

Inherits:
Object
  • Object
show all
Defined in:
utils/analytics/formula_history.rb

Overview

This class is part of a private API. This class may only be used in the Homebrew/brew repository. Third parties should avoid using this class if possible, as it may be removed or changed without warning.

Incremental stable-version history on core's first-parent branch.

Defined Under Namespace

Classes: Build, Change, Checkpoint, Commit, Package

Constant Summary collapse

METRICS =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

%w[added removed versions major minor patch other revision_bumps bottle_rebuilds unreadable].freeze
VERSION_PARTS =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

%w[major minor patch].freeze
BATCH_SIZE =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

100
MAX_RUN_SECONDS =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

600

Instance Method Summary collapse

Constructor Details

#initialize(path, cache: HOMEBREW_CACHE/"analytics/git/formula-history.json") ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

  • path (Pathname)
  • cache (Pathname) (defaults to: HOMEBREW_CACHE/"analytics/git/formula-history.json") 


57
58
59
60
 
# File 'utils/analytics/formula_history.rb', line 57

def initialize(path, cache: HOMEBREW_CACHE/"analytics/git/formula-history.json")
  @path = path
  @cache = cache
end

Instance Method Details

#apply(state, change, source, timestamp:) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:



206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
 
# File 'utils/analytics/formula_history.rb', line 206

def apply(state, change, source, timestamp:)
  previous = state.packages.delete(change.old_path)
  counts = state.years[Time.at(timestamp).utc.year.to_s] ||= {} if timestamp.positive?
  unless change.regular
    counts["removed"] = counts.fetch("removed", 0) + 1 if counts && previous
    return
  end
  current = if previous && previous.blob == change.blob
    previous.builds
  else
    builds(change.path, source || raise("Missing Git blob #{change.blob}"))
  end
  package = previous || Package.new(blob: change.blob, builds: {})
  observed = current.values.filter_map(&:version).uniq
  events = T.let([], T::Array[String])
  events << "added" unless previous
  changed = current.filter_map do |platform, build|
    old = package.builds[platform]
    next if old.nil?

    old_version = old.version
    version = build.version
    next if old_version.nil? || version.nil?

    if old_version != version
      before = Version.new(old_version)
      after = Version.new(version)
      if after > before
        VERSION_PARTS.find { |part| before.public_send(part) != after.public_send(part) } || "other"
      else
        "other"
      end
    elsif build.revision > old.revision
      "revision_bumps"
    elsif build.revision == old.revision && build.rebuild > old.rebuild
      "bottle_rebuilds"
    end
  end
  if (version_change = (%w[major minor patch other] & changed).first)
    events << version_change
  end
  events.concat(changed & %w[revision_bumps bottle_rebuilds])
  if current.values.any?(&:error)
    events << "unreadable"
    package.unknown_at = [package.unknown_at, timestamp].max
  end
  if events.intersect?(%w[added major minor patch other revision_bumps])
    package.updated_at = [package.updated_at, timestamp].max
  end
  if counts
    events.compact.each { |event| counts[event] = counts.fetch(event, 0) + 1 }
    counts["versions"] = counts.fetch("versions", 0) + (observed - package.versions).length
  end
  package.versions |= observed
  package.blob = change.blob
  package.builds = current
  state.packages[change.path] = package
end

#builds(path, source) ⇒ Hash{String => Build}

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
 
# File 'utils/analytics/formula_history.rb', line 184

def builds(path, source)
  independent = Homebrew::Vulns::PlatformIndependence.source?(source)
  result = T.let({}, T::Hash[String, Build])
  targets.each do |os, arch|
    result["#{os}/#{arch}"] = if independent && (shared = result.values.first)
      shared
    else
      Homebrew::SimulateSystem.with(os:, arch:) do
        versions = FormulaVersions.new(@path/path, repository: @path, simulated_target: true)
        versions.formula_at_revision(Digest::SHA256.hexdigest(source), path, source:) do |formula|
          Build.new(version: formula.stable&.version&.to_s, revision: formula.revision,
                    rebuild: formula.stable&.bottle_specification&.rebuild || 0)
        end || Build.new(version: nil, error: versions.load_error&.class&.name || "Unavailable formula")
      ensure
        Formulary.clear_cache
      end
    end
  end
  result
end

#commits(range) ⇒ Array<Commit>

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
 
# File 'utils/analytics/formula_history.rb', line 266

def commits(range)
  tokens = git("log", "--first-parent", "--reverse", "--raw", "--root", "--no-abbrev",
               "--diff-merges=first-parent", "--find-renames=50%", "--format=%x00commit%x00%H%x00%ct",
               "-z", range).split("\0")
  commits = T.let([], T::Array[Commit])
  while (token = tokens.shift)
    if token == "commit"
      commits << Commit.new(revision: tokens.shift.to_s, timestamp: tokens.shift.to_i)
    elsif token.lstrip.start_with?(":")
      _, mode, _, blob, status = token.split
      path = tokens.shift.to_s
      new_path = status&.start_with?("R") ? tokens.shift.to_s : path
      next if !formula_path?(path) && !formula_path?(new_path)

      commits.fetch(-1).changes << Change.new(
        path: new_path, old_path: path, blob: blob.to_s,
        regular: %w[100644 100755].include?(mode) && formula_path?(new_path)
      )
    end
  end
  commits
end

#formula_path?(path) ⇒ Boolean

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:

  • (Boolean) 


317
318
319
 
# File 'utils/analytics/formula_history.rb', line 317

def formula_path?(path)
  path.match?(%r{\A(?:Library/)?Formula/(?:[^/]+/)?[^/]+\.rb\z})
end

#git(*args) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



328
329
330
 
# File 'utils/analytics/formula_history.rb', line 328

def git(*args)
  Utils.safe_popen_read("git", "-C", @path.to_s, *args).strip
end

#report(start_year: nil, max_seconds: MAX_RUN_SECONDS) ⇒ Hash{String => T.anything}

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

A cold backfill may span runs; only complete reports describe today's catalogue.

Parameters:

  • start_year (Integer, nil) (defaults to: nil)
  • max_seconds (Integer) (defaults to: MAX_RUN_SECONDS)

Returns:



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
 
# File 'utils/analytics/formula_history.rb', line 64

def report(start_year: nil, max_seconds: MAX_RUN_SECONDS)
  if git("rev-parse", "--is-shallow-repository") == "true"
    raise "Formula history requires a complete Git history"
  end

  target = git("rev-parse", "HEAD")
  saved = JSON.parse(@cache.read) if @cache.exist?
  start_year ||= saved&.fetch("start_year", nil) || (Time.now.utc.year - 5)
  signature = Digest::SHA256.hexdigest([
    __FILE__, HOMEBREW_LIBRARY_PATH/"formula_versions.rb", HOMEBREW_LIBRARY_PATH/"version.rb",
    HOMEBREW_LIBRARY_PATH/"resource.rb", HOMEBREW_LIBRARY_PATH/"software_spec.rb",
    HOMEBREW_LIBRARY_PATH/"vulns/platform_independence.rb"
  ].map { |file| Digest::SHA256.file(file).hexdigest }.join + targets.to_s + start_year.to_s)
  if saved && saved["signature"] == signature
    # Sorbet's nested deserialiser resolves Utils against T::Props::Utils.
    saved.fetch("packages").transform_values! do |package|
      package.fetch("builds").transform_values! { |build| Build.from_hash(build) }
      Package.new(**package.transform_keys(&:to_sym))
    end
    state = Checkpoint.new(**saved.transform_keys(&:to_sym))
    begin
      git("merge-base", "--is-ancestor", state.head.presence || state.baseline.presence || target, target)
    rescue ErrorDuringExecution
      state = nil
    end
  end
  state ||= Checkpoint.new(signature:, start_year:, baseline: git(
    "rev-list", "--first-parent", "-1", "--before=#{start_year}-01-01T00:00:00Z", target
  ))
  deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + max_seconds
  Homebrew::API.with_no_api_env do
    unless state.seeded
      entries = state.baseline.empty? ? [] : git("ls-tree", "-rz", state.baseline).split("\0")
      pending = entries.filter_map do |entry|
        metadata, path = entry.split("\t", 2)
        next if path.nil? || !formula_path?(path) || state.packages.key?(path)
        next unless metadata&.start_with?("100644 blob ", "100755 blob ")

        Change.new(path:, old_path: path, blob: metadata.split.fetch(2), regular: true)
      end
      pending.each_slice(BATCH_SIZE) do |changes|
        break if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline

        sources = sources(changes.map(&:blob))
        changes.each { |change| apply(state, change, sources.fetch(change.blob), timestamp: 0) }
        save(state)
      end
      state.seeded = pending.all? { |change| state.packages.key?(change.path) }
      state.head = state.baseline if state.seeded
    end
    if state.seeded && state.head != target
      range = state.head.empty? ? target : "#{state.head}..#{target}"
      git("rev-list", "--first-parent", "--reverse",
          range).lines(chomp: true).each_slice(BATCH_SIZE) do |revisions|
        break if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline

        batch = commits(state.head.empty? ? revisions.fetch(-1) : "#{state.head}..#{revisions.fetch(-1)}")
        sources = sources(batch.flat_map(&:changes).select(&:regular).map(&:blob).uniq)
        batch.each do |commit|
          commit.changes.each do |change|
            apply(state, change, sources[change.blob], timestamp: commit.timestamp)
          end
          state.head = commit.revision
        end
        save(state)
      end
    end
  end
  save(state)
  complete = state.seeded && state.head == target
  cutoff_date = Time.now.utc.to_date << 12
  cutoff = Time.utc(cutoff_date.year, cutoff_date.month, cutoff_date.day).to_i
  processed_date = git("show", "-s", "--format=%cI", state.head) unless state.head.empty?
  {
    "generated_at"           => Time.now.utc.iso8601,
    "source"                 => "https://github.com/Homebrew/homebrew-core",
    "start_year"             => state.start_year,
    "revision"               => state.head.presence,
    "target_revision"        => target,
    "complete"               => complete,
    "processed_through"      => processed_date,
    "platforms"              => targets.map { |os, arch| "#{os}/#{arch}" },
    "maintenance"            => if complete
                                  state.packages.values.map do |package|
                                    if package.updated_at >= cutoff
                                      "updated"
                                    elsif package.unknown_at >= cutoff || package.builds.values.any?(&:error)
                                      "unknown"
                                    elsif package.builds.values.all? { |build| build.version.nil? }
                                      "no_stable_version"
                                    else
                                      "unchanged"
                                    end
                                  end.tally
    end,
    "maintenance_start_date" => Time.at(cutoff).utc.to_date.iso8601,
    "rows"                   => ["current", *Time.now.utc.year.downto(state.start_year)].map do |period|
      period = period.to_s
      METRICS.to_h do |metric|
        count = if period == "current"
          state.years.values.sum { |counts| counts.fetch(metric, 0) }
        else
          state.years.fetch(period, {}).fetch(metric, 0)
        end
        if !complete && (processed_date.nil? || (period != "current" && period.to_i > processed_date.to_i))
          count = nil
        end
        [metric, count]
      end.merge("period" => period)
    end,
  }
end

#save(state) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:



322
323
324
325
 
# File 'utils/analytics/formula_history.rb', line 322

def save(state)
  @cache.dirname.mkpath
  @cache.atomic_write(JSON.generate(state.serialize))
end

#sources(objects) ⇒ Hash{String => String}

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Fetch missing promisor objects together, avoiding one network fetch per old formula.

Parameters:

Returns:



291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
 
# File 'utils/analytics/formula_history.rb', line 291

def sources(objects)
  missing = SystemCommand.run!(
    "git", args: ["-C", @path.to_s, "cat-file", "--batch-check"],
    input: objects.map { |object| "#{object}\n" }, env: { "GIT_NO_LAZY_FETCH" => "1" }
  ).stdout.lines.filter_map { |line| line.split.first if line.end_with?(" missing\n") }
  missing.each_slice(500) do |batch|
    Utils.safe_popen_write("git", "-C", @path.to_s, "-c", "fetch.negotiationAlgorithm=noop",
                           "fetch", "--quiet", "--no-tags", "--no-write-fetch-head", "--filter=blob:none",
                           "--stdin", "origin") { |pipe| pipe.puts(batch) }
  end
  result = T.let({}, T::Hash[String, String])
  Utils.safe_popen_write("git", "-C", @path.to_s, "cat-file", "--batch") do |pipe|
    objects.each do |object|
      pipe.puts(object)
      pipe.flush
      _, type, length = pipe.gets.to_s.split
      raise "Missing Git blob #{object}" if type != "blob"

      result[object] = pipe.read(Integer(length.to_s, 10)).to_s.force_encoding(Encoding::UTF_8)
      pipe.read(1)
    end
  end
  result
end

#targets ⇒ Array<Array<(Symbol, Symbol)>>

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Returns:



178
179
180
181
 
# File 'utils/analytics/formula_history.rb', line 178

def targets
  macos = MacOSVersion.new((HOMEBREW_MACOS_NEWEST_UNSUPPORTED.to_i - 1).to_s).to_sym
  [[macos, :arm], [macos, :intel], [:linux, :arm], [:linux, :intel]]
end