Class: Utils::Analytics::Repository Private

Inherits:
Object
  • Object
show all
Defined in:
utils/analytics/repository.rb

Overview

This class is part of a private API. This class may only be used in the Homebrew/brew repository. Third parties should avoid using this class if possible, as it may be removed or changed without warning.

Repository analytics derived from local Git objects.

Constant Summary collapse

SPLIT_COMMITS =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Inherited commits refer to legacy-homebrew PR numbers before these split commits.

T.let({
  "Homebrew/brew"          => "c5132daf630",
  "Homebrew/homebrew-core" => "c491e31616b4",
}.freeze, T::Hash[String, String])
FIRST_LEGACY_PULL_REQUEST =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.

Legacy issue #2326 (1 September 2010) is the first surviving numbered PR.

2326

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(path, name) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:



24
25
26
27
28
 
# File 'utils/analytics/repository.rb', line 24

def initialize(path, name)
  @path = path
  @name = name
  @author_identities = T.let({}, T::Hash[String, T::Array[String]])
end

Instance Attribute Details

#author_identities ⇒ Hash{String => Array<String>} (readonly)

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Mailmapped email identities by UTC year, hex-encoded for non-UTF-8 history.

Returns:



21
22
23
 
# File 'utils/analytics/repository.rb', line 21

def author_identities
  @author_identities
end

Instance Method Details

#git(*args) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



220
221
222
 
# File 'utils/analytics/repository.rb', line 220

def git(*args)
  Utils.safe_popen_read("git", "-C", @path.to_s, *args).strip
end

#rows ⇒ Array<Hash{String => String, Integer, nil}>

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Returns:



31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
 
# File 'utils/analytics/repository.rb', line 31

def rows
  if git("rev-parse", "--is-shallow-repository") == "true"
    raise "Analytics require complete Git history for #{@name}"
  end

  head = git("rev-parse", "HEAD")
  mailmap = git("ls-tree", "HEAD", "--", ".mailmap")
  formula_repository = ["Homebrew/homebrew-core", "Homebrew/legacy-homebrew"].include?(@name)
  release_repository = ["Homebrew/brew", "Homebrew/legacy-homebrew"].include?(@name)
  tags = if release_repository
    git("for-each-ref", "--merged=HEAD",
        "--format=%(refname:strip=2)%09%(*committerdate:unix)%09%(committerdate:unix)", "refs/tags")
  else
    ""
  end
  cache = HOMEBREW_CACHE/"analytics/git/#{@name}.json"
  data = cache.exist? ? JSON.parse(cache.read) : {}
  generator = Digest::SHA256.file(__FILE__).hexdigest
  if data["head"] && data["mailmap"] == mailmap && data["generator"] == generator
    begin
      git("merge-base", "--is-ancestor", data.fetch("head"), "HEAD")
    rescue ErrorDuringExecution
      data = {}
    end
  else
    data = {}
  end
  year = Time.now.utc.year
  if data["head"] == head && data["year"] == year && data["tags"] == tags
    @author_identities = data.fetch("authors")
    return data.fetch("rows")
  end

  authors = data.fetch("authors", {})
  range = data["head"] ? "#{data.fetch("head")}..HEAD" : "HEAD"
  git("log", "--no-merges", "--format=%ct%x09%aE", range).each_line do |line|
    timestamp, email = line.chomp.split("\t", 2)
    next if timestamp.nil? || email.nil?

    commit_year = Time.at(timestamp.to_i).utc.year.to_s
    authors[commit_year] ||= []
    # Historical author addresses can contain non-UTF-8 bytes.
    authors[commit_year] << email.unpack1("H*")
  end
  authors.transform_values!(&:uniq)
  @author_identities = authors
  first_years = authors.keys.sort.each_with_object({}) do |period, first|
    authors.fetch(period).each { |identity| first[identity] ||= period }
  end.values.tally

  pull_requests = data.fetch("pull_requests", {})
  excluded = []
  if (split = SPLIT_COMMITS[@name]) &&
     system("git", "-C", @path.to_s, "cat-file", "-e", "#{split}^{commit}", out: File::NULL, err: File::NULL)
    excluded << "^#{split}^"
  end
  git("log", "--first-parent", "--reverse", "--format=%ct%x00%B%x00", range, *excluded)
    .split("\0").each_slice(2) do |timestamp, message|
      next if message.nil?

      subject = message.lines.first.to_s.chomp
      numbers = [subject[/\AMerge pull request #(\d+)\b/, 1], subject[/\(#(\d+)\)\z/, 1]].compact
      message.scan(/^Closes (.+?)\.?$/i).flatten.each do |reference|
        number = reference[/\A#(\d+)\z/, 1] ||
                 reference[/\A#{Regexp.escape(@name)}#(\d+)\z/i, 1] ||
                 reference[%r{\Ahttps://github\.com/#{Regexp.escape(@name)}/pull/(\d+)\z}i, 1]
        numbers << number if number
      end
      numbers.each do |number|
        next if @name == "Homebrew/legacy-homebrew" && number.to_i < FIRST_LEGACY_PULL_REQUEST

        pull_requests[number] ||= Time.at(timestamp.to_i).utc.year.to_s
      end
    end
  merged_years = pull_requests.values.tally

  changes = data.fetch("changes", {})
  if @name == "Homebrew/homebrew-cask"
    period = ""
    tokens = git("log", "--first-parent", "--reverse", "--raw", "--root", "--no-abbrev",
                 "--diff-merges=first-parent", "--find-renames=100%", "--format=%x00year%x00%ct",
                 "-z", range).split("\0")
    while (token = tokens.shift)
      if token == "year"
        period = Time.at(tokens.shift.to_i).utc.year.to_s
      elsif token.lstrip.start_with?(":")
        old_mode, new_mode, _, _, status = token.delete_prefix("\n").delete_prefix(":").split
        old_path = tokens.shift.to_s
        new_path = status&.start_with?("R") ? tokens.shift.to_s : old_path
        before = %w[100644 100755].include?(old_mode) && old_path.match?(%r{\ACasks?/(?:[^/]+/)*[^/]+\.rb\z})
        after = %w[100644 100755].include?(new_mode) && new_path.match?(%r{\ACasks?/(?:[^/]+/)*[^/]+\.rb\z})
        next if before == after

        metric = after ? "added" : "removed"
        changes[period] ||= {}
        changes[period][metric] = changes[period].fetch(metric, 0) + 1
      end
    end
  end

  revisions = data.fetch("year_end_revisions", {})
  missing_years = (2009...year).reject { |y| revisions.key?(y.to_s) }
  if missing_years.any?
    timeline = git("log", "--first-parent", "--format=%ct%x09%H", "HEAD").lines.map do |line|
      timestamp, revision = line.chomp.split("\t", 2)
      [timestamp.to_i, revision]
    end
    missing_years.each do |y|
      revisions[y.to_s] = timeline.find { |timestamp, _| timestamp < Time.utc(y + 1).to_i }&.last
    end
  end

  release_years = tags.lines.filter_map do |line|
    tag, annotated_date, commit_date = line.chomp.split("\t", -1)
    next unless tag&.match?(/\Av?\d+(?:\.\d+){1,2}\z/)

    Time.at((annotated_date.presence || commit_date).to_i).utc.year.to_s
  end.tally
  catalogues = data.fetch("catalogues", {})
  maintainers = data.fetch("maintainers", {})
  rows = ["current", *(2009..year).to_a.reverse.map(&:to_s)].map do |period|
    revision = (period == "current" || period == year.to_s) ? head : revisions[period]
    catalogue = if @name == "Homebrew/brew" || revision.nil?
      0
    else
      catalogues[revision] ||= git("ls-tree", "-rz", revision).split("\0").count do |entry|
        metadata, path = entry.split("\t", 2)
        next false unless metadata&.match?(/\A100(?:644|755) blob /)

        if formula_repository
          path&.match?(%r{\A(?:Library/)?Formula/(?:[^/]+/)?[^/]+\.rb\z})
        else
          path&.match?(%r{\ACasks?/(?:[^/]+/)*[^/]+\.rb\z})
        end
      end
    end
    if @name == "Homebrew/brew" && revision && !maintainers.key?(revision)
      readme = git("ls-tree", "--name-only", revision).lines(chomp: true).find do |file|
        file.match?(/\AREADME(?:\.md)?\z/)
      end
      identities = if readme
        git("show", "#{revision}:#{readme}").split(/\n\s*\n/).select do |paragraph|
          paragraph.match?(/\b(?:maintainers?|Project Leader|Technical Steering Committee)\b/i) &&
            !paragraph.match?(/\bformer\b/i)
        end.join("\n").scan(%r{\]\(https?://github\.com/([\w-]+)/?\)}).flatten.map(&:downcase).uniq
      end
      maintainers[revision] = identities&.presence&.size
    end
    contributors = ((period == "current") ? authors.values.flatten : authors.fetch(period, [])).uniq.size
    {
      "period"               => period,
      "revision"             => revision,
      "contributors"         => contributors,
      "new_contributors"     => (period == "current") ? contributors : first_years.fetch(period, 0),
      "maintainers"          => maintainers[revision],
      "pull_requests_merged" => (period == "current") ? pull_requests.size : merged_years.fetch(period, 0),
      "added"                => if @name == "Homebrew/homebrew-cask"
                                  if period == "current"
                                    changes.values.sum { |counts| counts.fetch("added", 0) }
                                  else
                                    changes.fetch(period, {}).fetch("added", 0)
                                  end
      end,
      "removed"              => if @name == "Homebrew/homebrew-cask"
                                  if period == "current"
                                    changes.values.sum { |counts| counts.fetch("removed", 0) }
                                  else
                                    changes.fetch(period, {}).fetch("removed", 0)
                                  end
      end,
      "formulae"             => formula_repository ? catalogue : nil,
      "casks"                => (@name == "Homebrew/homebrew-cask") ? catalogue : nil,
      "releases"             => if release_repository
                                  if period == "current"
                                    release_years.values.sum
                                  else
                                    release_years.fetch(period, 0)
                                  end
      end,
    }
  end
  cache.dirname.mkpath
  cache.atomic_write(JSON.generate({ head:, year:, mailmap:, tags:, authors:, generator:,
                                     year_end_revisions: revisions, catalogues:, maintainers:, changes:,
                                     pull_requests:, rows: }))
  rows
end