leandrof/u-boot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'buildman' of git://git.denx.de/u-boot-x86

Browse Source
This commit is contained in:
Tom Rini 2015-01-15 10:18:05 -05:00
commit d1c3310d40
9 changed files with 606 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

142
tools/buildman/README
View File

@ -141,8 +141,8 @@ $ git clone git://git.denx.de/u-boot.git .
$ git checkout -b my-branch origin/master
$ # Add some commits to the branch, reading for testing
2. Create ~/.buildman to tell buildman where to find tool chains. As an
example:
2. Create ~/.buildman to tell buildman where to find tool chains (see 'The
.buildman file' later for details). As an example:
# Buildman settings file
@ -171,7 +171,16 @@ The toolchain-alias section indicates that the i386 toolchain should be used
to build x86 commits.
2. Check the available toolchains
3. Make sure you have the require Python pre-requisites
Buildman uses multiprocessing, Queue, shutil, StringIO, ConfigParser and
urllib2. These should normally be available, but if you get an error like
this then you will need to obtain those modules:
ImportError: No module named multiprocessing
4. Check the available toolchains
Run this check to make sure that you have a toolchain for every architecture.
@ -301,6 +310,47 @@ You can see that everything is covered, even some strange ones that won't
be used (c88 and c99). This is a feature.
5. Install new toolchains if needed
You can download toolchains and update the [toolchain] section of the
settings file to find them.
To make this easier, buildman can automatically download and install
toolchains from kernel.org. First list the available architectures:
$ ./tools/buildman/buildman sandbox --fetch-arch list
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.2/
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.5.1/
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.2.4/
Available architectures: alpha am33_2.0 arm avr32 bfin cris crisv32 frv h8300
hppa hppa64 i386 ia64 m32r m68k mips mips64 or32 powerpc powerpc64 s390x sh4
sparc sparc64 tilegx x86_64 xtensa
Then pick one and download it:
$ ./tools/buildman/buildman sandbox --fetch-arch or32
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.2/
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.5.1/
Downloading: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.5.1//x86_64-gcc-4.5.1-nolibc_or32-linux.tar.xz
Unpacking to: /home/sjg/.buildman-toolchains
Testing
- looking in '/home/sjg/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/.'
- looking in '/home/sjg/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/bin'
- found '/home/sjg/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/bin/or32-linux-gcc'
Tool chain test: OK
Buildman should now be set up to use your new toolchain.
At the time of writing, U-Boot has these architectures:
arc, arm, avr32, blackfin, m68k, microblaze, mips, nds32, nios2, openrisc
powerpc, sandbox, sh, sparc, x86
Of these, only arc, microblaze and nds32 are not available at kernel.org..
How to run it
=============
@ -310,8 +360,9 @@ branch with a valid upstream)
$ ./tools/buildman/buildman -b <branch> -n
If it can't detect the upstream branch, try checking out the branch, and
doing something like 'git branch --set-upstream <branch> upstream/master'
or something similar.
doing something like 'git branch --set-upstream-to upstream/master'
or something similar. Buildman will try to guess a suitable upstream branch
if it can't find one (you will see a message like" Guessing upstream as ...).
As an example:
@ -665,28 +716,62 @@ It is common when refactoring code for the rodata to decrease as the text size
increases, and vice versa.
Providing 'make' flags
======================
The .buildman file
==================
U-Boot's build system supports a few flags (such as BUILD_TAG) which affect
the build product. These flags can be specified in the buildman settings
file. They can also be useful when building U-Boot against other open source
software.
The .buildman file provides information about the available toolchains and
also allows build flags to be passed to 'make'. It consists of several
sections, with the section name in square brackets. Within each section are
a set of (tag, value) pairs.
[make-flags]
at91-boards=ENABLE_AT91_TEST=1
snapper9260=${at91-boards} BUILD_TAG=442
snapper9g45=${at91-boards} BUILD_TAG=443
'[toolchain]' section
This will use 'make ENABLE_AT91_TEST=1 BUILD_TAG=442' for snapper9260
and 'make ENABLE_AT91_TEST=1 BUILD_TAG=443' for snapper9g45. A special
variable ${target} is available to access the target name (snapper9260 and
snapper9g20 in this case). Variables are resolved recursively. Note that
variables can only contain the characters A-Z, a-z, 0-9, hyphen (-) and
underscore (_).
This lists the available toolchains. The tag here doesn't matter, but
make sure it is unique. The value is the path to the toolchain. Buildman
will look in that path for a file ending in 'gcc'. It will then execute
it to check that it is a C compiler, passing only the --version flag to
it. If the return code is 0, buildman assumes that it is a valid C
compiler. It uses the first part of the name as the architecture and
strips off the last part when setting the CROSS_COMPILE environment
variable (parts are delimited with a hyphen).
It is expected that any variables added are dealt with in U-Boot's
config.mk file and documented in the README.
For example powerpc-linux-gcc will be noted as a toolchain for 'powerpc'
and CROSS_COMPILE will be set to powerpc-linux- when using it.
'[toolchain-alias]' section
This converts toolchain architecture names to U-Boot names. For example,
if an x86 toolchains is called i386-linux-gcc it will not normally be
used for architecture 'x86'. Adding 'x86: i386 x86_64' to this section
will tell buildman that the i386 and x86_64 toolchains can be used for
the x86 architecture.
'[make-flags]' section
U-Boot's build system supports a few flags (such as BUILD_TAG) which
affect the build product. These flags can be specified in the buildman
settings file. They can also be useful when building U-Boot against other
open source software.
[make-flags]
at91-boards=ENABLE_AT91_TEST=1
snapper9260=${at91-boards} BUILD_TAG=442
snapper9g45=${at91-boards} BUILD_TAG=443
This will use 'make ENABLE_AT91_TEST=1 BUILD_TAG=442' for snapper9260
and 'make ENABLE_AT91_TEST=1 BUILD_TAG=443' for snapper9g45. A special
variable ${target} is available to access the target name (snapper9260
and snapper9g20 in this case). Variables are resolved recursively. Note
that variables can only contain the characters A-Z, a-z, 0-9, hyphen (-)
and underscore (_).
It is expected that any variables added are dealt with in U-Boot's
config.mk file and documented in the README.
Note that you can pass ad-hoc options to the build using environment
variables, for example:
SOME_OPTION=1234 ./tools/buildman/buildman my_board
Quick Sanity Check
@ -698,6 +783,17 @@ build the selected boards and display build status as it runs (i.e. -v is
enabled automatically). Use -e to see errors/warnings as well.
Building Ranges
===============
You can build a range of commits by specifying a range instead of a branch
when using the -b flag. For example:
upstream/master..us-buildman
will build commits in us-buildman that are not in upstream/master.
Other options
=============

11
tools/buildman/bsettings.py
View File

@ -40,7 +40,16 @@ def GetItems(section):
try:
return settings.items(section)
except ConfigParser.NoSectionError as e:
print e
return []
except:
raise
def SetItem(section, tag, value):
"""Set an item and write it back to the settings file"""
global settings
global config_fname
settings.set(section, tag, value)
if config_fname is not None:
with open(config_fname, 'w') as fd:
settings.write(fd)

21
tools/buildman/builder.py
View File

@ -174,7 +174,8 @@ class Builder:
self.func_sizes = func_sizes
def __init__(self, toolchains, base_dir, git_dir, num_threads, num_jobs,
gnu_make='make', checkout=True, show_unknown=True, step=1):
gnu_make='make', checkout=True, show_unknown=True, step=1,
no_subdirs=False, full_path=False, verbose_build=False):
"""Create a new Builder object
Args:
@ -188,6 +189,11 @@ class Builder:
This is used for testing.
show_unknown: Show unknown boards (those not built) in summary
step: 1 to process every commit, n to process every nth commit
no_subdirs: Don't create subdirectories when building current
source for a single board
full_path: Return the full path in CROSS_COMPILE and don't set
PATH
verbose_build: Run build with V=1 and don't use 'make -s'
"""
self.toolchains = toolchains
self.base_dir = base_dir
@ -213,6 +219,9 @@ class Builder:
self._step = step
self.in_tree = False
self._error_lines = 0
self.no_subdirs = no_subdirs
self.full_path = full_path
self.verbose_build = verbose_build
self.col = terminal.Color()
@ -392,15 +401,17 @@ class Builder:
Args:
commit_upto: Commit number to use (0..self.count-1)
"""
commit_dir = None
if self.commits:
commit = self.commits[commit_upto]
subject = commit.subject.translate(trans_valid_chars)
commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1,
self.commit_count, commit.hash, subject[:20]))
else:
elif not self.no_subdirs:
commit_dir = 'current'
output_dir = os.path.join(self.base_dir, commit_dir)
return output_dir
if not commit_dir:
return self.base_dir
return os.path.join(self.base_dir, commit_dir)
def GetBuildDir(self, commit_upto, target):
"""Get the name of the build directory for a commit number
@ -1115,6 +1126,8 @@ class Builder:
create. Having left over directories is confusing when the user wants
to check the output manually.
"""
if not self.commits:
return
dir_list = []
for commit_upto in range(self.commit_count):
dir_list.append(self._GetOutputDir(commit_upto))

7
tools/buildman/builderthread.py
View File

@ -177,7 +177,7 @@ class BuilderThread(threading.Thread):
commit = 'current'
# Set up the environment and command line
env = self.toolchain.MakeEnvironment()
env = self.toolchain.MakeEnvironment(self.builder.full_path)
Mkdir(out_dir)
args = []
cwd = work_dir
@ -197,7 +197,8 @@ class BuilderThread(threading.Thread):
src_dir = os.getcwd()
else:
args.append('O=build')
args.append('-s')
if not self.builder.verbose_build:
args.append('-s')
if self.builder.num_jobs is not None:
args.extend(['-j', str(self.builder.num_jobs)])
config_args = ['%s_defconfig' % brd.target]
@ -284,7 +285,7 @@ class BuilderThread(threading.Thread):
print >>fd, 'path', result.toolchain.path
# Write out the image and function size information and an objdump
env = result.toolchain.MakeEnvironment()
env = result.toolchain.MakeEnvironment(self.builder.full_path)
lines = []
for fname in ['u-boot', 'spl/u-boot-spl']:
cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname]

10
tools/buildman/cmdline.py
View File

@ -36,6 +36,10 @@ def ParseArgs():
parser.add_option('-F', '--force-build-failures', dest='force_build_failures',
action='store_true', default=False,
help='Force build of previously-failed build')
parser.add_option('--fetch-arch', type='string',
help="Fetch a toolchain for architecture FETCH_ARCH ('list' to list)."
' You can also fetch several toolchains separate by comma, or'
" 'all' to download all")
parser.add_option('-g', '--git', type='string',
help='Git repo containing branch to build', default='.')
parser.add_option('-G', '--config-file', type='string',
@ -55,11 +59,15 @@ def ParseArgs():
help='List available tool chains')
parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
default=False, help="Do a dry run (describe actions, but do nothing)")
parser.add_option('-N', '--no-subdirs', action='store_true', dest='no_subdirs',
default=False, help="Don't create subdirectories when building current source for a single board")
parser.add_option('-o', '--output-dir', type='string',
dest='output_dir', default='..',
help='Directory where all builds happen and buildman has its workspace (default is ../)')
parser.add_option('-Q', '--quick', action='store_true',
default=False, help='Do a rough build, with limited warning resolution')
parser.add_option('-p', '--full-path', action='store_true',
default=False, help="Use full toolchain path in CROSS_COMPILE")
parser.add_option('-s', '--summary', action='store_true',
default=False, help='Show a build summary')
parser.add_option('-S', '--show-sizes', action='store_true',
@ -74,6 +82,8 @@ def ParseArgs():
default=False, help='Show boards with unknown build result')
parser.add_option('-v', '--verbose', action='store_true',
default=False, help='Show build results while the build progresses')
parser.add_option('-V', '--verbose-build', action='store_true',
default=False, help='Run make with V=1, showing all output')
parser.add_option('-x', '--exclude', dest='exclude',
type='string', action='append',
help='Specify a list of boards to exclude, separated by comma')

58
tools/buildman/control.py
View File

@ -118,21 +118,45 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
print
return 0
if options.fetch_arch:
if options.fetch_arch == 'list':
sorted_list = toolchains.ListArchs()
print 'Available architectures: %s\n' % ' '.join(sorted_list)
return 0
else:
fetch_arch = options.fetch_arch
if fetch_arch == 'all':
fetch_arch = ','.join(toolchains.ListArchs())
print 'Downloading toolchains: %s\n' % fetch_arch
for arch in fetch_arch.split(','):
ret = toolchains.FetchAndInstall(arch)
if ret:
return ret
return 0
# Work out how many commits to build. We want to build everything on the
# branch. We also build the upstream commit as a control so we can see
# problems introduced by the first commit on the branch.
col = terminal.Color()
count = options.count
has_range = options.branch and '..' in options.branch
if count == -1:
if not options.branch:
count = 1
else:
count = gitutil.CountCommitsInBranch(options.git_dir,
options.branch)
if has_range:
count, msg = gitutil.CountCommitsInRange(options.git_dir,
options.branch)
else:
count, msg = gitutil.CountCommitsInBranch(options.git_dir,
options.branch)
if count is None:
str = ("Branch '%s' not found or has no upstream" %
options.branch)
sys.exit(col.Color(col.RED, str))
sys.exit(col.Color(col.RED, msg))
elif count == 0:
sys.exit(col.Color(col.RED, "Range '%s' has no commits" %
options.branch))
if msg:
print col.Color(col.YELLOW, msg)
count += 1 # Build upstream commit also
if not count:
@ -172,8 +196,11 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
# to overwrite earlier ones by setting allow_overwrite=True
if options.branch:
if count == -1:
range_expr = gitutil.GetRangeInBranch(options.git_dir,
options.branch)
if has_range:
range_expr = options.branch
else:
range_expr = gitutil.GetRangeInBranch(options.git_dir,
options.branch)
upstream_commit = gitutil.GetUpstream(options.git_dir,
options.branch)
series = patchstream.GetMetaDataForList(upstream_commit,
@ -207,17 +234,22 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
if not gnu_make:
sys.exit('GNU Make not found')
# Create a new builder with the selected options
# Create a new builder with the selected options.
output_dir = options.output_dir
if options.branch:
dirname = options.branch.replace('/', '_')
else:
dirname = 'current'
output_dir = os.path.join(options.output_dir, dirname)
if clean_dir and os.path.exists(output_dir):
# As a special case allow the board directory to be placed in the
# output directory itself rather than any subdirectory.
if not options.no_subdirs:
output_dir = os.path.join(options.output_dir, dirname)
if (clean_dir and output_dir != options.output_dir and
os.path.exists(output_dir)):
shutil.rmtree(output_dir)
builder = Builder(toolchains, output_dir, options.git_dir,
options.threads, options.jobs, gnu_make=gnu_make, checkout=True,
show_unknown=options.show_unknown, step=options.step)
show_unknown=options.show_unknown, step=options.step,
no_subdirs=options.no_subdirs, full_path=options.full_path,
verbose_build=options.verbose_build)
builder.force_config_on_failure = not options.quick
if make_func:
builder.do_make = make_func

75
tools/buildman/test.py
View File

@ -24,6 +24,16 @@ import commit
import terminal
import toolchain
settings_data = '''
# Buildman settings file
[toolchain]
main: /usr/sbin
[toolchain-alias]
x86: i386 x86_64
'''
errors = [
'''main.c: In function 'main_loop':
main.c:260:6: warning: unused variable 'joe' [-Wunused-variable]
@ -83,6 +93,8 @@ boards = [
['Active', 'sandbox', 'sandbox', '', 'Tester', 'Sandbox board', 'board4', ''],
]
BASE_DIR = 'base'
class Options:
"""Class that holds build options"""
pass
@ -111,8 +123,11 @@ class TestBuild(unittest.TestCase):
self.boards.AddBoard(board.Board(*brd))
self.boards.SelectBoards([])
# Add some test settings
bsettings.Setup(None)
bsettings.AddFile(settings_data)
# Set up the toolchains
bsettings.Setup()
self.toolchains = toolchain.Toolchains()
self.toolchains.Add('arm-linux-gcc', test=False)
self.toolchains.Add('sparc-linux-gcc', test=False)
@ -341,6 +356,64 @@ class TestBuild(unittest.TestCase):
self.assertEqual(self.boards.SelectBoards(['sandbox sandbox',
'sandbox']),
{'all': 1, 'sandbox': 1})
def CheckDirs(self, build, dirname):
self.assertEqual('base%s' % dirname, build._GetOutputDir(1))
self.assertEqual('base%s/fred' % dirname,
build.GetBuildDir(1, 'fred'))
self.assertEqual('base%s/fred/done' % dirname,
build.GetDoneFile(1, 'fred'))
self.assertEqual('base%s/fred/u-boot.sizes' % dirname,
build.GetFuncSizesFile(1, 'fred', 'u-boot'))
self.assertEqual('base%s/fred/u-boot.objdump' % dirname,
build.GetObjdumpFile(1, 'fred', 'u-boot'))
self.assertEqual('base%s/fred/err' % dirname,
build.GetErrFile(1, 'fred'))
def testOutputDir(self):
build = builder.Builder(self.toolchains, BASE_DIR, None, 1, 2,
checkout=False, show_unknown=False)
build.commits = self.commits
build.commit_count = len(self.commits)
subject = self.commits[1].subject.translate(builder.trans_valid_chars)
dirname ='/%02d_of_%02d_g%s_%s' % (2, build.commit_count, commits[1][0],
subject[:20])
self.CheckDirs(build, dirname)
def testOutputDirCurrent(self):
build = builder.Builder(self.toolchains, BASE_DIR, None, 1, 2,
checkout=False, show_unknown=False)
build.commits = None
build.commit_count = 0
self.CheckDirs(build, '/current')
def testOutputDirNoSubdirs(self):
build = builder.Builder(self.toolchains, BASE_DIR, None, 1, 2,
checkout=False, show_unknown=False,
no_subdirs=True)
build.commits = None
build.commit_count = 0
self.CheckDirs(build, '')
def testToolchainAliases(self):
self.assertTrue(self.toolchains.Select('arm') != None)
with self.assertRaises(ValueError):
self.toolchains.Select('no-arch')
with self.assertRaises(ValueError):
self.toolchains.Select('x86')
self.toolchains = toolchain.Toolchains()
self.toolchains.Add('x86_64-linux-gcc', test=False)
self.assertTrue(self.toolchains.Select('x86') != None)
self.toolchains = toolchain.Toolchains()
self.toolchains.Add('i386-linux-gcc', test=False)
self.assertTrue(self.toolchains.Select('x86') != None)
def testToolchainDownload(self):
"""Test that we can download toolchains"""
self.assertEqual('https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/x86_64-gcc-4.6.3-nolibc_arm-unknown-linux-gnueabi.tar.xz',
self.toolchains.LocateArchUrl('arm'))
if __name__ == "__main__":
unittest.main()

270
tools/buildman/toolchain.py
View File

@ -5,11 +5,42 @@
import re
import glob
from HTMLParser import HTMLParser
import os
import sys
import tempfile
import urllib2
import bsettings
import command
# Simple class to collect links from a page
class MyHTMLParser(HTMLParser):
def __init__(self, arch):
"""Create a new parser
After the parser runs, self.links will be set to a list of the links
to .xz archives found in the page, and self.arch_link will be set to
the one for the given architecture (or None if not found).
Args:
arch: Architecture to search for
"""
HTMLParser.__init__(self)
self.arch_link = None
self.links = []
self._match = '_%s-' % arch
def handle_starttag(self, tag, attrs):
if tag == 'a':
for tag, value in attrs:
if tag == 'href':
if value and value.endswith('.xz'):
self.links.append(value)
if self._match in value:
self.arch_link = value
class Toolchain:
"""A single toolchain
@ -20,7 +51,6 @@ class Toolchain:
arch: Architecture of toolchain as determined from the first
component of the filename. E.g. arm-linux-gcc becomes arm
"""
def __init__(self, fname, test, verbose=False):
"""Create a new toolchain object.
@ -30,11 +60,18 @@ class Toolchain:
"""
self.gcc = fname
self.path = os.path.dirname(fname)
self.cross = os.path.basename(fname)[:-3]
# Find the CROSS_COMPILE prefix to use for U-Boot. For example,
# 'arm-linux-gnueabihf-gcc' turns into 'arm-linux-gnueabihf-'.
basename = os.path.basename(fname)
pos = basename.rfind('-')
self.cross = basename[:pos + 1] if pos != -1 else ''
# The architecture is the first part of the name
pos = self.cross.find('-')
self.arch = self.cross[:pos] if pos != -1 else 'sandbox'
env = self.MakeEnvironment()
env = self.MakeEnvironment(False)
# As a basic sanity check, run the C compiler with --version
cmd = [fname, '--version']
@ -74,15 +111,23 @@ class Toolchain:
return prio
return prio
def MakeEnvironment(self):
def MakeEnvironment(self, full_path):
"""Returns an environment for using the toolchain.
Thie takes the current environment, adds CROSS_COMPILE and
augments PATH so that the toolchain will operate correctly.
Thie takes the current environment and adds CROSS_COMPILE so that
the tool chain will operate correctly.
Args:
full_path: Return the full path in CROSS_COMPILE and don't set
PATH
"""
env = dict(os.environ)
env['CROSS_COMPILE'] = self.cross
env['PATH'] += (':' + self.path)
if full_path:
env['CROSS_COMPILE'] = os.path.join(self.path, self.cross)
else:
env['CROSS_COMPILE'] = self.cross
env['PATH'] = self.path + ':' + env['PATH']
return env
@ -101,18 +146,29 @@ class Toolchains:
self.paths = []
self._make_flags = dict(bsettings.GetItems('make-flags'))
def GetSettings(self):
def GetPathList(self):
"""Get a list of available toolchain paths
Returns:
List of strings, each a path to a toolchain mentioned in the
[toolchain] section of the settings file.
"""
toolchains = bsettings.GetItems('toolchain')
if not toolchains:
print ("Warning: No tool chains - please add a [toolchain] section"
" to your buildman config file %s. See README for details" %
bsettings.config_fname)
paths = []
for name, value in toolchains:
if '*' in value:
self.paths += glob.glob(value)
paths += glob.glob(value)
else:
self.paths.append(value)
paths.append(value)
return paths
def GetSettings(self):
self.paths += self.GetPathList()
def Add(self, fname, test=True, verbose=False):
"""Add a toolchain to our list
@ -132,6 +188,24 @@ class Toolchains:
if add_it:
self.toolchains[toolchain.arch] = toolchain
def ScanPath(self, path, verbose):
"""Scan a path for a valid toolchain
Args:
path: Path to scan
verbose: True to print out progress information
Returns:
Filename of C compiler if found, else None
"""
for subdir in ['.', 'bin', 'usr/bin']:
dirname = os.path.join(path, subdir)
if verbose: print " - looking in '%s'" % dirname
for fname in glob.glob(dirname + '/*gcc'):
if verbose: print " - found '%s'" % fname
return fname
return None
def Scan(self, verbose):
"""Scan for available toolchains and select the best for each arch.
@ -145,12 +219,9 @@ class Toolchains:
if verbose: print 'Scanning for tool chains'
for path in self.paths:
if verbose: print " - scanning path '%s'" % path
for subdir in ['.', 'bin', 'usr/bin']:
dirname = os.path.join(path, subdir)
if verbose: print " - looking in '%s'" % dirname
for fname in glob.glob(dirname + '/*gcc'):
if verbose: print " - found '%s'" % fname
self.Add(fname, True, verbose)
fname = self.ScanPath(path, verbose)
if fname:
self.Add(fname, True, verbose)
def List(self):
"""List out the selected toolchains for each architecture"""
@ -170,9 +241,11 @@ class Toolchains:
returns:
toolchain object, or None if none found
"""
for name, value in bsettings.GetItems('toolchain-alias'):
if arch == name:
arch = value
for tag, value in bsettings.GetItems('toolchain-alias'):
if arch == tag:
for alias in value.split():
if alias in self.toolchains:
return self.toolchains[alias]
if not arch in self.toolchains:
raise ValueError, ("No tool chain found for arch '%s'" % arch)
@ -247,3 +320,160 @@ class Toolchains:
else:
i += 1
return args
def LocateArchUrl(self, fetch_arch):
"""Find a toolchain available online
Look in standard places for available toolchains. At present the
only standard place is at kernel.org.
Args:
arch: Architecture to look for, or 'list' for all
Returns:
If fetch_arch is 'list', a tuple:
Machine architecture (e.g. x86_64)
List of toolchains
else
URL containing this toolchain, if avaialble, else None
"""
arch = command.OutputOneLine('uname', '-m')
base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
versions = ['4.6.3', '4.6.2', '4.5.1', '4.2.4']
links = []
for version in versions:
url = '%s/%s/%s/' % (base, arch, version)
print 'Checking: %s' % url
response = urllib2.urlopen(url)
html = response.read()
parser = MyHTMLParser(fetch_arch)
parser.feed(html)
if fetch_arch == 'list':
links += parser.links
elif parser.arch_link:
return url + parser.arch_link
if fetch_arch == 'list':
return arch, links
return None
def Download(self, url):
"""Download a file to a temporary directory
Args:
url: URL to download
Returns:
Tuple:
Temporary directory name
Full path to the downloaded archive file in that directory,
or None if there was an error while downloading
"""
print "Downloading: %s" % url
leaf = url.split('/')[-1]
tmpdir = tempfile.mkdtemp('.buildman')
response = urllib2.urlopen(url)
fname = os.path.join(tmpdir, leaf)
fd = open(fname, 'wb')
meta = response.info()
size = int(meta.getheaders("Content-Length")[0])
done = 0
block_size = 1 << 16
status = ''
# Read the file in chunks and show progress as we go
while True:
buffer = response.read(block_size)
if not buffer:
print chr(8) * (len(status) + 1), '\r',
break
done += len(buffer)
fd.write(buffer)
status = r"%10d MiB [%3d%%]" % (done / 1024 / 1024,
done * 100 / size)
status = status + chr(8) * (len(status) + 1)
print status,
sys.stdout.flush()
fd.close()
if done != size:
print 'Error, failed to download'
os.remove(fname)
fname = None
return tmpdir, fname
def Unpack(self, fname, dest):
"""Unpack a tar file
Args:
fname: Filename to unpack
dest: Destination directory
Returns:
Directory name of the first entry in the archive, without the
trailing /
"""
stdout = command.Output('tar', 'xvfJ', fname, '-C', dest)
return stdout.splitlines()[0][:-1]
def TestSettingsHasPath(self, path):
"""Check if builmand will find this toolchain
Returns:
True if the path is in settings, False if not
"""
paths = self.GetPathList()
return path in paths
def ListArchs(self):
"""List architectures with available toolchains to download"""
host_arch, archives = self.LocateArchUrl('list')
re_arch = re.compile('[-a-z0-9.]*_([^-]*)-.*')
arch_set = set()
for archive in archives:
# Remove the host architecture from the start
arch = re_arch.match(archive[len(host_arch):])
if arch:
arch_set.add(arch.group(1))
return sorted(arch_set)
def FetchAndInstall(self, arch):
"""Fetch and install a new toolchain
arch:
Architecture to fetch, or 'list' to list
"""
# Fist get the URL for this architecture
url = self.LocateArchUrl(arch)
if not url:
print ("Cannot find toolchain for arch '%s' - use 'list' to list" %
arch)
return 2
home = os.environ['HOME']
dest = os.path.join(home, '.buildman-toolchains')
if not os.path.exists(dest):
os.mkdir(dest)
# Download the tar file for this toolchain and unpack it
tmpdir, tarfile = self.Download(url)
if not tarfile:
return 1
print 'Unpacking to: %s' % dest,
sys.stdout.flush()
path = self.Unpack(tarfile, dest)
os.remove(tarfile)
os.rmdir(tmpdir)
print
# Check that the toolchain works
print 'Testing'
dirpath = os.path.join(dest, path)
compiler_fname = self.ScanPath(dirpath, True)
if not compiler_fname:
print 'Could not locate C compiler - fetch failed.'
return 1
toolchain = Toolchain(compiler_fname, True, True)
# Make sure that it will be found by buildman
if not self.TestSettingsHasPath(dirpath):
print ("Adding 'download' to config file '%s'" %
bsettings.config_fname)
tools_dir = os.path.dirname(dirpath)
bsettings.SetItem('toolchain', 'download', '%s/*' % tools_dir)
return 0

90
tools/patman/gitutil.py
View File

@ -61,6 +61,52 @@ def CountCommitsToBranch():
patch_count = int(stdout)
return patch_count
def NameRevision(commit_hash):
"""Gets the revision name for a commit
Args:
commit_hash: Commit hash to look up
Return:
Name of revision, if any, else None
"""
pipe = ['git', 'name-rev', commit_hash]
stdout = command.RunPipe([pipe], capture=True, oneline=True).stdout
# We expect a commit, a space, then a revision name
name = stdout.split(' ')[1].strip()
return name
def GuessUpstream(git_dir, branch):
"""Tries to guess the upstream for a branch
This lists out top commits on a branch and tries to find a suitable
upstream. It does this by looking for the first commit where
'git name-rev' returns a plain branch name, with no ! or ^ modifiers.
Args:
git_dir: Git directory containing repo
branch: Name of branch
Returns:
Tuple:
Name of upstream branch (e.g. 'upstream/master') or None if none
Warning/error message, or None if none
"""
pipe = [LogCmd(branch, git_dir=git_dir, oneline=True, count=100)]
result = command.RunPipe(pipe, capture=True, capture_stderr=True,
raise_on_error=False)
if result.return_code:
return None, "Branch '%s' not found" % branch
for line in result.stdout.splitlines()[1:]:
commit_hash = line.split(' ')[0]
name = NameRevision(commit_hash)
if '~' not in name and '^' not in name:
if name.startswith('remotes/'):
name = name[8:]
return name, "Guessing upstream as '%s'" % name
return None, "Cannot find a suitable upstream for branch '%s'" % branch
def GetUpstream(git_dir, branch):
"""Returns the name of the upstream for a branch
@ -69,7 +115,9 @@ def GetUpstream(git_dir, branch):
branch: Name of branch
Returns:
Name of upstream branch (e.g. 'upstream/master') or None if none
Tuple:
Name of upstream branch (e.g. 'upstream/master') or None if none
Warning/error message, or None if none
"""
try:
remote = command.OutputOneLine('git', '--git-dir', git_dir, 'config',
@ -77,13 +125,14 @@ def GetUpstream(git_dir, branch):
merge = command.OutputOneLine('git', '--git-dir', git_dir, 'config',
'branch.%s.merge' % branch)
except:
return None
upstream, msg = GuessUpstream(git_dir, branch)
return upstream, msg
if remote == '.':
return merge
elif remote and merge:
leaf = merge.split('/')[-1]
return '%s/%s' % (remote, leaf)
return '%s/%s' % (remote, leaf), None
else:
raise ValueError, ("Cannot determine upstream branch for branch "
"'%s' remote='%s', merge='%s'" % (branch, remote, merge))
@ -99,10 +148,29 @@ def GetRangeInBranch(git_dir, branch, include_upstream=False):
Expression in the form 'upstream..branch' which can be used to
access the commits. If the branch does not exist, returns None.
"""
upstream = GetUpstream(git_dir, branch)
upstream, msg = GetUpstream(git_dir, branch)
if not upstream:
return None
return '%s%s..%s' % (upstream, '~' if include_upstream else '', branch)
return None, msg
rstr = '%s%s..%s' % (upstream, '~' if include_upstream else '', branch)
return rstr, msg
def CountCommitsInRange(git_dir, range_expr):
"""Returns the number of commits in the given range.
Args:
git_dir: Directory containing git repo
range_expr: Range to check
Return:
Number of patches that exist in the supplied rangem or None if none
were found
"""
pipe = [LogCmd(range_expr, git_dir=git_dir, oneline=True)]
result = command.RunPipe(pipe, capture=True, capture_stderr=True,
raise_on_error=False)
if result.return_code:
return None, "Range '%s' not found or is invalid" % range_expr
patch_count = len(result.stdout.splitlines())
return patch_count, None
def CountCommitsInBranch(git_dir, branch, include_upstream=False):
"""Returns the number of commits in the given branch.
@ -114,14 +182,10 @@ def CountCommitsInBranch(git_dir, branch, include_upstream=False):
Number of patches that exist on top of the branch, or None if the
branch does not exist.
"""
range_expr = GetRangeInBranch(git_dir, branch, include_upstream)
range_expr, msg = GetRangeInBranch(git_dir, branch, include_upstream)
if not range_expr:
return None
pipe = [LogCmd(range_expr, git_dir=git_dir, oneline=True),
['wc', '-l']]
result = command.RunPipe(pipe, capture=True, oneline=True)
patch_count = int(result.stdout)
return patch_count
return None, msg
return CountCommitsInRange(git_dir, range_expr)
def CountCommits(commit_range):
"""Returns the number of commits in the given range.