[Notes] [Git][BuildStream/buildstream][valentindavid/git_force_fetch_tags] 5 commits: buildstream/data/projectconfig.yaml: Remove default strip-binaries

[Valentin David <gitlab mg gitlab com>]

Title: GitLab

Valentin David pushed to branch valentindavid/git_force_fetch_tags at BuildStream / buildstream

Commits:

• 56c2c2cc
  by Javier Jardón at 2018-12-09T16:59:26Z

  buildstream/data/projectconfig.yaml: Remove default strip-binaries
  
  They are too specific to be included by default
  
  Recommendation is if you are building in Linux is to use the
  ones begin used in the freedesktop-sdk project, for example
  
  See #645
  

• dcb6b5e8
  by Javier Jardón at 2018-12-09T16:59:26Z

  NEWS: Update for removal of default strip-commands
  

• ae0afce5
  by Javier Jardón at 2018-12-09T17:08:16Z

  buildstream/buildelement.py: Document where to find strip commands for Linux
  

• 873b618c
  by Javier Jardón at 2018-12-10T09:49:52Z

  Merge branch 'jjardon/strip-binaries-removal' into 'master'
  
  Remove default strip-commands
  
  Closes #645
  
  See merge request BuildStream/buildstream!987

• f2fcc2f6
  by Valentin David at 2018-12-11T12:12:08Z

  Force updating tags when fetching git repository
  
  When using aliases there are multiple remotes used in the cache
  repository. When fetching, tags are not updated if the were previously
  fetched from a different remote. Commits that not in a branch and only
  tagged do not get fetched if the tag is not fetched.
  
  Fixes #812
  

15 changed files:

• NEWS
• buildstream/buildelement.py
• buildstream/data/projectconfig.yaml
• buildstream/plugins/sources/git.py
• tests/cachekey/project/elements/build1.expected
• tests/cachekey/project/elements/build2.expected
• tests/cachekey/project/target.expected
• tests/examples/autotools.py
• tests/examples/developing.py
• tests/examples/flatpak-autotools.py
• tests/integration/autotools.py
• tests/integration/cmake.py
• tests/integration/compose.py
• tests/sources/git.py
• tests/testutils/repo/git.py

Changes:

NEWS

---

...	...	@@ -2,6 +2,10 @@
2	2	buildstream 1.3.1
3	3	=================
4	4
	5	+ o BREAKING CHANGE: Default strip-commands have been removed as they are too
	6	+ specific. Recommendation if you are building in Linux is to use the
	7	+ ones being used in freedesktop-sdk project, for example
	8	+
5	9	o All elements must now be suffixed with `.bst`
6	10	Attempting to use an element that does not have the `.bst` extension,
7	11	will result in a warning.

buildstream/buildelement.py

---

...	...	@@ -35,6 +35,14 @@ This section will give a brief summary of how some of the common features work,
35	35	some of them or the variables they use will be further detailed in the following
36	36	sections.
37	37
	38	+The `strip-binaries` variable
	39	+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	40	+The `strip-binaries` variable is by default **empty**. You need to use the
	41	+appropiate commands depending of the system you are building.
	42	+If you are targetting Linux, ones known to work are the ones used by the
	43	+`freedesktop-sdk <https://freedesktop-sdk.io/>`_, you can take a look to them in their
	44	+`project.conf <https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/freedesktop-sdk-18.08.21/project.conf#L74>`_
	45	+
38	46	Location for running commands
39	47	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
40	48	The ``command-subdir`` variable sets where the build commands will be executed,

buildstream/data/projectconfig.yaml

---

...	...	@@ -44,38 +44,8 @@ variables:
44	44	# Indicates the build installation directory in the sandbox
45	45	install-root: /buildstream-install
46	46
47		- # Arguments for tooling used when stripping debug symbols
48		- objcopy-link-args: --add-gnu-debuglink
49		- objcopy-extract-args: |
50		-
51		- --only-keep-debug --compress-debug-sections
52		-
53		- strip-args: |
54		-
55		- --remove-section=.comment --remove-section=.note --strip-unneeded
56		-
57		- # Generic implementation for stripping debugging symbols
58		- strip-binaries: |
59		-
60		- cd "%{install-root}" && find -type f \
61		- '(' -perm -111 -o -name '*.so*' \
62		- -o -name '*.cmxs' -o -name '*.node' ')' \
63		- -exec sh -ec \
64		- 'read -n4 hdr <"$1" # check for elf header
65		- case "$1" in
66		- %{install-root}%{debugdir}/*)
67		- exit 0
68		- ;;
69		- esac
70		- if [ "$hdr" != "$(printf \\x7fELF)" ]; then
71		- exit 0
72		- fi
73		- debugfile="%{install-root}%{debugdir}/$1"
74		- mkdir -p "$(dirname "$debugfile")"
75		- objcopy %{objcopy-extract-args} "$1" "$debugfile"
76		- chmod 644 "$debugfile"
77		- strip %{strip-args} "$1"
78		- objcopy %{objcopy-link-args} "$debugfile" "$1"' - {} ';'
	47	+ # You need to override this with the commands specific for your system
	48	+ strip-binaries: ""
79	49
80	50	# Generic implementation for reproducible python builds
81	51	fix-pyc-timestamps: |
...	...	@@ -196,4 +166,4 @@ shell:
196	166
197	167	# Command to run when `bst shell` does not provide a command
198	168	#
199		- command: [ 'sh', '-i' ]
		\ No newline at end of file
	169	+ command: [ 'sh', '-i' ]

buildstream/plugins/sources/git.py

---

...	...	@@ -247,7 +247,7 @@ class GitMirror(SourceFetcher):
247	247	else:
248	248	remote_name = "origin"
249	249
250		- self.source.call([self.source.host_git, 'fetch', remote_name, '--prune'],
	250	+ self.source.call([self.source.host_git, 'fetch', remote_name, '--prune', '--force', '--tags'],
251	251	fail="Failed to fetch from remote git repository: {}".format(url),
252	252	fail_temporarily=True,
253	253	cwd=self.mirror)

tests/cachekey/project/elements/build1.expected

---

1		-a0d000abc1dea8714cd27f348d0b798b35e7246c44e330c4b3f7912fabacc6db
		\ No newline at end of file
	1	+dadb8f86874f714b4f6d4c9025332934efb7e85c38f6a68b1267746ae8f43f24

tests/cachekey/project/elements/build2.expected

---

1		-79f546a78748d943a6958c99ab4ad03305f96fefd0b424b6b246b0c9816e00c6
		\ No newline at end of file
	1	+f81cefce283dd3581ba2fc865ff9c2763119274b114b12edb4e87196cfff8b2a

tests/cachekey/project/target.expected

---

1		-d6d283ed1fb0467fcfa5bf69f8596d0f0ac6638281bc9d8e52e1212e2ec0bcab
		\ No newline at end of file
	1	+92dae6a712b4f91f4fdbdf8dad732cf07ff4da092a319fa4f4b261a9287640de

tests/examples/autotools.py

---

...	...	@@ -29,9 +29,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
29	29	result.assert_success()
30	30
31	31	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
32		- '/usr/share', '/usr/lib/debug',
33		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
34		- '/usr/lib/debug/usr/bin/hello',
	32	+ '/usr/share',
35	33	'/usr/bin/hello',
36	34	'/usr/share/doc', '/usr/share/doc/amhello',
37	35	'/usr/share/doc/amhello/README'])

tests/examples/developing.py

---

...	...	@@ -30,9 +30,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
30	30	result.assert_success()
31	31
32	32	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
33		- '/usr/share', '/usr/lib/debug',
34		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
35		- '/usr/lib/debug/usr/bin/hello',
	33	+ '/usr/share',
36	34	'/usr/bin/hello'])
37	35
38	36

tests/examples/flatpak-autotools.py

---

...	...	@@ -48,9 +48,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
48	48	assert result.exit_code == 0
49	49
50	50	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
51		- '/usr/share', '/usr/lib/debug',
52		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
53		- '/usr/lib/debug/usr/bin/hello',
	51	+ '/usr/share',
54	52	'/usr/bin/hello', '/usr/share/doc',
55	53	'/usr/share/doc/amhello',
56	54	'/usr/share/doc/amhello/README'])

tests/integration/autotools.py

---

...	...	@@ -32,9 +32,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
32	32	assert result.exit_code == 0
33	33
34	34	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
35		- '/usr/share', '/usr/lib/debug',
36		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
37		- '/usr/lib/debug/usr/bin/hello',
	35	+ '/usr/share',
38	36	'/usr/bin/hello', '/usr/share/doc',
39	37	'/usr/share/doc/amhello',
40	38	'/usr/share/doc/amhello/README'])
...	...	@@ -57,9 +55,7 @@ def test_autotools_confroot_build(cli, tmpdir, datafiles):
57	55	assert result.exit_code == 0
58	56
59	57	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
60		- '/usr/share', '/usr/lib/debug',
61		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
62		- '/usr/lib/debug/usr/bin/hello',
	58	+ '/usr/share',
63	59	'/usr/bin/hello', '/usr/share/doc',
64	60	'/usr/share/doc/amhello',
65	61	'/usr/share/doc/amhello/README'])

tests/integration/cmake.py

---

...	...	@@ -28,10 +28,7 @@ def test_cmake_build(cli, tmpdir, datafiles):
28	28	result = cli.run(project=project, args=['checkout', element_name, checkout])
29	29	assert result.exit_code == 0
30	30
31		- assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello',
32		- '/usr/lib/debug', '/usr/lib/debug/usr',
33		- '/usr/lib/debug/usr/bin',
34		- '/usr/lib/debug/usr/bin/hello'])
	31	+ assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello'])
35	32
36	33
37	34	@pytest.mark.datafiles(DATA_DIR)
...	...	@@ -47,10 +44,7 @@ def test_cmake_confroot_build(cli, tmpdir, datafiles):
47	44	result = cli.run(project=project, args=['checkout', element_name, checkout])
48	45	assert result.exit_code == 0
49	46
50		- assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello',
51		- '/usr/lib/debug', '/usr/lib/debug/usr',
52		- '/usr/lib/debug/usr/bin',
53		- '/usr/lib/debug/usr/bin/hello'])
	47	+ assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello'])
54	48
55	49
56	50	@pytest.mark.datafiles(DATA_DIR)

tests/integration/compose.py

---

...	...	@@ -38,48 +38,40 @@ def create_compose_element(name, path, config={}):
38	38	@pytest.mark.datafiles(DATA_DIR)
39	39	@pytest.mark.parametrize("include_domains,exclude_domains,expected", [
40	40	# Test flat inclusion
41		- ([], [], ['/usr', '/usr/lib', '/usr/bin',
42		- '/usr/share', '/usr/lib/debug',
43		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
44		- '/usr/lib/debug/usr/bin/hello', '/usr/bin/hello',
	41	+ ([], [], ['/usr', '/usr/bin',
	42	+ '/usr/share',
	43	+ '/usr/bin/hello',
45	44	'/usr/share/doc', '/usr/share/doc/amhello',
46	45	'/usr/share/doc/amhello/README',
47	46	'/tests', '/tests/test']),
48	47	# Test only runtime
49		- (['runtime'], [], ['/usr', '/usr/lib', '/usr/share',
	48	+ (['runtime'], [], ['/usr', '/usr/share',
50	49	'/usr/bin', '/usr/bin/hello']),
51	50	# Test with runtime and doc
52		- (['runtime', 'doc'], [], ['/usr', '/usr/lib', '/usr/share',
	51	+ (['runtime', 'doc'], [], ['/usr', '/usr/share',
53	52	'/usr/bin', '/usr/bin/hello',
54	53	'/usr/share/doc', '/usr/share/doc/amhello',
55	54	'/usr/share/doc/amhello/README']),
56	55	# Test with only runtime excluded
57		- ([], ['runtime'], ['/usr', '/usr/lib', '/usr/share',
58		- '/usr/lib/debug', '/usr/lib/debug/usr',
59		- '/usr/lib/debug/usr/bin',
60		- '/usr/lib/debug/usr/bin/hello',
	56	+ ([], ['runtime'], ['/usr', '/usr/share',
61	57	'/usr/share/doc', '/usr/share/doc/amhello',
62	58	'/usr/share/doc/amhello/README',
63	59	'/tests', '/tests/test']),
64	60	# Test with runtime and doc excluded
65		- ([], ['runtime', 'doc'], ['/usr', '/usr/lib', '/usr/share',
66		- '/usr/lib/debug', '/usr/lib/debug/usr',
67		- '/usr/lib/debug/usr/bin',
68		- '/usr/lib/debug/usr/bin/hello',
	61	+ ([], ['runtime', 'doc'], ['/usr', '/usr/share',
69	62	'/tests', '/tests/test']),
70	63	# Test with runtime simultaneously in- and excluded
71		- (['runtime'], ['runtime'], ['/usr', '/usr/lib', '/usr/share']),
	64	+ (['runtime'], ['runtime'], ['/usr', '/usr/share']),
72	65	# Test with runtime included and doc excluded
73		- (['runtime'], ['doc'], ['/usr', '/usr/lib', '/usr/share',
	66	+ (['runtime'], ['doc'], ['/usr', '/usr/share',
74	67	'/usr/bin', '/usr/bin/hello']),
75	68	# Test including a custom 'test' domain
76		- (['test'], [], ['/usr', '/usr/lib', '/usr/share',
	69	+ (['test'], [], ['/usr', '/usr/share',
77	70	'/tests', '/tests/test']),
78	71	# Test excluding a custom 'test' domain
79		- ([], ['test'], ['/usr', '/usr/lib', '/usr/bin',
80		- '/usr/share', '/usr/lib/debug',
81		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
82		- '/usr/lib/debug/usr/bin/hello', '/usr/bin/hello',
	72	+ ([], ['test'], ['/usr', '/usr/bin',
	73	+ '/usr/share',
	74	+ '/usr/bin/hello',
83	75	'/usr/share/doc', '/usr/share/doc/amhello',
84	76	'/usr/share/doc/amhello/README'])
85	77	])

tests/sources/git.py

---

...	...	@@ -23,6 +23,7 @@
23	23	import os
24	24	import pytest
25	25	import subprocess
	26	+import shutil
26	27
27	28	from buildstream._exceptions import ErrorDomain
28	29	from buildstream import _yaml
...	...	@@ -920,3 +921,100 @@ def test_default_do_not_track_tags(cli, tmpdir, datafiles):
920	921
921	922	element = _yaml.load(element_path)
922	923	assert 'tags' not in element['sources'][0]
	924	+
	925	+
	926	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	927	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	928	+def test_overwrite_rogue_tag_multiple_remotes(cli, tmpdir, datafiles):
	929	+ """When using multiple remotes in cache (i.e. when using aliases), we
	930	+ need to make sure we override tags. This is not allowed to fetch
	931	+ tags that were present from different origins
	932	+ """
	933	+
	934	+ project = str(datafiles)
	935	+
	936	+ repofiles = os.path.join(str(tmpdir), 'repofiles')
	937	+ os.makedirs(repofiles, exist_ok=True)
	938	+ file0 = os.path.join(repofiles, 'file0')
	939	+ with open(file0, 'w') as f:
	940	+ f.write('test\n')
	941	+
	942	+ repo = create_repo('git', str(tmpdir))
	943	+
	944	+ top_commit = repo.create(repofiles)
	945	+
	946	+ repodir, reponame = os.path.split(repo.repo)
	947	+ project_config = _yaml.load(os.path.join(project, 'project.conf'))
	948	+ project_config['aliases'] = {
	949	+ 'repo': 'http://example.com/'
	950	+ }
	951	+ project_config['mirrors'] = [
	952	+ {
	953	+ 'name': 'middle-earth',
	954	+ 'aliases': {
	955	+ 'repo': ['file://{}/'.format(repodir)]
	956	+ }
	957	+ }
	958	+ ]
	959	+ _yaml.dump(_yaml.node_sanitize(project_config), os.path.join(project, 'project.conf'))
	960	+
	961	+ repo.add_annotated_tag('tag', 'tag')
	962	+
	963	+ file1 = os.path.join(repofiles, 'file1')
	964	+ with open(file1, 'w') as f:
	965	+ f.write('test\n')
	966	+
	967	+ ref = repo.add_file(file1)
	968	+
	969	+ config = repo.source_config(ref=ref)
	970	+ del config['track']
	971	+ config['url'] = 'repo:{}'.format(reponame)
	972	+
	973	+ # Write out our test target
	974	+ element = {
	975	+ 'kind': 'import',
	976	+ 'sources': [
	977	+ config
	978	+ ],
	979	+ }
	980	+ element_path = os.path.join(project, 'target.bst')
	981	+ _yaml.dump(element, element_path)
	982	+
	983	+ result = cli.run(project=project, args=['build', 'target.bst'])
	984	+ result.assert_success()
	985	+
	986	+ repo.checkout(top_commit)
	987	+
	988	+ file2 = os.path.join(repofiles, 'file2')
	989	+ with open(file2, 'w') as f:
	990	+ f.write('test\n')
	991	+
	992	+ new_ref = repo.add_file(file2)
	993	+
	994	+ repo.delete_tag('tag')
	995	+ repo.add_annotated_tag('tag', 'tag')
	996	+ repo.checkout('master')
	997	+
	998	+ otherpath = os.path.join(str(tmpdir), 'other_path')
	999	+ shutil.copytree(repo.repo,
	1000	+ os.path.join(otherpath, 'repo'))
	1001	+ new_repo = create_repo('git', otherpath)
	1002	+
	1003	+ repodir, reponame = os.path.split(repo.repo)
	1004	+
	1005	+ _yaml.dump(_yaml.node_sanitize(project_config), os.path.join(project, 'project.conf'))
	1006	+
	1007	+ config = repo.source_config(ref=new_ref)
	1008	+ del config['track']
	1009	+ config['url'] = 'repo:{}'.format(reponame)
	1010	+
	1011	+ element = {
	1012	+ 'kind': 'import',
	1013	+ 'sources': [
	1014	+ config
	1015	+ ],
	1016	+ }
	1017	+ _yaml.dump(element, element_path)
	1018	+
	1019	+ result = cli.run(project=project, args=['build', 'target.bst'])
	1020	+ result.assert_success()

tests/testutils/repo/git.py

---

...	...	@@ -99,12 +99,15 @@ class Git(Repo):
99	99	return config
100	100
101	101	def latest_commit(self):
102		- output = self._run_git('rev-parse', 'master', stdout=subprocess.PIPE).stdout
	102	+ output = self._run_git('rev-parse', 'HEAD', stdout=subprocess.PIPE).stdout
103	103	return output.decode('UTF-8').strip()
104	104
105	105	def branch(self, branch_name):
106	106	self._run_git('checkout', '-b', branch_name)
107	107
	108	+ def delete_tag(self, tag_name):
	109	+ self._run_git('tag', '-d', tag_name)
	110	+
108	111	def checkout(self, commit):
109	112	self._run_git('checkout', commit)
110	113

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.