mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
Martin VogelandAnand Aiyer
18edfa0023
vendored/nomic/code_vectors_blob.S is a GENERATED file that was hand-edited.
1f674c80 added the ELF .note.GNU-stack section to it after finding that every
Linux binary we had ever shipped requested an executable stack: an object
carrying no such note tells GNU ld nothing about its stack requirement, and ld
then assumes the worst for the entire link. That fix went into the artifact
only. scripts/extract_nomic_vectors.py still emitted the Mach-O branch alone,
so the next regeneration would have overwritten the tracked .S, dropped both
the ELF and COFF branches, and put GNU_STACK RWE back into every release --
silently, because nothing compared the generator against its own output.
write_blob_s() now emits the tracked wrapper verbatim, and the new contract
test fails if the two ever drift again. The template is compared rather than
executed so the test does not need the torch/transformers import the extraction
script pulls in. The write is pinned to UTF-8 because the WHY comment carries
an em dash and the default encoding is locale-dependent.
check-binary-composition.sh already fails a release whose binary has an
executable stack; this closes the same hole at edit time, where the diff is
still small enough to read.
Reported and originally fixed by Anand Aiyer in #1151, which also carried the
COFF branch. That PR sat unreviewed for a month while the .S half was
rediscovered and landed independently without credit.
Co-authored-by: Anand Aiyer <anand@aiyer.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
67 lines
2.8 KiB
Bash
67 lines
2.8 KiB
Bash
#!/usr/bin/env bash
|
|
# Contract: scripts/extract_nomic_vectors.py must still emit the tracked
|
|
# vendored/nomic/code_vectors_blob.S byte for byte.
|
|
#
|
|
# WHY this exists: the .S is a GENERATED file that was hand-edited to add the
|
|
# ELF .note.GNU-stack section. Without that note GNU ld assumes the whole link
|
|
# needs an executable stack, and every Linux release binary shipped GNU_STACK
|
|
# RWE because of the omission. The generator kept emitting only the Mach-O
|
|
# branch, so the next regeneration would have silently reverted the hardening
|
|
# and reintroduced the defect. check-binary-composition.sh catches that at
|
|
# release time; this catches it at edit time.
|
|
set -euo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
GEN="$ROOT/scripts/extract_nomic_vectors.py"
|
|
TRACKED="$ROOT/vendored/nomic/code_vectors_blob.S"
|
|
|
|
for f in "$GEN" "$TRACKED"; do
|
|
if [ ! -f "$f" ]; then
|
|
echo "FAIL: missing $f" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
# The generator holds the .S as an f-string whose only placeholder is the
|
|
# incbin path, so the tracked file with that path re-parameterized must appear
|
|
# in it verbatim. Comparing the template rather than running the generator
|
|
# keeps this test free of the torch/transformers import the script needs.
|
|
python3 - "$GEN" "$TRACKED" <<'PYEOF'
|
|
import sys
|
|
|
|
gen_path, tracked_path = sys.argv[1], sys.argv[2]
|
|
|
|
with open(gen_path, encoding="utf-8") as fh:
|
|
generator = fh.read()
|
|
with open(tracked_path, encoding="utf-8") as fh:
|
|
tracked = fh.read()
|
|
|
|
INCBIN = "vendored/nomic/code_vectors.bin"
|
|
if INCBIN not in tracked:
|
|
sys.exit("FAIL: %s no longer references %s" % (tracked_path, INCBIN))
|
|
if "{" in tracked or "}" in tracked:
|
|
sys.exit("FAIL: %s gained a brace; the generator f-string would need "
|
|
"escaping and this contract can no longer compare them directly"
|
|
% tracked_path)
|
|
|
|
template = tracked.replace(INCBIN, "{incbin_path}")
|
|
|
|
if template not in generator:
|
|
sys.exit(
|
|
"FAIL: scripts/extract_nomic_vectors.py no longer emits the tracked\n"
|
|
" vendored/nomic/code_vectors_blob.S.\n"
|
|
" Regenerating the blob wrapper would overwrite the tracked file\n"
|
|
" with different content. Update write_blob_s() so its f-string\n"
|
|
" matches the tracked .S with the incbin path parameterized.")
|
|
|
|
# The note is the whole point; assert it explicitly so a future edit that keeps
|
|
# the files in sync while dropping the note still fails loudly.
|
|
NOTE = '.section .note.GNU-stack,"",@progbits'
|
|
for label, blob in (("tracked .S", tracked), ("generator template", generator)):
|
|
if NOTE not in blob:
|
|
sys.exit("FAIL: %s lost the ELF %s section -- Linux binaries would ship "
|
|
"an executable stack again" % (label, NOTE))
|
|
|
|
print("ok: generator template is byte-identical to the tracked blob wrapper")
|
|
PYEOF
|