nprasad2077/airflow
1
0 Fork 0
mirror of https://github.com/apache/airflow.git synced 2026-11-04 07:12:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
5308480f2eeb3f006bd92c7f748a48d38f4df0c8
airflow/airflow-core/.pre-commit-config.yaml
T
Jarek PotiukandWei Lee 0cde487d42 Stop parallel prek hooks racing on __pycache__ cleanup (#73823) 
* Stop parallel prek hooks racing on __pycache__ cleanup

prek runs hooks in parallel and every breeze invocation removes all
__pycache__ directories in the checkout on start-up. Any other hook
process importing from the source tree (breeze itself, prek hook
scripts, local mypy runs) can write a fresh .pyc between rmtree emptying
a directory and removing it, which fails the breeze hook with
"OSError: [Errno 39] Directory not empty" - a flaky Static checks
failure in CI.

Redirecting bytecode of every hook into a hidden, gitignored .build/
directory keeps the source tree free of new .pyc files while hooks run,
and a new check keeps future hooks from reintroducing the race. The
cleanup additionally tolerates ENOTEMPTY for writers prek does not
control, such as an IDE.

Generated-by: Claude Opus 5

* Set PYTHONPYCACHEPREFIX on the hooks added since the branch was cut

Eight hooks that landed on main after this branch was created did not redirect their bytecode, so the new check-prek-hooks-pycache-prefix hook failed after the rebase.

Generated-by: Claude Opus 5

* Update scripts/ci/prek/check_prek_hooks_pycache_prefix.py

Co-authored-by: Wei Lee <hello@wei-lee.me>

---------

Co-authored-by: Wei Lee <hello@wei-lee.me>
2026-10-21 00:58:38 +02:00

392 lines
16 KiB
YAML
Raw Blame History

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
---
default_stages: [pre-commit, pre-push]
minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
golang: 1.24.0
repos:
- repo: local
hooks:
- id: check-first-limit
name: Check single-row lookups have an explicit limit
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_first_limit.py
language: python
files: ^src/airflow/.*\.py$
pass_filenames: true
- id: check-airflow-k8s-not-used
name: Check airflow.kubernetes imports are not used
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
files: ^src/airflow/.*\.py$
require_serial: true
exclude: ^src/airflow/kubernetes/
entry: ../scripts/ci/prek/check_airflow_imports.py
--pattern '^airflow\.kubernetes'
--message "You should only import kubernetes code from `airflow.providers.cncf.kubernetes`."
- id: check-common-compat-used-for-openlineage
name: Check common.compat is used for OL deprecated classes
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
files: ^src/airflow/.*\.py$
require_serial: true
exclude: ^src/airflow/openlineage/.*
entry: ../scripts/ci/prek/check_airflow_imports.py
--pattern '^openlineage\.client\.(facet|run)'
--message "You should import from `airflow.providers.common.compat.openlineage.facet` instead."
- id: check-cncf-k8s-only-for-executors
name: Check cncf.kubernetes imports used for executors only
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
files: ^src/airflow/.*\.py$
require_serial: true
exclude: >
(?x)
^src/airflow/exceptions\.py$|
^src/airflow/models/renderedtifields\.py$|
^src/airflow/serialization/serialized_objects\.py$|
^src/airflow/serialization/serializers/kubernetes\.py$|
^src/airflow/utils/sqlalchemy\.py$
entry: ../scripts/ci/prek/check_airflow_imports.py
--pattern '^airflow\.providers\.cncf\.kubernetes'
--message
"Only few k8s executors exceptions are allowed to use `airflow.providers.cncf.kubernetes`."
- id: check-extra-packages-references
name: Checks setup extra packages
env:
PYTHONPYCACHEPREFIX: .build/pycache
description: Checks if extras defined in hatch_build.py are listed in extra-packages-ref.rst file
language: python
files: ^docs/extra-packages-ref\.rst$|^hatch_build\.py$
pass_filenames: false
entry: ../scripts/ci/prek/check_extra_packages_ref.py
- id: check-no-providers-in-core-examples
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: pygrep
name: No providers imports in core example DAGs
description: The core example DAGs have no dependencies other than standard provider or core Airflow
entry: "^\\s*from airflow\\.providers.(?!standard.)"
pass_filenames: true
files: ^src/airflow/example_dags/.*\.py$
- id: check-only-new-session-with-provide-session
name: Check NEW_SESSION is only used with @provide_session
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/new_session_in_provide_session.py
pass_filenames: true
files: ^src/airflow/.+\.py$
exclude: ^src/airflow/serialization/pydantic/.*
- id: check-base-operator-partial-arguments
name: Check BaseOperator and partial() arguments
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/check_base_operator_partial_arguments.py
pass_filenames: false
files: ^src/airflow/models/(?:base|mapped)operator\.py$
- id: check-decorated-operator-implements-custom-name
name: Check @task decorator implements custom_operator_name
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/decorator_operator_implements_custom_name.py
pass_filenames: true
files: ^src/airflow/.*\.py$
- id: compile-ui-assets
name: Compile ui assets (manual)
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: node
stages: ['manual']
types_or: [javascript, ts, tsx]
files:
(?x)
^src/airflow/ui/|
^src/airflow/api_fastapi/auth/managers/simple/ui/
entry: ../scripts/ci/prek/compile_ui_assets.py
pass_filenames: false
additional_dependencies: ['pnpm@10.25.0']
- id: compile-ui-assets-dev
name: Compile ui assets in dev mode (manual)
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: node
stages: ['manual']
types_or: [javascript, ts, tsx]
files:
(?x)
^src/airflow/ui/|
^src/airflow/api_fastapi/auth/managers/simple/ui/
entry: ../scripts/ci/prek/compile_ui_assets_dev.py
pass_filenames: false
additional_dependencies: ['pnpm@10.25.0']
- id: check-tests-in-the-right-folders
name: Check if tests are in the right folders
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_tests_in_right_folders.py
language: python
files: ^tests/.*\.py$
pass_filenames: true
require_serial: true
- id: lint-json-schema
name: Lint config_templates/config.yml
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/lint_json_schema.py
args:
- --spec-file
- src/airflow/config_templates/config.yml.schema.json
language: python
pass_filenames: true
files: ^src/airflow/config_templates/config\.yml$
require_serial: true
- id: check-code-deprecations
name: Check deprecations categories in decorators
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_deprecations.py
language: python
pass_filenames: true
files: ^src/airflow/.*\.py$
- id: check-http-exception-import-from-fastapi
name: Check HTTPException is imported from fastapi
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_http_exception_import_from_fastapi.py
language: python
pass_filenames: true
files: >
(?x)
^src/airflow/api_fastapi/.*\.py$|
^src/airflow/utils/serve_logs/.*\.py$|
^tests/unit/api_fastapi/.*\.py$|
^tests/unit/utils/test_serve_logs\.py$
- id: check-openapi-exception-doc-in-sync
name: Check API routes declare the HTTP statuses they raise
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_openapi_exception_doc_in_sync.py
language: python
pass_filenames: true
files: ^src/airflow/api_fastapi/.*/routes/.*\.py$
- id: create-missing-init-py-files-tests
name: Create missing init.py files in tests
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_init_in_tests.py
language: python
pass_filenames: false
files: ^tests/.*\.py$
- id: check-tests-unittest-testcase
name: Unit tests do not inherit from unittest.TestCase
env:
PYTHONPYCACHEPREFIX: .build/pycache
description: Check that unit tests do not inherit from unittest.TestCase
entry: ../scripts/ci/prek/unittest_testcase.py
language: python
pass_filenames: true
files: ^tests/.*\.py$
- id: check-no-fab-migrations
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: pygrep
name: Check no migration is done on FAB related table
description: >
FAB tables are no longer used in core Airflow but in FAB provider.
As such, it is forbidden to create migrations related to FAB tables in core Airflow.
Such migrations should be in FAB provider. To achieve this, a new capability must be implemented:
support migrations for providers. In other words, providers need to be able to specify migrations
so that, any FAB related migration (besides the legacy ones) is defined in FAB provider.
See https://github.com/apache/airflow/issues/32210
entry: >
(?ix)
\bab_permission\b|
\bab_view_menu\b|
\bab_role\b|
\bab_permission_view\b|
\bab_permission_view_role\b|
\bab_user\b|
\bab_user_role\b|
\bab_register_user\b
pass_filenames: true
files: ^src/airflow/migrations/versions/.*\.py$
exclude:
(?x)
^src/airflow/migrations/versions/0000_2_6_2_squashed_migrations\.py$|
^src/airflow/migrations/versions/0028_3_0_0_drop_ab_user_id_foreign_key\.py$
- id: ts-compile-lint-ui
name: Compile / format / lint UI
env:
PYTHONPYCACHEPREFIX: .build/pycache
description: TS types generation / ESLint / Prettier new UI files
language: node
files: |
(?x)
^src/airflow/ui/.*\.(js|ts|tsx|yaml|css|json)$|
^src/airflow/api_fastapi/core_api/openapi/.*\.yaml$
exclude: |
(?x)
^src/airflow/ui/node-modules/.*|
^src/airflow/ui/.pnpm-store
entry: ../scripts/ci/prek/ts_compile_lint_ui.py
additional_dependencies: ['pnpm@10.25.0']
pass_filenames: true
require_serial: true
- id: ts-compile-lint-simple-auth-manager-ui
name: Compile / format / lint simple auth manager UI
env:
PYTHONPYCACHEPREFIX: .build/pycache
description: TS types generation / ESLint / Prettier new UI files
language: node
files: |
(?x)
^src/airflow/api_fastapi/auth/managers/simple/ui/.*\.(js|ts|tsx|yaml|css|json)$|
^src/airflow/api_fastapi/core_api/openapi/.*\.yaml$|
^src/airflow/api_fastapi/auth/managers/simple/openapi/.*\.yaml$
exclude: |
(?x)
^src/airflow/api_fastapi/node-modules/.*|
^src/airflow/api_fastapi/.pnpm-store
entry: ../scripts/ci/prek/ts_compile_lint_simple_auth_manager_ui.py
additional_dependencies: ['pnpm@10.25.0']
pass_filenames: true
require_serial: true
- id: mypy-airflow-core
name: Run mypy for airflow-core
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py airflow-core
pass_filenames: false
files: ^.*\.py$
require_serial: true
## ADD MOST PREK HOOK ABOVE THAT LINE
# The below prek hooks are those requiring CI image to be built
- id: generate-openapi-spec
name: Generate the FastAPI API spec
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/generate_openapi_spec.py
pass_filenames: false
files:
(?x)
^src/airflow/api_fastapi/.*\.py$|
^src/airflow/api_fastapi/auth/managers/simple/.*\.py$
exclude: ^src/airflow/api_fastapi/execution_api/.*
- id: check-i18n-json
name: Check i18n files validity
env:
PYTHONPYCACHEPREFIX: .build/pycache
description: Check i18n files are valid json, have no TODOs, and auto-format them
language: python
files: ^src/airflow/ui/public/i18n/locales/.*\.json$
entry: ../scripts/ci/prek/check_i18n_json.py
pass_filenames: false
- id: update-migration-references
name: Update migration ref doc
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/migration_reference.py
args: ["--app", "airflow"]
pass_filenames: false
files:
(?x)
^src/airflow/migrations/versions/.*\.py$|
^docs/migrations-ref\.rst$
- id: migration-round-trip
name: Migration round-trip SQLite FK enforcement
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: python
entry: ../scripts/ci/prek/migration_round_trip.py
pass_filenames: false
require_serial: true
stages: [manual]
files: ^src/airflow/migrations/versions/.*\.py$
- id: check-default-configuration
name: Check the default configuration
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_default_configuration.py
language: python
require_serial: true
pass_filenames: false
files: ^src/airflow/config_templates/config\.yml$
- id: check-security-doc-constants
name: Check security docs match config.yml constants
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_security_doc_constants.py
language: python
pass_filenames: false
files: >
(?x)
^src/airflow/config_templates/config\.yml$|
^docs/security/jwt_token_authentication\.rst$|
^docs/security/security_model\.rst$
- id: check-airflow-version-checks-in-core
env:
PYTHONPYCACHEPREFIX: .build/pycache
language: pygrep
name: No AIRFLOW_V_* imports in airflow-core
entry: "import AIRFLOW_V_"
files: ^.*\.py$
pass_filenames: true
# TODO (@amoghrajesh): revisit last few in this list as they all rely on versioned
# secrets masker imports
exclude: >
(?x)
^tests/integration/otel/dags/otel_test_dag_with_pause_between_tasks\.py$|
^tests/integration/otel/dags/otel_test_dag_with_pause_in_task\.py$|
^tests/integration/otel/test_otel\.py$|
^tests/unit/core/test_configuration\.py$|
^tests/unit/models/test_renderedtifields\.py$|
^tests/unit/models/test_variable\.py$
- id: check-sdk-imports-in-core
name: Check for SDK imports in core files
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_sdk_imports_in_core.py
language: python
types: [python]
files: ^src/airflow/
## ONLY ADD PREK HOOKS HERE THAT REQUIRE CI IMAGE
- id: check-schema-defaults
name: Check schema defaults match server-side defaults
env:
PYTHONPYCACHEPREFIX: .build/pycache
entry: ../scripts/ci/prek/check_schema_defaults.py
language: python
files:
(?x)
^src/airflow/serialization/schema\.json$|
^src/airflow/serialization/serialized_objects\.py$
pass_filenames: false
require_serial: true
Reference in new issue View Git Blame Copy Permalink