* Update docs for the Python 3.11 baseline
Contributing docs, Breeze docs, Docker stack docs and the user-facing
installation pages used Python 3.10 as the example and default
version. Examples and defaults now use 3.11; historical changelog
entries and ADRs keep their original wording.
* Re-pad the image naming table after dropping Python 3.10
* Fix stale Python 3.10 examples left in the Python 3.11 docs update
Review found commands that still pin a 3.10.x interpreter, which can no
longer install Breeze or Airflow, shell prompts and pytest transcripts
that still show 3.10, and a note that tied the supported Python range to
Airflow 3.2.0 even though the 3.2 to 3.4 release lines still accept 3.10.
Two Breeze examples used options that do not exist. The python-client
release notes presented `breeze --python 3.11` as equivalent to
`UV_PYTHON=3.11`, but the reproducibility guard checks the interpreter
Breeze itself runs on, so only the `UV_PYTHON` form works. The Docker
stack build examples now refer to the default Python version instead of
a hard-coded one, so they stop drifting with every bump.
* Fix remaining Python 3.10 references in docs and image examples
Review found more shell prompts that still show a 3.10 Breeze image.
The air-gapped image example pinned Airflow 2.5.3, which publishes no
3.11 constraints, so its download step fetched a 404 page and failed;
it also used extras that Airflow 3 no longer defines. The contributor
quick start called 3.10 end-of-life before its October 2026 end-of-life
date.
* Update Python version in database setup guide
Updated Python version in setup instructions from 3.8.10 to 3.11.16.
* Fix stale Python 3.10 constraints examples in INSTALL
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The INSTALL guide only covered Debian Bookworm, leaving Ubuntu Noble
users to guess the package names that differ between the two (libldap2
vs libldap-2.5-0). It also omitted the build headers needed to compile
optional dependencies from source. Document the Noble apt line and the
extra build packages so a source install works on both distributions.
Add a new Breeze CLI command that helps maintainers efficiently triage
open PRs from non-collaborators that don't meet minimum quality criteria.
The command fetches open PRs via GitHub GraphQL API with optimized chunked
queries, runs deterministic CI checks (failures, merge conflicts, missing
test workflows), optionally runs LLM-based quality assessment, and presents
flagged PRs interactively for maintainer review with author profiles and
contribution history.
Key features:
- Optimized GraphQL queries with chunking to avoid GitHub timeout errors
- Deterministic CI failure detection with categorized fix instructions
- LLM assessment via `claude` or `codex` CLI for content quality
- Interactive review with Rich panels, clickable links, and author context
- "maintainer-accepted" label to skip PRs on future runs
- Workflow approval support for first-time contributor PRs awaiting CI runs
- Merge conflict and commits-behind detection with rebase guidance
Update dev/breeze/src/airflow_breeze/commands/pr_commands.py
Update dev/breeze/src/airflow_breeze/utils/llm_utils.py
Update contributing-docs/05_pull_requests.rst
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There were a couple of problems with genearted documentation for
installation from sources for "apache-airflow" and "apache-airflow-ctl":
* apache-airflow-ctl links in source installation were broken
* apache-airflow did not have link to sdist and whl distributions
of task-sdk (which is currently released together with apache-airflow
and apache-airflow-core)
* airflow-ctl was missing in development index generated locally
* INSTALL that is common for all distribution sources only contained
information on building apache-airflow, but did not have
instrunctions on building other distributions
The pre-commit is a fantastic tool, and we heavily used it for years,
but generally the tool stagnated and is not showing a sign of adapting
to our needs. For years we tried to convince pre-commit maintainers that
things like autocomplete are necessary - but it met with pretty much
resistance (if not hostility) from the maintainer. Also there was no
chance for them to accept expectations of bigger projects like ours,
where we have a huge monorepo and not only multiple needs but also
different parts of the repo needing different language support (golang,
typescript soon) - and apparenty the maintainer of pre-commit does not
think monorepo is a good thing at all. Similarly - they did not recognize
the raise of `uv` and the only way to use `uv` with pre-commit is to patch
it by installing `pre-comit-uv` that essentialy patches pre-commit with
uv support. This is not really sustainable and the tool lags behind many
of our needs.
Luckily - we have new project in town - prek - which rewrites pre-commit
that is 100% compatible (now), 10x faster (because rust), uses `uv`
natively, supports auto-complete already and they have very friendly
maintainer who is not only supporting us but also very happily works
on improving `prek` to close all the gaps, and plans to implement (with
our support of course and cooperation) monorepo support - that will
allow us to modularise our pre-commits.
This PR switches our pre-commit support to use prek exclusively:
* breeze static checks command is completely removed
* custom auto-complete code in breeze as well
* instructions are updated to setup prek instead of precommit
* CI is updated to run prek instead of pre-commmit
* documentation for static checks is reviewed and new features that
prek enables are added
The lazy consensus decision has been made at the devlist to switch
entirely to `uv` as development tool:
link: https://lists.apache.org/thread/6xxdon9lmjx3xh8zw09xc5k9jxb2n256
This PR implements that decision and removes a lot of baggage connected
to using `pip` additionally to uv to install and sync the environment.
It also introduces more consistency in the way how distribution
packages are used in airflow sources - basicaly switching all internal
distributions to use `pyproject.toml` approach and linking them all
together via `uv`'s workspace feature.
This enables much more streamlined development workflows, where any
part of airflow development is manageable using `uv sync` in the right
distribution - opening the way to moving more of the "sub-worfklows"
from the CI image to local virtualenv environment.
Unfortunately, such change cannot be done incrementally, really, because
any change in the project layout drags with itself a lot of changes
in the test/CI/management scripts, so we have to implement one big
PR covering the move.
This PR is "safe" in terms of the airflow and provider's code - it
does not **really** (except occasional imports and type hint changes
resulting from better isolation of packages) change Airflow code nor
it should not affect any airflow or provider code, because it does
not move any of the folder where airflow or provider's code is modified.
It does move the test code - in a number of "auxiliary" distributions
we have. It also moves the `docs` generation code to `devel-common`
and introduces separate conf.py files for every doc package.
What is still NOT done after that move and will be covered in the
follow-up changes:
* isolating docs-building to have separate configuraiton for docs
building per distribution - allowing to run doc build locally
with it's own conf.py file
* moving some of the tests and checks out from breeze container
image up to the local environment (for example mypy checks) and
likely isolating them per-provider
* Constraints are still generated using `pip freeze` and automatically
managed by our custom scripts in `canary` builds - this will be
replaced later by switching to `uv.lock` mechanism.
* potentially, we could merge `devel-common` and `dev` - to be
considered as a follow-up.
* PROD image is stil build with `pip` by default when using
`PyPI` or distribution packages - but we do not support building
the source image with `pip` - when building from sources, uv
is forced internally to install packages. Currently we have
no plans to change default PROD building to use `uv`.
This is the detailed list of changes implemented in this PR:
* uv is now mandatory to install as pre-requisite in order to
develop airflow. We do not support installing airflow for
development with `pip` - there will be a lot of cases where
it will not work for development - including development
dependencies and installing several distributions together.
* removed meta-package `hatch_build.py' and replacing it with
pre-commit automatically modifying declarative pyproject.toml
* stripped down `hatch_build_airflow_core.py` to only cover custom
git and asset build hooks (and renaming the file to `hatch_build.py`
and moving all airflow dependencies to `pyproject.toml`
* converted "loose" packages in airflow repo into distributions:
* docker-tests
* kubernetes-tests
* helm-tests
* dev (here we do not have `src` subfolder - sources are directly
in the distribution, which is for-now inconsistent with other
distributions).
The names of the `_tests` distribution folders have been renamed to
the `-tests` convention to make sure the imports are always
referring to base of each distribution and are not used from the
content root.
* Each eof the distributions (on top of already existing airflow-core,
task-sdk, devel-common and 90+providers has it's own set of
dependencies, and the top-level meta-package workspace root brings
those distributions together allowing to install them all tegether
with a simple `uv sync --all-packages` command and come up with
consistent set of dependencies that are good for all those
packages (yay!). This is used to build CI image with single
common environment to run the tests (with some quirks due to
constraints use where we have to manually list all distributions
until we switch to `uv.lock` mechanism)
* `doc` code is moved to `devel-common` distribution. The `doc` folder
only keeps README informing where the other doc code is, the
spelling_wordlist.txt and start_docs_server.sh. The documentation is
generated in `generated/generated-docs/` folder which is entirely
.gitignored.
* the documentation is now fully moved to:
* `airflow-core/docs` - documentation for Airflow Core
* `providers/**/docs` - documentation for Providers
* `chart/docs` - documentation for Helm Chart
* `task-sdk/docs` - documentation for Task SDK (new format not yet published)
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
* `versions` are not dynamically retrieved from `__init__.py` all
of them are synchronized directly to pyproject.toml files - this
way - except the custom build hook - we have no dynamic components
in our `pyproject.toml` properties.
* references to extras were removed from INSTALL and other places,
the only references to extras remains in the user documentation - we
stop using extras for local development, we switch to using
dependency groups.
* backtracking command was removed from breeze - we did not need it
since we started using `uv`
* internal commands (except constraint generation) have been moved to
`uv` from `pip`
* breeze requires `uv` to be installed and expects to be installed by
`uv tool install -e ./dev/breeze`
* pyproject.tomls are dynamically modified when we add a version
suffix dynamically (`--version-suffix-for-pypi`) - only for the
time of building the versions with updated suffix
* `mypy` checks are now consistently used across all the different
distributions and for consistency (and to fix some of the issues
with namespace packages) rather than using "folder" approach
when running mypy checks, even if we run mypy for whole
distribution, we run check on individual files rather than on
a folder. That adds consistency in execution of mypy heursistics.
Rather than using in-container mypy script all the logic of
selection and parameters passed to mypy are in pre-commit code.
For now we are still using CI image to run mypy because mypy is
very sensitive to version of dependencies installed, we should
be able to switch to running mypy locally once we have the
`uv.lock` mechanism incorporated in our workflows.
* lower bounds for dependencies have been set consistently across
all the distributions. With `uv sync` and dependabot, those
should be generally kept consistently for the future
* the `devel-common` dependencies have been groupped together in
`devel-common` extras - including `basic`, `doc`, `doc-gen`, and
`all` which will make it easier to install them for some OS-es
(basic is used as default set of dependencies to cover most
common set of development dependencies to be used for development)
* generated/provider_dependencies.json are not committed to the
repository any longer. They are .gitignored and geberated
on-the-flight as needed (breeze will generate them automatically
when empty and pre-commit will always regenerate them to be
consistent with provider's pyproject.toml files.
* `chart-utils` have been noved to `helm-tests` from `devel-common`
as they were only used there.
* for k8s tests we are using the `uv` main `.venv` environment
rather than creating our own `.build` environment and we use
`uv sync` to keep it in sync
* Updated `uv` version to 0.6.10
* We are using `uv sync` to perform "upgrade to newer depencies"
in `canary` builds and locally
* leveldb has been turned into "dependency group" and removed from
apache-airflow and apache-airflow-core extras, it is now only
available by google provider's leveldb optional extra to install
with `pip`
* Move `GitDagBundle` to providers
This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.
* fixup! Move `GitDagBundle` to providers
* Update documentation
* Fix static checks
* add git to compatibility tests matrix
* Update doctree
* Update with recent changes
* Fix doc, update version and add provider to prod image preinstalled providers
* fixup! Fix doc, update version and add provider to prod image preinstalled providers
* Fix doc
* update breeze command hash
* Fix incompatible providers regex removal syntax
* Remove providers mistakenly added in pyproject.toml
* Update providers/git/pyproject.toml
This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:
* apache-airflow - becomes an empty, meta no-code distribution that
only has dependencies to apache-airflow-core and task-sdk
distributions and it has preinstalled provider distributions
added in standard "wheel" distribution. All "extras" lead
either to "apache-airflow-core" extras or to providers - the
dependencies and optional dependencies are calculated differently
depending on "editable" or "standard" mode - in editable mode,
just provider dependencies are installed for preinstalled providers
in standard mode - those preinstalled providers are dependencies.
* the apache-airflow-core distribution contains all airflow core
sources (previously in apache-airflow) and it has no provider
extras. Thanks to that apache-airflow distribution does not
have any dynamically calculated dependencies.
* the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
build hooks that add custom build target and implement custom
cleanup in order to implement compiling assets as part of the build.
* During the move, the following changes were applied for consistency:
* packages when used in context of distribution packages have been
renamed to "distributions" - including all documentations and
commands in breeze to void confusion with import packages
(see
https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)
* all tests in `airflow-core` follow now the same convention
where tests are in `unit`, `system` and `integration` package.
no extra package has been as second level, because all the
provider tests have "<PROVIDER>" there, so we just have to avoid
naming airflow unit."<PROVIDER>" with the same name as provider.
* all tooling in CI/DEV have been updated to follow the new
structure. We should always build to packages now when we
are building them using `breeze`.
The change #47281 introduced ``devel-common`` and simplified
hatch_build.py with removal of unneeded bundle extras, but it
removed ``all-core`` extra that was actually used in constraint
generation.
This is a follow-up for it, fixing the problem and restoring
``all-core`` - also it synchronizes documentation and adds
newsfragment describing the removal and explaining how users
who used ``all`` extra before (for example in their ``CI``)
with ``uv pip install --all-extras``.
This PR moves all commo tests to a "devel-common" sub-project in
Airlfow repo.
This project is now part of the airflow's workspace, which means
that it is installed by default when `uv sync` is run.
The following changes have been implemented:
* removed pytest options from providers and moved to common
pyproject.toml
* devel-common now keeps all the test dependencies that are used by
other packages - they are automatically installed when `uv sync`
is run. We do not need devel-tests extra any more
* providers_src_folder fixture was effectively not used and replaced
by __file__ deriving the source of where the appropriate package
is imported from.
* pytest init options are removed from provider's pyproject.toml because
they overwrote the options defined in the main pyproject.toml. Instead
--ignore-glob command line is passed to addopts
* devel dependencies from task_sdk are removed and devel-common package
is added as required dependency as well as standard and common.sql
package - both needed to run task_sdk tests. This allows to run tests
in task_sdk tests from withing task_sdk package.
* all dev dependency group settings for all providers contain the
dependent sources thaat allow to treat each provider separately.
* all devel and bundle dependencies and deprecated dependencies have
been removed complely - the devel dependencies have been incorporated
into "devel-common" distribution - so you can install all development
dependencies (except docs) by installing `devel-common`. Doc deps
will be later extracted to a separate distriubution.
This is provider package similar to those and is intended to be an abstraction over Apache Kafka, Amazon SQL, and Google PubSub to begin with. It can then be expanded to other messaging systems based on community adoption.
The initial goal with this is to provide a simple abstraction for integrating Event Driven Scheduling coming with Airflow 3 to message notification systems such as Kafka, currently being used to publish data availability.
---------
Co-authored-by: vincbeck <vincbeck@amazon.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
The ``pipx`` has still been used and recommended in a few places
in our documentation and in a few scripts/github actions.
We also installed pipx in the container image to be able to run
mmsql-cli, but this is not needed any more as mssql is not
supported as metadata backend.
This PR updates the remaining parts.
The PythonVirtualenvOperator has been using virtualenv in order
to support Python 2.7 and pre Python 3.5, however we do not need
those any more - and we can use built-in venv instead.
Fixes: #40420
* Add simple instructions for installing providers in editable mode
After #42505, you need to get through extra hoops to develop
providers in Airflow's monorepo.
This is a simple (not yet uv-specific) documentation on how to
install providers in editable mode when you want to develop
providers, so that you can run unit test.
Copied mostly from https://github.com/apache/airflow/pull/43082
It's not yet full set of docs explaining how to use workspaces
and UV. This shoudl be handled via #43200
* Apply suggestions from code review
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
---------
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
* add core.time provider
* add source-date-epoch
* change core to essentials
* revert external task sensor location
* add provider to airflow_providers_bug_report list
* change new provider name to standard
* change changelog
* add integration
* add standard provider to pre-install list
* revert hatch_build
* move examples back to airflow core
* fixing mock
* add title to commit
* fix time.rst
* change sensors example dags paths
* revert example get_test_run addition
* remove init
* revert howto docs
* fix docs
* update standard provider airflow minimum version, remove standarad from python 2.8 and 2.9 competability
* change provider as not-ready
* add changelog
* move how to docs to provider
* fix docs images
* fix import error
* revert global constant
* change time_sensors.py to time.py
* revert min version to be 2.10
* remove providers from 2.9 and 2.8
* revert global constant
* add operators guid to provider yaml
Amazon provider with auth manager requires SAML onelogin import and
it starts to be more and more problematic to skip the related tests
for compatibility.
It seems appropriate to move saml to be a required dependency of
Amazon provider in this case.
Since saml is only used by Amazon provider, we can also safely
remove optional extra for it.
Corrected several typographical and grammatical errors in the INSTALL documentation.
These changes improve the clarity and readability of the
instructions. Specific changes include correcting "CPyhon" to "CPython"
and standardizing the capitalization of "Airflow".
While currently hatchling and pip nicely supports dynamic replacement of
the dependencies even if they are statically defined, this is not proper
according to EP 621. When property of the project is set to be dynamic,
it also contains static values. It's either static or dynamic.
This is not a problem for wheel packages when installed, by any
standard tool, because the wheel package has all the metadata added
to wheel (and does not contain pyproject.toml) but in various cases
(such as installing airflow via Github URL or from sdist, it can
make a difference - depending whether the tool installing airflow will
use directly pyproject.toml for optimization, or whether it will run
build hooks to prepare the dependencies).
This change makes all optional dependencies dynamici - rather than
bake them in the pyproject.toml, we mark them as dynamic, so that
any tool that uses pyproject.toml or sdist PKG-INFO will know that
it has to run build hooks to get the actual optional dependencies.
There are a few consequences of that:
* our pyproject.toml will not contain automatically generated
part - which is actually good, as it caused some confusion
* all dynamic optional dependencies of ours are either present in
hatch_build.py or calculated there - this is a bit new
but sounds reasonable - and those dynamic dependencies are not
really updated often, so thish is not an issue to maintain them
there
* the pre-commits that manage the optional dependencies got a lot
simpler now - a lot of code has been removed.
This PR introduces a joint way to treat the .local (--user) folder
as both - venv and `--user` package installation. It fixes a number
of problems the `--user` installation created us in the past and
does it in fully backwards compatible way.
This improves both "production" use for end user as well as local
iteration on the PROD image during tests - but also for CI.
Improvements for "end user":
* user does not have to use `pip install --user` to install new
packages any more and it is not enabled by default with PIP_USER
flag.
* users can use uv to install packages when they extend the image
(but it's not obligatory - pip continues working as it did)
* users can use `uv` to build custom production image, which gives
40%-50% saving for image build time compring to `pip`.
* python -m venv --system-site-packages continues to use the
.local packages from the .local installation (and not uses
them if --system-site-packages is not used) - so we have full
compatibility with previous images.
Improvements for development:
* when image is built from sources (no --use-docker-context-files
are specified), airflow is installed in --editable mode, which
means that airflow + all providers are installed locally from
airflow sources, not from packages - which means that both
airflow and providers have the latest version inside the
prod image.
* when local sources changes and you want to run k8s tests locally,
it is now WAY faster (several minutes) to iterate with your changes
because you do not have to rebuild the base image - the only thing
needed is to copy sources to the PROD image to "/opt/airflow" which
is where editable installlation is done from. You only need to
rebuild the image if dependencies change.
* By default `uv` is used for local source build for k8s tests so
even if you have to rebuild it, it is way faster (60%-80%) during
iterating with the image.
CI/DEV tooling improvements:
* this PR switches to use `uv` by default for most prod images we
build in CI, but it adds a check if the image still builds with `pip`.
* we also switch to more PEP standard way of installing packages
from local filesystem (package-name @ file:///FILE)
Fixes: #37785Fixes: #37815
Update contributing-docs/testing/k8s_tests.rst
Update contributing-docs/testing/k8s_tests.rst
Update docs/docker-stack/build.rst
Update docs/docker-stack/build.rst
Update docs/docker-stack/build.rst
Update docs/docker-stack/build.rst
Update docs/docker-stack/build.rst
Update scripts/docker/install_airflow.sh
Update docs/docker-stack/changelog.rst
Update docs/docker-stack/build.rst
Co-authored-by: Niko Oliveira <onikolas@amazon.com>
We have a "gosu" binary installed in Airflow image but it does not
seem to be used and it brings in a number of security vulnerabilities
because gosu uses an older go stdlib library.
This PR removes the gosu binary from the image altogether.
We've been internally using pydantic for internal API and it caused
some compatibility issues, because Pydantic is so popular and currently
still users of Pydantic are somewhat split between Pydantic 1 and
Pydantic 2. The popularity of Pydantic works against us, and since we
are not yet using it in "production" (and in the future we will only
actually use it for Internal API), it seems that turning Pydantic into
an optional dependency is the best way we can proceed.
It's as simple as converting all the direct imports into a common util
imports that have a fallback mechanism when import is not found.
This should enable less conflicts when installing 3rd-party libraries
with Airflow.
Added test where pydantic is removed. Also made sure that the special
cases we have tests for run full suite of tests - non-db and db.
Following #36936 and the fact that GitHub stopped rendering big .rst
files, we also split CONTRIBUTING.rst into multiplet files. It will be
much easier to follow and it will render in GitHub.
This PR changes Airflow installation and build backend to use new
standard Python ways of building Python applications.
We've been trying to do it for quite a while. Airflow tranditionally
has been using complex and convoluted build process based on
setuptools and (extremely) custom setup.py file. It survived
migration to Airflow 2.0 and splitting Airlfow monorepo into
Airflow and Providers, adding pre-installed providers and switching
providers to use flit (and follow build standards).
So far tooling in Python ecosystme had not been able to fuflill our
needs and we refrained to develop our own tooling, but finally with
appearance of Hatch (managed by Python Packaging Authority) and
few recent advancements there we are finally able to swtich to
Python standard ways of managing project dependnecy configuration
and project build setup (with a few customizations).
This PR makes airflow build process follow those standard PEPs:
* Airflow has all build configuration stored in pyproject.toml
following PEP 518 which allows any fronted (`pip`, `poetry`,
`hatch`, `flit`, or whatever other frontend is used to
install required build dependendencies to install Airflow
locally and to build distribution pacakges (sdist/wheel)
* Hatchling backend follows PEP 517 for standard source tree and build
backend implementation that allows to execute the build in a
frontend-independent way
* We store all project metadata in pyprooject.toml - following
PEP 621 where all necessary project metadata components were
defined.
* We plug-in into Hatchling "editable build" hooks following
PEP 660. Hatchling internally builds editable wheel that
is used as ephemeral step and communication between backend
and frontend (and this ephemeral wheel is used to make
editable installation of the projeect - suitable for fast
iteration of code without reinstalling the package.
With Airflow having many provider packages in single source tree
where we want to be able to install and develop airflow and
providers together, this is not a small feat to implement the
case wher editable installation has to behave quite a bit
differently when it comes to packaging and dependencies for
editable install (when you want to edit sources directly) and
installable package (where you want to have separate Airflow
package and provider packages). Fortunately the standardisation
efforts in the Python Packaging community and tooling implementing
it had finally made it possible.
Some of the important ways bow this has been achieved:
* We continue using provider.yaml in providers as the single source
of trutgh for per-provider dependencies. We added a possibility
to specify "devel-dependencies" in provider.yaml so that all
per-provider dependencies in `generated/provider_dependencies.json`
and `pyproject.toml` are generated from those dependencies via
update-providers-dependencies pre-commit.
* Pyproject.toml is generally managed manually, but the part where
provider dependencies and bundle dependencies are used is
automatically updated by a pre-commit whenever provider
dependencies change. Those generated provider dependencies contain
just dependencies of providers - not the provider packages, but
in the final "standard" wheel file they are replaced with
"apache-airflow-providers-PROVIDER" dependencies - so that the
wheel package will only install the provider and use the
dependencies of that version of provider it installs.
* We are utilising custom hatchiling build hooks (PEP 660 standard)
that allow to modify 'standard' wheel package on-the-fly when
the wheel is being prepared by adding preinstalled package
dependencies (which are not needed in editable build) and by
removing all devel extras (that are not needed in the PyPI
distributed wheel package). This allows to solve the conundrum
of having different "editable" and "standard" behaviour while
keeping the same project specification in pyproject.toml.
* We added description of how `Hatch` can be employed as build
frontend in order to manage local virtualenv and install Airflow
in editable way easily - while keeping all properties of the
installed application (including working airflow cli and
package metadata discovery) as well as how to use PEP-standard
ways of bulding wheel and sdist packages.
* We have a custom step (following PEP-standards) to inject
airflow-specific build steps - compiling www assets and
generating git commit hash version to display it in the UI
* We also show how all this makes it possible to make it easy to
manage local virtualenvs and editable installations for Airflow
contributors - without vendor lock-in of the build tools as
by following standard PEPs Airflow can be locally and editably
installed by anyone using any build front-end tools following
the standards - whether you use `pip`, `poetry`, `Hatch`, `flit`
or any other frontent build tools, Airflow local installation
and package building will work the same way for all of them,
where both "editable" and "standard" package prepration is
managed by `hatchling` backend in the same way.
* Previously our extras contained a "." which is not normalized
name for extras - `pip` and other tools replaced it automatically
with `_'. This change updates the extra names to contain
'-' rather than '.' in the name, following PEP-685. This should be
fully backwards compatible, users will still be able to use "." but it
will be normalized to "-" in Airflow packages. This is also future
proof as it is expected that all package managers and tools
will eventually use PEP-685 applied to extras, even if currently
some of the tools (pip + setuptools) might generate warnings.
* Additionally, this change organizes the documentation around
the extras and dependencies, explaining the reasoning behind
all the different extras we have.
* As a bonus (and this is what we used to test it all) we are
documenting how to use Hatch frontend to:
* manage multiple Python installations
* manage multiple Pythob virtualenv environments
* build Airflow packages for release management
The `graphviz` dependency has been problematic as Airflow required
dependency - especially for ARM-based installations. Graphviz
packages require binary graphviz libraries - which is already a
limitation, but they also require to install graphviz Python
bindings to be build and installed. This does not work for older
Linux installation but - more importantly - when you try
to install Graphviz libraries for Python 3.8, 3.9 for ARM M1
MacBooks, the packages fail to install because Python bindings
compilation for M1 can only work for Python 3.10+.
There is not an easy solution for that except commenting out
graphviz dependency from setup.py, when you want to install Airflow
for Python 3.8, 3.9 for MacBook M1.
However Graphviz is really used in two places:
* when you want to render DAGs wia airflow CLI - either to an image
or directly to terminal (for terminals/systems supporting imgcat)
* when you want to render ER diagram after you modified Airflow
models
The latter is a development-only feature, the former is production
feature, however it is a very niche one.
This PR turns rendering of the images in Airflow in optional feature
(only working when graphviz python bindings are installed) and
effectively turns graphviz into an optional extra (and removes it
from requirements).
This is not a breaking change technically - the CLIs to render the
DAGs is still there and IF you already have graphviz installed, it
will continue working as it did before. The only problem when it
does not work is where you do not have graphviz installed for
fresh installation and it will raise an error and inform that you need it.
Graphviz will remain to be installed for most users:
* the Airflow Image will still contain graphviz library, because
it is added there as extra
* when previous version of Airflow has been installed already, then
graphviz library is already installed there and Airflow will
continue working as it did
The only change will be a new installation of new version of Airflow
from the scratch, where graphviz will need to be specified as extra
or installed separately in order to enable DAG rendering option.
Taking into account this behaviour (which only requires to install
a graphviz package), this should not be considered as a breaking
change.
Extracted from: #36537
* Add OpenAI Provider
* Apply suggestions from code review
Co-authored-by: Phani Kumar <94376113+phanikumv@users.noreply.github.com>
* Remove create_completions method from hook
* Change type of input_text param
Since the upstream API accepts str ot list of tokens, we accept the similar inputs from user.
* Updated min-airflow version to 2.5.0
* Updated the interface and fix docs and static files
* Fix tests
* Fix tests
* Change the version
Because of OpenAI SDK not being production ready
* Add embedding_kwargs as a param to operator
* Update tests/providers/openai/hooks/test_openai.py
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
* Remove unwanted params in docstring
* Update Changelog
* Add security.rst file
* Update docs/apache-airflow-providers-openai/index.rst
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
* Add host field for connections
* Update docs/apache-airflow-providers-openai/index.rst
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
* Add changelog.rst file to docs
* Change version to 1.0.0
* Resolve conflicts
* Fix tests
* Fixed tests
* Fix test
* Resolve Conflict
---------
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Phani Kumar <94376113+phanikumv@users.noreply.github.com>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
This PR is part of our larger effort to add first-class integrations to support LLMOps that was presented at Airflow Summit.
In this PR we are adding Pinecone Provider. Pinecone is a fully managed vector database that makes it easy to add vector search to production applications. The primary objective of this Provider is to present users with an alternative Vector Database.
---------
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
Co-authored-by: Hussein Awala <hussein@awala.fr>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
Shahar EpsteinandJarek Potiuk
Jens Scheffler
Ephraim Anierobi
Jed Cunningham
rom sharon
Elad Kalif
Pierre Jeambrun
Yoonji Heo
uzhastik
Pankaj Koti
VladaZakharovaandMaksim Moiseenkov
SatishChGit
