![]() In your config.yaml file, copy the complete podTemplate configuration from your version of the Astronomer Airflow Helm chart. See Configure the worker Pod for a specific task. You can then override this default at the task level using a pod_override file. You can modify the default podTemplate to configure the default worker Pods for all Deployments using the Kubernetes executor on your Astronomer Software installation. Configure the default worker Pod for all Deploymentsīy default, the Kubernetes executor launches workers based on a podTemplate configuration in the Astronomer Airflow Helm chart. To learn more about different executor types, see Airflow executors explained. For more information on configuring an executor, see Configure a Deployment. You must have an Airflow Deployment on Astronomer running with the Kubernetes executor. For more information on configuring Pod template values, reference the Kubernetes documentation. To specify a Pod template for many or all of your tasks, you can write a helper function to construct a pod_override in your DAGs or configure a global setting. To configure these resources for a given task's Pod, you specify a pod_override in your DAG code. It also means you can configure the following for each individual Airflow task: You can customize your Kubernetes Pods to scale depending on how many Airflow tasks you're running at a given time. For each task that needs to run, the executor works with the Kubernetes API and dynamically launches Pods which terminate when the task is completed. The Kubernetes Executor creates individual Pods that dynamically delegate work and resources to individual tasks. ![]() Run and configure the Kubernetes executor on Astronomer.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |