![]() Observable Bottlenecks - Things to Look For The non_pooled_task_slot_count is removed as of Airflow 1.10.4 in favor of an explicit default_pool, which functions similarly, but is governed in a different way through configuration. When not using pools, tasks are run in the "default pool", whose size is guided by this config element. While the Airflow documentation is not overly descriptive, the important point here is that this is a task concurrency limitation set and applied at a per-DAG level. The number of task instances allowed to run concurrently by the scheduler. This defines how often the scheduler should run (in seconds). The scheduler constantly tries to trigger new tasks (look at the scheduler section in the docs for more information). This defines how many threads will run in parallel when Airflow schedules DAGs. This defines the number of task instances that a worker will take, so size up your workers based on the resources on your worker box and the nature of your tasks. This defines the max number of task instances that should run simultaneously on this airflow installation. We’ll reiterate a few of the definitions from Airflow’s documentation here, and make some useful comments about the parameter’s usage. The relationship between those parameters, the infrastructure of your Airflow deployment, and the desired result based on the observed bottlenecks is what’s described here. They can be set in a few ways, but the general precedence is that configuration parameters set using environment variables will take priority over a setting set in the configuration file.įor the topic at hand, we’re interested only in the Airflow configuration settings that govern the capacity of Airflow to run tasks and schedule DAGs. ![]() Outlining the Relevant Airflow Configuration SettingsĪirflow has a number of configuration settings. Secondly, the discussion is based on Airflow 1.10.x versions, as Airflow 2.0 will introduce significant changes. Much of what we discuss regarding Airflow worker settings are grounded within that context. First, we will assume you’ve already got a distributed Airflow deployment using the CeleryExecutor. Further, we’ll be equipping our mental models of understanding with the ‘why’.įinally, before we dive in, we’ll make two assumptions. Ultimately, we hope to achieve the understanding of ‘which’ Airflow control levers to pull and ‘when’. The discussion concludes on additional Airflow functionality supporting further tuning and monitoring for these situations. A walkthrough of several theoretical Airflow bottlenecks and the role those parameters play in fixing them follows. For example, we’ll call out the appropriate Airflow scheduler settings in the event of a scheduler bottleneck. To that end, we’ll begin by identifying a subset of Airflow’s configuration options that are relevant to the Airflow processes experiencing a bottleneck. We will frame the types of problems that can be identified and tuned for by discussing several broad examples. We mean that in a sense beyond Airflow’s own notion of configuration - the machine specs running our Airflow processes can also matter. Generally, problems of this nature are a result of either improper or insufficient Airflow configuration. The goal of this article is to connect questions like the above to various control levers we have in Airflow. If you’re reading this, maybe you’ve had and answered questions like this for yourself before. Ultimately, if my Airflow DAG performance is being negatively affected, what do I actually need to look at and fix? Why is my task scheduling latency increasing? Why are more tasks not being run, even after I add workers? You’ll have many questions and, frustratingly, fewer answers. ![]() You run the risk of developing a fragmented and mystical understanding of Airflow’s goings-on. You’ll apply solutions to those as they arise. You’ll discover and experience sporadic bottlenecks in Airflow-wide and DAG-level execution. You can easily shoot yourself in the foot if you don’t do it right at first. Properly configuring and tuning your Airflow deployment to be generally performant can be a nontrivial endeavor. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |