API Reference

Operators

Operators allow for generation of certain types of tasks that become nodes in the DAG when instantiated. All operators derive from BaseOperator and inherit many attributes and methods that way. Refer to the BaseOperator documentation for more details.

There are 3 main types of operators:

• Operators that performs an action, or tell another system to perform an action
• Transfer operators move data from one system to another
• Sensors are a certain type of operator that will keep running until a certain criterion is met. Examples include a specific file landing in HDFS or S3, a partition appearing in Hive, or a specific time of the day. Sensors are derived from BaseSensorOperator and run a poke method at a specified poke_interval until it returns True.

BaseOperator

All operators are derived from BaseOperator and acquire much functionality through inheritance. Since this is the core of the engine, it’s worth taking the time to understand the parameters of BaseOperator to understand the primitive features that can be leveraged in your DAGs.

class airflow.models.BaseOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, task_concurrency=None, *args, **kwargs)

Abstract base class for all operators. Since operators create objects that become node in the dag, BaseOperator contains many recursive methods for dag crawling behavior. To derive this class, you are expected to override the constructor as well as the ‘execute’ method.

Operators derived from this class should perform or trigger certain tasks synchronously (wait for completion). Example of operators could be an operator the runs a Pig job (PigOperator), a sensor operator that waits for a partition to land in Hive (HiveSensorOperator), or one that moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these operators (tasks) target specific operations, running specific scripts, functions or data transfers.

This class is abstract and shouldn’t be instantiated. Instantiating a class derived from this one results in the creation of a task object, which ultimately becomes a node in DAG objects. Task dependencies should be set by using the set_upstream and/or set_downstream methods.

Parameters:

• task_id (string) – a unique, meaningful id for the task
• owner (string) – the owner of the task, using the unix username is recommended
• retries (int) – the number of retries that should be performed before failing the task
• retry_delay (timedelta) – delay between retries
• retry_exponential_backoff (bool) – allow progressive longer waits between retries by using exponential backoff algorithm on retry delay (delay will be converted into seconds)
• max_retry_delay (timedelta) – maximum delay interval between retries
• start_date (datetime) – The start_date for the task, determines the execution_date for the first task instance. The best practice is to have the start_date rounded to your DAG’s schedule_interval. Daily jobs have their start_date some day at 00:00:00, hourly jobs have their start_date at 00:00 of a specific hour. Note that Airflow simply looks at the latest execution_date and adds the schedule_interval to determine the next execution_date. It is also very important to note that different tasks’ dependencies need to line up in time. If task A depends on task B and their start_date are offset in a way that their execution_date don’t line up, A’s dependencies will never be met. If you are looking to delay a task, for example running a daily task at 2AM, look into the TimeSensor and TimeDeltaSensor. We advise against using dynamic start_date and recommend using fixed ones. Read the FAQ entry about start_date for more information.
• end_date (datetime) – if specified, the scheduler won’t go beyond this date
• depends_on_past (bool) – when set to true, task instances will run sequentially while relying on the previous task’s schedule to succeed. The task instance for the start_date is allowed to run.
• wait_for_downstream (bool) – when set to true, an instance of task X will wait for tasks immediately downstream of the previous instance of task X to finish successfully before it runs. This is useful if the different instances of a task X alter the same asset, and this asset is used by tasks downstream of task X. Note that depends_on_past is forced to True wherever wait_for_downstream is used.
• queue (str) – which queue to target when running this job. Not all executors implement queue management, the CeleryExecutor does support targeting specific queues.
• dag (DAG) – a reference to the dag the task is attached to (if any)
• priority_weight (int) – priority weight of this task against other task. This allows the executor to trigger higher priority tasks before others when things get backed up.
• pool (str) – the slot pool this task should run in, slot pools are a way to limit concurrency for certain tasks
• sla (datetime.timedelta) – time by which the job is expected to succeed. Note that this represents the timedelta after the period is closed. For example if you set an SLA of 1 hour, the scheduler would send dan email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance has not succeeded yet. The scheduler pays special attention for jobs with an SLA and sends alert emails for sla misses. SLA misses are also recorded in the database for future reference. All tasks that share the same SLA time get bundled in a single email, sent soon after that time. SLA notification are sent once and only once for each task instance.
• execution_timeout (datetime.timedelta) – max time allowed for the execution of this task instance, if it goes beyond it will raise and fail.
• on_failure_callback (callable) – a function to be called when a task instance of this task fails. a context dictionary is passed as a single parameter to this function. Context contains references to related objects to the task instance and is documented under the macros section of the API.
• on_retry_callback – much like the on_failure_callback except that it is executed when retries occur.
• on_success_callback (callable) – much like the on_failure_callback except that it is executed when the task succeeds.
• trigger_rule (str) – defines the rule by which dependencies are applied for the task to get triggered. Options are: { all_success | all_failed | all_done | one_success | one_failed | dummy} default is all_success. Options can be set as string or using the constants defined in the static class airflow.utils.TriggerRule
• resources (dict) – A map of resource parameter names (the argument names of the Resources constructor) to their values.
• run_as_user (str) – unix username to impersonate while running the task
• task_concurrency (int) – When set, a task will be able to limit the concurrent runs across execution_dates

BaseSensorOperator

All sensors are derived from BaseSensorOperator. All sensors inherit the timeout and poke_interval on top of the BaseOperator attributes.

class airflow.operators.sensors.BaseSensorOperator(poke_interval=60, timeout=604800, soft_fail=False, *args, **kwargs)

Sensor operators are derived from this class an inherit these attributes.

Sensor operators keep executing at a time interval and succeed when

a criteria is met and fail if and when they time out.

Parameters:

• soft_fail (bool) – Set to true to mark the task as SKIPPED on failure
• poke_interval (int) – Time in seconds that the job should wait in between each tries
• timeout (int) – Time, in seconds before the task times out and fails.

Operator API

Importer that dynamically loads a class and module from its parent. This allows Airflow to support from airflow.operators import BashOperator even though BashOperator is actually in airflow.operators.bash_operator.

The importer also takes over for the parent_module by wrapping it. This is required to support attribute-based usage:

from airflow import operators
operators.BashOperator(...)

class airflow.operators.BashOperator(bash_command, xcom_push=False, env=None, output_encoding='utf-8', *args, **kwargs)

Bases: airflow.models.BaseOperator

Execute a Bash script, command or set of commands.

Parameters:

• bash_command (string) – The command, set of commands or reference to a bash script (must be ‘.sh’) to be executed.
• xcom_push (bool) – If xcom_push is True, the last line written to stdout will also be pushed to an XCom when the bash command completes.
• env (dict) – If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process environment, which is the default behavior. (templated)

execute(context)

Execute the bash command in a temporary directory which will be cleaned afterwards

class airflow.operators.BranchPythonOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)

Bases: python_operator.PythonOperator, airflow.models.SkipMixin

Allows a workflow to “branch” or follow a single path following the execution of this task.

It derives the PythonOperator and expects a Python function that returns the task_id to follow. The task_id returned should point to a task directly downstream from {self}. All other “branches” or directly downstream tasks are marked with a state of skipped so that these paths can’t move forward. The skipped states are propageted downstream to allow for the DAG state to fill up and the DAG run’s state to be inferred.

Note that using tasks with depends_on_past=True downstream from BranchPythonOperator is logically unsound as skipped status will invariably lead to block tasks that depend on their past successes. skipped states propagates where all directly upstream tasks are skipped.

class airflow.operators.TriggerDagRunOperator(trigger_dag_id, python_callable, *args, **kwargs)

Bases: airflow.models.BaseOperator

Triggers a DAG run for a specified dag_id if a criteria is met

Parameters:

• trigger_dag_id (str) – the dag_id to trigger
• python_callable (python callable) – a reference to a python function that will be called while passing it the context object and a placeholder object obj for your callable to fill and return if you want a DagRun created. This obj object contains a run_id and payload attribute that you can modify in your function. The run_id should be a unique identifier for that DAG run, and the payload has to be a picklable object that will be made available to your tasks while executing that DAG run. Your function header should look like def foo(context, dag_run_obj):

class airflow.operators.DummyOperator(*args, **kwargs)

Bases: airflow.models.BaseOperator

Operator that does literally nothing. It can be used to group tasks in a DAG.

class airflow.operators.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, mime_subtype='mixed', *args, **kwargs)

Bases: airflow.models.BaseOperator

Sends an email.

Parameters:

• to (list or string (comma or semicolon delimited)) – list of emails to send the email to
• subject (string) – subject line for the email (templated)
• html_content (string) – content of the email (templated), html markup is allowed
• files (list) – file names to attach in email
• cc (list or string (comma or semicolon delimited)) – list of recipients to be added in CC field
• bcc (list or string (comma or semicolon delimited)) – list of recipients to be added in BCC field

class airflow.operators.ExternalTaskSensor(external_dag_id, external_task_id, allowed_states=None, execution_delta=None, execution_date_fn=None, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits for a task to complete in a different DAG

Parameters:

• external_dag_id (string) – The dag_id that contains the task you want to wait for
• external_task_id (string) – The task_id that contains the task you want to wait for
• allowed_states (list) – list of allowed states, default is ['success']
• execution_delta (datetime.timedelta) – time difference with the previous execution to look at, the default is the same execution_date as the current task. For yesterday, use [positive!] datetime.timedelta(days=1). Either execution_delta or execution_date_fn can be passed to ExternalTaskSensor, but not both.
• execution_date_fn (callable) – function that receives the current execution date and returns the desired execution dates to query. Either execution_delta or execution_date_fn can be passed to ExternalTaskSensor, but not both.

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.GenericTransfer(sql, destination_table, source_conn_id, destination_conn_id, preoperator=None, *args, **kwargs)

Bases: airflow.models.BaseOperator

Moves data from a connection to another, assuming that they both provide the required methods in their respective hooks. The source hook needs to expose a get_records method, and the destination a insert_rows method.

This is meant to be used on small-ish datasets that fit in memory.

Parameters:

• sql (str) – SQL query to execute against the source database
• destination_table (str) – target table
• source_conn_id (str) – source connection
• destination_conn_id (str) – source connection
• preoperator (str or list of str) – sql statement or list of statements to be executed prior to loading the data

class airflow.operators.HdfsSensor(filepath, hdfs_conn_id='hdfs_default', ignored_ext=['_COPYING_'], ignore_copying=True, file_size=None, hook=<class 'airflow.hooks.hdfs_hook.HDFSHook'>, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits for a file or folder to land in HDFS

static filter_for_filesize(result, size=None)

Will test the filepath result and test if its size is at least self.filesize

Parameters:

• result – a list of dicts returned by Snakebite ls
• size – the file size in MB a file should be at least to trigger True

Returns:

(bool) depending on the matching criteria

static filter_for_ignored_ext(result, ignored_ext, ignore_copying)

Will filter if instructed to do so the result to remove matching criteria

Parameters:

• result – (list) of dicts returned by Snakebite ls
• ignored_ext – (list) of ignored extensions
• ignore_copying – (bool) shall we ignore ?

Returns:

(list) of dicts which were not removed

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.HivePartitionSensor(table, partition="ds='{{ ds }}'", metastore_conn_id='metastore_default', schema='default', poke_interval=180, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits for a partition to show up in Hive.

Note: Because partition supports general logical operators, it can be inefficient. Consider using NamedHivePartitionSensor instead if you don’t need the full flexibility of HivePartitionSensor.

Parameters:

• table (string) – The name of the table to wait for, supports the dot notation (my_database.my_table)
• partition (string) – The partition clause to wait for. This is passed as is to the metastore Thrift client get_partitions_by_filter method, and apparently supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01"
• metastore_conn_id (str) – reference to the metastore thrift service connection id

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.SimpleHttpOperator(endpoint, method='POST', data=None, headers=None, response_check=None, extra_options=None, xcom_push=False, http_conn_id='http_default', *args, **kwargs)

Bases: airflow.models.BaseOperator

Calls an endpoint on an HTTP system to execute an action

Parameters:

• http_conn_id (string) – The connection to run the sensor against
• endpoint (string) – The relative part of the full url
• method (string) – The HTTP method to use, default = “POST”
• data (For POST/PUT, depends on the content-type parameter, for GET a dictionary of key/value string pairs) – The data to pass. POST-data in POST/PUT and params in the URL for a GET request.
• headers (a dictionary of string key/value pairs) – The HTTP headers to be added to the GET request
• response_check (A lambda or defined function.) – A check against the ‘requests’ response object. Returns True for ‘pass’ and False otherwise.
• extra_options (A dictionary of options, where key is string and value depends on the option that's being modified.) – Extra options for the ‘requests’ library, see the ‘requests’ documentation (options to modify timeout, ssl, etc.)

class airflow.operators.HttpSensor(endpoint, http_conn_id='http_default', method='GET', request_params=None, headers=None, response_check=None, extra_options=None, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Executes a HTTP get statement and returns False on failure:

404 not found or response_check function returned False

Parameters:

• http_conn_id (string) – The connection to run the sensor against
• method (string) – The HTTP request method to use
• endpoint (string) – The relative part of the full url
• request_params (a dictionary of string key/value pairs) – The parameters to be added to the GET url
• headers (a dictionary of string key/value pairs) – The HTTP headers to be added to the GET request
• response_check (A lambda or defined function.) – A check against the ‘requests’ response object. Returns True for ‘pass’ and False otherwise.
• extra_options (A dictionary of options, where key is string and value depends on the option that's being modified.) – Extra options for the ‘requests’ library, see the ‘requests’ documentation (options to modify timeout, ssl, etc.)

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.MetastorePartitionSensor(table, partition_name, schema='default', mysql_conn_id='metastore_mysql', *args, **kwargs)

Bases: sensors.SqlSensor

An alternative to the HivePartitionSensor that talk directly to the MySQL db. This was created as a result of observing sub optimal queries generated by the Metastore thrift service when hitting subpartitioned tables. The Thrift service’s queries were written in a way that wouldn’t leverage the indexes.

Parameters:

• schema (str) – the schema
• table (str) – the table
• partition_name (str) – the partition name, as defined in the PARTITIONS table of the Metastore. Order of the fields does matter. Examples: ds=2016-01-01 or ds=2016-01-01/sub=foo for a sub partitioned table
• mysql_conn_id (str) – a reference to the MySQL conn_id for the metastore

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.NamedHivePartitionSensor(partition_names, metastore_conn_id='metastore_default', poke_interval=180, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits for a set of partitions to show up in Hive.

Parameters:

• partition_names (list of strings) – List of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01. This is passed as is to the metastore Thrift client get_partitions_by_name method. Note that you cannot use logical or comparison operators as in HivePartitionSensor.
• metastore_conn_id (str) – reference to the metastore thrift service connection id

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.PythonOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)

Bases: airflow.models.BaseOperator

Executes a Python callable

Parameters:

• python_callable (python callable) – A reference to an object that is callable
• op_kwargs (dict) – a dictionary of keyword arguments that will get unpacked in your function
• op_args (list) – a list of positional arguments that will get unpacked when calling your callable
• provide_context (bool) – if set to true, Airflow will pass a set of keyword arguments that can be used in your function. This set of kwargs correspond exactly to what you can use in your jinja templates. For this to work, you need to define **kwargs in your function header.
• templates_dict (dict of str) – a dictionary where the values are templates that will get templated by the Airflow engine sometime between __init__ and execute takes place and are made available in your callable’s context after the template has been applied
• templates_exts – a list of file extensions to resolve while processing templated fields, for examples ['.sql', '.hql']

class airflow.operators.S3KeySensor(bucket_key, bucket_name=None, wildcard_match=False, aws_conn_id='aws_default', *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits for a key (a file-like instance on S3) to be present in a S3 bucket. S3 being a key/value it does not support folders. The path is just a key a resource.

Parameters:

• bucket_key (str) – The key being waited on. Supports full s3:// style url or relative path from root level.
• bucket_name (str) – Name of the S3 bucket
• wildcard_match (bool) – whether the bucket_key should be interpreted as a Unix wildcard pattern
• aws_conn_id (str) – a reference to the s3 connection

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.ShortCircuitOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)

Bases: python_operator.PythonOperator, airflow.models.SkipMixin

Allows a workflow to continue only if a condition is met. Otherwise, the workflow “short-circuits” and downstream tasks are skipped.

The ShortCircuitOperator is derived from the PythonOperator. It evaluates a condition and short-circuits the workflow if the condition is False. Any downstream tasks are marked with a state of “skipped”. If the condition is True, downstream tasks proceed as normal.

The condition is determined by the result of python_callable.

class airflow.operators.SqlSensor(conn_id, sql, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Runs a sql statement until a criteria is met. It will keep trying while sql returns no row, or if the first cell in (0, ‘0’, ‘’).

Parameters:

• conn_id (string) – The connection to run the sensor against
• sql – The sql to run. To pass, it needs to return at least one cell that contains a non-zero / empty string value.

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.TimeSensor(target_time, *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits until the specified time of the day.

Parameters:target_time (datetime.time) – time after which the job succeeds

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.WebHdfsSensor(filepath, webhdfs_conn_id='webhdfs_default', *args, **kwargs)

Bases: sensors.BaseSensorOperator

Waits for a file or folder to land in HDFS

poke(context)

Function that the sensors defined while deriving this class should override.

class airflow.operators.docker_operator.DockerOperator(image, api_version=None, command=None, cpus=1.0, docker_url='unix://var/run/docker.sock', environment=None, force_pull=False, mem_limit=None, network_mode=None, tls_ca_cert=None, tls_client_cert=None, tls_client_key=None, tls_hostname=None, tls_ssl_version=None, tmp_dir='/tmp/airflow', user=None, volumes=None, working_dir=None, xcom_push=False, xcom_all=False, docker_conn_id=None, *args, **kwargs)

Execute a command inside a docker container.

A temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. The path to the mounted directory can be accessed via the environment variable AIRFLOW_TMP_DIR.

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

Parameters:

• image (str) – Docker image from which to create the container.
• api_version (str) – Remote API version. Set to auto to automatically detect the server’s version.
• command (str or list) – Command to be run in the container.
• cpus (float) – Number of CPUs to assign to the container. This value gets multiplied with 1024. See https://docs.docker.com/engine/reference/run/#cpu-share-constraint
• docker_url (str) – URL of the host running the docker daemon. Default is unix://var/run/docker.sock
• environment (dict) – Environment variables to set in the container.
• force_pull (bool) – Pull the docker image on every run. Default is false.
• mem_limit (float or str) – Maximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like 128m or 1g.
• network_mode (str) – Network mode for the container.
• tls_ca_cert (str) – Path to a PEM-encoded certificate authority to secure the docker connection.
• tls_client_cert (str) – Path to the PEM-encoded certificate used to authenticate docker client.
• tls_client_key (str) – Path to the PEM-encoded key used to authenticate docker client.
• tls_hostname (str or bool) – Hostname to match against the docker server certificate or False to disable the check.
• tls_ssl_version (str) – Version of SSL to use when communicating with docker daemon.
• tmp_dir (str) – Mount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable AIRFLOW_TMP_DIR inside the container.
• user (int or str) – Default user inside the docker container.
• volumes – List of volumes to mount into the container, e.g. ['/host/path:/container/path', '/host/path2:/container/path2:ro'].
• working_dir (str) – Working directory to set on the container (equivalent to the -w switch the docker client)
• xcom_push (bool) – Does the stdout will be pushed to the next step using XCom. The default is False.
• xcom_all (bool) – Push all the stdout or just the last line. The default is False (last line).
• docker_conn_id (str) – ID of the Airflow connection to use

Community-contributed Operators

Importer that dynamically loads a class and module from its parent. This allows Airflow to support from airflow.operators import BashOperator even though BashOperator is actually in airflow.operators.bash_operator.

The importer also takes over for the parent_module by wrapping it. This is required to support attribute-based usage:

from airflow import operators
operators.BashOperator(...)

class airflow.contrib.operators.bigquery_operator.BigQueryOperator(bql, destination_dataset_table=False, write_disposition='WRITE_EMPTY', allow_large_results=False, bigquery_conn_id='bigquery_default', delegate_to=None, udf_config=False, use_legacy_sql=True, maximum_billing_tier=None, create_disposition='CREATE_IF_NEEDED', query_params=None, *args, **kwargs)

Executes BigQuery SQL queries in a specific BigQuery database

Parameters:

• bql (Can receive a str representing a sql statement, a list of str (sql statements), or reference to a template file. Template reference are recognized by str ending in '.sql') – the sql code to be executed
• destination_dataset_table (string) – A dotted (<project>.|<project>:)<dataset>.<table> that, if set, will store the results of the query.
• write_disposition (string) – Specifies the action that occurs if the destination table already exists. (default: ‘WRITE_EMPTY’)
• create_disposition (string) – Specifies whether the job is allowed to create new tables. (default: ‘CREATE_IF_NEEDED’)
• bigquery_conn_id (string) – reference to a specific BigQuery hook.
• delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
• udf_config (list) – The User Defined Function configuration for the query. See https://cloud.google.com/bigquery/user-defined-functions for details.
• use_legacy_sql (boolean) – Whether to use legacy SQL (true) or standard SQL (false).
• maximum_billing_tier (integer) – Positive integer that serves as a multiplier of the basic price. Defaults to None, in which case it uses the value set in the project.
• query_params (dict) – a dictionary containing query parameter types and values, passed to BigQuery.

class airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator(source_project_dataset_table, destination_cloud_storage_uris, compression='NONE', export_format='CSV', field_delimiter=', ', print_header=True, bigquery_conn_id='bigquery_default', delegate_to=None, *args, **kwargs)

Transfers a BigQuery table to a Google Cloud Storage bucket.

See here:

https://cloud.google.com/bigquery/docs/reference/v2/jobs

For more details about these parameters.

Parameters:

• source_project_dataset_table (string) – The dotted (<project>.|<project>:)<dataset>.<table> BigQuery table to use as the source data. If <project> is not included, project will be the project defined in the connection json.
• destination_cloud_storage_uris (list) – The destination Google Cloud Storage URI (e.g. gs://some-bucket/some-file.txt). Follows convention defined here: https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple
• compression (string) – Type of compression to use.
• export_format – File format to export.
• field_delimiter (string) – The delimiter to use when extracting to a CSV.
• print_header (boolean) – Whether to print a header for a CSV file extract.
• bigquery_conn_id (string) – reference to a specific BigQuery hook.
• delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

class airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator(json=None, spark_jar_task=None, notebook_task=None, new_cluster=None, existing_cluster_id=None, libraries=None, run_name=None, timeout_seconds=None, databricks_conn_id='databricks_default', polling_period_seconds=30, databricks_retry_limit=3, **kwargs)

Submits an Spark job run to Databricks using the api/2.0/jobs/runs/submit API endpoint.

There are two ways to instantiate this operator.

In the first way, you can take the JSON payload that you typically use to call the api/2.0/jobs/runs/submit endpoint and pass it directly to our DatabricksSubmitRunOperator through the json parameter. For example

json = {
  'new_cluster': {
    'spark_version': '2.1.0-db3-scala2.11',
    'num_workers': 2
  },
  'notebook_task': {
    'notebook_path': '/Users/airflow@example.com/PrepareData',
  },
}
notebook_run = DatabricksSubmitRunOperator(task_id='notebook_run', json=json)

Another way to accomplish the same thing is to use the named parameters of the DatabricksSubmitRunOperator directly. Note that there is exactly one named parameter for each top level parameter in the runs/submit endpoint. In this method, your code would look like this:

new_cluster = {
  'spark_version': '2.1.0-db3-scala2.11',
  'num_workers': 2
}
notebook_task = {
  'notebook_path': '/Users/airflow@example.com/PrepareData',
}
notebook_run = DatabricksSubmitRunOperator(
    task_id='notebook_run',
    new_cluster=new_cluster,
    notebook_task=notebook_task)

In the case where both the json parameter AND the named parameters are provided, they will be merged together. If there are conflicts during the merge, the named parameters will take precedence and override the top level json keys.

Currently the named parameters that DatabricksSubmitRunOperator supports are

• spark_jar_task
• notebook_task
• new_cluster
• existing_cluster_id
• libraries
• run_name
• timeout_seconds

Parameters:

• json (dict) –

  A JSON object containing API parameters which will be passed directly to the api/2.0/jobs/runs/submit endpoint. The other named parameters (i.e. spark_jar_task, notebook_task..) to this operator will be merged with this json dictionary if they are provided. If there are conflicts during the merge, the named parameters will take precedence and override the top level json keys. This field will be templated.

  See also

  For more information about templating see Jinja Templating. https://docs.databricks.com/api/latest/jobs.html#runs-submit

• spark_jar_task (dict) –

  The main class and parameters for the JAR task. Note that the actual JAR is specified in the libraries. EITHER spark_jar_task OR notebook_task should be specified. This field will be templated.

  See also

  https://docs.databricks.com/api/latest/jobs.html#jobssparkjartask

• notebook_task (dict) –

  The notebook path and parameters for the notebook task. EITHER spark_jar_task OR notebook_task should be specified. This field will be templated.

  See also

  https://docs.databricks.com/api/latest/jobs.html#jobsnotebooktask

• new_cluster (dict) –

  Specs for a new cluster on which this task will be run. EITHER new_cluster OR existing_cluster_id should be specified. This field will be templated.

  See also

  https://docs.databricks.com/api/latest/jobs.html#jobsclusterspecnewcluster

• existing_cluster_id (string) – ID for existing cluster on which to run this task. EITHER new_cluster OR existing_cluster_id should be specified. This field will be templated.
• libraries (list of dicts) –

  Libraries which this run will use. This field will be templated.

  See also

  https://docs.databricks.com/api/latest/libraries.html#managedlibrarieslibrary

• run_name (string) – The run name used for this task. By default this will be set to the Airflow task_id. This task_id is a required parameter of the superclass BaseOperator. This field will be templated.
• timeout_seconds (int32) – The timeout for this run. By default a value of 0 is used which means to have no timeout. This field will be templated.
• databricks_conn_id (string) – The name of the Airflow connection to use. By default and in the common case this will be databricks_default. To use token based authentication, provide the key token in the extra field for the connection.
• polling_period_seconds (int) – Controls the rate which we poll for the result of this run. By default the operator will poll every 30 seconds.
• databricks_retry_limit (int) – Amount of times retry if the Databricks backend is unreachable. Its value must be greater than or equal to 1.

class airflow.contrib.operators.ecs_operator.ECSOperator(task_definition, cluster, overrides, aws_conn_id=None, region_name=None, **kwargs)

Execute a task on AWS EC2 Container Service

Parameters:

• task_definition (str) – the task definition name on EC2 Container Service
• cluster (str) – the cluster name on EC2 Container Service
• aws_conn_id (str) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used (http://boto3.readthedocs.io/en/latest/guide/configuration.html).
• region_name – region name to use in AWS Hook. Override the region_name in connection (if provided)

Param:

overrides: the same parameter that boto3 will receive: http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task

Type:

overrides: dict

class airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator(bucket, object, filename=False, store_to_xcom_key=False, google_cloud_storage_conn_id='google_cloud_storage_default', delegate_to=None, *args, **kwargs)

Downloads a file from Google Cloud Storage.

Parameters:

• bucket (string) – The Google cloud storage bucket where the object is.
• object (string) – The name of the object to download in the Google cloud storage bucket.
• filename (string) – The file path on the local file system (where the operator is being executed) that the file should be downloaded to. If false, the downloaded data will not be stored on the local file system.
• store_to_xcom_key (string) – If this param is set, the operator will push the contents of the downloaded file to XCom with the key set in this parameter. If false, the downloaded data will not be pushed to XCom.
• google_cloud_storage_conn_id (string) – The connection ID to use when connecting to Google cloud storage.
• delegate_to (string) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

class airflow.contrib.operators.hipchat_operator.HipChatAPIOperator(token, base_url='https://api.hipchat.com/v2', *args, **kwargs)

Base HipChat Operator. All derived HipChat operators reference from HipChat’s official REST API documentation at https://www.hipchat.com/docs/apiv2. Before using any HipChat API operators you need to get an authentication token at https://www.hipchat.com/docs/apiv2/auth. In the future additional HipChat operators will be derived from this class as well.

Parameters:

• token (str) – HipChat REST API authentication token
• base_url (str) – HipChat REST API base url.

class airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator(room_id, message, *args, **kwargs)

Send notification to a specific HipChat room. More info: https://www.hipchat.com/docs/apiv2/method/send_room_notification

Parameters:

• room_id (str) – Room in which to send notification on HipChat
• message (str) – The message body
• frm (str) – Label to be shown in addition to sender’s name
• message_format (str) – How the notification is rendered: html or text
• color (str) – Background color of the msg: yellow, green, red, purple, gray, or random
• attach_to (str) – The message id to attach this notification to
• notify (bool) – Whether this message should trigger a user notification
• card (dict) – HipChat-defined card object

Macros

Here’s a list of variables and macros that can be used in templates

Default Variables

The Airflow engine passes a few variables by default that are accessible in all templates

Variable	Description
{{ ds }}	the execution date as YYYY-MM-DD
{{ ds_nodash }}	the execution date as YYYYMMDD
{{ yesterday_ds }}	yesterday’s date as YYYY-MM-DD
{{ yesterday_ds_nodash }}	yesterday’s date as YYYYMMDD
{{ tomorrow_ds }}	tomorrow’s date as YYYY-MM-DD
{{ tomorrow_ds_nodash }}	tomorrow’s date as YYYYMMDD
{{ ts }}	same as execution_date.isoformat()
{{ ts_nodash }}	same as ts without - and :
{{ execution_date }}	the execution_date, (datetime.datetime)
{{ prev_execution_date }}	the previous execution date (if available) (datetime.datetime)
{{ next_execution_date }}	the next execution date (datetime.datetime)
{{ dag }}	the DAG object
{{ task }}	the Task object
{{ macros }}	a reference to the macros package, described below
{{ task_instance }}	the task_instance object
{{ end_date }}	same as {{ ds }}
{{ latest_date }}	same as {{ ds }}
{{ ti }}	same as {{ task_instance }}
{{ params }}	a reference to the user-defined params dictionary
{{ var.value.my_var }}	global defined variables represented as a dictionary
{{ var.json.my_var.path }}	global defined variables represented as a dictionary with deserialized JSON object, append the path to the key within the JSON object
{{ task_instance_key_str }}	a unique, human-readable key to the task instance formatted {dag_id}_{task_id}_{ds}
{{ conf }}	the full configuration object located at airflow.configuration.conf which represents the content of your airflow.cfg
{{ run_id }}	the run_id of the current DAG run
{{ dag_run }}	a reference to the DagRun object
{{ test_mode }}	whether the task instance was called using the CLI’s test subcommand

Note that you can access the object’s attributes and methods with simple dot notation. Here are some examples of what is possible: {{ task.owner }}, {{ task.task_id }}, {{ ti.hostname }}, … Refer to the models documentation for more information on the objects’ attributes and methods.

The var template variable allows you to access variables defined in Airflow’s UI. You can access them as either plain-text or JSON. If you use JSON, you are also able to walk nested structures, such as dictionaries like: {{ var.json.my_dict_var.key1 }}

Macros

Macros are a way to expose objects to your templates and live under the macros namespace in your templates.

A few commonly used libraries and methods are made available.

Variable	Description
macros.datetime	The standard lib’s datetime.datetime
macros.timedelta	The standard lib’s datetime.timedelta
macros.dateutil	A reference to the dateutil package
macros.time	The standard lib’s time
macros.uuid	The standard lib’s uuid
macros.random	The standard lib’s random

Some airflow specific macros are also defined:

airflow.macros.ds_add(ds, days)

Add or subtract days from a YYYY-MM-DD

Parameters:

• ds (str) – anchor date in YYYY-MM-DD format to add to
• days (int) – number of days to add to the ds, you can use negative values

>>> ds_add('2015-01-01', 5)
'2015-01-06'
>>> ds_add('2015-01-06', -5)
'2015-01-01'

airflow.macros.ds_format(ds, input_format, output_format)

Takes an input string and outputs another string as specified in the output format

Parameters:

• ds (str) – input string which contains a date
• input_format (str) – input string format. E.g. %Y-%m-%d
• output_format (str) – output string format E.g. %Y-%m-%d

>>> ds_format('2015-01-01', "%Y-%m-%d", "%m-%d-%y")
'01-01-15'
>>> ds_format('1/5/2015', "%m/%d/%Y",  "%Y-%m-%d")
'2015-01-05'

airflow.macros.hive.closest_ds_partition(table, ds, before=True, schema='default', metastore_conn_id='metastore_default')

This function finds the date in a list closest to the target date. An optional parameter can be given to get the closest before or after.

Parameters:

• table (str) – A hive table name
• ds (datetime.date list) – A datestamp %Y-%m-%d e.g. yyyy-mm-dd
• before (bool or None) – closest before (True), after (False) or either side of ds

Returns:

The closest date

Return type:

str or None

>>> tbl = 'airflow.static_babynames_partitioned'
>>> closest_ds_partition(tbl, '2015-01-02')
'2015-01-01'

airflow.macros.hive.max_partition(table, schema='default', field=None, filter=None, metastore_conn_id='metastore_default')

Gets the max partition for a table.

Parameters:

• schema (string) – The hive schema the table lives in
• table (string) – The hive table you are interested in, supports the dot notation as in “my_database.my_table”, if a dot is found, the schema param is disregarded
• hive_conn_id (string) – The hive connection you are interested in. If your default is set you don’t need to use this parameter.
• filter (string) – filter on a subset of partition as in sub_part=’specific_value’
• field – the field to get the max value from. If there’s only one partition field, this will be inferred

>>> max_partition('airflow.static_babynames_partitioned')
'2015-01-01'

Models

Models are built on top of the SQLAlchemy ORM Base class, and instances are persisted in the database.

class airflow.models.DAG(dag_id, description=u'', schedule_interval=datetime.timedelta(1), start_date=None, end_date=None, full_filepath=None, template_searchpath=None, user_defined_macros=None, user_defined_filters=None, default_args=None, concurrency=16, max_active_runs=16, dagrun_timeout=None, sla_miss_callback=None, default_view=u'tree', orientation='LR', catchup=True, params=None)

Bases: airflow.dag.base_dag.BaseDag, airflow.utils.log.logging_mixin.LoggingMixin

A dag (directed acyclic graph) is a collection of tasks with directional dependencies. A dag also has a schedule, a start end an end date (optional). For each schedule, (say daily or hourly), the DAG needs to run each individual tasks as their dependencies are met. Certain tasks have the property of depending on their own past, meaning that they can’t run until their previous schedule (and upstream tasks) are completed.

DAGs essentially act as namespaces for tasks. A task_id can only be added once to a DAG.

Parameters:

• dag_id (string) – The id of the DAG
• description (string) – The description for the DAG to e.g. be shown on the webserver
• schedule_interval (datetime.timedelta or dateutil.relativedelta.relativedelta or str that acts as a cron expression) – Defines how often that DAG runs, this timedelta object gets added to your latest task instance’s execution_date to figure out the next schedule
• start_date (datetime.datetime) – The timestamp from which the scheduler will attempt to backfill
• end_date (datetime.datetime) – A date beyond which your DAG won’t run, leave to None for open ended scheduling
• template_searchpath (string or list of stings) – This list of folders (non relative) defines where jinja will look for your templates. Order matters. Note that jinja/airflow includes the path of your DAG file by default
• user_defined_macros (dict) – a dictionary of macros that will be exposed in your jinja templates. For example, passing dict(foo='bar') to this argument allows you to {{ foo }} in all jinja templates related to this DAG. Note that you can pass any type of object here.
• user_defined_filters (dict) – a dictionary of filters that will be exposed in your jinja templates. For example, passing dict(hello=lambda name: 'Hello %s' % name) to this argument allows you to {{ 'world' | hello }} in all jinja templates related to this DAG.
• default_args (dict) – A dictionary of default parameters to be used as constructor keyword parameters when initialising operators. Note that operators have the same hook, and precede those defined here, meaning that if your dict contains ‘depends_on_past’: True here and ‘depends_on_past’: False in the operator’s call default_args, the actual value will be False.
• params (dict) – a dictionary of DAG level parameters that are made accessible in templates, namespaced under params. These params can be overridden at the task level.
• concurrency (int) – the number of task instances allowed to run concurrently
• max_active_runs (int) – maximum number of active DAG runs, beyond this number of DAG runs in a running state, the scheduler won’t create new active DAG runs
• dagrun_timeout (datetime.timedelta) – specify how long a DagRun should be up before timing out / failing, so that new DagRuns can be created
• sla_miss_callback (types.FunctionType) – specify a function to call when reporting SLA timeouts.
• default_view (string) – Specify DAG default view (tree, graph, duration, gantt, landing_times)
• orientation (string) – Specify DAG orientation in graph view (LR, TB, RL, BT)
• catchup (bool) – Perform scheduler catchup (or only run latest)? Defaults to True

add_task(task)

Add a task to the DAG

Parameters:task (task) – the task you want to add

add_tasks(tasks)

Add a list of tasks to the DAG

Parameters:tasks (list of tasks) – a lit of tasks you want to add

clear(start_date=None, end_date=None, only_failed=False, only_running=False, confirm_prompt=False, include_subdags=True, reset_dag_runs=True, dry_run=False)

Clears a set of task instances associated with the current dag for a specified date range.

cli()

Exposes a CLI specific to this DAG

concurrency_reached

Returns a boolean indicating whether the concurrency limit for this DAG has been reached

create_dagrun(**kwargs)

Creates a dag run from this dag including the tasks associated with this dag. Returns the dag run.

Parameters:

• run_id (string) – defines the the run id for this dag run
• execution_date (datetime) – the execution date of this dag run
• state (State) – the state of the dag run
• start_date (datetime) – the date this dag run should be evaluated
• external_trigger (bool) – whether this dag run is externally triggered
• session (Session) – database session

static deactivate_stale_dags(*args, **kwargs)

Deactivate any DAGs that were last touched by the scheduler before the expiration date. These DAGs were likely deleted.

Parameters:expiration_date (datetime) – set inactive DAGs that were touched before this time Returns:None

static deactivate_unknown_dags(*args, **kwargs)

Given a list of known DAGs, deactivate any other DAGs that are marked as active in the ORM

Parameters:active_dag_ids (list[unicode]) – list of DAG IDs that are active Returns:None

filepath

File location of where the dag object is instantiated

folder

Folder location of where the dag object is instantiated

get_active_runs(**kwargs)

Returns a list of dag run execution dates currently running

Parameters:session – Returns:List of execution dates

get_dagrun(**kwargs)

Returns the dag run for a given execution date if it exists, otherwise none.

Parameters:

• execution_date – The execution date of the DagRun to find.
• session –

Returns:

The DagRun if found, otherwise None.

get_last_dagrun(**kwargs)

Returns the last dag run for this dag, None if there was none. Last dag run can be any type of run eg. scheduled or backfilled. Overridden DagRuns are ignored

get_num_active_runs(**kwargs)

Returns the number of active “running” dag runs

Parameters:

• external_trigger (bool) – True for externally triggered active dag runs
• session –

Returns:

number greater than 0 for active dag runs

static get_num_task_instances(*args, **kwargs)

Returns the number of task instances in the given DAG.

Parameters:

• session – ORM session
• dag_id (unicode) – ID of the DAG to get the task concurrency of
• task_ids (list[unicode]) – A list of valid task IDs for the given DAG
• states (list[state]) – A list of states to filter by if supplied

Returns:

The number of running tasks

Return type:

int

get_run_dates(start_date, end_date=None)

Returns a list of dates between the interval received as parameter using this dag’s schedule interval. Returned dates can be used for execution dates.

Parameters:

• start_date (datetime) – the start date of the interval
• end_date (datetime) – the end date of the interval, defaults to datetime.utcnow()

Returns:

a list of dates within the interval following the dag’s schedule

Return type:

list

get_template_env()

Returns a jinja2 Environment while taking into account the DAGs template_searchpath, user_defined_macros and user_defined_filters

is_paused

Returns a boolean indicating whether this DAG is paused

latest_execution_date

Returns the latest date for which at least one dag run exists

normalize_schedule(dttm)

Returns dttm + interval unless dttm is first interval then it returns dttm

run(start_date=None, end_date=None, mark_success=False, include_adhoc=False, local=False, executor=None, donot_pickle=False, ignore_task_deps=False, ignore_first_depends_on_past=False, pool=None, delay_on_limit_secs=1.0)

Runs the DAG.

Parameters:

• start_date (datetime) – the start date of the range to run
• end_date (datetime) – the end date of the range to run
• mark_success (bool) – True to mark jobs as succeeded without running them
• include_adhoc (bool) – True to include dags with the adhoc parameter
• local (bool) – True to run the tasks using the LocalExecutor
• executor (BaseExecutor) – The executor instance to run the tasks
• donot_pickle (bool) – True to avoid pickling DAG object and send to workers
• ignore_task_deps (bool) – True to skip upstream tasks
• ignore_first_depends_on_past (bool) – True to ignore depends_on_past dependencies for the first set of tasks only
• pool (string) – Resource pool to use
• delay_on_limit_secs (float) – Time in seconds to wait before next attempt to run dag run when max_active_runs limit has been reached

set_dependency(upstream_task_id, downstream_task_id)

Simple utility method to set dependency between two tasks that already have been added to the DAG using add_task()

sub_dag(task_regex, include_downstream=False, include_upstream=True)

Returns a subset of the current dag as a deep copy of the current dag based on a regex that should match one or many tasks, and includes upstream and downstream neighbours based on the flag passed.

subdags

Returns a list of the subdag objects associated to this DAG

sync_to_db(**kwargs)

Save attributes about this DAG to the DB. Note that this method can be called for both DAGs and SubDAGs. A SubDag is actually a SubDagOperator.

Parameters:

• dag (DAG) – the DAG object to save to the DB
• sync_time (datetime) – The time that the DAG should be marked as sync’ed

Returns:

None

topological_sort()

Sorts tasks in topographical order, such that a task comes after any of its upstream dependencies.

Heavily inspired by: http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/

Returns:list of tasks in topological order

tree_view()

Shows an ascii tree representation of the DAG

class airflow.models.BaseOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, task_concurrency=None, *args, **kwargs)

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Abstract base class for all operators. Since operators create objects that become node in the dag, BaseOperator contains many recursive methods for dag crawling behavior. To derive this class, you are expected to override the constructor as well as the ‘execute’ method.

Operators derived from this class should perform or trigger certain tasks synchronously (wait for completion). Example of operators could be an operator the runs a Pig job (PigOperator), a sensor operator that waits for a partition to land in Hive (HiveSensorOperator), or one that moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these operators (tasks) target specific operations, running specific scripts, functions or data transfers.

This class is abstract and shouldn’t be instantiated. Instantiating a class derived from this one results in the creation of a task object, which ultimately becomes a node in DAG objects. Task dependencies should be set by using the set_upstream and/or set_downstream methods.

Parameters:

• task_id (string) – a unique, meaningful id for the task
• owner (string) – the owner of the task, using the unix username is recommended
• retries (int) – the number of retries that should be performed before failing the task
• retry_delay (timedelta) – delay between retries
• retry_exponential_backoff (bool) – allow progressive longer waits between retries by using exponential backoff algorithm on retry delay (delay will be converted into seconds)
• max_retry_delay (timedelta) – maximum delay interval between retries
• start_date (datetime) – The start_date for the task, determines the execution_date for the first task instance. The best practice is to have the start_date rounded to your DAG’s schedule_interval. Daily jobs have their start_date some day at 00:00:00, hourly jobs have their start_date at 00:00 of a specific hour. Note that Airflow simply looks at the latest execution_date and adds the schedule_interval to determine the next execution_date. It is also very important to note that different tasks’ dependencies need to line up in time. If task A depends on task B and their start_date are offset in a way that their execution_date don’t line up, A’s dependencies will never be met. If you are looking to delay a task, for example running a daily task at 2AM, look into the TimeSensor and TimeDeltaSensor. We advise against using dynamic start_date and recommend using fixed ones. Read the FAQ entry about start_date for more information.
• end_date (datetime) – if specified, the scheduler won’t go beyond this date
• depends_on_past (bool) – when set to true, task instances will run sequentially while relying on the previous task’s schedule to succeed. The task instance for the start_date is allowed to run.
• wait_for_downstream (bool) – when set to true, an instance of task X will wait for tasks immediately downstream of the previous instance of task X to finish successfully before it runs. This is useful if the different instances of a task X alter the same asset, and this asset is used by tasks downstream of task X. Note that depends_on_past is forced to True wherever wait_for_downstream is used.
• queue (str) – which queue to target when running this job. Not all executors implement queue management, the CeleryExecutor does support targeting specific queues.
• dag (DAG) – a reference to the dag the task is attached to (if any)
• priority_weight (int) – priority weight of this task against other task. This allows the executor to trigger higher priority tasks before others when things get backed up.
• pool (str) – the slot pool this task should run in, slot pools are a way to limit concurrency for certain tasks
• sla (datetime.timedelta) – time by which the job is expected to succeed. Note that this represents the timedelta after the period is closed. For example if you set an SLA of 1 hour, the scheduler would send dan email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance has not succeeded yet. The scheduler pays special attention for jobs with an SLA and sends alert emails for sla misses. SLA misses are also recorded in the database for future reference. All tasks that share the same SLA time get bundled in a single email, sent soon after that time. SLA notification are sent once and only once for each task instance.
• execution_timeout (datetime.timedelta) – max time allowed for the execution of this task instance, if it goes beyond it will raise and fail.
• on_failure_callback (callable) – a function to be called when a task instance of this task fails. a context dictionary is passed as a single parameter to this function. Context contains references to related objects to the task instance and is documented under the macros section of the API.
• on_retry_callback – much like the on_failure_callback except that it is executed when retries occur.
• on_success_callback (callable) – much like the on_failure_callback except that it is executed when the task succeeds.
• trigger_rule (str) – defines the rule by which dependencies are applied for the task to get triggered. Options are: { all_success | all_failed | all_done | one_success | one_failed | dummy} default is all_success. Options can be set as string or using the constants defined in the static class airflow.utils.TriggerRule
• resources (dict) – A map of resource parameter names (the argument names of the Resources constructor) to their values.
• run_as_user (str) – unix username to impersonate while running the task
• task_concurrency (int) – When set, a task will be able to limit the concurrent runs across execution_dates

clear(start_date=None, end_date=None, upstream=False, downstream=False)

Clears the state of task instances associated with the task, following the parameters specified.

dag

Returns the Operator’s DAG if set, otherwise raises an error

deps

Returns the list of dependencies for the operator. These differ from execution context dependencies in that they are specific to tasks and can be extended/overridden by subclasses.

detect_downstream_cycle(task=None)

When invoked, this routine will raise an exception if a cycle is detected downstream from self. It is invoked when tasks are added to the DAG to detect cycles.

downstream_list

@property: list of tasks directly downstream

execute(context)

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_direct_relatives(upstream=False)

Get the direct relatives to the current task, upstream or downstream.

get_flat_relatives(upstream=False, l=None)

Get a flat list of relatives, either upstream or downstream.

get_task_instances(session, start_date=None, end_date=None)

Get a set of task instance related to this task for a specific date range.

has_dag()

Returns True if the Operator has been assigned to a DAG.

on_kill()

Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.

post_execute(context, result=None)

This hook is triggered right after self.execute() is called. It is passed the execution context and any results returned by the operator.

pre_execute(context)

This hook is triggered right before self.execute() is called.

prepare_template()

Hook that is triggered after the templated fields get replaced by their content. If you need your operator to alter the content of the file before the template is rendered, it should override this method to do so.

render_template(attr, content, context)

Renders a template either from a file or directly in a field, and returns the rendered result.

render_template_from_field(attr, content, context, jinja_env)

Renders a template from a field. If the field is a string, it will simply render the string and return the result. If it is a collection or nested set of collections, it will traverse the structure and render all strings in it.

run(start_date=None, end_date=None, ignore_first_depends_on_past=False, ignore_ti_state=False, mark_success=False)

Run a set of task instances for a date range.

schedule_interval

The schedule interval of the DAG always wins over individual tasks so that tasks within a DAG always line up. The task still needs a schedule_interval as it may not be attached to a DAG.

set_downstream(task_or_task_list)

Set a task, or a task task to be directly downstream from the current task.

set_upstream(task_or_task_list)

Set a task, or a task task to be directly upstream from the current task.

upstream_list

@property: list of tasks directly upstream

xcom_pull(context, task_ids, dag_id=None, key=u'return_value', include_prior_dates=None)

See TaskInstance.xcom_pull()

xcom_push(context, key, value, execution_date=None)

See TaskInstance.xcom_push()

class airflow.models.TaskInstance(task, execution_date, state=None)

Bases: sqlalchemy.ext.declarative.api.Base, airflow.utils.log.logging_mixin.LoggingMixin

Task instances store the state of a task instance. This table is the authority and single source of truth around what tasks have run and the state they are in.

The SqlAlchemy model doesn’t have a SqlAlchemy foreign key to the task or dag model deliberately to have more control over transactions.

Database transactions on this table should insure double triggers and any confusion around what task instances are or aren’t ready to run even while multiple schedulers may be firing task instances.

are_dependencies_met(**kwargs)

Returns whether or not all the conditions are met for this task instance to be run given the context for the dependencies (e.g. a task instance being force run from the UI will ignore some dependencies).

Parameters:

• dep_context (DepContext) – The execution context that determines the dependencies that should be evaluated.
• session (Session) – database session
• verbose (boolean) – whether or not to print details on failed dependencies

are_dependents_done(**kwargs)

Checks whether the dependents of this task instance have all succeeded. This is meant to be used by wait_for_downstream.

This is useful when you do not want to start processing the next schedule of a task until the dependents are done. For instance, if the task DROPs and recreates a table.

clear_xcom_data(**kwargs)

Clears all XCom data from the database for the task instance

command(mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)

Returns a command that can be executed anywhere where airflow is installed. This command is part of the message sent to executors by the orchestrator.

command_as_list(mark_success=False, ignore_all_deps=False, ignore_task_deps=False, ignore_depends_on_past=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)

Returns a command that can be executed anywhere where airflow is installed. This command is part of the message sent to executors by the orchestrator.

current_state(**kwargs)

Get the very latest state from the database, if a session is passed, we use and looking up the state becomes part of the session, otherwise a new session is used.

error(**kwargs)

Forces the task instance’s state to FAILED in the database.

static generate_command(dag_id, task_id, execution_date, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, file_path=None, raw=False, job_id=None, pool=None, cfg_path=None)

Generates the shell command required to execute this task instance.

Parameters:

• dag_id (unicode) – DAG ID
• task_id (unicode) – Task ID
• execution_date (datetime) – Execution date for the task
• mark_success (bool) – Whether to mark the task as successful
• ignore_all_deps (boolean) – Ignore all ignorable dependencies. Overrides the other ignore_* parameters.
• ignore_depends_on_past (boolean) – Ignore depends_on_past parameter of DAGs (e.g. for Backfills)
• ignore_task_deps (boolean) – Ignore task-specific dependencies such as depends_on_past and trigger rule
• ignore_ti_state (boolean) – Ignore the task instance’s previous failure/success
• local (bool) – Whether to run the task locally
• pickle_id (unicode) – If the DAG was serialized to the DB, the ID associated with the pickled DAG
• file_path – path to the file containing the DAG definition
• raw – raw mode (needs more details)
• job_id – job ID (needs more details)
• pool (unicode) – the Airflow pool that the task should run in

Returns:

shell command that can be used to run the task instance

get_dagrun(**kwargs)

Returns the DagRun for this TaskInstance

Parameters:session – Returns:DagRun

init_on_load()

Initialize the attributes that aren’t stored in the DB.

is_premature

Returns whether a task is in UP_FOR_RETRY state and its retry interval has elapsed.

key

Returns a tuple that identifies the task instance uniquely

next_retry_datetime()

Get datetime of the next retry if the task instance fails. For exponential backoff, retry_delay is used as base and will be converted to seconds.

pool_full(**kwargs)

Returns a boolean as to whether the slot pool has room for this task to run

previous_ti

The task instance for the task that ran before this task instance

ready_for_retry()

Checks on whether the task instance is in the right state and timeframe to be retried.

refresh_from_db(**kwargs)

Refreshes the task instance from the database based on the primary key

Parameters:lock_for_update – if True, indicates that the database should lock the TaskInstance (issuing a FOR UPDATE clause) until the session is committed.

try_number

Return the try number that this task number will be when it is acutally run.

If the TI is currently running, this will match the column in the databse, in all othercases this will be incremenetd

xcom_pull(task_ids, dag_id=None, key=u'return_value', include_prior_dates=False)

Pull XComs that optionally meet certain criteria.

The default value for key limits the search to XComs that were returned by other tasks (as opposed to those that were pushed manually). To remove this filter, pass key=None (or any desired value).

If a single task_id string is provided, the result is the value of the most recent matching XCom from that task_id. If multiple task_ids are provided, a tuple of matching values is returned. None is returned whenever no matches are found.

Parameters:

• key (string) – A key for the XCom. If provided, only XComs with matching keys will be returned. The default key is ‘return_value’, also available as a constant XCOM_RETURN_KEY. This key is automatically given to XComs returned by tasks (as opposed to being pushed manually). To remove the filter, pass key=None.
• task_ids (string or iterable of strings (representing task_ids)) – Only XComs from tasks with matching ids will be pulled. Can pass None to remove the filter.
• dag_id (string) – If provided, only pulls XComs from this DAG. If None (default), the DAG of the calling task is used.
• include_prior_dates (bool) – If False, only XComs from the current execution_date are returned. If True, XComs from previous dates are returned as well.

xcom_push(key, value, execution_date=None)

Make an XCom available for tasks to pull.

Parameters:

• key (string) – A key for the XCom
• value (any pickleable object) – A value for the XCom. The value is pickled and stored in the database.
• execution_date (datetime) – if provided, the XCom will not be visible until this date. This can be used, for example, to send a message to a task on a future date without it being immediately visible.

class airflow.models.DagBag(dag_folder=None, executor=None, include_examples=True)

Bases: airflow.dag.base_dag.BaseDagBag, airflow.utils.log.logging_mixin.LoggingMixin

A dagbag is a collection of dags, parsed out of a folder tree and has high level configuration settings, like what database to use as a backend and what executor to use to fire off tasks. This makes it easier to run distinct environments for say production and development, tests, or for different teams or security profiles. What would have been system level settings are now dagbag level so that one system can run multiple, independent settings sets.

Parameters:

• dag_folder (unicode) – the folder to scan to find DAGs
• executor – the executor to use when executing task instances in this DagBag
• include_examples (bool) – whether to include the examples that ship with airflow or not

bag_dag(dag, parent_dag, root_dag)

Adds the DAG into the bag, recurses into sub dags.

collect_dags(dag_folder=None, only_if_updated=True)

Given a file path or a folder, this method looks for python modules, imports them and adds them to the dagbag collection.

Note that if a .airflowignore file is found while processing, the directory, it will behaves much like a .gitignore does, ignoring files that match any of the regex patterns specified in the file.

dagbag_report()

Prints a report around DagBag loading stats

get_dag(dag_id)

Gets the DAG out of the dictionary, and refreshes it if expired

kill_zombies(**kwargs)

Fails tasks that haven’t had a heartbeat in too long

process_file(filepath, only_if_updated=True, safe_mode=True)

Given a path to a python module or zip file, this method imports the module and look for dag objects within it.

size()

Returns:the amount of dags contained in this dagbag

class airflow.models.Connection(conn_id=None, conn_type=None, host=None, login=None, password=None, schema=None, port=None, extra=None, uri=None)

Bases: sqlalchemy.ext.declarative.api.Base, airflow.utils.log.logging_mixin.LoggingMixin

Placeholder to store information about different database instances connection information. The idea here is that scripts use references to database instances (conn_id) instead of hard coding hostname, logins and passwords when using operators or hooks.

extra_dejson

Returns the extra property by deserializing json.

Hooks

Importer that dynamically loads a class and module from its parent. This allows Airflow to support from airflow.operators import BashOperator even though BashOperator is actually in airflow.operators.bash_operator.

The importer also takes over for the parent_module by wrapping it. This is required to support attribute-based usage:

from airflow import operators
operators.BashOperator(...)

class airflow.hooks.DbApiHook(*args, **kwargs)

Bases: airflow.hooks.base_hook.BaseHook

Abstract base class for sql hooks.

bulk_dump(table, tmp_file)

Dumps a database table into a tab-delimited file

Parameters:

• table (str) – The name of the source table
• tmp_file (str) – The path of the target file

bulk_load(table, tmp_file)

Loads a tab-delimited file into a database table

Parameters:

• table (str) – The name of the target table
• tmp_file (str) – The path of the file to load into the table

get_conn()

Returns a connection object

get_cursor()

Returns a cursor

get_first(sql, parameters=None)

Executes the sql and returns the first resulting row.

Parameters:

• sql (str or list) – the sql statement to be executed (str) or a list of sql statements to execute
• parameters (mapping or iterable) – The parameters to render the SQL query with.

get_pandas_df(sql, parameters=None)

Executes the sql and returns a pandas dataframe

Parameters:

• sql (str or list) – the sql statement to be executed (str) or a list of sql statements to execute
• parameters (mapping or iterable) – The parameters to render the SQL query with.

get_records(sql, parameters=None)

Executes the sql and returns a set of records.

Parameters:

• sql (str or list) – the sql statement to be executed (str) or a list of sql statements to execute
• parameters (mapping or iterable) – The parameters to render the SQL query with.

insert_rows(table, rows, target_fields=None, commit_every=1000)

A generic way to insert a set of tuples into a table, a new transaction is created every commit_every rows

Parameters:

• table (str) – Name of the target table
• rows (iterable of tuples) – The rows to insert into the table
• target_fields (iterable of strings) – The names of the columns to fill in the table
• commit_every (int) – The maximum number of rows to insert in one transaction. Set to 0 to insert all rows in one transaction.

run(sql, autocommit=False, parameters=None)

Runs a command or a list of commands. Pass a list of sql statements to the sql parameter to get them to execute sequentially

Parameters:

• sql (str or list) – the sql statement to be executed (str) or a list of sql statements to execute
• autocommit (bool) – What to set the connection’s autocommit setting to before executing the query.
• parameters (mapping or iterable) – The parameters to render the SQL query with.

class airflow.hooks.HttpHook(method='POST', http_conn_id='http_default')

Bases: airflow.hooks.base_hook.BaseHook

Interact with HTTP servers.

get_conn(headers)

Returns http session for use with requests

run(endpoint, data=None, headers=None, extra_options=None)

Performs the request

run_and_check(session, prepped_request, extra_options)

Grabs extra options like timeout and actually runs the request, checking for the result

class airflow.hooks.DruidHook(druid_ingest_conn_id='druid_ingest_default', timeout=1, max_ingestion_time=18000)

Bases: airflow.hooks.base_hook.BaseHook

Connection to Druid

Parameters:

• druid_ingest_conn_id (string) – The connection id to the Druid overlord machine which accepts index jobs
• timeout (int) – The interval between polling the Druid job for the status of the ingestion job
• max_ingestion_time (int) – The maximum ingestion time before assuming the job failed

class airflow.hooks.S3Hook(aws_conn_id='aws_default')

Bases: airflow.contrib.hooks.aws_hook.AwsHook

Interact with AWS S3, using the boto3 library.

check_for_bucket(bucket_name)

Check if bucket_name exists.

Parameters:bucket_name (str) – the name of the bucket

check_for_key(key, bucket_name=None)

Checks if a key exists in a bucket

Parameters:

• key (str) – S3 key that will point to the file
• bucket_name (str) – Name of the bucket in which the file is stored

check_for_prefix(bucket_name, prefix, delimiter)

Checks that a prefix exists in a bucket

check_for_wildcard_key(wildcard_key, bucket_name=None, delimiter='')

Checks that a key matching a wildcard expression exists in a bucket

get_bucket(bucket_name)

Returns a boto3.S3.Bucket object

Parameters:bucket_name (str) – the name of the bucket

get_key(key, bucket_name=None)

Returns a boto3.s3.Object

Parameters:

• key (str) – the path to the key
• bucket_name (str) – the name of the bucket

get_wildcard_key(wildcard_key, bucket_name=None, delimiter='')

Returns a boto3.s3.Object object matching the regular expression

Parameters:

• regex_key (str) – the path to the key
• bucket_name (str) – the name of the bucket

list_keys(bucket_name, prefix='', delimiter='')

Lists keys in a bucket under prefix and not containing delimiter

Parameters:

• bucket_name (str) – the name of the bucket
• prefix (str) – a key prefix
• delimiter (str) – the delimiter marks key hierarchy.

list_prefixes(bucket_name, prefix='', delimiter='')

Lists prefixes in a bucket under prefix

Parameters:

• bucket_name (str) – the name of the bucket
• prefix (str) – a key prefix
• delimiter (str) – the delimiter marks key hierarchy.

load_file(filename, key, bucket_name=None, replace=False, encrypt=False)

Loads a local file to S3

Parameters:

• filename (str) – name of the file to load.
• key (str) – S3 key that will point to the file
• bucket_name (str) – Name of the bucket in which to store the file
• replace (bool) – A flag to decide whether or not to overwrite the key if it already exists. If replace is False and the key exists, an error will be raised.
• encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_string(string_data, key, bucket_name=None, replace=False, encrypt=False, encoding='utf-8')

Loads a string to S3

This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.

Parameters:

• string_data (str) – string to set as content for the key.
• key (str) – S3 key that will point to the file
• bucket_name (str) – Name of the bucket in which to store the file
• replace (bool) – A flag to decide whether or not to overwrite the key if it already exists
• encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

read_key(key, bucket_name=None)

Reads a key from S3

Parameters:

• key (str) – S3 key that will point to the file
• bucket_name (str) – Name of the bucket in which the file is stored

class airflow.hooks.SqliteHook(*args, **kwargs)

Bases: airflow.hooks.dbapi_hook.DbApiHook

Interact with SQLite.

get_conn()

Returns a sqlite connection object

class airflow.hooks.WebHDFSHook(webhdfs_conn_id='webhdfs_default', proxy_user=None)

Bases: airflow.hooks.base_hook.BaseHook

Interact with HDFS. This class is a wrapper around the hdfscli library.

check_for_path(hdfs_path)

Check for the existence of a path in HDFS by querying FileStatus.

get_conn()

Returns a hdfscli InsecureClient object.

load_file(source, destination, overwrite=True, parallelism=1, **kwargs)

Uploads a file to HDFS

Parameters:

• source (str) – Local path to file or folder. If a folder, all the files inside of it will be uploaded (note that this implies that folders empty of files will not be created remotely).
• destination (str) – PTarget HDFS path. If it already exists and is a directory, files will be uploaded inside.
• overwrite (bool) – Overwrite any existing file or directory.
• parallelism (int) – Number of threads to use for parallelization. A value of 0 (or negative) uses as many threads as there are files.
• **kwargs – Keyword arguments forwarded to upload().

Community contributed hooks

Importer that dynamically loads a class and module from its parent. This allows Airflow to support from airflow.operators import BashOperator even though BashOperator is actually in airflow.operators.bash_operator.

The importer also takes over for the parent_module by wrapping it. This is required to support attribute-based usage:

from airflow import operators
operators.BashOperator(...)

class airflow.contrib.hooks.BigQueryHook(bigquery_conn_id='bigquery_default', delegate_to=None)

Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook, airflow.hooks.dbapi_hook.DbApiHook, airflow.utils.log.logging_mixin.LoggingMixin

Interact with BigQuery. This hook uses the Google Cloud Platform connection.

get_conn()

Returns a BigQuery PEP 249 connection object.

get_pandas_df(bql, parameters=None, dialect='legacy')

Returns a Pandas DataFrame for the results produced by a BigQuery query. The DbApiHook method must be overridden because Pandas doesn’t support PEP 249 connections, except for SQLite. See:

https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447 https://github.com/pydata/pandas/issues/6900

Parameters:

• bql (string) – The BigQuery SQL to execute.
• parameters (mapping or iterable) – The parameters to render the SQL query with (not used, leave to override superclass method)
• dialect (string in {'legacy', 'standard'}, default 'legacy') – Dialect of BigQuery SQL – legacy SQL or standard SQL

get_service()

Returns a BigQuery service object.

insert_rows(table, rows, target_fields=None, commit_every=1000)

Insertion is currently unsupported. Theoretically, you could use BigQuery’s streaming API to insert rows into a table, but this hasn’t been implemented.

table_exists(project_id, dataset_id, table_id)

Checks for the existence of a table in Google BigQuery.

Parameters:project_id – The Google cloud project in which to look for the table. The connection supplied to the hook

must provide access to the specified project. :type project_id: string :param dataset_id: The name of the dataset in which to look for the table.

storage bucket.

Parameters:table_id (string) – The name of the table to check the existence of.

class airflow.contrib.hooks.GoogleCloudStorageHook(google_cloud_storage_conn_id='google_cloud_storage_default', delegate_to=None)

Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook

Interact with Google Cloud Storage. This hook uses the Google Cloud Platform connection.

copy(source_bucket, source_object, destination_bucket=None, destination_object=None)

Copies an object from a bucket to another, with renaming if requested.

destination_bucket or destination_object can be omitted, in which case source bucket/object is used, but not both.

Parameters:

• bucket (string) – The bucket of the object to copy from.
• object (string) – The object to copy.
• destination_bucket (string) – The destination of the object to copied to. Can be omitted; then the same bucket is used.
• destination_object – The (renamed) path of the object if given. Can be omitted; then the same name is used.

delete(bucket, object, generation=None)

Delete an object if versioning is not enabled for the bucket, or if generation parameter is used.

Parameters:

• bucket (string) – name of the bucket, where the object resides
• object (string) – name of the object to delete
• generation (string) – if present, permanently delete the object of this generation

Returns:

True if succeeded

download(bucket, object, filename=False)

Get a file from Google Cloud Storage.

Parameters:

• bucket (string) – The bucket to fetch from.
• object (string) – The object to fetch.
• filename (string) – If set, a local file path where the file should be written to.

exists(bucket, object)

Checks for the existence of a file in Google Cloud Storage.

Parameters:

• bucket (string) – The Google cloud storage bucket where the object is.
• object (string) – The name of the object to check in the Google cloud storage bucket.

get_conn()

Returns a Google Cloud Storage service object.

is_updated_after(bucket, object, ts)

Checks if an object is updated in Google Cloud Storage.

Parameters:

• bucket (string) – The Google cloud storage bucket where the object is.
• object (string) – The name of the object to check in the Google cloud storage bucket.
• ts (datetime) – The timestamp to check against.

list(bucket, versions=None, maxResults=None, prefix=None)

List all objects from the bucket with the give string prefix in name

Parameters:

• bucket (string) – bucket name
• versions (boolean) – if true, list all versions of the objects
• maxResults (integer) – max count of items to return in a single page of responses
• prefix (string) – prefix string which filters objects whose name begin with this prefix

Returns:

a stream of object names matching the filtering criteria

upload(bucket, object, filename, mime_type='application/octet-stream')

Uploads a local file to Google Cloud Storage.

Parameters:

• bucket (string) – The bucket to upload to.
• object (string) – The object name to set when uploading the local file.
• filename (string) – The local file path to the file to be uploaded.
• mime_type (string) – The MIME type to set when uploading the file.

class airflow.contrib.hooks.FTPHook(ftp_conn_id='ftp_default')

Bases: airflow.hooks.base_hook.BaseHook, airflow.utils.log.logging_mixin.LoggingMixin

Interact with FTP.

Errors that may occur throughout but should be handled downstream.

close_conn()

Closes the connection. An error will occur if the connection wasn’t ever opened.

create_directory(path)

Creates a directory on the remote system.

Parameters:path (str) – full path to the remote directory to create

delete_directory(path)

Deletes a directory on the remote system.

Parameters:path (str) – full path to the remote directory to delete

delete_file(path)

Removes a file on the FTP Server.

Parameters:path (str) – full path to the remote file

describe_directory(path)

Returns a dictionary of {filename: {attributes}} for all files on the remote system (where the MLSD command is supported).

Parameters:path (str) – full path to the remote directory

get_conn()

Returns a FTP connection object

list_directory(path, nlst=False)

Returns a list of files on the remote system.

Parameters:path (str) – full path to the remote directory to list

rename(from_name, to_name)

Rename a file.

Parameters:

• from_name – rename file from name
• to_name – rename file to name

retrieve_file(remote_full_path, local_full_path_or_buffer)

Transfers the remote file to a local location.

If local_full_path_or_buffer is a string path, the file will be put at that location; if it is a file-like buffer, the file will be written to the buffer but not closed.

Parameters:

• remote_full_path (str) – full path to the remote file
• local_full_path_or_buffer – full path to the local file or a file-like buffer

store_file(remote_full_path, local_full_path_or_buffer)

Transfers a local file to the remote location.

If local_full_path_or_buffer is a string path, the file will be read from that location; if it is a file-like buffer, the file will be read from the buffer but not closed.

Parameters:

• remote_full_path (str) – full path to the remote file
• local_full_path_or_buffer (str or file-like buffer) – full path to the local file or a file-like buffer

class airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook(google_cloud_storage_conn_id='google_cloud_storage_default', delegate_to=None)

Interact with Google Cloud Storage. This hook uses the Google Cloud Platform connection.

Executors

Executors are the mechanism by which task instances get run.

class airflow.executors.LocalExecutor(parallelism=32)

Bases: airflow.executors.base_executor.BaseExecutor

LocalExecutor executes tasks locally in parallel. It uses the multiprocessing Python library and queues to parallelize the execution of tasks.

end()

This method is called when the caller is done submitting job and is wants to wait synchronously for the job submitted previously to be all done.

execute_async(key, command, queue=None)

This method will execute the command asynchronously.

start()

Executors may need to get things started. For example LocalExecutor starts N workers.

sync()

Sync will get called periodically by the heartbeat method. Executors should override this to perform gather statuses.

class airflow.executors.SequentialExecutor

Bases: airflow.executors.base_executor.BaseExecutor

This executor will only run one task instance at a time, can be used for debugging. It is also the only executor that can be used with sqlite since sqlite doesn’t support multiple connections.

Since we want airflow to work out of the box, it defaults to this SequentialExecutor alongside sqlite as you first install it.

end()

This method is called when the caller is done submitting job and is wants to wait synchronously for the job submitted previously to be all done.

execute_async(key, command, queue=None)

This method will execute the command asynchronously.

sync()

Sync will get called periodically by the heartbeat method. Executors should override this to perform gather statuses.

Community-contributed executors

class airflow.contrib.executors.mesos_executor.MesosExecutor(parallelism=32)

MesosExecutor allows distributing the execution of task instances to multiple mesos workers.

Apache Mesos is a distributed systems kernel which abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. See http://mesos.apache.org/