Resources#

Resources are used in Dagster to make external resources (like a database connection) available to solids during pipeline execution. When used alongside modes, resources make it easier to write pipelines that can be developed locally and tested.

Relevant APIs#

Name	Description
@resource	The decorator used to define resources. The decorated function is called a resource_fn. The decorator returns a ResourceDefinition
ResourceDefinition	Base class for solids. You almost never want to use initialize this class directly. Instead, you should use the @resource which returns a ResourceDefinition

Overview#

We recommend learning about Modes before learning about resources.

Resources are pipeline-scoped, and are typically used to expose features of the execution environment (like EMR, Redshift, Spark, BigQuery, etc.) to solids during pipeline execution. Resources bound to pipeline execution through Modes, and can also clean up after execution resolves. They are typically defined using the @resource decorator or using the ResourceDefinition class directly.

The resource system enables:

• Lightweight testing: A resource is an intuitive and logical seam where you can mock and fake implementations for testing.
• Business logic and environment separation You can use this same seam to execute the same business logic in different environments and clouds.
• Easy configuration Resources declare their configuration in a self-describing way. You can configure them in the typeahead in Dagit and with high-quality error messages.

Defining a Resource#

To define a resource, we use the @resource decorator:

class DatabaseConnection():
    def execute_query():
        pass

@resource
def db_resource(init_context):
    return DatabaseConnection()

The @resource decorator decorates a function that returns a resource. The object returned from the resource function itself can be any standard python object - it doesn't need to inherit any particular interface or implement any particular functionality.

Resource Configuration#

ResourceDefinitions can have a config schema, which allows you to customize behavior at runtime through pipeline configuration.

For example, let's say we wanted to pass a connection string to our DatabaseConnection resource.

class DatabaseConnection():
    def __init__(connection: str):
        self.connection = connection

@resource(config_schema={"connection": str})
def db_resource(init_context):
    connection = init_context.resource_config["connection"]
    return DatabaseConnection(connection)

Resource to Resource Dependencies#

Resources can depend upon other resources. If a resource key is included in the required_resource_keys set provided to the resource definition, then the resource initializer can access a required resource via the "resources" attribute of its context object.

Default Resources#