docs: Update documentation and skill

docs/api/collection/metadata.rst

@@ -10,6 +10,8 @@ Metadata
     Collection.member_schemas
     Collection.required_members
     Collection.optional_members
+    Collection.lazy_members
+    Collection.eager_members
     Collection.non_ignored_members
     Collection.ignored_members
     Collection.common_primary_key

docs/api/columns/index.rst

@@ -12,6 +12,7 @@ Columns
     Binary
     Bool
     Categorical
+    Column
     Date
     Datetime
     Decimal

docs/api/filter_result/failure_info.rst

@@ -18,6 +18,7 @@ Inspection
     :toctree: _gen/
 
     FailureInfo.invalid
+    FailureInfo.details
     FailureInfo.counts
     FailureInfo.cooccurrence_counts
     FailureInfo.__len__

docs/api/misc/index.rst

@@ -7,6 +7,7 @@ Miscellaneous
     :toctree: _gen/
     :nosignatures:
 
+    CollectionMember
     Config
     random.Generator
     testing.create_schema

docs/api/schema/conversion.rst

@@ -7,3 +7,4 @@ Conversion
     :toctree: _gen/
 
     Schema.to_sqlalchemy_columns
+    Schema.to_pydantic_model

docs/conf.py

@@ -95,7 +95,6 @@
 
 ## myst_parser
 myst_parser_config = {"myst_enable_extensions": ["rst_eval_roles"]}
-nitpick_ignore = [("myst", "group-rules")]
 
 ## numpydoc
 numpydoc_class_members_toctree = False

docs/guides/faq.md

@@ -5,25 +5,19 @@ thinking, please add it here.
 
 ## How do I define additional unique keys in a {class}`~dataframely.Schema`?
 
-By default, `dataframely` only supports defining a single non-nullable (composite) primary key in {class}
-`~dataframely.Schema`.
-However, in some scenarios it may be useful to define additional unique keys (which support nullable fields and/or which
-are additionally unique).
+`dataframely` only supports defining a single non-nullable (composite) primary key in {class}`~dataframely.Schema`.
+However, in some scenarios it may be useful to define additional unique keys.
 
-Consider the following example, which demonstrates two rules: one for validating that a field is entirely unique, and
-another for validating that a field, when provided, is unique.
+For individual columns, set `unique=True` to require that the column's values are unique. Note that multiple `null`
+values are considered duplicates. To require uniqueness only for non-null values, or uniqueness across a combination of
+columns, use a custom rule instead:
 
 ```python
 class UserSchema(dy.Schema):
-    user_id = dy.UInt64(primary_key=True, nullable=False)
-    username = dy.String(nullable=False)
+    user_id = dy.UInt64(primary_key=True)
+    username = dy.String(unique=True)
     email = dy.String(nullable=True)  # Must be unique, or null.
 
-    @dy.rule()
-    def unique_username(cls) -> pl.Expr:
-        """Username, a non-nullable field, must be total unique."""
-        return (pl.len() == 1).over("username")
-
     @dy.rule()
     def unique_email_or_null(cls) -> pl.Expr:
         """Email must be unique, if provided."""
@@ -32,7 +26,7 @@ class UserSchema(dy.Schema):
 
 ## How do I fix the ruff error `First argument of a method should be named self`?
 
-See our documentation on [group rules](./quickstart.md#group-rules).
+See our documentation on [custom rules](./quickstart.md#custom-rules).
 
 ## How do I get a {class}`~dataframely.Schema` for my dataframe?
 

docs/guides/features/column-metadata.md

@@ -26,6 +26,13 @@ class UserSchema(dy.Schema):
 Metadata are never read by `dataframely` and merely enable users to provide custom information
 in a structured way.
 
+For human-readable column descriptions, all column types additionally provide a dedicated `description` parameter:
+
+```python
+class UserSchema(dy.Schema):
+    id = dy.String(primary_key=True, description="The unique identifier of the user.")
+```
+
 ```{note}
 User-provided metadata can be useful for code generation. For instance, one could specify metadata for columns, such as database column types or constraints, and override the built-in SQLAlchemy generation for more tailored SQL output.
 ```

docs/guides/features/data-generation.md

@@ -54,7 +54,7 @@ class InvoiceSchema(dy.Schema):
 
     @dy.rule()
     def discharge_after_admission(cls) -> pl.Expr:
-        return InvoiceSchema.discharge_date.col >= InvoiceSchema.admission_date.col
+        return cls.discharge_date.col >= cls.admission_date.col
 
 
 # `@dy.rule`s will be respected as well for data generation.
@@ -86,8 +86,8 @@ df: dy.DataFrame[InvoiceSchema] = InvoiceSchema.sample(
     overrides={
         # Use either <schema>.<column>.name or just the column name as a string.
         InvoiceSchema.invoice_id.name: ["1234567890", "2345678901", "3456789012"],
-        # Dataframely will automatically infer the number of rows based on the longest given
-        # sequence of values and broadcast all other columns to that shape.
+        # Dataframely will automatically infer the number of rows based on the length of the
+        # given sequences of values and broadcast scalar values to that shape.
         "admission_date": date(2025, 1, 1),
     }
 )
@@ -110,7 +110,7 @@ df: dy.DataFrame[InvoiceSchema] = InvoiceSchema.sample(
 
 ### Providing custom column overrides
 
-Complex validation rules (such as dependencies between columns or ordering criteria) can cause sampling to run excessively long or fail. Specifying additional `overrides` individually in such cases can be tedious. Instead, the {meth}`~dataframely.Schema._sampling_overrides` hook on a schema can be used to specify polars expressions for columns in the schema that will be applied before generated data is filtered during sampling.
+Complex validation rules (such as dependencies between columns or ordering criteria) can cause sampling to run excessively long or fail. Specifying additional `overrides` individually in such cases can be tedious. Instead, the `Schema._sampling_overrides` hook on a schema can be used to specify polars expressions for columns in the schema that will be applied before generated data is filtered during sampling.
 
 ```python
 import polars as pl
@@ -174,16 +174,16 @@ class HospitalInvoiceData(dy.Collection):
     diagnosis: dy.LazyFrame[DiagnosisSchema]
 
     @dy.filter()
-    def at_least_one_diagnosis(cls) -> pl.Expr:
+    def at_least_one_diagnosis(self) -> pl.LazyFrame:
         return dy.functional.require_relationship_one_to_at_least_one(
-            cls.invoice,
-            cls.diagnosis,
+            self.invoice,
+            self.diagnosis,
             on="invoice_id",
         )
 ```
 
 In this case, calling {meth}`~dataframely.Collection.sample` will fail, because dataframely does not parse the body of `at_least_one_diagnosis` which may contain arbitrary polars expressions.
-To address the problem, one can override {meth}`~dataframely.Collection._preprocess_sample` to generate a random number of diagnoses per invoice:
+To address the problem, one can override `Collection._preprocess_sample` to generate a random number of diagnoses per invoice:
 
 ```python
 from random import random
@@ -197,10 +197,10 @@ class HospitalInvoiceData(dy.Collection):
     diagnosis: dy.LazyFrame[DiagnosisSchema]
 
     @dy.filter()
-    def at_least_one_diagnosis(cls) -> pl.Expr:
+    def at_least_one_diagnosis(self) -> pl.LazyFrame:
         return dy.functional.require_relationship_one_to_at_least_one(
-            cls.invoice,
-            cls.diagnosis,
+            self.invoice,
+            self.diagnosis,
             on="invoice_id",
         )
 
@@ -240,7 +240,7 @@ class OutputSchema(dy.Schema):
 
 # function under test
 def get_diabetes_invoice_amounts(
-    invoice_data: HospitalClaims,
+    invoice_data: HospitalInvoiceData,
 ) -> dy.LazyFrame[OutputSchema]:
     return OutputSchema.cast(
         invoice_data.diagnosis.filter(DiagnosisSchema.code.col.str.starts_with("E11"))

docs/guides/features/lazy-validation.md

@@ -3,39 +3,47 @@
 In many cases, dataframely's capability to validate and filter input data is used at core application boundaries.
 As a result, `validate` and `filter` are generally expected to be used at points where `collect` is called on a lazy
 frame. However, there may be situations where validation or filtering should simply be added to the lazy computation
-graph. Starting in dataframely v2, this is supported via a custom polars plugin.
+graph. This is supported via a custom polars plugin.
 
-## The `eager` parameter
+## Schemas: lazy-in-lazy-out
 
-All of the following methods expose an `eager: bool` parameter:
+{meth}`Schema.validate() <dataframely.Schema.validate>` and {meth}`Schema.filter() <dataframely.Schema.filter>` follow
+the same "eager-in-eager-out" and "lazy-in-lazy-out" semantics as polars: if a {class}`~polars.DataFrame` is passed,
+validation and filtering are performed immediately. If a {class}`~polars.LazyFrame` is passed, the validation or
+filtering operation is simply appended to the lazy frame:
 
-- {meth}`Schema.validate() <dataframely.Schema.validate>`
-- {meth}`Schema.filter() <dataframely.Schema.filter>`
-- {meth}`Collection.validate() <dataframely.Collection.validate>`
-- {meth}`Collection.filter() <dataframely.Collection.filter>`
+```python
+def validate_lf(lf: pl.LazyFrame) -> dy.LazyFrame[MySchema]:
+    return lf.pipe(MySchema.validate)
+```
+
+For lazy inputs, validation is only run once the lazy frame is collected. If input data does not satisfy the schema,
+no error is raised here, yet.
+
+## Collections: the `lazy` parameter
 
-By default, `eager=True`. However, users may decide to set `eager=False` in order to simply append the validation or
-the filtering operation to the lazy frame. For example, one might decide to run validation lazily:
+{meth}`Collection.validate() <dataframely.Collection.validate>` and
+{meth}`Collection.filter() <dataframely.Collection.filter>` collect their results by default, regardless of whether the
+members are passed as data frames or lazy frames. Set `lazy=True` to keep the results lazy:
 
 ```python
-def validate_lf(lf: pl.LazyFrame) -> pl.LazyFrame:
-    return lf.pipe(MySchema.validate, eager=False)
+collection = MyCollection.validate(data, lazy=True)
 ```
 
-When `eager=False`, validation is only run once the lazy frame is collected. If input data does not satisfy the schema,
-no error is raised here, yet.
+Note that `lazy=True` cannot be used on a collection that defines at least one eager member, i.e. a member annotated
+with `dy.DataFrame`.
 
 ## Error Types
 
 Due to current limitations in polars plugins, the type of error that is being raised from the `validate` function (both
-for schemas and collections) is dependent on the value of the `eager` parameter:
+for schemas and collections) is dependent on whether validation is performed eagerly or lazily:
 
-- When `eager=True`, a {class}`~dataframely.ValidationError` is raised from the `validate` function
-- When `eager=False`, a {class}`~polars.exceptions.ComputeError` is raised from the `collect` function
+- When validating eagerly, a {class}`~dataframely.exc.ValidationError` is raised from the `validate` function
+- When validating lazily, a {class}`~polars.exceptions.ComputeError` is raised from the `collect` function
 
 ```{note}
 For schemas, the error _message_ itself is equivalent.
-For collections, the error message for `eager=False` is limited and non-deterministic: the error message only includes
+For collections, the error message for lazy validation is limited and non-deterministic: the error message only includes
 information about a single member and, if multiple members fail validation, the member that the error message refers to
 may vary across executions.
 ```
@@ -51,7 +59,7 @@ in the error message and the specific failure surfaced may be non-deterministic
 By default, validation error messages report only the name of each failing rule and the number of rows that violated it.
 For easier debugging, dataframely can additionally include a few example rows for each failing rule. This is configured
 via {meth}`~dataframely.Config.set_max_failure_examples` (or the `max_failure_examples` keyword on the
-{class}`~dataframely.Config` context manager) and applies to both `eager=True` and `eager=False`:
+{class}`~dataframely.Config` context manager) and applies to both eager and lazy validation:
 
 ```python
 import dataframely as dy

docs/guides/features/primary-keys.md

@@ -42,6 +42,18 @@ class LineItemSchema(dy.Schema):
 
 Validation will now ensure that all pairs of (`invoice_id`, `item_id`) are unique.
 
+### Unique columns
+
+In addition to the primary key, individual columns can be required to be unique by setting `unique=True`. Unlike
+`primary_key=True`, setting `unique=True` on multiple columns does not form a composite constraint: each column is
+checked for uniqueness independently.
+
+```python
+class UserSchema(dy.Schema):
+    id = dy.String(primary_key=True)
+    email = dy.String(unique=True)
+```
+
 ## Primary keys in {class}`~dataframely.Collection`
 
 The central idea behind {class}`~dataframely.Collection` is to unify multiple tables relating to the same set of

docs/guides/features/sql-generation.md

@@ -64,15 +64,15 @@ df.write_database(connection=engine, table_name=my_table.name, if_table_exists="
 ```
 
 ```{note}
-**Implementation:** The choice of `sqlalchemy` type is implemented in {meth}`~dataframely.Column.sqlalchemy_dtype`, which is overwritten by each of the subtypes of {class}`~dataframely.Column`. For example, the implementation for {class}`~dataframely.Date` is {meth}`~dataframely.Date.sqlalchemy_dtype`.
+**Implementation:** The choice of `sqlalchemy` type is implemented in `Column.sqlalchemy_dtype`, which is overwritten by each of the subtypes of {class}`~dataframely.Column`. For example, the implementation for {class}`~dataframely.Date` is `Date.sqlalchemy_dtype`.
 ```
 
 ```{note}
-**Constraints:** The nullability and primary key constraints you define in `dataframely` are translated to SQL. Custom filters and rules are not.
+**Constraints:** The nullability, primary key and `unique` constraints you define in `dataframely` are translated to SQL. Custom filters and rules are not.
 ```
 
 ```{note}
-**Length of string columns:** For string columns, `dataframely` will attempt to pass information about the maximal length into the SQL definition. This is trivial if `max_length` is set. Otherwise, if a `regex` is provided,
+**Length of string columns:** For string columns, `dataframely` will attempt to pass information about the maximal length into the SQL definition. This is trivial if `max_length` is set. Otherwise, if a fully anchored `regex` (i.e. starting with `^` and ending with `$`) is provided,
 the maximal length of the string is inferred from the regular expression if possible. Note that having inferable
 maximal lengths can be particularly important for primary key columns. Some database systems, such as Microsoft SQL Server, do not allow `VARCHAR(max)` columns (unbounded strings) to be used as primary keys.
 ```
@@ -126,5 +126,5 @@ for name, dy_schema in MyCollection.member_schemas().items():
         meta,
         *dy_schema.to_sqlalchemy_columns(dialect=engine.dialect),
     )
-meta.create_all()
+meta.create_all(engine)
 ```

docs/guides/migration/v1-v2.md

@@ -12,7 +12,7 @@ Dataframely v2 finally implements lazy validation and filtering using a custom p
 
 ### Lazy `scan` operations
 
-With lazy validation, all `scan_*` methods (e.g. {meth}`Schema.scan_parquet`, {meth}`Collection.scan_delta`, ...) are
+With lazy validation, all `scan_*` methods (e.g. `Schema.scan_parquet`, `Collection.scan_delta`, ...) are
 now truly lazy, even if validation is necessary. Previously, this required collecting the input and running validation
 eagerly.
 

docs/guides/quickstart.md

@@ -38,13 +38,15 @@ This translates into the following expectations on our data:
 ## Custom rules
 
 While parameters in the column-initializers allow for defining expectations on a single column (e.g. `min_length` for
-string columns), this might not always by sufficient.
+string columns), this might not always be sufficient.
 In many cases, we want to check expectations across columns: for example, the ratio between the number of bathrooms and
 bedrooms should not be too high.
 
 In `dataframely`, we can do this by adding a custom rule to our schema:
 
 ```python
+import polars as pl
+
 import dataframely as dy
 
 
@@ -65,7 +67,30 @@ The decorator `@dy.rule()` "registers" the function as a rule using its name (i.
 The returned expression provides a boolean value for each row of the data which evaluates to `True` whenever the data
 are valid with respect to this rule.
 
-Note that, for more complex rules that should evaluate across _rows_, you can simply use an `over` expression.
+For defining even more complex rules that should evaluate across _rows_, you can simply use an `over` expression.
+For our housing data, this allows us to specify, for example, that we want to observe at least two houses per zip code:
+
+```python
+import polars as pl
+
+import dataframely as dy
+
+
+class HouseSchema(dy.Schema):
+    zip_code = dy.String(nullable=False, min_length=3)
+    num_bedrooms = dy.UInt8(nullable=False)
+    num_bathrooms = dy.UInt8(nullable=False)
+    price = dy.Float64(nullable=False)
+
+    @dy.rule()
+    def reasonable_bathroom_to_bedroom_ratio(cls) -> pl.Expr:
+        ratio = pl.col("num_bathrooms") / pl.col("num_bedrooms")
+        return (ratio >= 1 / 3) & (ratio <= 3)
+
+    @dy.rule()
+    def minimum_zip_code_count(cls) -> pl.Expr:
+        return (pl.len() >= 2).over("zip_code")
+```
 
 ````{note}
 If you are using [`ruff`](https://docs.astral.sh/ruff/) to lint your code, you'll need to tell `ruff` to treat rules like classmethods. To this end, you can add the following to your `pyproject.toml`:
@@ -101,17 +126,20 @@ validated_df = HouseSchema.validate(df, cast=True)
 
 If any row in `df` is invalid, i.e., any rule defined on individual columns or the entire schema evaluates to
 `False`, a validation exception is raised.
-Here, we have invalid data in the `num_bedrooms` and `zip_code` columns.
+Here, we have invalid data in the `num_bedrooms` and `zip_code` columns, and some rows violate the custom rules.
 
-    RuleValidationError: 2 rules failed validation:
-    * Column 'num_bedrooms' failed validation for 1 rules:
-    - 'nullability' failed for 2 rows
-    * Column 'zip_code' failed validation for 1 rules:
-    - 'min_length' failed for 1 rows
+    ValidationError: 4 rules failed validation:
+     - 'reasonable_bathroom_to_bedroom_ratio' failed for 1 rows
+     - 'minimum_zip_code_count' failed for 2 rows
+     * Column 'num_bedrooms' failed validation for 1 rules:
+       - 'nullability' failed for 2 rows
+     * Column 'zip_code' failed validation for 1 rules:
+       - 'min_length' failed for 1 rows
 
 Otherwise, if all rows in `df` are valid, `validate` returns a validated data frame of type
 `dy.DataFrame[HouseSchema]`.
-The same applies when a `pl.LazyFrame` is passed to `validate`.
+When a `pl.LazyFrame` is passed to `validate`, a `dy.LazyFrame[HouseSchema]` is returned instead and validation only
+runs once the lazy frame is collected (see [lazy validation](features/lazy-validation.md)).
 The generic data frame types allow for more readable function signatures to express
 expectations on the schema of the data frame, e.g.:
 
@@ -220,9 +248,8 @@ Lastly, `dataframely` schemas can be used to integrate with external tools:
   allowing to easily translate the dataframely schema into other packages' schema definitions:
   - `polars.Schema(HouseSchema)` can be used to construct a polars schema
   - `pyarrow.schema(HouseSchema)` can be used to construct a [pyarrow](https://arrow.apache.org/docs/python/index.html) schema
-- `HouseSchema.to_sqlalchemy_columns()` provides a list of [sqlalchemy](https://www.sqlalchemy.org) columns that can be
-  used to
-  create SQL tables using types and constraints in line with the schema
+- `HouseSchema.to_sqlalchemy_columns(dialect)` provides a list of [sqlalchemy](https://www.sqlalchemy.org) columns
+  that can be used to create SQL tables using types and constraints in line with the schema
 - You can use `dy.DataFrame[HouseSchema]` (or the `LazyFrame` equivalent) as fields in
   [pydantic](https://pydantic.dev) models, including support for validation and serialization. Integration with
   pydantic is unstable.