polars

# Polars

## Overview

Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.

## Quick Start

### Installation and Basic Usage

Install Polars:
```python
uv pip install polars
```

Basic DataFrame creation and operations:
```python
import polars as pl

# Create DataFrame
df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "city": ["NY", "LA", "SF"]
})

# Select columns
df.select("name", "age")

# Filter rows
df.filter(pl.col("age") > 25)

# Add computed columns
df.with_columns(
    age_plus_10=pl.col("age") + 10
)
```

## Core Concepts

### Expressions

Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.

**Key principles:**
- Use `pl.col("column_name")` to reference columns
- Chain methods to build complex transformations
- Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)

**Example:**
```python
# Expression-based computation
df.select(
    pl.col("name"),
    (pl.col("age") * 12).alias("age_in_months")
)
```

### Lazy vs Eager Evaluation

**Eager (DataFrame):** Operations execute immediately
```python
df = pl.read_csv("file.csv")  # Reads immediately
result = df.filter(pl.col("age") > 25)  # Executes immediately
```

**Lazy (LazyFrame):** Operations build a query plan, optimized before execution
```python
lf = pl.scan_csv("file.csv")  # Doesn't read yet
result = lf.filter(pl.col("age") > 25).select("name", "age")
df = result.collect()  # Now executes optimized query
```

**When to use lazy:**
- Working with large datasets
- Complex query pipelines
- When only some columns/rows are needed
- Performance is critical

**Benefits of lazy evaluation:**
- Automatic query optimization
- Predicate pushdown
- Projection pushdown
- Parallel execution

For detailed concepts, load `references/core_concepts.md`.

## Common Operations

### Select
Select and manipulate columns:
```python
# Select specific columns
df.select("name", "age")

# Select with expressions
df.select(
    pl.col("name"),
    (pl.col("age") * 2).alias("double_age")
)

# Select all columns matching a pattern
df.select(pl.col("^.*_id$"))
```

### Filter
Filter rows by conditions:
```python
# Single condition
df.filter(pl.col("age") > 25)

# Multiple conditions (cleaner than using &)
df.filter(
    pl.col("age") > 25,
    pl.col("city") == "NY"
)

# Complex conditions
df.filter(
    (pl.col("age") > 25) | (pl.col("city") == "LA")
)
```

### With Columns
Add or modify columns while preserving existing ones:
```python
# Add new columns
df.with_columns(
    age_plus_10=pl.col("age") + 10,
    name_upper=pl.col("name").str.to_uppercase()
)

# Parallel computation (all columns computed in parallel)
df.with_columns(
    pl.col("value") * 10,
    pl.col("value") * 100,
)
```

### Group By and Aggregations
Group data and compute aggregations:
```python
# Basic grouping
df.group_by("city").agg(
    pl.col("age").mean().alias("avg_age"),
    pl.len().alias("count")
)

# Multiple group keys
df.group_by("city", "department").agg(
    pl.col("salary").sum()
)

# Conditional aggregations
df.group_by("city").agg(
    (pl.col("age") > 30).sum().alias("over_30")
)
```

For detailed operation patterns, load `references/operations.md`.

## Aggregations and Window Functions

### Aggregation Functions
Common aggregations within `group_by` context:
- `pl.len()` - count rows
- `pl.col("x").sum()` - sum values
- `pl.col("x").mean()` - average
- `pl.col("x").min()` / `pl.col("x").max()` - extremes
- `pl.first()` / `pl.last()` - first/last values

### Window Functions with `over()`
Apply aggregations while preserving row count:
```python
# Add group statistics to each row
df.with_columns(
    avg_age_by_city=pl.col("age").mean().over("city"),
    rank_in_city=pl.col("salary").rank().over("city")
)

# Multiple grouping columns
df.with_columns(
    group_avg=pl.col("value").mean().over("category", "region")
)
```

**Mapping strategies:**
- `group_to_rows` (default): Preserves original row order
- `explode`: Faster but groups rows together
- `join`: Creates list columns

## Data I/O

### Supported Formats
Polars supports reading and writing:
- CSV, Parquet, JSON, Excel
- Databases (via connectors)
- Cloud storage (S3, Azure, GCS)
- Google BigQuery
- Multiple/partitioned files

### Common I/O Operations

**CSV:**
```python
# Eager
df = pl.read_csv("file.csv")
df.write_csv("output.csv")

# Lazy (preferred for large files)
lf = pl.scan_csv("file.csv")
result = lf.filter(...).select(...).collect()
```

**Parquet (recommended for performance):**
```python
df = pl.read_parquet("file.parquet")
df.write_parquet("output.parquet")
```

**JSON:**
```python
df = pl.read_json("file.json")
df.write_json("output.json")
```

For comprehensive I/O documentation, load `references/io_guide.md`.

## Transformations

### Joins
Combine DataFrames:
```python
# Inner join
df1.join(df2, on="id", how="inner")

# Left join
df1.join(df2, on="id", how="left")

# Join on different column names
df1.join(df2, left_on="user_id", right_on="id")
```

### Concatenation
Stack DataFrames:
```python
# Vertical (stack rows)
pl.concat([df1, df2], how="vertical")

# Horizontal (add columns)
pl.concat([df1, df2], how="horizontal")

# Diagonal (union with different schemas)
pl.concat([df1, df2], how="diagonal")
```

### Pivot and Unpivot
Reshape data:
```python
# Pivot (wide format)
df.pivot(values="sales", index="date", columns="product")

# Unpivot (long format)
df.unpivot(index="id", on=["col1", "col2"])
```

For detailed transformation examples, load `references/transformations.md`.

## Pandas Migration

Polars offers significant performance improvements over pandas with a cleaner API. Key differences:

### Conceptual Differences
- **No index**: Polars uses integer positions only
- **Strict typing**: No silent type conversions
- **Lazy evaluation**: Available via LazyFrame
- **Parallel by default**: Operations parallelized automatically

### Common Operation Mappings

| Operation | Pandas | Polars |
|-----------|--------|--------|
| Select column | `df["col"]` | `df.select("col")` |
| Filter | `df[df["col"] > 10]` | `df.filter(pl.col("col") > 10)` |
| Add column | `df.assign(x=...)` | `df.with_columns(x=...)` |
| Group by | `df.groupby("col").agg(...)` | `df.group_by("col").agg(...)` |
| Window | `df.groupby("col").transform(...)` | `df.with_columns(...).over("col")` |

### Key Syntax Patterns

**Pandas sequential (slow):**
```python
df.assign(
    col_a=lambda df_: df_.value * 10,
    col_b=lambda df_: df_.value * 100
)
```

**Polars parallel (fast):**
```python
df.with_columns(
    col_a=pl.col("value") * 10,
    col_b=pl.col("value") * 100,
)
```

For comprehensive migration guide, load `references/pandas_migration.md`.

## Best Practices

### Performance Optimization

1. **Use lazy evaluation for large datasets:**
   ```python
   lf = pl.scan_csv("large.csv")  # Don't use read_csv
   result = lf.filter(...).select(...).collect()
   ```

2. **Avoid Python functions in hot paths:**
   - Stay within expression API for parallelization
   - Use `.map_elements()` only when necessary
   - Prefer native Polars operations

3. **Use streaming for very large data:**
   ```python
   lf.collect(streaming=True)
   ```

4. **Select only needed columns early:**
   ```python
   # Good: Select columns early
   lf.select("col1", "col2").filter(...)

   # Bad: Filter on all columns first
   lf.filter(...).select("col1", "col2")
   ```

5. **Use appropriate data types:**
   - Categorical for low-cardinality strings
   - Appropriate integer sizes (i32 vs i64)
   - Date types for temporal data

### Expression Patterns

**Conditional operations:**
```python
pl.when(condition).then(value).otherwise(other_value)
```

**Column operations across multiple columns:**
```python
df.select(pl.col("^.*_value$") * 2)  # Regex pattern
```

**Null handling:**
```python
pl.col("x").fill_null(0)
pl.col("x").is_null()
pl.col("x").drop_nulls()
```

For additional best practices and patterns, load `references/best_practices.md`.

## Resources

This skill includes comprehensive reference documentation:

### references/
- `core_concepts.md` - Detailed explanations of expressions, lazy evaluation, and type system
- `operations.md` - Comprehensive guide to all common operations with examples
- `pandas_migration.md` - Complete migration guide from pandas to Polars
- `io_guide.md` - Data I/O operations for all supported formats
- `transformations.md` - Joins, concatenation, pivots, and reshaping operations
- `best_practices.md` - Performance optimization tips and common patterns

Load these references as needed when users require detailed information about specific topics.

{
  "format": "aiskill/v1",
  "skill": {
    "name": "polars",
    "slug": "polars",
    "version": "1.0.0",
    "description": "Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.",
    "category": "Research",
    "tags": [
      "research-science"
    ],
    "platforms": [
      "chatgpt",
      "claude-code",
      "cursor"
    ],
    "tested_environment": "Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and compatible agents",
    "license": "HTTPS://GITHUB.COM/POLA-RS/POLARS/BLOB/MAIN/LICENSE"
  },
  "prompts": {
    "system": "# Polars\r\n\r\n## Overview\r\n\r\nPolars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.\r\n\r\n## Quick Start\r\n\r\n### Installation and Basic Usage\r\n\r\nInstall Polars:\r\n```python\r\nuv pip install polars\r\n```\r\n\r\nBasic DataFrame creation and operations:\r\n```python\r\nimport polars as pl\r\n\r\n# Create DataFrame\r\ndf = pl.DataFrame({\r\n    \"name\": [\"Alice\", \"Bob\", \"Charlie\"],\r\n    \"age\": [25, 30, 35],\r\n    \"city\": [\"NY\", \"LA\", \"SF\"]\r\n})\r\n\r\n# Select columns\r\ndf.select(\"name\", \"age\")\r\n\r\n# Filter rows\r\ndf.filter(pl.col(\"age\") > 25)\r\n\r\n# Add computed columns\r\ndf.with_columns(\r\n    age_plus_10=pl.col(\"age\") + 10\r\n)\r\n```\r\n\r\n## Core Concepts\r\n\r\n### Expressions\r\n\r\nExpressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.\r\n\r\n**Key principles:**\r\n- Use `pl.col(\"column_name\")` to reference columns\r\n- Chain methods to build complex transformations\r\n- Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)\r\n\r\n**Example:**\r\n```python\r\n# Expression-based computation\r\ndf.select(\r\n    pl.col(\"name\"),\r\n    (pl.col(\"age\") * 12).alias(\"age_in_months\")\r\n)\r\n```\r\n\r\n### Lazy vs Eager Evaluation\r\n\r\n**Eager (DataFrame):** Operations execute immediately\r\n```python\r\ndf = pl.read_csv(\"file.csv\")  # Reads immediately\r\nresult = df.filter(pl.col(\"age\") > 25)  # Executes immediately\r\n```\r\n\r\n**Lazy (LazyFrame):** Operations build a query plan, optimized before execution\r\n```python\r\nlf = pl.scan_csv(\"file.csv\")  # Doesn't read yet\r\nresult = lf.filter(pl.col(\"age\") > 25).select(\"name\", \"age\")\r\ndf = result.collect()  # Now executes optimized query\r\n```\r\n\r\n**When to use lazy:**\r\n- Working with large datasets\r\n- Complex query pipelines\r\n- When only some columns/rows are needed\r\n- Performance is critical\r\n\r\n**Benefits of lazy evaluation:**\r\n- Automatic query optimization\r\n- Predicate pushdown\r\n- Projection pushdown\r\n- Parallel execution\r\n\r\nFor detailed concepts, load `references/core_concepts.md`.\r\n\r\n## Common Operations\r\n\r\n### Select\r\nSelect and manipulate columns:\r\n```python\r\n# Select specific columns\r\ndf.select(\"name\", \"age\")\r\n\r\n# Select with expressions\r\ndf.select(\r\n    pl.col(\"name\"),\r\n    (pl.col(\"age\") * 2).alias(\"double_age\")\r\n)\r\n\r\n# Select all columns matching a pattern\r\ndf.select(pl.col(\"^.*_id$\"))\r\n```\r\n\r\n### Filter\r\nFilter rows by conditions:\r\n```python\r\n# Single condition\r\ndf.filter(pl.col(\"age\") > 25)\r\n\r\n# Multiple conditions (cleaner than using &)\r\ndf.filter(\r\n    pl.col(\"age\") > 25,\r\n    pl.col(\"city\") == \"NY\"\r\n)\r\n\r\n# Complex conditions\r\ndf.filter(\r\n    (pl.col(\"age\") > 25) | (pl.col(\"city\") == \"LA\")\r\n)\r\n```\r\n\r\n### With Columns\r\nAdd or modify columns while preserving existing ones:\r\n```python\r\n# Add new columns\r\ndf.with_columns(\r\n    age_plus_10=pl.col(\"age\") + 10,\r\n    name_upper=pl.col(\"name\").str.to_uppercase()\r\n)\r\n\r\n# Parallel computation (all columns computed in parallel)\r\ndf.with_columns(\r\n    pl.col(\"value\") * 10,\r\n    pl.col(\"value\") * 100,\r\n)\r\n```\r\n\r\n### Group By and Aggregations\r\nGroup data and compute aggregations:\r\n```python\r\n# Basic grouping\r\ndf.group_by(\"city\").agg(\r\n    pl.col(\"age\").mean().alias(\"avg_age\"),\r\n    pl.len().alias(\"count\")\r\n)\r\n\r\n# Multiple group keys\r\ndf.group_by(\"city\", \"department\").agg(\r\n    pl.col(\"salary\").sum()\r\n)\r\n\r\n# Conditional aggregations\r\ndf.group_by(\"city\").agg(\r\n    (pl.col(\"age\") > 30).sum().alias(\"over_30\")\r\n)\r\n```\r\n\r\nFor detailed operation patterns, load `references/operations.md`.\r\n\r\n## Aggregations and Window Functions\r\n\r\n### Aggregation Functions\r\nCommon aggregations within `group_by` context:\r\n- `pl.len()` - count rows\r\n- `pl.col(\"x\").sum()` - sum values\r\n- `pl.col(\"x\").mean()` - average\r\n- `pl.col(\"x\").min()` / `pl.col(\"x\").max()` - extremes\r\n- `pl.first()` / `pl.last()` - first/last values\r\n\r\n### Window Functions with `over()`\r\nApply aggregations while preserving row count:\r\n```python\r\n# Add group statistics to each row\r\ndf.with_columns(\r\n    avg_age_by_city=pl.col(\"age\").mean().over(\"city\"),\r\n    rank_in_city=pl.col(\"salary\").rank().over(\"city\")\r\n)\r\n\r\n# Multiple grouping columns\r\ndf.with_columns(\r\n    group_avg=pl.col(\"value\").mean().over(\"category\", \"region\")\r\n)\r\n```\r\n\r\n**Mapping strategies:**\r\n- `group_to_rows` (default): Preserves original row order\r\n- `explode`: Faster but groups rows together\r\n- `join`: Creates list columns\r\n\r\n## Data I/O\r\n\r\n### Supported Formats\r\nPolars supports reading and writing:\r\n- CSV, Parquet, JSON, Excel\r\n- Databases (via connectors)\r\n- Cloud storage (S3, Azure, GCS)\r\n- Google BigQuery\r\n- Multiple/partitioned files\r\n\r\n### Common I/O Operations\r\n\r\n**CSV:**\r\n```python\r\n# Eager\r\ndf = pl.read_csv(\"file.csv\")\r\ndf.write_csv(\"output.csv\")\r\n\r\n# Lazy (preferred for large files)\r\nlf = pl.scan_csv(\"file.csv\")\r\nresult = lf.filter(...).select(...).collect()\r\n```\r\n\r\n**Parquet (recommended for performance):**\r\n```python\r\ndf = pl.read_parquet(\"file.parquet\")\r\ndf.write_parquet(\"output.parquet\")\r\n```\r\n\r\n**JSON:**\r\n```python\r\ndf = pl.read_json(\"file.json\")\r\ndf.write_json(\"output.json\")\r\n```\r\n\r\nFor comprehensive I/O documentation, load `references/io_guide.md`.\r\n\r\n## Transformations\r\n\r\n### Joins\r\nCombine DataFrames:\r\n```python\r\n# Inner join\r\ndf1.join(df2, on=\"id\", how=\"inner\")\r\n\r\n# Left join\r\ndf1.join(df2, on=\"id\", how=\"left\")\r\n\r\n# Join on different column names\r\ndf1.join(df2, left_on=\"user_id\", right_on=\"id\")\r\n```\r\n\r\n### Concatenation\r\nStack DataFrames:\r\n```python\r\n# Vertical (stack rows)\r\npl.concat([df1, df2], how=\"vertical\")\r\n\r\n# Horizontal (add columns)\r\npl.concat([df1, df2], how=\"horizontal\")\r\n\r\n# Diagonal (union with different schemas)\r\npl.concat([df1, df2], how=\"diagonal\")\r\n```\r\n\r\n### Pivot and Unpivot\r\nReshape data:\r\n```python\r\n# Pivot (wide format)\r\ndf.pivot(values=\"sales\", index=\"date\", columns=\"product\")\r\n\r\n# Unpivot (long format)\r\ndf.unpivot(index=\"id\", on=[\"col1\", \"col2\"])\r\n```\r\n\r\nFor detailed transformation examples, load `references/transformations.md`.\r\n\r\n## Pandas Migration\r\n\r\nPolars offers significant performance improvements over pandas with a cleaner API. Key differences:\r\n\r\n### Conceptual Differences\r\n- **No index**: Polars uses integer positions only\r\n- **Strict typing**: No silent type conversions\r\n- **Lazy evaluation**: Available via LazyFrame\r\n- **Parallel by default**: Operations parallelized automatically\r\n\r\n### Common Operation Mappings\r\n\r\n| Operation | Pandas | Polars |\r\n|-----------|--------|--------|\r\n| Select column | `df[\"col\"]` | `df.select(\"col\")` |\r\n| Filter | `df[df[\"col\"] > 10]` | `df.filter(pl.col(\"col\") > 10)` |\r\n| Add column | `df.assign(x=...)` | `df.with_columns(x=...)` |\r\n| Group by | `df.groupby(\"col\").agg(...)` | `df.group_by(\"col\").agg(...)` |\r\n| Window | `df.groupby(\"col\").transform(...)` | `df.with_columns(...).over(\"col\")` |\r\n\r\n### Key Syntax Patterns\r\n\r\n**Pandas sequential (slow):**\r\n```python\r\ndf.assign(\r\n    col_a=lambda df_: df_.value * 10,\r\n    col_b=lambda df_: df_.value * 100\r\n)\r\n```\r\n\r\n**Polars parallel (fast):**\r\n```python\r\ndf.with_columns(\r\n    col_a=pl.col(\"value\") * 10,\r\n    col_b=pl.col(\"value\") * 100,\r\n)\r\n```\r\n\r\nFor comprehensive migration guide, load `references/pandas_migration.md`.\r\n\r\n## Best Practices\r\n\r\n### Performance Optimization\r\n\r\n1. **Use lazy evaluation for large datasets:**\r\n   ```python\r\n   lf = pl.scan_csv(\"large.csv\")  # Don't use read_csv\r\n   result = lf.filter(...).select(...).collect()\r\n   ```\r\n\r\n2. **Avoid Python functions in hot paths:**\r\n   - Stay within expression API for parallelization\r\n   - Use `.map_elements()` only when necessary\r\n   - Prefer native Polars operations\r\n\r\n3. **Use streaming for very large data:**\r\n   ```python\r\n   lf.collect(streaming=True)\r\n   ```\r\n\r\n4. **Select only needed columns early:**\r\n   ```python\r\n   # Good: Select columns early\r\n   lf.select(\"col1\", \"col2\").filter(...)\r\n\r\n   # Bad: Filter on all columns first\r\n   lf.filter(...).select(\"col1\", \"col2\")\r\n   ```\r\n\r\n5. **Use appropriate data types:**\r\n   - Categorical for low-cardinality strings\r\n   - Appropriate integer sizes (i32 vs i64)\r\n   - Date types for temporal data\r\n\r\n### Expression Patterns\r\n\r\n**Conditional operations:**\r\n```python\r\npl.when(condition).then(value).otherwise(other_value)\r\n```\r\n\r\n**Column operations across multiple columns:**\r\n```python\r\ndf.select(pl.col(\"^.*_value$\") * 2)  # Regex pattern\r\n```\r\n\r\n**Null handling:**\r\n```python\r\npl.col(\"x\").fill_null(0)\r\npl.col(\"x\").is_null()\r\npl.col(\"x\").drop_nulls()\r\n```\r\n\r\nFor additional best practices and patterns, load `references/best_practices.md`.\r\n\r\n## Resources\r\n\r\nThis skill includes comprehensive reference documentation:\r\n\r\n### references/\r\n- `core_concepts.md` - Detailed explanations of expressions, lazy evaluation, and type system\r\n- `operations.md` - Comprehensive guide to all common operations with examples\r\n- `pandas_migration.md` - Complete migration guide from pandas to Polars\r\n- `io_guide.md` - Data I/O operations for all supported formats\r\n- `transformations.md` - Joins, concatenation, pivots, and reshaping operations\r\n- `best_practices.md` - Performance optimization tips and common patterns\r\n\r\nLoad these references as needed when users require detailed information about specific topics.",
    "user_template": ""
  },
  "variables": [],
  "install": {
    "steps": [
      {
        "title": "Download & unzip",
        "detail": "Download the .zip package from SkillVault and unzip it. The polars/ folder contains SKILL.md plus any companion scripts, references and assets."
      },
      {
        "title": "Copy into your agent skills directory",
        "detail": "Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code), .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or reload the session."
      },
      {
        "title": "Use it",
        "detail": "Describe the task in natural language; the agent loads the skill automatically when the request matches the skill description."
      }
    ]
  },
  "usage": {
    "steps": []
  },
  "metadata": {
    "author": "SkillVault",
    "source": "https://skillvault.top/skills/polars",
    "exported_at": "2026-11-03T07:44:19.825Z"
  }
}

format: aiskill/v1
skill:
  name: polars
  slug: polars
  version: 1.0.0
  description: >-
    Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but
    data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for
    1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or
    vaex.
  category: Research
  tags:
    - research-science
  platforms:
    - chatgpt
    - claude-code
    - cursor
  tested_environment: >-
    Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and
    compatible agents
  license: HTTPS://GITHUB.COM/POLA-RS/POLARS/BLOB/MAIN/LICENSE
prompts:
  system: "# Polars\r\n\r\n## Overview\r\n\r\nPolars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.\r\n\r\n## Quick Start\r\n\r\n### Installation and Basic Usage\r\n\r\nInstall Polars:\r\n```python\r\nuv pip install polars\r\n```\r\n\r\nBasic DataFrame creation and operations:\r\n```python\r\nimport polars as pl\r\n\r\n# Create DataFrame\r\ndf = pl.DataFrame({\r\n    \"name\": [\"Alice\", \"Bob\", \"Charlie\"],\r\n    \"age\": [25, 30, 35],\r\n    \"city\": [\"NY\", \"LA\", \"SF\"]\r\n})\r\n\r\n# Select columns\r\ndf.select(\"name\", \"age\")\r\n\r\n# Filter rows\r\ndf.filter(pl.col(\"age\") > 25)\r\n\r\n# Add computed columns\r\ndf.with_columns(\r\n    age_plus_10=pl.col(\"age\") + 10\r\n)\r\n```\r\n\r\n## Core Concepts\r\n\r\n### Expressions\r\n\r\nExpressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.\r\n\r\n**Key principles:**\r\n- Use `pl.col(\"column_name\")` to reference columns\r\n- Chain methods to build complex transformations\r\n- Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)\r\n\r\n**Example:**\r\n```python\r\n# Expression-based computation\r\ndf.select(\r\n    pl.col(\"name\"),\r\n    (pl.col(\"age\") * 12).alias(\"age_in_months\")\r\n)\r\n```\r\n\r\n### Lazy vs Eager Evaluation\r\n\r\n**Eager (DataFrame):** Operations execute immediately\r\n```python\r\ndf = pl.read_csv(\"file.csv\")  # Reads immediately\r\nresult = df.filter(pl.col(\"age\") > 25)  # Executes immediately\r\n```\r\n\r\n**Lazy (LazyFrame):** Operations build a query plan, optimized before execution\r\n```python\r\nlf = pl.scan_csv(\"file.csv\")  # Doesn't read yet\r\nresult = lf.filter(pl.col(\"age\") > 25).select(\"name\", \"age\")\r\ndf = result.collect()  # Now executes optimized query\r\n```\r\n\r\n**When to use lazy:**\r\n- Working with large datasets\r\n- Complex query pipelines\r\n- When only some columns/rows are needed\r\n- Performance is critical\r\n\r\n**Benefits of lazy evaluation:**\r\n- Automatic query optimization\r\n- Predicate pushdown\r\n- Projection pushdown\r\n- Parallel execution\r\n\r\nFor detailed concepts, load `references/core_concepts.md`.\r\n\r\n## Common Operations\r\n\r\n### Select\r\nSelect and manipulate columns:\r\n```python\r\n# Select specific columns\r\ndf.select(\"name\", \"age\")\r\n\r\n# Select with expressions\r\ndf.select(\r\n    pl.col(\"name\"),\r\n    (pl.col(\"age\") * 2).alias(\"double_age\")\r\n)\r\n\r\n# Select all columns matching a pattern\r\ndf.select(pl.col(\"^.*_id$\"))\r\n```\r\n\r\n### Filter\r\nFilter rows by conditions:\r\n```python\r\n# Single condition\r\ndf.filter(pl.col(\"age\") > 25)\r\n\r\n# Multiple conditions (cleaner than using &)\r\ndf.filter(\r\n    pl.col(\"age\") > 25,\r\n    pl.col(\"city\") == \"NY\"\r\n)\r\n\r\n# Complex conditions\r\ndf.filter(\r\n    (pl.col(\"age\") > 25) | (pl.col(\"city\") == \"LA\")\r\n)\r\n```\r\n\r\n### With Columns\r\nAdd or modify columns while preserving existing ones:\r\n```python\r\n# Add new columns\r\ndf.with_columns(\r\n    age_plus_10=pl.col(\"age\") + 10,\r\n    name_upper=pl.col(\"name\").str.to_uppercase()\r\n)\r\n\r\n# Parallel computation (all columns computed in parallel)\r\ndf.with_columns(\r\n    pl.col(\"value\") * 10,\r\n    pl.col(\"value\") * 100,\r\n)\r\n```\r\n\r\n### Group By and Aggregations\r\nGroup data and compute aggregations:\r\n```python\r\n# Basic grouping\r\ndf.group_by(\"city\").agg(\r\n    pl.col(\"age\").mean().alias(\"avg_age\"),\r\n    pl.len().alias(\"count\")\r\n)\r\n\r\n# Multiple group keys\r\ndf.group_by(\"city\", \"department\").agg(\r\n    pl.col(\"salary\").sum()\r\n)\r\n\r\n# Conditional aggregations\r\ndf.group_by(\"city\").agg(\r\n    (pl.col(\"age\") > 30).sum().alias(\"over_30\")\r\n)\r\n```\r\n\r\nFor detailed operation patterns, load `references/operations.md`.\r\n\r\n## Aggregations and Window Functions\r\n\r\n### Aggregation Functions\r\nCommon aggregations within `group_by` context:\r\n- `pl.len()` - count rows\r\n- `pl.col(\"x\").sum()` - sum values\r\n- `pl.col(\"x\").mean()` - average\r\n- `pl.col(\"x\").min()` / `pl.col(\"x\").max()` - extremes\r\n- `pl.first()` / `pl.last()` - first/last values\r\n\r\n### Window Functions with `over()`\r\nApply aggregations while preserving row count:\r\n```python\r\n# Add group statistics to each row\r\ndf.with_columns(\r\n    avg_age_by_city=pl.col(\"age\").mean().over(\"city\"),\r\n    rank_in_city=pl.col(\"salary\").rank().over(\"city\")\r\n)\r\n\r\n# Multiple grouping columns\r\ndf.with_columns(\r\n    group_avg=pl.col(\"value\").mean().over(\"category\", \"region\")\r\n)\r\n```\r\n\r\n**Mapping strategies:**\r\n- `group_to_rows` (default): Preserves original row order\r\n- `explode`: Faster but groups rows together\r\n- `join`: Creates list columns\r\n\r\n## Data I/O\r\n\r\n### Supported Formats\r\nPolars supports reading and writing:\r\n- CSV, Parquet, JSON, Excel\r\n- Databases (via connectors)\r\n- Cloud storage (S3, Azure, GCS)\r\n- Google BigQuery\r\n- Multiple/partitioned files\r\n\r\n### Common I/O Operations\r\n\r\n**CSV:**\r\n```python\r\n# Eager\r\ndf = pl.read_csv(\"file.csv\")\r\ndf.write_csv(\"output.csv\")\r\n\r\n# Lazy (preferred for large files)\r\nlf = pl.scan_csv(\"file.csv\")\r\nresult = lf.filter(...).select(...).collect()\r\n```\r\n\r\n**Parquet (recommended for performance):**\r\n```python\r\ndf = pl.read_parquet(\"file.parquet\")\r\ndf.write_parquet(\"output.parquet\")\r\n```\r\n\r\n**JSON:**\r\n```python\r\ndf = pl.read_json(\"file.json\")\r\ndf.write_json(\"output.json\")\r\n```\r\n\r\nFor comprehensive I/O documentation, load `references/io_guide.md`.\r\n\r\n## Transformations\r\n\r\n### Joins\r\nCombine DataFrames:\r\n```python\r\n# Inner join\r\ndf1.join(df2, on=\"id\", how=\"inner\")\r\n\r\n# Left join\r\ndf1.join(df2, on=\"id\", how=\"left\")\r\n\r\n# Join on different column names\r\ndf1.join(df2, left_on=\"user_id\", right_on=\"id\")\r\n```\r\n\r\n### Concatenation\r\nStack DataFrames:\r\n```python\r\n# Vertical (stack rows)\r\npl.concat([df1, df2], how=\"vertical\")\r\n\r\n# Horizontal (add columns)\r\npl.concat([df1, df2], how=\"horizontal\")\r\n\r\n# Diagonal (union with different schemas)\r\npl.concat([df1, df2], how=\"diagonal\")\r\n```\r\n\r\n### Pivot and Unpivot\r\nReshape data:\r\n```python\r\n# Pivot (wide format)\r\ndf.pivot(values=\"sales\", index=\"date\", columns=\"product\")\r\n\r\n# Unpivot (long format)\r\ndf.unpivot(index=\"id\", on=[\"col1\", \"col2\"])\r\n```\r\n\r\nFor detailed transformation examples, load `references/transformations.md`.\r\n\r\n## Pandas Migration\r\n\r\nPolars offers significant performance improvements over pandas with a cleaner API. Key differences:\r\n\r\n### Conceptual Differences\r\n- **No index**: Polars uses integer positions only\r\n- **Strict typing**: No silent type conversions\r\n- **Lazy evaluation**: Available via LazyFrame\r\n- **Parallel by default**: Operations parallelized automatically\r\n\r\n### Common Operation Mappings\r\n\r\n| Operation | Pandas | Polars |\r\n|-----------|--------|--------|\r\n| Select column | `df[\"col\"]` | `df.select(\"col\")` |\r\n| Filter | `df[df[\"col\"] > 10]` | `df.filter(pl.col(\"col\") > 10)` |\r\n| Add column | `df.assign(x=...)` | `df.with_columns(x=...)` |\r\n| Group by | `df.groupby(\"col\").agg(...)` | `df.group_by(\"col\").agg(...)` |\r\n| Window | `df.groupby(\"col\").transform(...)` | `df.with_columns(...).over(\"col\")` |\r\n\r\n### Key Syntax Patterns\r\n\r\n**Pandas sequential (slow):**\r\n```python\r\ndf.assign(\r\n    col_a=lambda df_: df_.value * 10,\r\n    col_b=lambda df_: df_.value * 100\r\n)\r\n```\r\n\r\n**Polars parallel (fast):**\r\n```python\r\ndf.with_columns(\r\n    col_a=pl.col(\"value\") * 10,\r\n    col_b=pl.col(\"value\") * 100,\r\n)\r\n```\r\n\r\nFor comprehensive migration guide, load `references/pandas_migration.md`.\r\n\r\n## Best Practices\r\n\r\n### Performance Optimization\r\n\r\n1. **Use lazy evaluation for large datasets:**\r\n   ```python\r\n   lf = pl.scan_csv(\"large.csv\")  # Don't use read_csv\r\n   result = lf.filter(...).select(...).collect()\r\n   ```\r\n\r\n2. **Avoid Python functions in hot paths:**\r\n   - Stay within expression API for parallelization\r\n   - Use `.map_elements()` only when necessary\r\n   - Prefer native Polars operations\r\n\r\n3. **Use streaming for very large data:**\r\n   ```python\r\n   lf.collect(streaming=True)\r\n   ```\r\n\r\n4. **Select only needed columns early:**\r\n   ```python\r\n   # Good: Select columns early\r\n   lf.select(\"col1\", \"col2\").filter(...)\r\n\r\n   # Bad: Filter on all columns first\r\n   lf.filter(...).select(\"col1\", \"col2\")\r\n   ```\r\n\r\n5. **Use appropriate data types:**\r\n   - Categorical for low-cardinality strings\r\n   - Appropriate integer sizes (i32 vs i64)\r\n   - Date types for temporal data\r\n\r\n### Expression Patterns\r\n\r\n**Conditional operations:**\r\n```python\r\npl.when(condition).then(value).otherwise(other_value)\r\n```\r\n\r\n**Column operations across multiple columns:**\r\n```python\r\ndf.select(pl.col(\"^.*_value$\") * 2)  # Regex pattern\r\n```\r\n\r\n**Null handling:**\r\n```python\r\npl.col(\"x\").fill_null(0)\r\npl.col(\"x\").is_null()\r\npl.col(\"x\").drop_nulls()\r\n```\r\n\r\nFor additional best practices and patterns, load `references/best_practices.md`.\r\n\r\n## Resources\r\n\r\nThis skill includes comprehensive reference documentation:\r\n\r\n### references/\r\n- `core_concepts.md` - Detailed explanations of expressions, lazy evaluation, and type system\r\n- `operations.md` - Comprehensive guide to all common operations with examples\r\n- `pandas_migration.md` - Complete migration guide from pandas to Polars\r\n- `io_guide.md` - Data I/O operations for all supported formats\r\n- `transformations.md` - Joins, concatenation, pivots, and reshaping operations\r\n- `best_practices.md` - Performance optimization tips and common patterns\r\n\r\nLoad these references as needed when users require detailed information about specific topics."
  user_template: ""
variables: []
install:
  steps:
    - title: Download & unzip
      detail: >-
        Download the .zip package from SkillVault and unzip it. The polars/ folder contains SKILL.md
        plus any companion scripts, references and assets.
    - title: Copy into your agent skills directory
      detail: >-
        Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code),
        .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or
        reload the session.
    - title: Use it
      detail: >-
        Describe the task in natural language; the agent loads the skill automatically when the
        request matches the skill description.
usage:
  steps: []
metadata:
  author: SkillVault
  source: https://skillvault.top/skills/polars
  exported_at: "2026-11-03T07:44:19.825Z"