vaex

# Vaex

## Overview

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

## When to Use This Skill

Use Vaex when:
- Processing tabular datasets larger than available RAM (gigabytes to terabytes)
- Performing fast statistical aggregations on massive datasets
- Creating visualizations and heatmaps of large datasets
- Building machine learning pipelines on big data
- Converting between data formats (CSV, HDF5, Arrow, Parquet)
- Needing lazy evaluation and virtual columns to avoid memory overhead
- Working with astronomical data, financial time series, or other large-scale scientific datasets

## Core Capabilities

Vaex provides six primary capability areas, each documented in detail in the references directory:

### 1. DataFrames and Data Loading

Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference `references/core_dataframes.md` for:
- Opening large files efficiently
- Converting from pandas/NumPy/Arrow
- Working with example datasets
- Understanding DataFrame structure

### 2. Data Processing and Manipulation

Perform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference `references/data_processing.md` for:
- Filtering and selections
- Virtual columns and expressions
- Groupby operations and aggregations
- String operations and datetime handling
- Working with missing data

### 3. Performance and Optimization

Leverage Vaex's lazy evaluation, caching strategies, and memory-efficient operations. Reference `references/performance.md` for:
- Understanding lazy evaluation
- Using `delay=True` for batching operations
- Materializing columns when needed
- Caching strategies
- Asynchronous operations

### 4. Data Visualization

Create interactive visualizations of large datasets including heatmaps, histograms, and scatter plots. Reference `references/visualization.md` for:
- Creating 1D and 2D plots
- Heatmap visualizations
- Working with selections
- Customizing plots and subplots

### 5. Machine Learning Integration

Build ML pipelines with transformers, encoders, and integration with scikit-learn, XGBoost, and other frameworks. Reference `references/machine_learning.md` for:
- Feature scaling and encoding
- PCA and dimensionality reduction
- K-means clustering
- Integration with scikit-learn/XGBoost/CatBoost
- Model serialization and deployment

### 6. I/O Operations

Efficiently read and write data in various formats with optimal performance. Reference `references/io_operations.md` for:
- File format recommendations
- Export strategies
- Working with Apache Arrow
- CSV handling for large files
- Server and remote data access

## Quick Start Pattern

For most Vaex tasks, follow this pattern:

```python
import vaex

# 1. Open or create DataFrame
df = vaex.open('large_file.hdf5')  # or .csv, .arrow, .parquet
# OR
df = vaex.from_pandas(pandas_df)

# 2. Explore the data
print(df)  # Shows first/last rows and column info
df.describe()  # Statistical summary

# 3. Create virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y

# 4. Filter with selections
df_filtered = df[df.age > 25]

# 5. Compute statistics (fast, lazy evaluation)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})

# 6. Visualize
df.plot1d(df.x, limits=[0, 100])
df.plot(df.x, df.y, limits='99.7%')

# 7. Export if needed
df.export_hdf5('output.hdf5')
```

## Working with References

The reference files contain detailed information about each capability area. Load references into context based on the specific task:

- **Basic operations**: Start with `references/core_dataframes.md` and `references/data_processing.md`
- **Performance issues**: Check `references/performance.md`
- **Visualization tasks**: Use `references/visualization.md`
- **ML pipelines**: Reference `references/machine_learning.md`
- **File I/O**: Consult `references/io_operations.md`

## Best Practices

1. **Use HDF5 or Apache Arrow formats** for optimal performance with large datasets
2. **Leverage virtual columns** instead of materializing data to save memory
3. **Batch operations** using `delay=True` when performing multiple calculations
4. **Export to efficient formats** rather than keeping data in CSV
5. **Use expressions** for complex calculations without intermediate storage
6. **Profile with `df.stat()`** to understand memory usage and optimize operations

## Common Patterns

### Pattern: Converting Large CSV to HDF5
```python
import vaex

# Open large CSV (processes in chunks automatically)
df = vaex.from_csv('large_file.csv')

# Export to HDF5 for faster future access
df.export_hdf5('large_file.hdf5')

# Future loads are instant
df = vaex.open('large_file.hdf5')
```

### Pattern: Efficient Aggregations
```python
# Use delay=True to batch multiple operations
mean_x = df.x.mean(delay=True)
std_y = df.y.std(delay=True)
sum_z = df.z.sum(delay=True)

# Execute all at once
results = vaex.execute([mean_x, std_y, sum_z])
```

### Pattern: Virtual Columns for Feature Engineering
```python
# No memory overhead - computed on the fly
df['age_squared'] = df.age ** 2
df['full_name'] = df.first_name + ' ' + df.last_name
df['is_adult'] = df.age >= 18
```

## Resources

This skill includes reference documentation in the `references/` directory:

- `core_dataframes.md` - DataFrame creation, loading, and basic structure
- `data_processing.md` - Filtering, expressions, aggregations, and transformations
- `performance.md` - Optimization strategies and lazy evaluation
- `visualization.md` - Plotting and interactive visualizations
- `machine_learning.md` - ML pipelines and model integration
- `io_operations.md` - File formats and data import/export

{
  "format": "aiskill/v1",
  "skill": {
    "name": "vaex",
    "slug": "vaex",
    "version": "1.0.0",
    "description": "Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that do not fit in memory.",
    "category": "Research",
    "tags": [
      "research-science",
      "data-viz",
      "documents"
    ],
    "platforms": [
      "chatgpt",
      "claude-code",
      "cursor"
    ],
    "tested_environment": "Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and compatible agents",
    "license": "MIT LICENSE"
  },
  "prompts": {
    "system": "# Vaex\r\n\r\n## Overview\r\n\r\nVaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.\r\n\r\n## When to Use This Skill\r\n\r\nUse Vaex when:\r\n- Processing tabular datasets larger than available RAM (gigabytes to terabytes)\r\n- Performing fast statistical aggregations on massive datasets\r\n- Creating visualizations and heatmaps of large datasets\r\n- Building machine learning pipelines on big data\r\n- Converting between data formats (CSV, HDF5, Arrow, Parquet)\r\n- Needing lazy evaluation and virtual columns to avoid memory overhead\r\n- Working with astronomical data, financial time series, or other large-scale scientific datasets\r\n\r\n## Core Capabilities\r\n\r\nVaex provides six primary capability areas, each documented in detail in the references directory:\r\n\r\n### 1. DataFrames and Data Loading\r\n\r\nLoad and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference `references/core_dataframes.md` for:\r\n- Opening large files efficiently\r\n- Converting from pandas/NumPy/Arrow\r\n- Working with example datasets\r\n- Understanding DataFrame structure\r\n\r\n### 2. Data Processing and Manipulation\r\n\r\nPerform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference `references/data_processing.md` for:\r\n- Filtering and selections\r\n- Virtual columns and expressions\r\n- Groupby operations and aggregations\r\n- String operations and datetime handling\r\n- Working with missing data\r\n\r\n### 3. Performance and Optimization\r\n\r\nLeverage Vaex's lazy evaluation, caching strategies, and memory-efficient operations. Reference `references/performance.md` for:\r\n- Understanding lazy evaluation\r\n- Using `delay=True` for batching operations\r\n- Materializing columns when needed\r\n- Caching strategies\r\n- Asynchronous operations\r\n\r\n### 4. Data Visualization\r\n\r\nCreate interactive visualizations of large datasets including heatmaps, histograms, and scatter plots. Reference `references/visualization.md` for:\r\n- Creating 1D and 2D plots\r\n- Heatmap visualizations\r\n- Working with selections\r\n- Customizing plots and subplots\r\n\r\n### 5. Machine Learning Integration\r\n\r\nBuild ML pipelines with transformers, encoders, and integration with scikit-learn, XGBoost, and other frameworks. Reference `references/machine_learning.md` for:\r\n- Feature scaling and encoding\r\n- PCA and dimensionality reduction\r\n- K-means clustering\r\n- Integration with scikit-learn/XGBoost/CatBoost\r\n- Model serialization and deployment\r\n\r\n### 6. I/O Operations\r\n\r\nEfficiently read and write data in various formats with optimal performance. Reference `references/io_operations.md` for:\r\n- File format recommendations\r\n- Export strategies\r\n- Working with Apache Arrow\r\n- CSV handling for large files\r\n- Server and remote data access\r\n\r\n## Quick Start Pattern\r\n\r\nFor most Vaex tasks, follow this pattern:\r\n\r\n```python\r\nimport vaex\r\n\r\n# 1. Open or create DataFrame\r\ndf = vaex.open('large_file.hdf5')  # or .csv, .arrow, .parquet\r\n# OR\r\ndf = vaex.from_pandas(pandas_df)\r\n\r\n# 2. Explore the data\r\nprint(df)  # Shows first/last rows and column info\r\ndf.describe()  # Statistical summary\r\n\r\n# 3. Create virtual columns (no memory overhead)\r\ndf['new_column'] = df.x ** 2 + df.y\r\n\r\n# 4. Filter with selections\r\ndf_filtered = df[df.age > 25]\r\n\r\n# 5. Compute statistics (fast, lazy evaluation)\r\nmean_val = df.x.mean()\r\nstats = df.groupby('category').agg({'value': 'sum'})\r\n\r\n# 6. Visualize\r\ndf.plot1d(df.x, limits=[0, 100])\r\ndf.plot(df.x, df.y, limits='99.7%')\r\n\r\n# 7. Export if needed\r\ndf.export_hdf5('output.hdf5')\r\n```\r\n\r\n## Working with References\r\n\r\nThe reference files contain detailed information about each capability area. Load references into context based on the specific task:\r\n\r\n- **Basic operations**: Start with `references/core_dataframes.md` and `references/data_processing.md`\r\n- **Performance issues**: Check `references/performance.md`\r\n- **Visualization tasks**: Use `references/visualization.md`\r\n- **ML pipelines**: Reference `references/machine_learning.md`\r\n- **File I/O**: Consult `references/io_operations.md`\r\n\r\n## Best Practices\r\n\r\n1. **Use HDF5 or Apache Arrow formats** for optimal performance with large datasets\r\n2. **Leverage virtual columns** instead of materializing data to save memory\r\n3. **Batch operations** using `delay=True` when performing multiple calculations\r\n4. **Export to efficient formats** rather than keeping data in CSV\r\n5. **Use expressions** for complex calculations without intermediate storage\r\n6. **Profile with `df.stat()`** to understand memory usage and optimize operations\r\n\r\n## Common Patterns\r\n\r\n### Pattern: Converting Large CSV to HDF5\r\n```python\r\nimport vaex\r\n\r\n# Open large CSV (processes in chunks automatically)\r\ndf = vaex.from_csv('large_file.csv')\r\n\r\n# Export to HDF5 for faster future access\r\ndf.export_hdf5('large_file.hdf5')\r\n\r\n# Future loads are instant\r\ndf = vaex.open('large_file.hdf5')\r\n```\r\n\r\n### Pattern: Efficient Aggregations\r\n```python\r\n# Use delay=True to batch multiple operations\r\nmean_x = df.x.mean(delay=True)\r\nstd_y = df.y.std(delay=True)\r\nsum_z = df.z.sum(delay=True)\r\n\r\n# Execute all at once\r\nresults = vaex.execute([mean_x, std_y, sum_z])\r\n```\r\n\r\n### Pattern: Virtual Columns for Feature Engineering\r\n```python\r\n# No memory overhead - computed on the fly\r\ndf['age_squared'] = df.age ** 2\r\ndf['full_name'] = df.first_name + ' ' + df.last_name\r\ndf['is_adult'] = df.age >= 18\r\n```\r\n\r\n## Resources\r\n\r\nThis skill includes reference documentation in the `references/` directory:\r\n\r\n- `core_dataframes.md` - DataFrame creation, loading, and basic structure\r\n- `data_processing.md` - Filtering, expressions, aggregations, and transformations\r\n- `performance.md` - Optimization strategies and lazy evaluation\r\n- `visualization.md` - Plotting and interactive visualizations\r\n- `machine_learning.md` - ML pipelines and model integration\r\n- `io_operations.md` - File formats and data import/export",
    "user_template": ""
  },
  "variables": [],
  "install": {
    "steps": [
      {
        "title": "Download & unzip",
        "detail": "Download the .zip package from SkillVault and unzip it. The vaex/ folder contains SKILL.md plus any companion scripts, references and assets."
      },
      {
        "title": "Copy into your agent skills directory",
        "detail": "Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code), .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or reload the session."
      },
      {
        "title": "Use it",
        "detail": "Describe the task in natural language; the agent loads the skill automatically when the request matches the skill description."
      }
    ]
  },
  "usage": {
    "steps": []
  },
  "metadata": {
    "author": "SkillVault",
    "source": "https://skillvault.top/skills/vaex",
    "exported_at": "2026-11-03T07:41:07.876Z"
  }
}

format: aiskill/v1
skill:
  name: vaex
  slug: vaex
  version: 1.0.0
  description: >-
    Use this skill for processing and analyzing large tabular datasets (billions of rows) that
    exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast
    aggregations, efficient visualization of big data, and machine learning on large datasets. Apply
    when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on
    massive datasets, create visualizations of big data, or build ML pipelines that do not fit in
    memory.
  category: Research
  tags:
    - research-science
    - data-viz
    - documents
  platforms:
    - chatgpt
    - claude-code
    - cursor
  tested_environment: >-
    Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and
    compatible agents
  license: MIT LICENSE
prompts:
  system: "# Vaex\r\n\r\n## Overview\r\n\r\nVaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.\r\n\r\n## When to Use This Skill\r\n\r\nUse Vaex when:\r\n- Processing tabular datasets larger than available RAM (gigabytes to terabytes)\r\n- Performing fast statistical aggregations on massive datasets\r\n- Creating visualizations and heatmaps of large datasets\r\n- Building machine learning pipelines on big data\r\n- Converting between data formats (CSV, HDF5, Arrow, Parquet)\r\n- Needing lazy evaluation and virtual columns to avoid memory overhead\r\n- Working with astronomical data, financial time series, or other large-scale scientific datasets\r\n\r\n## Core Capabilities\r\n\r\nVaex provides six primary capability areas, each documented in detail in the references directory:\r\n\r\n### 1. DataFrames and Data Loading\r\n\r\nLoad and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference `references/core_dataframes.md` for:\r\n- Opening large files efficiently\r\n- Converting from pandas/NumPy/Arrow\r\n- Working with example datasets\r\n- Understanding DataFrame structure\r\n\r\n### 2. Data Processing and Manipulation\r\n\r\nPerform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference `references/data_processing.md` for:\r\n- Filtering and selections\r\n- Virtual columns and expressions\r\n- Groupby operations and aggregations\r\n- String operations and datetime handling\r\n- Working with missing data\r\n\r\n### 3. Performance and Optimization\r\n\r\nLeverage Vaex's lazy evaluation, caching strategies, and memory-efficient operations. Reference `references/performance.md` for:\r\n- Understanding lazy evaluation\r\n- Using `delay=True` for batching operations\r\n- Materializing columns when needed\r\n- Caching strategies\r\n- Asynchronous operations\r\n\r\n### 4. Data Visualization\r\n\r\nCreate interactive visualizations of large datasets including heatmaps, histograms, and scatter plots. Reference `references/visualization.md` for:\r\n- Creating 1D and 2D plots\r\n- Heatmap visualizations\r\n- Working with selections\r\n- Customizing plots and subplots\r\n\r\n### 5. Machine Learning Integration\r\n\r\nBuild ML pipelines with transformers, encoders, and integration with scikit-learn, XGBoost, and other frameworks. Reference `references/machine_learning.md` for:\r\n- Feature scaling and encoding\r\n- PCA and dimensionality reduction\r\n- K-means clustering\r\n- Integration with scikit-learn/XGBoost/CatBoost\r\n- Model serialization and deployment\r\n\r\n### 6. I/O Operations\r\n\r\nEfficiently read and write data in various formats with optimal performance. Reference `references/io_operations.md` for:\r\n- File format recommendations\r\n- Export strategies\r\n- Working with Apache Arrow\r\n- CSV handling for large files\r\n- Server and remote data access\r\n\r\n## Quick Start Pattern\r\n\r\nFor most Vaex tasks, follow this pattern:\r\n\r\n```python\r\nimport vaex\r\n\r\n# 1. Open or create DataFrame\r\ndf = vaex.open('large_file.hdf5')  # or .csv, .arrow, .parquet\r\n# OR\r\ndf = vaex.from_pandas(pandas_df)\r\n\r\n# 2. Explore the data\r\nprint(df)  # Shows first/last rows and column info\r\ndf.describe()  # Statistical summary\r\n\r\n# 3. Create virtual columns (no memory overhead)\r\ndf['new_column'] = df.x ** 2 + df.y\r\n\r\n# 4. Filter with selections\r\ndf_filtered = df[df.age > 25]\r\n\r\n# 5. Compute statistics (fast, lazy evaluation)\r\nmean_val = df.x.mean()\r\nstats = df.groupby('category').agg({'value': 'sum'})\r\n\r\n# 6. Visualize\r\ndf.plot1d(df.x, limits=[0, 100])\r\ndf.plot(df.x, df.y, limits='99.7%')\r\n\r\n# 7. Export if needed\r\ndf.export_hdf5('output.hdf5')\r\n```\r\n\r\n## Working with References\r\n\r\nThe reference files contain detailed information about each capability area. Load references into context based on the specific task:\r\n\r\n- **Basic operations**: Start with `references/core_dataframes.md` and `references/data_processing.md`\r\n- **Performance issues**: Check `references/performance.md`\r\n- **Visualization tasks**: Use `references/visualization.md`\r\n- **ML pipelines**: Reference `references/machine_learning.md`\r\n- **File I/O**: Consult `references/io_operations.md`\r\n\r\n## Best Practices\r\n\r\n1. **Use HDF5 or Apache Arrow formats** for optimal performance with large datasets\r\n2. **Leverage virtual columns** instead of materializing data to save memory\r\n3. **Batch operations** using `delay=True` when performing multiple calculations\r\n4. **Export to efficient formats** rather than keeping data in CSV\r\n5. **Use expressions** for complex calculations without intermediate storage\r\n6. **Profile with `df.stat()`** to understand memory usage and optimize operations\r\n\r\n## Common Patterns\r\n\r\n### Pattern: Converting Large CSV to HDF5\r\n```python\r\nimport vaex\r\n\r\n# Open large CSV (processes in chunks automatically)\r\ndf = vaex.from_csv('large_file.csv')\r\n\r\n# Export to HDF5 for faster future access\r\ndf.export_hdf5('large_file.hdf5')\r\n\r\n# Future loads are instant\r\ndf = vaex.open('large_file.hdf5')\r\n```\r\n\r\n### Pattern: Efficient Aggregations\r\n```python\r\n# Use delay=True to batch multiple operations\r\nmean_x = df.x.mean(delay=True)\r\nstd_y = df.y.std(delay=True)\r\nsum_z = df.z.sum(delay=True)\r\n\r\n# Execute all at once\r\nresults = vaex.execute([mean_x, std_y, sum_z])\r\n```\r\n\r\n### Pattern: Virtual Columns for Feature Engineering\r\n```python\r\n# No memory overhead - computed on the fly\r\ndf['age_squared'] = df.age ** 2\r\ndf['full_name'] = df.first_name + ' ' + df.last_name\r\ndf['is_adult'] = df.age >= 18\r\n```\r\n\r\n## Resources\r\n\r\nThis skill includes reference documentation in the `references/` directory:\r\n\r\n- `core_dataframes.md` - DataFrame creation, loading, and basic structure\r\n- `data_processing.md` - Filtering, expressions, aggregations, and transformations\r\n- `performance.md` - Optimization strategies and lazy evaluation\r\n- `visualization.md` - Plotting and interactive visualizations\r\n- `machine_learning.md` - ML pipelines and model integration\r\n- `io_operations.md` - File formats and data import/export"
  user_template: ""
variables: []
install:
  steps:
    - title: Download & unzip
      detail: >-
        Download the .zip package from SkillVault and unzip it. The vaex/ folder contains SKILL.md
        plus any companion scripts, references and assets.
    - title: Copy into your agent skills directory
      detail: >-
        Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code),
        .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or
        reload the session.
    - title: Use it
      detail: >-
        Describe the task in natural language; the agent loads the skill automatically when the
        request matches the skill description.
usage:
  steps: []
metadata:
  author: SkillVault
  source: https://skillvault.top/skills/vaex
  exported_at: "2026-11-03T07:41:07.876Z"