Pandas API for DuckDB, PostgreSQL & ClickHouse — keeping computation inside the database[P]

Reddit r/MachineLearning Tools

Summary

memFrame is an open-source dataframe API that translates Python DataFrame operations to SQL, enabling computation to stay within databases like DuckDB, PostgreSQL, and ClickHouse, and includes a multiagent architecture for natural language data interaction.

I've been building memFrame — an open-source dataframe API that compiles operations to SQL. The idea: **Python/DataFrame API → SQL → DuckDB / PostgreSQL / ClickHouse** Instead of pulling data into Python and doing everything in pandas, memFrame tries to keep computation inside the database. Currently it has operations for inspection, selection, cleaning, statistics, arithmetic, visualization, etc. I'm intentionally releasing analytics operations incrementally , because first I want the currently released features to be well tested in public and once all the issues on the current features are solved then — groupby, window functions, sorting, filtering are on the way. It comes with built-in multiagent architecture for chatting with your data in natural languages. Would love some technical feedback & suggestions to improve this from people working with pandas/SQL/DuckDB/Postgres. github || docs If the idea sounds useful, a ⭐ helps a lot.
Original Article

Similar Articles

pumaDB

Product Hunt

pumaDB is a small hosted memory layer designed for AI agents.

Pandas Should Go Extinct

Lobsters Hottest

The article argues that Pandas, a Python DataFrame library, should be replaced by more efficient alternatives like Polars and DuckDB due to its inefficiencies, based on analysis of data sizes and adoption patterns.