When I first started out in Finance, there was one undisputed hero: Excel. We pushed it to its limits: macros, array formulas, custom-built VBA scripts, and then some. We used it for financial modelling, data cleansing, forecasting, even as a poor man's database. We demanded more power, more rows, more reliability. Eventually, Microsoft responded.
Power Query and Power Pivot were born, not as replacements for Excel, but as bolt-ons to meet the growing hunger for more robust data processing within the familiar spreadsheet interface. We wanted Excel to do more, and Microsoft delivered. But while these tools were revolutionary, they were still built around the same core idea: a spreadsheet interface.
Now, years later, we're in the age of big data and AI. And the tool that feels like Excel's true successor isn't even from Microsoft. It's DuckDB.
Why DuckDB Is the Next Logical Step
DuckDB is a fast, in-process SQL OLAP database. It reads Parquet and CSV files directly, supports complex SQL queries, and integrates with tools like Python and R, but what's game-changing is that it runs entirely on your local machine with no setup, no cloud dependency, and no IT gatekeeping.
If DuckDB had existed back during my early finance days, I probably wouldn't have jumped ship to IT and gone deep into data engineering. That leap happened mostly because I outgrew the tools available to me as a finance professional. I needed versioning, scale, traceability, and the ability to work with more than a million rows, something Excel simply couldn't provide.
Excel
~1M rows, formula-based logic, crash-prone with big data, UI-first
DuckDB
Billions of rows, SQL-based, runs locally, tiny file, no server needed
A Foot in Both Worlds, Until Now
These days, I'm fully immersed in the tech world. But I look back and see finance professionals today doing what data analysts are doing (data wrangling, reconciliations, KPI tracking) all centred around large, complex datasets. The only difference? Finance folks are still doing it in Excel.
Imagine reconciling intercompany transactions from two 5-million-row CSVs, a task that would crash Excel but is a simple join query in DuckDB.
The honest truth: Relying on Excel's row limits or multi-tab nightmares in a big data world is like using a flip phone in the age of smartphones. You can do it, but you're making life harder for yourself.
Power Query Is Great, But It's Not a Database
I have enormous respect for Power Query. It's one of the best data ingestion and transformation tools available, especially for non-coders. But, and this is key, it's not a database. It doesn't store data. It doesn't manage data in a structured, ACID-compliant way. You can't write data back to it in a controlled manner.
DuckDB changes that. It's not just read-only. You can write data back. You can insert, update, delete, and store data in tables, just like a real database. It supports transactions, joins, window functions, and even local Delta Lake-style workflows using Parquet files.
Why this matters emotionally: Accountants like to own the numbers. Power Query lets you look at data. DuckDB lets you own it, at scale.
| Feature | Excel | Power Query | DuckDB |
|---|---|---|---|
| Max rows (practical) | ~1 million | ~10 million | Billions |
| Write data back | ✓ | ✗ | ✓ |
| Runs locally, no server | ✓ | ✓ | ✓ |
| Logic separate from data | ✗ | Partial | ✓ (.sql files) |
| ACID transactions | ✗ | ✗ | ✓ |
| Reads Parquet/CSV directly | ✗ | ✓ | ✓ |
| Version-controllable logic | ✗ | Partial | ✓ (plain .sql) |
It's Simple Enough to Start Today
Here's what working with DuckDB can look like. No VBA. No Power Query GUI. Just clean, understandable SQL:
SELECT CustomerID,
SUM(InvoiceAmount) AS total_invoiced
FROM 'invoices.parquet'
GROUP BY CustomerID;
And even if you've never written SQL before, AI tools like ChatGPT or GitHub Copilot can help you write queries just by describing what you want. The learning curve has never been lower.
Install DuckDB
Visit duckdb.org and follow their 5-minute install guide for Python or the command line. Or ask IT to set it up.
Get an IDE
Install DBeaver. It's free, connects to DuckDB, and looks just like a professional database client. You can point it at your .db file directly.
Point it at your data
Have a CSV or Parquet file? Just reference it in your SQL query. DuckDB reads it directly, no importing required.
Ask AI to help write your queries
Describe what you want in plain English to ChatGPT. It'll write the SQL. You review, tweak, run.
Safe by Design
One of DuckDB's biggest hidden advantages is its isolation. It doesn't touch production systems. It doesn't connect to the company-wide SQL Server. And that's exactly why it's safe.
By running locally and only interacting with the user's own .db file or flat files, DuckDB avoids the typical "oops" moments that make IT nervous: no accidental deletes from a shared database, no broken relationships, no unintended orphan records. Your changes affect your file, and nothing more.
And since DuckDB files are tiny (just like Excel workbooks), versioning is simple. SharePoint or OneDrive can track every saved version of your .db file. Need to roll back to yesterday's state? Just click "Restore previous version." It's audit-ready without needing DevOps.
Separating Logic from Data: SQL's Hidden Strength
One of the most overlooked benefits of working with SQL instead of Excel is the clear separation between logic and data. In SQL, your transformation and aggregation logic lives in a separate .sql file. The data lives in a database or flat file. That makes audits much simpler. You can just send someone the SQL script and the data source.
This avoids the classic Excel scenario: "What the heck is wrong with this formula?", only to discover someone accidentally deleted a row, broke a reference, or altered cell A1. SQL removes those risks entirely.
-- Author: Maha | Last updated: 2025-06-01
-- Data source: /data/ar_export_jun2025.parquet
WITH invoiced AS (
SELECT CustomerID, SUM(Amount) AS total_invoiced
FROM 'invoices.parquet'
WHERE YEAR(InvoiceDate) = YEAR(CURRENT_DATE)
GROUP BY CustomerID
),
paid AS (
SELECT CustomerID, SUM(Amount) AS total_paid
FROM 'payments.parquet'
GROUP BY CustomerID
)
SELECT i.CustomerID,
i.total_invoiced,
COALESCE(p.total_paid, 0) AS total_paid,
i.total_invoiced - COALESCE(p.total_paid, 0) AS outstanding
FROM invoiced i
LEFT JOIN paid p ON i.CustomerID = p.CustomerID
ORDER BY outstanding DESC;
The Rise of the Digital Accountant
Let's be clear: in 2025, digital skills are no longer a "nice-to-have" for accountants. They're essential. AI can now generate reconciliations, explain SQL code, and automate tedious processes. The differentiator isn't who can use Excel faster. It's who can work with data at scale, and who can automate tedious workflows.
DuckDB gives accountants a serious seat at the data table, without needing to become full-fledged engineers. And best of all, the skills you gain in DuckDB don't just live inside one tool. They transfer. To PostgreSQL. To Snowflake. To SQL Server. To Databricks. You're not learning in isolation. You're building a foundation for long-term growth.
If Excel is like driving a car with manual gears, DuckDB is like piloting a Tesla on autopilot. You're still in control, but now you can go further, faster, and safer.
Maha KepakisanSo if you're an accountant tired of stretching Excel to its breaking point, give DuckDB a try. You might just fall in love with data again.