Fala galera do mundo dos dados! Já estamos cansados de saber que a análise de dados é uma etapa fundamental em projetos de ciência de dados e inteligência artificial. Já vimos aqui no Dados ao Cubo como fazer uma análise de dados poderosa com Polars em Python, mas sabemos que também podemos utilizar o Pandas para análise de dados. São duas bibliotecas para manipulação e análise de dados, uma já consolidada e outra chegando com tudo. Então, vamos comparar Polars vs. Pandas e explorar as principais funções dessas bibliotecas. Dessa forma, você aprenderá como realizar tarefas comuns de análise de dados usando ambas as bibliotecas, além de identificar as diferenças entre elas.
Instalação e Importação das Bibliotecas
Primeiramente, é importante instalar as bibliotecas necessárias. Para o Polars, você pode usar o seguinte comando.
pip install polars
Para o Pandas, use o comando.
pip install pandas
Em seguida, importe as bibliotecas em seu código.
# Polarimport polars as pl# Pandasimport pandas as pd
Agora, faremos a carga de um conjunto de dados.
Carregando Dados
Ambas as bibliotecas possuem funções para carregar dados de diferentes fontes, como arquivos CSV ou bancos de dados. Aqui está um exemplo de como carregar um arquivo CSV com o Polars.
# Polarspl_df = pl.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv')E com o Pandas:
# Pandaspd_df = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv')Com os dados carregados, já podemos visualizar uma prévia deles.
Visualizando os Dados
Para visualizar os dados, o Pandas oferece a função head, que retorna as primeiras linhas do DataFrame. Já o Polars possui a função limit. Veja os exemplos de código abaixo.
# Polarspl_df.limit(5)
O resultado da função limit é exibido na imagem abaixo.
Confere a visualização no Pandas.
# Pandaspd_df.head
O resultado da função head é exibido na imagem abaixo.
Assim, visualizamos todas as colunas. E como selecionar algumas colunas?
Selecionando Colunas
Ambas as bibliotecas permitem selecionar colunas específicas de um DataFrame. Com o Pandas, você pode usar a sintaxe de colchetes [], enquanto com o Polars é possível usar a função select. Veja o exemplo abaixo.
# Polarspl_df.select('total_bill')Confere o resultado da seleção do Polars na imagem abaixo.
Seleção de colunas no Pandas.
# Pandaspd_df['total_bill']
Confere o resultado da seleção do Pandas na imagem abaixo.
Dessa forma, fizemos a seleção de linha. E como filtrar linhas específicas?
Filtrando Dados
Para filtrar dados com base em condições, o Pandas usa a sintaxe de colchetes [] combinada com uma expressão booleana. O Polars oferece a função filter. Veja o código Pyhon a seguir.
# Polarspl_df.filter(pl.col('total_bill') > 50)Dados filtrados com o Polars.
O filtro no Pandas.
# Pandaspd_df[pd_df['total_bill'] > 50]
Dados filtrados com o Pandas na imagem abaixo.
Na sequência a agregação de dados.
Agregando de Dados
Ambas as bibliotecas oferecem funcionalidades para realizar agregações de dados, como soma, média e contagem. Veja o exemplo da soma abaixo.
# Polarspl_df['total_bill'].sum()
Dessa forma, no Polars, temos como resultado o valor resultante da soma de: 4827.7699999999995
# Pandaspd_df['total_bill'].sum()
Dessa forma, no Pandas, temos como resultado o valor resultante da soma de: 4827.77
Também, podemos fazer agregações mais complexas utilizando a função groupby em ambas as bibliotecas. Confere os exemplos de código.
# Polarspl_df.groupby('sex').agg(pl.col('total_bill').mean)Agregação do groupby no Polars.
Utilizando o groupby no Pandas.
# Pandaspd_df.groupby('sex').agg({'total_bill': 'mean'})Agregação do groupby no Pandas.
Da agregação para a alteração do nome das colunas.
Renomeando colunas
Ambas as bibliotecas possuem a função rename. Aqui está um exemplo de como renomear colunas tanto no Polars, quanto no Pandas.
# Polarspl_df.rename({'total_bill':'total'})Na imagem abaixo já temos a primeira coluna renomeada, conforme código Python.
Função rename no Pandas.
# Pandaspd_df.rename(columns={'total_bill':'total'})Agora com o Pandas, também temos a primeira coluna renomeada, conforme código Python.
Depois de renomear, na sequência, como ordenar o conjunto de dados.
Ordenando os dados
Para ordenar os dados, o Pandas oferece a função sort_values, ordenando pela coluna informada do DataFrame. Já o Polars possui a função sort, que trabalha da mesma forma. Veja os exemplos de código abaixo.
# Polarspl_df.sort('total_bill')A imagem abaixo é o resultado da ordenação da coluna total_bill pelo Polars.
Confere a função de ordenação do Pandas.
# Pandaspd_df.sort_values('total_bill')Aqui está o resultado da ordenação da coluna total_bill pelo Pandas.
Dados ordenados, confere como deletar linhas duplicada.
Removendo duplicatas
Ambas as bibliotecas permitem remover dados duplicados em um DataFrame. Com o Pandas, você pode usar a função drop_duplicates, enquanto com o Polars é possível usar a função unique. A seguir exemplos das funções.
Código Polars para remover dados duplicados.
# Polarspl_df.unique()
E o código Pandas para remover dados duplicados.
# Pandaspd_df.drop_duplicates()
Depois dos duplicados, agora o tratamento de dados nulos.
Preenchendo valores nulos
Para preencher valores nulos, primeiro vamos identificar as colunas e quantidades de dados nulos e depois podemos fazer o preenchimento. No Pandas usa a função fillna para identificar os nulos e a função fillna para preencher valores nulos. O Polars oferece a função null_count e a função fill_null para preencher valores nulos. Veja os códigos Pyhon a seguir.
# Polarspl_df.null_count()
Na imagem abaixo a contagem dos dados nulos de todas as colunas no Polars.
Se houver necessidade de substituição de dados nulos, podemos utilizar o código Python abaixo.
# Polarspl_df.fill_null(value='novo_valor')
Contagem dos dados nulos com o Pandas.
# Pandaspd_df.isna.sum()
A imagem mostra o resultado, bem semelhante ao Polars, só muda o formato dos dados.
Para o Pandas, se houver necessidade de substituição de dados nulos, podemos utilizar o código Python abaixo.
# Pandaspd_df.fillna('novo_valor')Agora vamos ver, como aplicar funções em colunas do conjunto de dados.
Aplicando uma função a uma coluna
Para aplicar função em uma coluna, o Pandas oferece a função apply, e o Polars possui a função de mesmo nome. Veja os exemplos de código abaixo.
# Polarspl_df.select('total_bill').apply(lambda x: x[0] * 2)A imagem a seguir mostra a função sendo aplicada na coluna total_bill do dataframe Polars.
A mesma função apply, agora no Pandas.
# Pandaspd_df['total_bill'].apply(lambda x: x * 2)
Da mesma forma a função sendo aplicada na coluna total_bill do dataframe Pandas.
A seguir, algumas estatísticas básicas do conjunto de dados.
Estatísticas descritivas
Ambas as bibliotecas oferecem funcionalidades para estatísticas descritivas dos dados, exibindo os principais indicadores através da função describe. Confere os exemplos a seguir.
# Polarspl_df.describe()
Esse é o resultado da função describe no dataframe Polars.
Agora a estatística descritiva no Pandas.
# Pandaspd_df.describe()
Já no Pandas, temos esse resultado da função describe, que é bem similar.
E para encerrar nossas comparações, como contar a frequência dentro do conjunto de dados.
Contando de valores únicos
Para cotar valores únicos no DataFrames, o Pandas possui a função value_counts, e o Polars oferece a função de mesmo nome. Veja o exemplo abaixo.
# Polarspl_df.select('sex').to_series.value_counts()Na imagem a seguir a frequência da coluna sex no dataframe Polars.
Utilizando a mesma função value_counts no Pandas.
# Pandaspd_df.value_counts('sex')E no Pandas, não é muito diferente, confere a imagem abaixo.
E assim finalizamos essa comparação de algumas funções do Polars vs. Pandas.
Polars vs. Pandas ao Cubo
Sendo assim, tanto Polars, quanto Pandas são bibliotecas poderosas para análise de dados em Python. Ambas possuem uma ampla gama de funções e métodos para manipulação e análise de DataFrames. No entanto, existem algumas diferenças sutis entre as duas.
O Polars se destaca por sua capacidade de processamento paralelo e execução eficiente em grandes conjuntos de dados. Ele também oferece uma sintaxe expressiva e familiar para quem já está acostumado com o Pandas. Por outro lado, o Pandas é amplamente utilizado e possui uma vasta quantidade de recursos e funcionalidades avançadas. É uma escolha popular para análise de dados em tarefas de menor escala ou quando a velocidade de execução não é uma preocupação crítica.
Experimente as diferentes funcionalidades de cada biblioteca, explore a documentação e os exemplos de código, e escolha a que melhor atenda às suas necessidades e objetivos de análise de dados. Com o conhecimento adequado e a prática contínua, você se tornará um especialista em análise de dados com Python. O importante é estar sempre atualizado, então para ficar sempre ligado nas novidades assine a nossa Newsletter. Um abraço e até a próxima!!!
Conteúdos ao Cubo
Se você curtiu o conteúdo, aqui no Dados ao Cubo tem muito mais. Então, deixo algumas sugestões de conteúdos que você pode encontrar. Sempre falando sobre o mundo dos dados!
- Time de Dados na Prática
- Etapas para Análise de Dados
- Tipos de Análise de Dados
- Dicas para Visualização de Dados
- Análise de Dados com Airbyte e Metabase
- Importar CSV no PostgreSQL com o DBeaver
Finalizo com um convite para você ser Parceiro de Publicação Dados ao Cubo e escrever o próximo artigo, compartilhando conhecimento para toda a comunidade de dados.
Tiago Dias
Baiano, apaixonado por dados e tecnologia, amante das inovações tecnológicas que facilitam a vida humana! Formado em Engenharia da Computação e com MBA em Gestão da Informação e Business Intelligence e especialização em Data Science. Atualmente atua como Data Analytics Specialist na Lopes, além de Professor na área de dados e nas horas vagas cria uns modelos de Machine Learning com Python e soluções diversas com dados!
Dados Python
Gostou? Compartilhe!
FAQs
Should I use polars instead of Pandas? ›
Will Polars replace Pandas? The main advantage of Polars over Pandas is its speed. If you need to do a lot of data processing on large datasets, you should definitely try Polars. The main advantage of Polars over Pandas is its speed.
Is polars more memory efficient than Pandas? ›In a similar way, the Polars library is designed to efficiently handle large datasets in a way that is optimized for the modern computing environment. Compared to Pandas, Polars is designed to be faster and more memory-efficient, making it an attractive option for data analysis tasks.
What is the difference between Pandas arrow and polars? ›Polars represents data internally using Apache Arrow arrays while Pandas stores data internally using NumPy arrays. Apache Arrow arrays is much more efficient in areas like load time, memory usage, and computation. Polars supports more parallel operations than Pandas.
Why is polar faster than Pandas? ›Polars is much faster than libraries that try to implement concurrency using python, like Pandas. That's because Polars is written with Rust, and Rust is much better than Python at implementing concurrency.
What is the faster alternative to pandas DataFrame? ›The Polars dataframe library is a potential solution. While Polars is mostly known for running faster than Pandas, if you use it right it can sometimes also significantly reduce memory usage compared to Pandas.
Should I learn pandas for machine learning? ›pandas is one of the first Python packages you should learn because it's easy to use, open source, and will allow you to work with large quantities of data. It allows fast and efficient data manipulation, data aggregation and pivoting, flexible time series functionality, and more.
Is Pandas good for big data? ›Pandas is has became the de-facto python library for data scientist and analyst due to its intuitive data structure and rich APIs. Pandas uses in-memory computation which makes it ideal for small to medium sized datasets. However, Pandas ability to process big datasets is limited due to out-of-memory errors.
Why Pandas is better than PySpark? ›In very simple words Pandas run operations on a single machine whereas PySpark runs on multiple machines. If you are working on a Machine Learning application where you are dealing with larger datasets, PySpark is a best fit which could processes operations many times(100x) faster than Pandas.
Is Pandas efficient for large data sets? ›The default pandas data types are not the most memory efficient. This is especially true for text data columns with relatively few unique values (commonly referred to as “low-cardinality” data). By using more efficient data types, you can store larger datasets in memory.
Is Dask better than Pandas? ›Pandas is better suited for small-to-medium-sized datasets that fit into memory, while Dask is designed to handle larger-than-memory datasets with distributed computing. While Pandas is easier to use, Dask's performance and scalability make it a better choice for handling larger datasets.
How fast is NumPy vs Polars? ›
Again, numpy takes 226 microseconds, whereas polars takes 673 microseconds, about three times slower.
What is the Python Pandas equivalent in Rust? ›One big difference between Pandas and Rust is that Rust filtering uses Closures (eq. lambda function in python) whereas Pandas filtering uses Pandas API based on columns. This means Rust can make more complex filters compared to Pandas.
Why is Pandas more efficient? ›Pandas has the capability to run calculations on an entire vector instead of single values. A vector is a one-dimensional array of numbers or objects, in this case, a row or a column. Using vector operations lets Pandas do the entire calculation in its optimized form within the pandas' library.
Why is Pandas so slow compared to NumPy? ›Pandas is more user-friendly, but NumPy is faster. Pandas has a lot more options for handling missing data, but NumPy has better performance on large datasets. Pandas uses Python objects internally, making it easier to work with than NumPy (which uses C arrays).
Is Polars faster in Rust or Python? ›Built-in Support for Rust: Polars is written in Rust. Rust's ability to be immediately compiled into machine code without the use of an interpreter can make it faster than Python.
What is the fastest iteration DataFrame? ›Vectorization is always the first and best choice. You can convert the data frame to NumPy array or into dictionary format to speed up the iteration workflow. Iterating through the key-value pair of dictionaries comes out to be the fastest way with around 280x times speed up for 20 million records.
Which is faster Pandas or NumPy? ›Pandas DataFrames are typically going to be slower than a NumPy array if you want to perform mathematical operations like computing the mean, the dot product, and other similar tasks.
What is better than Pandas Python? ›Panda, NumPy, R Language, Apache Spark, and PySpark are the most popular alternatives and competitors to Pandas.
Should I learn SQL or Pandas? ›Both SQL and Pandas are important tools for Data Analysis. The logic behind most of the functions is similar in both of them with just a few minor syntactical changes. If you want just to access/modify the data using some filter then SQL will efficient option. Pandas can perform complex grouping operations easily.
Which Python is best for machine learning? ›- NumPy.
- SciPy.
- Scikit-learn.
- Theano.
- TensorFlow.
- Keras. Best Machine Learning Courses & AI Courses Online.
- PyTorch.
- Pandas. In-demand Machine Learning Skills.
Do I need to learn NumPy if I know Pandas? ›
First, you should learn Numpy. It is the most fundamental module for scientific computing with Python. Numpy provides the support of highly optimized multidimensional arrays, which are the most basic data structure of most Machine Learning algorithms. Next, you should learn Pandas.
Is pandas a good ETL tool? ›However, it is time-consuming as you would have to write your own code. It can be used to write simple scripts easily, and is one of the widely used Python ETL tools. However, when it comes to in-memory and scalability, Pandas' performance may not keep up with expectations.
How much RAM do you need for pandas? ›"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB dataset, you should really have about 64, preferably 128 GB of RAM if you want to avoid memory management problems."
Do data engineers need pandas? ›Pandas. Pandas is the Python library popular among data analysts and data scientists. It is equally useful for data engineers, who often use it for reading, writing, querying, and manipulating data.
Should I learn PySpark or pandas? ›In summary, use PySpark for large datasets and complex tasks that are not feasible with pandas, and use pandas for small datasets and simple tasks that can be handled on a single machine.
Why pandas is better than Excel? ›Because it is built on NumPy (Numerical Python), Pandas boasts several advantages over Excel: Scalability - Pandas is only limited by hardware and can manipulate larger quantities of data. Speed - Pandas is much faster than Excel, which is especially noticeable when working with larger quantities of data.
What is the equivalent of pandas in PySpark? ›PySpark users can access the full PySpark APIs by calling DataFrame. to_spark() . pandas-on-Spark DataFrame and Spark DataFrame are virtually interchangeable. However, note that a new default index is created when pandas-on-Spark DataFrame is created from Spark DataFrame.
Can Pandas handle 2 million rows of data? ›Typically, Pandas find its' sweet spot in usage in low- to medium-sized datasets up to a few million rows. Beyond this, more distributed frameworks such as Spark or Dask are usually preferred. It is, however, possible to scale pandas much beyond this point.
How big is too big for a Pandas Dataframe? ›The short answer is yes, there is a size limit for pandas DataFrames, but it's so large you will likely never have to worry about it. The long answer is the size limit for pandas DataFrames is 100 gigabytes (GB) of memory instead of a set number of cells.
Which database is best for large datasets Python? ›SQLite can be used as a parallel solution for client/server RDBMS testing. If you need a quick connection to your data, there's no need to connect to a server to use SQLite, reflecting the library has low latency. Hence, SQLite is known as the best python database.
Is Pandas better than Openpyxl? ›
Developers describe openpyxl as "A Python library to read/write Excel 2010 xlsx/xlsm files". A Python library to read/write Excel 2010 xlsx/xlsm files. On the other hand, pandas is detailed as "Powerful data structures for data analysis". Powerful data structures for data analysis, time series, and statistics.
Is Openpyxl faster than Pandas? ›Step 3: Load with Openpyxl
The file is loaded to memory but data is loaded through a generator which allows mapped-retrieval of values. Still slow but a tiny drop faster than Pandas.
Let's start with the simplest operation — read a single CSV file. To my surprise, we can already see a huge difference in the most basic operation. Datatable is 70% faster than pandas while dask is 500% faster! The outcomes are all sorts of DataFrame objects which have very identical interfaces.
Should I use Pandas or NumPy? ›Pandas is mostly used for data analysis tasks in Python. NumPy is mostly used for working with Numerical values as it makes it easy to apply mathematical functions. Pandas library works well for numeric, alphabets, and heterogeneous types of data simultaneously.
Is NumPy vectorize faster than loops? ›Vectorized implementations (numpy) are much faster and more efficient as compared to for-loops. To really see HOW large the difference is, let's try some simple operations used in most machine learnign algorithms (especially deep learning).
Why is NumPy vectorize fast? ›The concept of vectorized operations on NumPy allows the use of more optimal and pre-compiled functions and mathematical operations on NumPy array objects and data sequences. The Output and Operations will speed up when compared to simple non-vectorized operations.
Is Rust replacing Python? ›Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Can Python Pandas replace Excel? ›Can Python Replace Excel? You can replace Excel with Python, by using the Pandas library which allows you to deal with DataFrames in a similar, but more powerful, way as what you would do with Excel tables.
Which is easier to learn Rust or Python? ›Regarding ease of use and learning, Python is ahead of the Rust language. As mentioned earlier, Python has become one of the top programming languages used worldwide because of its ease of learning. If someone is learning to code for the first time, they should pick up Python than Rust.
Why is Pandas so difficult? ›Pandas is Powerful but Difficult to use
Some reasons for this include: There are often multiple ways to complete common tasks. There are over 240 DataFrame attributes and methods. There are several methods that are aliases (reference the same exact underlying code) of each other.
Why does everyone use Pandas? ›
Pandas is a flexible and easy-to-use open source data analysis and manipulation tool written for the Python programming language. It offers users a vast library of data to explore and is a common resource for data scientists and analysts.
Can Python handle 1 billion rows? ›When dealing with 1 billion rows, things can get slow, quickly. And native Python isn't optimized for this sort of processing. Fortunately numpy is really great at handling large quantities of numeric data.
Why is Pandas preferred over NumPy? ›The Pandas module mainly works with the tabular data, whereas the NumPy module works with the numerical data. The Pandas provides some sets of powerful tools like DataFrame and Series that mainly used for analyzing the data, whereas in NumPy module offers a powerful object called Array.
Is there anything faster than NumPy? ›pandas provides a bunch of C or Cython optimized functions that can be faster than the NumPy equivalent function (e.g. reading text from text files).
Why NumPy is better than array? ›There are two main reasons why we would use NumPy array instead of lists in Python. These reasons are: Less memory usage: The Python NumPy array consumes less memory than lists. Less execution time: The NumPy array is pretty fast in terms of execution, as compared to lists in Python.
Which is the fastest type of Python? ›- PyPy. PyPy is one of the most popular alternative compilers which used by the Python developer to gain more speed. ...
- CPython. The CPython is the most commonly used compiler of Python that written in C. ...
- JPython or JPython. ...
- IronPython. ...
- Nuitka.
Polars is going to be faster on a single machine. I mean I could spend a bunch of time comparing the features between Polars and Spark, and maybe I will if enough people ask for it.
When should I use NumPy instead of pandas? ›Pandas is mostly used for data analysis tasks in Python. NumPy is mostly used for working with Numerical values as it makes it easy to apply mathematical functions. Pandas library works well for numeric, alphabets, and heterogeneous types of data simultaneously.
What is the most efficient way to add rows in pandas? ›By using append() function you can add or insert a row to existing pandas DataFrame from the dict. This method is required to take ignore_index=True in order to add a dict as a row to DataFrame, not using this will get you an error. This method returns the new DataFrame with the newly added row.
When to use SQL instead of pandas? ›Both SQL and Pandas are important tools for Data Analysis. The logic behind most of the functions is similar in both of them with just a few minor syntactical changes. If you want just to access/modify the data using some filter then SQL will efficient option. Pandas can perform complex grouping operations easily.
How much faster is polars than Pandas? ›
In terms of performance, Polars is 2–5 times faster for numerical filter operations, whereas Pandas requires less code to be written.
Does Polars use GPU? ›Polar can perform operations on dataframes on the GPU, which can further improve performance.
Why is Spark 100x faster? ›Commercially Spark is said to 100x faster than Hadoop. Apache Spark works faster when the data resides in memory, Spark processes data in memory which makes it faster in processing. While MapReduce pushes data to disk after processing it.
Is Pandas the best Python library? ›So, Which Python Library Is Better? Pandas is more user-friendly, but NumPy is faster. Pandas has a lot more options for handling missing data, but NumPy has better performance on large datasets. Pandas uses Python objects internally, making it easier to work with than NumPy (which uses C arrays).
Should I learn Pandas or NumPy first? ›That is exactly what Numpy and Pandas do. First, you should learn Numpy. It is the most fundamental module for scientific computing with Python. Numpy provides the support of highly optimized multidimensional arrays, which are the most basic data structure of most Machine Learning algorithms.
Should I import NumPy or Pandas first? ›You don't need to import it specifically when working with Pandas. And when you install Pandas you can see that your package manager will automatically install the Numpy package if you have not installed NumPy before.
How to get top 5 rows Pandas? ›head() function is used to get the first N rows of Pandas DataFrame. It allows an argument N to the method (which is the first n number of rows we want to get from the start). If the argument is not specified, this function returns the topmost 5 rows from the given DataFrame.
What is the max rows to display Pandas? ›The default number is 60. As shown, if we have a data frame with more than 60 rows, 50 rows in the middle will be truncated. If we set the option larger than the number of rows of our data frame, all the rows will be displayed.
What is the efficient way to concatenate Dataframes? ›When concatenating datasets vertically, assuming the dataframes have the same column names and the order of the columns is the same, we can simply use the pandas. concat() method to perform the concatenation.
Will Python replace SQL? ›Python and SQL can perform some overlapping functions, but developers typically use SQL when working directly with databases and use Python for more general programming applications. Choosing which language to use depends on the query you need to complete.
Why not use Pandas instead of SQL? ›
Pandas allows you to transform metadata (column/row labels) flexibly; in SQL you cannot. And poof, just like that, it's gone! Unfortunately, SQL doesn't give you the ability to operate on column names in the same way as Pandas. You'll need to manually specify how each column name will change.