Epython Lab

Time Complexity https://youtu.be/ri1WCuTwxWY

🤖 Behind every AI application is more than just a machine learning model.

We often focus on model architectures, neural networks, training algorithms, and performance metrics. But the foundations of efficient AI systems also depend on something fundamental: Data Structures.

As an AI developer, understanding how to organize and manage data is important when building real-world applications.

Consider where different data structures become useful in AI development:

🔹 Lists: Managing sequences, records, and collections of data.

🔹 Dictionaries: Mapping feature names, model configurations, and lookup results.

🔹 Sets: Removing duplicates and maintaining unique identifiers or labels.

🔹 Queues: Managing incoming requests in asynchronous data and inference pipelines.

🔹 Heaps: Finding top-ranked results and implementing priority-based processing.

🔹 Trees: Supporting decision trees, hierarchical data, and search operations.

Understanding these structures also helps you reason about time complexity, memory usage, and system performance.

Building AI is not just about choosing the right model. It's also about building efficient systems around it.

🔗 Explore more: https://www.youtube.com/watch?v=9LImjTMUHX0

#ArtificialIntelligence #AIEngineering #Python #MachineLearning #DataStructures #Algorithms #SoftwareEngineering

𝐈 𝐬𝐩𝐞𝐞𝐝 𝐮𝐩 𝐩𝐚𝐧𝐝𝐚𝐬 𝐛𝐲 𝟏𝟎𝟎𝐱, 𝐚𝐧𝐝 𝐡𝐞𝐫𝐞 𝐢𝐬 𝐞𝐱𝐚𝐜𝐭𝐥𝐲 𝐡𝐨𝐰 𝐲𝐨𝐮 𝐜𝐚𝐧 𝐭𝐨𝐨. 🚀

If your DataFrame manipulation feels lightning-fast on a small sample but crawls to a halt on a million rows, you are likely falling into a few common performance traps.

Here are the 3 core optimizations that drove a massive speedup:



1️⃣ 𝐃𝐢𝐭𝐜𝐡 .𝚒𝚝𝚎𝚛𝚛𝚘𝚠𝚜() 𝐟𝐨𝐫 𝐕𝐞𝐜𝐭𝐨𝐫𝐢𝐳𝐚𝐭𝐢𝐨𝐧

Row-by-row looping forces Pandas to create a Series object for every row, bypassing C-level optimizations. Let NumPy and Pandas handle your array math natively.



❌ 𝐒𝐥𝐨𝐰 (𝐑𝐨𝐰-𝐛𝐲-𝐫𝐨𝐰 𝐢𝐭𝐞𝐫𝐚𝐭𝐢𝐨𝐧):

𝚏𝚘𝚛 𝚒, 𝚛𝚘𝚠 𝚒𝚗 𝚍𝚏.𝚒𝚝𝚎𝚛𝚛𝚘𝚠𝚜():

𝚍𝚏.𝚕𝚘𝚌[𝚒, "𝚝𝚘𝚝𝚊𝚕"] = 𝚛𝚘𝚠["𝚙𝚛𝚒𝚌𝚎"] * 𝚛𝚘𝚠["𝚚𝚞𝚊𝚗𝚝𝚒𝚝𝚢"]



✅ 𝟏𝟎𝟎𝐱 𝐅𝐚𝐬𝐭𝐞𝐫 (𝐕𝐞𝐜𝐭𝐨𝐫𝐢𝐳𝐞𝐝 𝐨𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧):

𝚍𝚏["𝚝𝚘𝚝𝚊𝚕"] = 𝚍𝚏["𝚙𝚛𝚒𝚌𝚎"] * 𝚍𝚏["𝚚𝚞𝚊𝚗𝚝𝚒𝚝𝚢"]



2️⃣ 𝐒𝐮𝐩𝐞𝐫𝐜𝐡𝐚𝐫𝐠𝐞 𝐌𝐮𝐥𝐭𝐢-𝐂𝐨𝐥𝐮𝐦𝐧 𝐌𝐚𝐭𝐡 𝐰𝐢𝐭𝐡 𝚙𝚍.𝚎𝚟𝚊𝚕()

# 𝙴𝚕𝚒𝚖𝚒𝚗𝚊𝚝𝚎𝚜 𝚝𝚎𝚖𝚙𝚘𝚛𝚊𝚛𝚢 𝚖𝚎𝚖𝚘𝚛𝚢 𝚊𝚕𝚕𝚘𝚌𝚊𝚝𝚒𝚘𝚗 𝚘𝚟𝚎𝚛𝚑𝚎𝚊𝚍

𝚍𝚏["𝚛𝚎𝚜𝚞𝚕𝚝"] = 𝚙𝚍.𝚎𝚟𝚊𝚕("𝚌𝚘𝚕𝟷 + 𝚌𝚘𝚕𝟸 * 𝚌𝚘𝚕𝟹 - 𝚌𝚘𝚕𝟺", 𝚝𝚊𝚛𝚐𝚎𝚝=𝚍𝚏)



3️⃣ 𝐃𝐨𝐰𝐧𝐜𝐚𝐬𝐭 𝐃𝐚𝐭𝐚 𝐓𝐲𝐩𝐞𝐬 & 𝐔𝐬𝐞 𝐂𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐜𝐚𝐥𝐬

By default, Pandas assigns 64-bit integers/floats and treats string columns as bloated 𝚘𝚋𝚓𝚎𝚌𝚝 types. Downcasting your numbers and converting repeating strings to 𝚌𝚊𝚝𝚎𝚐𝚘𝚛𝚢 can shrink your memory footprint by up to 80%—drastically accelerating every subsequent operation.



# 𝙳𝚘𝚠𝚗𝚌𝚊𝚜𝚝 𝚗𝚞𝚖𝚎𝚛𝚒𝚌𝚜 & 𝚖𝚊𝚙 𝚜𝚝𝚛𝚒𝚗𝚐𝚜 𝚝𝚘 𝚌𝚊𝚝𝚎𝚐𝚘𝚛𝚢

𝚍𝚏["𝚒𝚍"] = 𝚙𝚍.𝚝𝚘_𝚗𝚞𝚖𝚎𝚛𝚒𝚌(𝚍𝚏["𝚒𝚍"], 𝚍𝚘𝚠𝚗𝚌𝚊𝚜𝚝="𝚒𝚗𝚝𝚎𝚐𝚎𝚛")

𝚍𝚏["𝚛𝚎𝚐𝚒𝚘𝚗"] = 𝚍𝚏["𝚛𝚎𝚐𝚒𝚘𝚗"].𝚊𝚜𝚝𝚢𝚙𝚎("𝚌𝚊𝚝𝚎𝚐𝚘𝚛𝚢")



💡 𝐓𝐡𝐞 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲: Optimize your code at the vector and memory level before throwing expensive compute clusters or hardware upgrades at the problem.



What is your favorite performance trick for handling large datasets? Let’s share notes in the comments below! 👇



Explore more https://youtu.be/3vX1S2qxDu8



#Python #Pandas #DataScience #DataEngineering #CodeOptimization #MachineLearning #Programming