Pandas: Removing Duplicates

If you want to discover duplicates on the dataset, we can use duplicated() method. It gives you a boolean values (True or False).

 

Example 

print (df.duplicated())

 

• Removing Duplicates 

To remove duplicates, use the drop_duplicates() method. 

 

Example 

df.drop_duplicates(inplace = True)

 

Exercises 

Explain the following code.

# Count duplicate student names
duplicate_count = df[df.duplicated(subset=["StudentName"], keep=False)].shape[0]

# Display the count
print("Count of duplicate student names:", duplicate_count)

• 'duplicated()' was used to discover the duplicates.
• subset=["StudentName"]: checks for duplicate values in the specified subset in this case, "StudentName".
• The keep=False parameter means that all occurrences of duplicates will be marked as True.
• .shape[0]: This part retrieves the number of rows in the filtered DataFrame, which corresponds to the count of duplicate student names.