About the Course
This course will introduce the learner to the basics of the python programming environment, including fundamental python programming techniques such as lambdas, reading and manipulating csv files, and the numpy library. The course will introduce data manipulation and cleaning techniques using the popular python pandas data science library and introduce the abstraction of the Series and DataFrame as the central data structures for data analysis, along with tutorials on how to use functions such as groupby, merge, and pivot tables effectively. By the end of this course, students will be able to take tabular data, clean it, manipulate it, and run basic inferential statistical analyses. This course should be taken before any of the other Applied Data Science with Python courses: Applied Plotting, Charting & Data Representation in Python, Applied Machine Learning in Python, Applied Text Mining in Python, Applied Social Network Analysis in Python....
Top reviews
PK
May 9, 2020
The course had helped in understanding the concepts of NumPy and pandas. The assignments were so helpful to apply these concepts which provide an in-depth understanding of the Numpy as well as pandans
YY
Sep 28, 2021
This is the practical course.There is some concepts and assignments like: pandas, data-frame, merge and time. The asg 3 and asg4 are difficult but I think that it's very useful and improve my ability.
I have some problem with Assignment 3 [//github.com/AparaV/intro-to-data-science-with-python/tree/master/assignment-03]. I expect there are no 'Nan' in result. but there are 'Nan' value. this is my first time to study programming language. It would be great if someone could tell me what is wrong with my python code. here is the picture of wanted result and result using codes below
import pandas as pd
import numpy as np
import re
def answer_one[]:
def energy[]:
energy = pd.read_excel['Energy Indicators.xls',sheet_name='Energy']
energy = energy.iloc[16:243]
energy.drop[['Unnamed: 0','Unnamed: 1'], axis ='columns',inplace=True]
energy.columns = ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']
energy = energy.replace['...',np.nan]
energy['Energy Supply'] = energy['Energy Supply']*1000000
energy = energy.replace['Republic of Korea','South Korea']
energy = energy.replace['United States of America','United States']
energy = energy.replace['United Kingdom of Great Britain and Northern Ireland','United Kingdom']
energy = energy.replace['China, Hong Kong Special Administrative Region','Hong Kong']
energy['Country'] = energy['Country'].str.replace[" \[.*\]",""]
energy = energy.reset_index[]
energy = energy[['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']]
return energy
def GDP[]:
GDP= pd.read_csv['world_bank.csv']
s=[GDP.iloc[3].values][:4].astype[str].tolist[] + [GDP.iloc[3].values][4:].astype[int].astype[str].tolist[]
GDP = GDP[4:]
GDP.columns = s
GDP = GDP[['Country Name','2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']]
GDP.columns = ['Country','2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']
GDP = GDP.replace['Korea, Rep.','South Korea']
GDP = GDP.replace['Iran, Islamic Rep','Iran']
GDP = GDP.replace['Hong Kong SAR, China','Hong Kong']
return GDP
def ScimEn[]:
ScimEn = pd.read_excel['scimagojr-3.xlsx']
return ScimEn
e = energy[]
g = GDP[]
s = ScimEn[]
df=pd.merge[e,g,how = 'outer',left_on='Country',right_on='Country']
df=pd.merge[s,df,how='outer',left_on='Country',right_on='Country']
df.sort_values[by=['Rank'], inplace = True]
df.set_index['Country',inplace=True]
res = df.head[15]
return res
answer_one[]
preface
This chapter is the third Assignment of Introduction to Data Science in Python. The main task of this chapter is to select the appropriate method of summarizing and merging data tables based on the three tables given and on the understanding of various merging processes, that is, row data summary, clean and merge the data, and cluster the data by using the function of DataFrame, Calculate and view the statistical description value, find the maximum and minimum items and other data processing, which requires skilled use of various technologies.
1, Q1
1.1 problem description
Load the energy data from the file assets/Energy Indicators.xls, which is a list of indicators of energy supply and renewable electricity production from the United
Nations for the year 2013, and should be put into a DataFrame with the variable name of Energy.
Keep in mind that this is an Excel file, and not a comma separated values file. Also, make sure to exclude the footer and header information from the datafile. The first two columns are unneccessary, so you should get rid of them, and you should change the column labels so that the columns are:
['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable]
Convert Energy Supply to gigajoules [Note: there are 1,000,000 gigajoules in a petajoule]. For all countries which have missing data [e.g. data with "..."] make sure this is reflected as np.NaN values.
Rename the following list of countries [for use in later questions]:
"Republic of Korea": "South Korea", "United States of America": "United States", "United Kingdom of Great Britain and Northern Ireland": "United Kingdom", "China, Hong Kong Special Administrative Region": "Hong Kong"
There are also several countries with parenthesis in their name. Be sure to remove these, e.g. 'Bolivia [Plurinational State of]' should be 'Bolivia'.
Next, load the GDP data from the file assets/world_bank.csv, which is a csv containing countries' GDP from 1960 to 2015 from World Bank. Call this DataFrame GDP.
Make sure to skip the header, and rename the following list of countries:
"Korea, Rep.": "South Korea", "Iran, Islamic Rep.": "Iran", "Hong Kong SAR, China": "Hong Kong"
Finally, load the Sciamgo Journal and Country Rank data for Energy Engineering and Power Technology from the file assets/scimagojr-3.xlsx, which ranks countries based on their journal contributions in the aforementioned area. Call this DataFrame ScimEn.
Join the three datasets: GDP, Energy, and ScimEn into a new dataset [using the intersection of country names]. Use only the last 10 years [2006-2015] of GDP data and only the top 15 countries by Scimagojr 'Rank' [Rank 1 through 15].
The index of this DataFrame should be the name of the country, and the columns should
be ['Rank', 'Documents', 'Citable documents', 'Citations', 'Self-citations',
'Citations per document', 'H index', 'Energy Supply',
'Energy Supply per Capita', '% Renewable', '2006', '2007', '2008',
'2009', '2010', '2011', '2012', '2013', '2014', '2015'].
This function should return a DataFrame with 20 columns and 15 entries, and the rows of the DataFrame should be sorted by "Rank".
1.2 problem analysis
The first question mainly introduces the data contents of three
data sets, namely:
- Energy: the data is taken from 'energy indicators' Xls', energy index, the task takes the fields ['Country', 'Energy Supply', 'Energy Supply per capital', 'Renewable'].
- GDP: data from 'world'_ bank. CSV ', this task takes the GDP of the top 10 countries from 2006 to 2015.
- ScimEn: the data is taken from 'scimagojr-3 Xlsx ', ranking them mainly according to the journal contributions of countries in the above fields, and taking all fields.
Next, standardize the data of each data set, normalize the names in the Energy and GDP tables, for example, change "Republic of Korea" to "South Korea", remove the [] and [] comments in the country names in the Energy table, convert the units of 'Energy Supply', change the column names of Energy and GDP, and finally merge the three tables together.
1.3 code and detailed explanation
#Set the function to change the name
def change_country_values[item]:
dicts={"Republic of Korea": "South Korea",
"United States of America": "United States",
"United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
"China, Hong Kong Special Administrative Region": "Hong Kong",
"Korea, Rep.": "South Korea",
"Iran, Islamic Rep.": "Iran",
"Hong Kong SAR, China": "Hong Kong"}
#Using the iterator of dicts for matching
for key,values in dicts.items[]:
if item==key:
return values
return item
def answer_one[]:
#Create Energy's DataFrame
Energy=pd.read_excel['assets/Energy Indicators.xls',skiprows=16,usecols=[2,3,4,5]]
Energy=Energy[1:228]
#Change column name
Energy.columns=['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']
#Remove the tail patch
Energy.replace[to_replace=' \[.*\]$',value='',regex=True,inplace=True]
Energy.replace[to_replace='[\d]+$',value='',regex=True,inplace=True]
Energy.replace[to_replace='^[\.]+$',value=np.nan,regex=True,inplace=True]
#Change unit
Energy['Energy Supply']=Energy['Energy Supply'].apply[lambda x:x*1000000]
#Change country name
Energy['Country']=Energy['Country'].apply[change_country_values]
#print[Energy]
#Create DataFrame for GDP
GDP=pd.read_csv['assets/world_bank.csv',skiprows=4]
GDP['Country Name']=GDP['Country Name'].apply[change_country_values]
GDP.rename[columns={'Country Name':'Country'},inplace=True]
#print[GDP.head[]]
#Create ScimEn's DataFrame
ScimEn=pd.read_excel['assets/scimagojr-3.xlsx']
#print[ScimEn]
#Summarize the data columns to go into a list
year=list[range[2006,2016]]
GDP_year=[str[i] for i in year]
GDP_year.append['Country']
#Take out the data from 2006 to 2015
GDP=GDP.loc[:,GDP_year]
#print[GDP_year]
#Merge DataFrame by country name
merge1=pd.merge[Energy,GDP,left_on='Country',right_on='Country',how='inner']
#print[merge1.columns]
#Merge DataFrame by country name
merge2=pd.merge[ScimEn,merge1,on='Country',how='inner']
#print[len[merge2]]
#Take the top 15 after merger
merge2=merge2[merge2['Rank']