> ## Content Index > Fetch the complete content index at: https://datascientyst.com/llms.txt > Use this file to discover other available public pages before exploring further. # FIFA World Cup 2022: Data-Driven Analyze (Twitter) - URL: https://datascientyst.com/data-driven-approach-to-analyze-fifa-world-cup-2022-twitter/ - Published: 2022-11-20T15:53:06.000Z - Updated: 2023-03-19T09:31:53.000Z - Author: John D K - Tags: Project Today is the first day of the World Cup 2022 which takes place in Qatar from 20-th of November to 18-th of December. 32 teams will compete in eight groups for the prize. In this post we will use data-driven approach to analyze the teams and what people twit for the new football event. What we can learn from this post is how to extract tweets with Python library [tweepy](https://pypi.org/project/tweepy/?ref=datascientyst.com) and use Pandas to analyse them. This article will walk through how to collect meta data, scrape tweets and read all tweets into a Pandas DataFrame. Because the Twitter setup and authentication process is complex, we will not cover it in details. This is a sample data science project for confident beginners :) ## Requirements Libraries needed for this project: ``` pip install tweepy pip install python-dotenv pip install pycountry pip install emoji-country-flag ``` ## 1\. Collect meta data The first part of the process is collecting finalist countries and their flags. If you haven't used Pandas to scrape web before, use the following links: - [pandas.read\_html](https://pandas.pydata.org/docs/reference/api/pandas.read%5Fhtml.html?ref=datascientyst.com) - [Easily extract tables from websites with Pandas and Python](https://www.youtube.com/watch?v=OXA%5FZD1gR6A&ref=datascientyst.com) ### 1.1 Collect finalists Once we find the page where we can collect data, we can use it in the next Python code: ```python from pandas.io.html import read_html page = 'https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_qualification' wikitables = read_html(page) len(wikitables) ``` Method `read_html` reads HTML tables into a list of DataFrame objects. The result is 30. As we can see the page: [2022 FIFA World Cup qualification](https://en.wikipedia.org/wiki/2022%5FFIFA%5FWorld%5FCup%5Fqualification?ref=datascientyst.com) contains 30 tables. The one which contains our data is available under number 1 - we can find that by inspecting the page or displaying DataFrames: ```python df_fin = wikitables[1] df_fin.head() ``` Now we have all countries which plays on the World Cup 2022 as Pandas DataFrame: | | Team | Method ofqualification | Date ofqualification | Totaltimesqualified | Lasttimequalified | Currentconsecutiveappearances | Previous bestperformance | | - | ------- | ---------------------- | -------------------- | ------------------- | ----------------- | ----------------------------- | -------------------------------------- | | 0 | Qatar | Hosts | 2 December 2010 | 1 | โ€“ | 1 | โ€“ | | 1 | Germany | UEFA Group J winners | 11 October 2021 | 20\[a\] | 2018 | 18 | Winners (1954, 1974, 1990, 2014) | | 2 | Denmark | UEFA Group F winners | 12 October 2021 | 6 | 2018 | 2 | Quarter-finals (1998) | | 3 | Brazil | CONMEBOL winners | 11 November 2021 | 22 | 2018 | 22 | Winners (1958, 1962, 1970, 1994, 2002) | | 4 | France | UEFA Group D winners | 13 November 2021 | 16 | 2018 | 7 | Winners (1998, 2018) | ### 1.2 Get country flags Python offers elegant way to collect information about counrtries like: - flag - abbreviations - official name The first library is named: [pycountry](https://pypi.org/project/pycountry/?ref=datascientyst.com). This Python library provides the ISO databases for the standards: - Languages - Countries - Subdivisions of countries - Currencies Currently contains about 250 countries. From this library we would use fuzzy matching to collect the country details: ```python import pycountry pycountry.countries.search_fuzzy('Brazil') ``` which results into: ``` [Country(alpha_2='BR', alpha_3='BRA', flag='๐Ÿ‡ง๐Ÿ‡ท', name='Brazil', numeric='076', official_name='Federative Republic of Brazil')] ``` For our needs we will collect the flags for the 32 teams on the World Cup: ```python countries = [] for fin in finalists: try: country = pycountry.countries.search_fuzzy(fin)[0] flag = country.flag country = {fin:flag} except: print(fin) countries.append(country) print(countries) ``` Which results into: ``` England [{'Qatar': '๐Ÿ‡ถ๐Ÿ‡ฆ'}, {'Germany': '๐Ÿ‡ฉ๐Ÿ‡ช'}, {'Denmark': '๐Ÿ‡ฉ๐Ÿ‡ฐ'}, {'Brazil': '๐Ÿ‡ง๐Ÿ‡ท'}, {'France': '๐Ÿ‡ซ๐Ÿ‡ท'}, {'Belgium': '๐Ÿ‡ง๐Ÿ‡ช'}, {'Serbia': '๐Ÿ‡ท๐Ÿ‡ธ'}, {'Spain': '๐Ÿ‡ช๐Ÿ‡ธ'}, {'Croatia': '๐Ÿ‡ญ๐Ÿ‡ท'}, {'Switzerland': '๐Ÿ‡จ๐Ÿ‡ญ'}, {'Switzerland': '๐Ÿ‡จ๐Ÿ‡ญ'}, {'Netherlands': '๐Ÿ‡ณ๐Ÿ‡ฑ'}, {'Argentina': '๐Ÿ‡ฆ๐Ÿ‡ท'}, {'Iran': '๐Ÿ‡ฎ๐Ÿ‡ท'}, {'South Korea': '๐Ÿ‡ฐ๐Ÿ‡ท'}, {'Saudi Arabia': '๐Ÿ‡ธ๐Ÿ‡ฆ'}, {'Japan': '๐Ÿ‡ฏ๐Ÿ‡ต'}, {'Uruguay': '๐Ÿ‡บ๐Ÿ‡พ'}, {'Ecuador': '๐Ÿ‡ช๐Ÿ‡จ'}, {'Canada': '๐Ÿ‡จ๐Ÿ‡ฆ'}, {'Ghana': '๐Ÿ‡ฌ๐Ÿ‡ญ'}, {'Senegal': '๐Ÿ‡ธ๐Ÿ‡ณ'}, {'Poland': '๐Ÿ‡ต๐Ÿ‡ฑ'}, {'Portugal': '๐Ÿ‡ต๐Ÿ‡น'}, {'Tunisia': '๐Ÿ‡น๐Ÿ‡ณ'}, {'Morocco': '๐Ÿ‡ฒ๐Ÿ‡ฆ'}, {'Cameroon': '๐Ÿ‡จ๐Ÿ‡ฒ'}, {'United States': '๐Ÿ‡บ๐Ÿ‡ธ'}, {'Mexico': '๐Ÿ‡ฒ๐Ÿ‡ฝ'}, {'Wales': '๐Ÿ‡ฆ๐Ÿ‡บ'}, {'Australia': '๐Ÿ‡ฆ๐Ÿ‡บ'}, {'Costa Rica': '๐Ÿ‡จ๐Ÿ‡ท'}] ``` There is open issue for England: [search\_fuzzy fails for 'England'](https://github.com/flyingcircusio/pycountry/issues/126?ref=datascientyst.com) Alternatively we can use another library called: [emoji-country-flag](https://pypi.org/project/emoji-country-flag/?ref=datascientyst.com). ```python import flag flag.flag("FR") ``` result: ``` '๐Ÿ‡ซ๐Ÿ‡ท' ``` Finally we need to get English flag as dict `{'England': '๓ ง๓ ข๓ ฅ๓ ฎ๓ ง๐Ÿด๓ ง๓ ข๓ ฅ๓ ฎ๓ ง๓ ฟ'}` and append it to the list. ## 2\. Extract Tweets For this step we are going to use library: [tweepy](https://pypi.org/project/tweepy/?ref=datascientyst.com). Depending on the Twitter access we can use two different approaches to extract tweets: - Essential access - Twitter API v2 endpoints only - Elevated access - everything ### 2.1 Setup a Twitter Developer Account To setup new Twitter Developer Account we can use these articles as a reference: - [tweepy authentication](https://docs.tweepy.org/en/latest/authentication.html?ref=datascientyst.com#twitter-api-v2) - [How To Extract Data From The Twitter API Using Python](https://towardsdatascience.com/how-to-extract-data-from-the-twitter-api-using-python-b6fbd7129a33?ref=datascientyst.com). ### 2.2 Essential access Using this approach we need only BEARER\_TOKEN from Twitter. So we can search for tweets by query `'#FIFAWorldCup'`. To make this code work you need to replace `os.environ["BEARER_TOKEN"]` with your token. Alternatively you can create `.env` file with the following syntax - the file should be in the same folder as the script: ``` API_KEY="xxx" API_KEY_SECRET="xxx" BEARER_TOKEN="xxx" ACCESS_TOKEN="xxx" ACCESS_TOKEN_SECRET="xxx" ``` Then to extract tweets run the following code: ```python import tweepy import pandas as pd bearer_token = os.environ["BEARER_TOKEN"] client = tweepy.Client(bearer_token=bearer_token) """ More examples https://github.com/twitterdev/getting-started-with-the-twitter-api-v2-for-academic-research/blob/main/modules/5-how-to-write-search-queries.md """ query = '#FIFAWorldCup' tweets = client.search_recent_tweets(query=query, tweet_fields=['context_annotations', 'created_at'], max_results=10) pd.DataFrame(tweets.data) ``` By default the Twitter API will return max of 100 results. In order to get more than 100 tweets we are using built-in tweepy.Paginator: ```python import tweepy client = tweepy.Client(bearer_token=bearer_token) query = '#FIFAWorldCup' tweets = tweepy.Paginator(client.search_recent_tweets, query=query, tweet_fields=['context_annotations', 'created_at'], max_results=100).flatten(limit=10000) ls = [] for tweet in tweets: ls.append(tweet) import pandas as pd df = pd.DataFrame(ls) ``` The tweets stored as Pandas DataFrame: | | created\_at | id | text | | - | ------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | 2022-11-20 11:14:41+00:00 | 1594288115992043520 | RT @sushimmii: @rahmdess27 @BTS\_twt DREAMERS BY JUNGKOOK\\nDREAMERS STREAMING PARTY\\n\\n#Dreamers2022 #FIFAWorldCup @BTS\_twt #DreamersByJungkooโ€ฆ | | 1 | 2022-11-20 11:14:41+00:00 | 1594288115895914496 | RT @FIFAWorldCup: "See you at the Opening!" - Jung Kook \\n\\nToday, 5.30pm local time. \\n\\n#Qatar2022 \| #FIFAWorldCup https://t.co/3oulTUthBV | | 2 | 2022-11-20 11:14:41+00:00 | 1594288115635539969 | @kingnyamjoon DREAMERS BY JUNGKOOK\\nDREAMERS STREAMING PARTY\\n\\n#Dreamers2022 #FIFAWorldCup @BTS\_twt #DreamersByJungkook #JungKook #์ •๊ตญ | | 3 | 2022-11-20 11:14:41+00:00 | 1594288115581358081 | RT @btsarmy39521947: Jungkook ๐Ÿ’œ\\nIโ€™m so addicted with #Jungkook of \\n@BTS\_twt\\n's new single "Dreamers" for the #FIFAWorldCup ! The lyriโ€ฆ | | 4 | 2022-11-20 11:14:41+00:00 | 1594288115283214336 | RT @Bangtan\_twt\_com: @btssomma DREAMERS BY JUNGKOOK\\nDREAMERS STREAMING PARTY\\n\\n#Dreamers2022 #FIFAWorldCup #DreamersByJungkook #JungKโ€ฆ | ### 2.3 Elevated access You need to apply for Elevated access to work with the code below. Otherwise you will face an error: > Forbidden: 403 Forbidden > 453 - You currently have Essential access which includes access to Twitter API v2 endpoints only. If you need access to this endpoint, youโ€™ll need to apply for Elevated access via the Developer Portal. You can learn more here: [https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api?ref=datascientyst.com#v2-access-leve) Code: ```python import os import tweepy from dotenv import load_dotenv, find_dotenv from pathlib import Path path='./conf/.env' load_dotenv(dotenv_path=path,verbose=True) consumer_key = os.environ["API_KEY"] consumer_secret = os.environ["API_KEY_SECRET"] access_token = os.environ["ACCESS_TOKEN"] access_token_secret = os.environ["ACCESS_TOKEN_SECRET"] auth = tweepy.OAuth1UserHandler( consumer_key, consumer_secret, access_token, access_token_secret ) api = tweepy.API(auth) tweets = api.search_tweets("World cup", tweet_mode="extended") for tweet in tweets: try: print(tweet.retweeted_status.full_text) print("=====") except AttributeError: print(tweet.full_text) print("=====") ``` ## 3\. Analyse Extracted Data Finally we will use data collected in previous two steps. ### 3.1\. Finding the most mentioned countries To find the most mentioned countries we can use loop over the teams and count the mentions: ```python count_team = [] for team in df_fin['Team']: count_team.append(df[df['text'].str.contains(team)].shape[0]) df_fin['count'] = count_team df_fin.sort_values('count', ascending=False).set_index('Team').tail(31)['count' ].plot(kind='bar') ``` We exlude Qatar from the chart as it has more than 3500 mentions: ![data-driven-approach-to-analyze-fifa-world-cup-2022-twitter](https://datascientyst.com/content/images/2022/11/data-driven-approach-to-analyze-fifa-world-cup-2022-twitter.webp) We can see that the other top mentioned country is Ecuador. This is expected as this is going to be the first game: Qatar - Ecuador. ### 3.2\. Searching by flags We can use flags to search in the tweets - as we know emojis are very popular in the Twitter jargon: ```python df[df.text.str.contains(flag.flag("FR"))].shape df[df.text.str.contains(flag.flag("BR"))].shape ``` So we get that Brazile is mentioned 48 times while France is mentioned 45 times. ## Conclusion There are several new ideas we learned with this post:. - how to easily extract tabular data with Pandas - several new Python libraries to make your live much easier - how to scrape tweets with few line of codes - data-driven approach to solve everyday problems Finally, I love the motto of 2022 World Cup: > "Expect Amazing"