Facebook Groups Data Mining and Analysis Guide

Overview

Documentation of Facebook Groups data scraping, preparation, and analysis procedures as part of Monthly MG Reporting Process, CIDP Reporting Process, and quarterly reporting. This method will allow you to scrape hundreds to thousands of Facebook Groups posts and comments at a time, and then import the data into Quid for analysis.

Important Info and Links

All underlying data should be managed through the Google Sheet.

Requirements

• Facebook account with access to any private groups that are necessary
• Chromium-based internet browser
  • Preferred: Arc Browser
  • Backup option: Brave or Chrome
• Custom stylesheet plugin
  • If using Arc, use its built-in Boosts feature and install my custom boost
• Data Scraper chromium extension with all permissions enabled (see below screenshots)
• OpenRefine

Add Posts to Collections

1. Open Scraping Groups and Search URLs from the Google Spreadsheet.

2. Open a URL that is listed on that sheet to show a list of posts. Scroll through and add posts to a collection that you’ve named by quarter or month i.e. Q2 2023 or August 2023. Don’t worry about reading through each post, just add as many as you can do your collection quickly, as long as the posts fall within your reporting date range. The post text will be automatically analyzed in later steps.

3. Repeat for each URL listed on step 1. NOTE: Facebook may block you from using the feature if you do too many, too fast. I am not sure of the threshold, but it happened to me.

Scrape Collection Posts

1. Goto your collection of posts:

2. In Data Miner, run recipe `July 2023 Facebook Post Collections Scrape

3. Export as XLSX

4. Upload to the Google Spreadsheet - this will be your ORIGINAL POSTS.

Run Scrape Crawl on Posts to Extract Comments

1. In the new sheet you just added from step 4 of Scrape collection posts, select and copy 20 or so URLs:

2. Open new tab to Google.com. Then open the Data Miner extension and click the blue Scrape This Page button:

3. Click Crawl Scrape on left, then on the right under Start a new crawl, click Create New Crawl. Then select Type URLs, and paste in the URLs you copied from step 1. Then Click the blue Check it button:

4. Select My Recipes tab, then the “Feb 2024 Facebook Groups Posts Scrape” recipe (or the most recent version of it). It will run a test, like below. The Next Step button will then turn blue, click it:

5. Click Start Crawl and let it run. Do NOT interrupt, go have a coffee for a little bit:

6. When it’s done, goto the Download tab and download as XLSX. Repeat in batches as necessary. I tested up to 50 URLs at a time and it worked for me, but YMMV.

Import Comments and Merge Data

1. Import your crawled COMMENTS spreadsheet data as sheets within the same file (repeat as necessary for each excel file):

2. Combine any of the imported crawled comment spreadsheets into one, repeating below screenshots as necessary (and deleting the sheets that have been copied over):

3. Still in your COMMENTS sheet, do the following edits:

4. Goto your POSTS sheet, and copy A2:E

5. Delete sheet and post to the bottom of the COMMMENTS sheet:

6. Rename the sheet to something like “Q2 2023 Myasthenia Gravis Facebook Groups Data”:

7. While still on the sheet, File > Download > CSV

Process and Refine Data

1. Start OpenRefine, and upload your newly exported CSV:

2. Navigate to apply operation histroy

3. Copy and paste json history of operations from this file into the window that pops up, then click the Perform Operations button

4. Wait, it will take awhile to process. Then export as CSV:

Import Data Into Quid

1. Goto the data uploader in Quid from the main page:

2. Select topic to upload to:

3. Upload the CSV file you downloaded from OpenRefine.

4. Click Next, Confirm, and Confirm.

5. Select the V2- Facebook Groups Data

6. If needed, map the appropriate fields, the click Save and Verify.

7. Click Apply Mapping, then Confirm.

8. Done!

Analyze and Apply Sentiment

The following steps outline how to review and apply sentiment to Facebook Group Posts for reporting.

1. Goto Edit on the top, then click on Myasthenia Gravis.
2. Select date range and tag SOURCE=FACEBOOK GROUPS then click Apply

Can further drill down to Vyvgart themes:

3. Set number of sound bites to show to max:
4. Assign sentiments, then Save and Close: